2024-02-20T07:37:26.318 INFO:root:teuthology version: 0.0.1.dev260+gb1dac55 2024-02-20T07:37:26.318 DEBUG:teuthology.run:Teuthology command: teuthology --verbose --name pdonnell-2024-02-20_07:23:03-fs:upgrade:mds_upgrade_sequence-wip-batrick-testing-20240220.022152-distro-default-smithi --owner scheduled_pdonnell@teuthology --description fs:upgrade:mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 0a 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client/kclient 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} --archive /home/teuthworker/archive/pdonnell-2024-02-20_07:23:03-fs:upgrade:mds_upgrade_sequence-wip-batrick-testing-20240220.022152-distro-default-smithi/7568541 -- /home/teuthworker/archive/pdonnell-2024-02-20_07:23:03-fs:upgrade:mds_upgrade_sequence-wip-batrick-testing-20240220.022152-distro-default-smithi/7568541/orig.config.yaml 2024-02-20T07:37:26.376 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-02-20T07:37:26.474 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/pdonnell-2024-02-20_07:23:03-fs:upgrade:mds_upgrade_sequence-wip-batrick-testing-20240220.022152-distro-default-smithi/7568541 branch: wip-batrick-testing-20240220.022152 description: fs:upgrade:mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 0a 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client/kclient 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} email: pdonnell@redhat.com first_in_suite: false job_id: '7568541' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi meta: - desc: 'setup ceph/quincy ' name: pdonnell-2024-02-20_07:23:03-fs:upgrade:mds_upgrade_sequence-wip-batrick-testing-20240220.022152-distro-default-smithi no_nested_subset: false nuke-on-error: true os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-batrick-testing-20240220.022152 ceph: conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 rados mon op timeout: 900 rados osd op timeout: 900 global: mon pg warn min per osd: 0 mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon op complaint time: 120 osd: bdev async discard: true bdev enable discard: true bluestore allocator: bitmap bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug ms: 1 debug osd: 20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - FS_DEGRADED - FS_INLINE_DATA_DEPRECATED - FS_WITH_FAILED_MDS - MDS_ALL_DOWN - MDS_DAMAGE - MDS_DEGRADED - MDS_FAILED - MDS_INSUFFICIENT_STANDBY - MDS_UP_LESS_THAN_MAX - POOL_APP_NOT_ENABLED - overall HEALTH_ - Replacing daemon - OSD_DOWN - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding sha1: 91ff50e903dcc99ee3ecf9523050878eb452da5c ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: flavor: default sha1: 91ff50e903dcc99ee3ecf9523050878eb452da5c kclient: syntax: v1 selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-batrick-testing-20240220.022152 sha1: 3d72bcfd7affbb4a923c191d212814b4934cfeaa owner: scheduled_pdonnell@teuthology priority: 50 repo: https://git.ceph.com/ceph-ci.git roles: - - host.a - client.0 - osd.0 - osd.1 - osd.2 - - host.b - client.1 - osd.3 - osd.4 - osd.5 seed: 9910 sha1: 91ff50e903dcc99ee3ecf9523050878eb452da5c sleep_before_teardown: 0 suite: fs:upgrade:mds_upgrade_sequence suite_branch: wip-batrick-testing-20240220.022152 suite_path: /home/teuthworker/src/github.com_batrick_ceph_3d72bcfd7affbb4a923c191d212814b4934cfeaa/qa suite_relpath: qa suite_repo: https://github.com/batrick/ceph.git suite_sha1: 3d72bcfd7affbb4a923c191d212814b4934cfeaa targets: smithi133.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLkjOQOC8d7yhry/bHm5pROTzCvqDQM+A7aVk/xdrmEzKhMlapDJ8ADcoYOUv7PtjtrLT3zeo0UbgMBXaX5hyO8= smithi183.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEgZmL9WQMs3I6qqnHNxQrqtYMMn1nYmIxZNjI4HotJ6cvNlZcY79uT1EOzikzEjiglVbto/qdQL+mWRmUzefIU= tasks: - install: branch: quincy exclude_packages: - ceph-volume - print: '**** done install task...' - cephadm: cephadm_branch: quincy cephadm_git_url: https://github.com/ceph/ceph conf: osd: osd_class_default_list: '*' osd_class_load_list: '*' image: quay.ceph.io/ceph-ci/ceph:quincy roleless: true - print: '**** done end installing quincy cephadm ...' - cephadm.shell: host.a: - ceph config set mgr mgr/cephadm/use_repo_digest true --force - print: '**** done cephadm.shell ceph config set mgr...' - cephadm.shell: host.a: - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - full_sequential_finally: - sleep: duration: 3600 - cephadm.shell: host.a: - ceph fs volume create cephfs --placement=4 - ceph fs dump - cephadm.shell: host.a: - ceph fs set cephfs max_mds 2 - cephadm.shell: host.a: - ceph fs set cephfs allow_standby_replay true - cephadm.shell: host.a: - ceph fs set cephfs inline_data true --yes-i-really-really-mean-it - cephadm.shell: host.a: - ceph fs dump - ceph --format=json fs dump | jq -e ".filesystems | length == 1" - while ! ceph --format=json mds versions | jq -e ". | add == 4"; do sleep 1; done - fs.pre_upgrade_save: null - kclient: null - print: '**** done client' - parallel: - upgrade-tasks - workload-tasks - cephadm.shell: host.a: - ceph fs dump - fs.post_upgrade_checks: null teuthology: fragments_dropped: [] meta: {} postmerge: [] variables: fail_fs: true teuthology_branch: main teuthology_sha1: b1dac5519c57c269ea98a22fb7729016a1d0e4d2 timestamp: 2024-02-20_07:23:03 tube: smithi upgrade-tasks: sequential: - cephadm.shell: env: - sha1 host.a: - ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force - ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force - ceph config set global log_to_journald false --force - ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 --daemon-types mgr - while ceph orch upgrade status | jq '.in_progress' | grep true && ! ceph orch upgrade status | jq '.message' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; sleep 30 ; done - ceph versions | jq -e '.mgr | length == 1' - ceph versions | jq -e '.mgr | keys' | grep $sha1 - ceph versions | jq -e '.overall | length == 2' - ceph orch upgrade check quay.ceph.io/ceph-ci/ceph:$sha1 | jq -e '.up_to_date | length == 2' - ceph orch ps - cephadm.shell: env: - sha1 host.a: - ceph config set mgr mgr/orchestrator/fail_fs true - cephadm.shell: env: - sha1 host.a: - ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force - ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force - ceph config set global log_to_journald false --force - ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 - cephadm.shell: env: - sha1 host.a: - while ceph orch upgrade status | jq '.in_progress' | grep true && ! ceph orch upgrade status | jq '.message' | grep Error ; do ceph orch ps ; ceph versions ; ceph fs dump; ceph orch upgrade status ; ceph health detail ; sleep 30 ; done - ceph orch ps - ceph orch upgrade status - ceph health detail - ceph versions - echo "wait for servicemap items w/ changing names to refresh" - sleep 60 - ceph orch ps - ceph versions - ceph versions | jq -e '.overall | length == 1' - ceph versions | jq -e '.overall | keys' | grep $sha1 user: pdonnell verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 workload-tasks: sequential: - workunit: clients: all: - suites/fsstress.sh 2024-02-20T07:37:26.475 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_batrick_ceph_3d72bcfd7affbb4a923c191d212814b4934cfeaa/qa; will attempt to use it 2024-02-20T07:37:26.475 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_batrick_ceph_3d72bcfd7affbb4a923c191d212814b4934cfeaa/qa/tasks 2024-02-20T07:37:26.476 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-02-20T07:37:26.477 INFO:teuthology.task.internal:Checking packages... 2024-02-20T07:37:26.501 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '91ff50e903dcc99ee3ecf9523050878eb452da5c' 2024-02-20T07:37:26.501 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-02-20T07:37:26.501 INFO:teuthology.packaging:ref: None 2024-02-20T07:37:26.501 INFO:teuthology.packaging:tag: None 2024-02-20T07:37:26.501 INFO:teuthology.packaging:branch: wip-batrick-testing-20240220.022152 2024-02-20T07:37:26.501 INFO:teuthology.packaging:sha1: 91ff50e903dcc99ee3ecf9523050878eb452da5c 2024-02-20T07:37:26.502 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-batrick-testing-20240220.022152 2024-02-20T07:37:26.718 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-1587.g91ff50e9 2024-02-20T07:37:26.720 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-02-20T07:37:26.727 INFO:teuthology.task.internal:no buildpackages task found 2024-02-20T07:37:26.727 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-02-20T07:37:26.764 INFO:teuthology.task.internal:Saving configuration 2024-02-20T07:37:26.787 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-02-20T07:37:26.793 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-02-20T07:37:26.819 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi133.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2024-02-20_07:23:03-fs:upgrade:mds_upgrade_sequence-wip-batrick-testing-20240220.022152-distro-default-smithi/7568541', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-02-20 07:32:27.090664', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLkjOQOC8d7yhry/bHm5pROTzCvqDQM+A7aVk/xdrmEzKhMlapDJ8ADcoYOUv7PtjtrLT3zeo0UbgMBXaX5hyO8='} 2024-02-20T07:37:26.840 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi183.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2024-02-20_07:23:03-fs:upgrade:mds_upgrade_sequence-wip-batrick-testing-20240220.022152-distro-default-smithi/7568541', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-02-20 07:32:27.088682', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEgZmL9WQMs3I6qqnHNxQrqtYMMn1nYmIxZNjI4HotJ6cvNlZcY79uT1EOzikzEjiglVbto/qdQL+mWRmUzefIU='} 2024-02-20T07:37:26.840 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-02-20T07:37:26.847 INFO:teuthology.task.internal:roles: ubuntu@smithi133.front.sepia.ceph.com - ['host.a', 'client.0', 'osd.0', 'osd.1', 'osd.2'] 2024-02-20T07:37:26.847 INFO:teuthology.task.internal:roles: ubuntu@smithi183.front.sepia.ceph.com - ['host.b', 'client.1', 'osd.3', 'osd.4', 'osd.5'] 2024-02-20T07:37:26.847 INFO:teuthology.run_tasks:Running task console_log... 2024-02-20T07:37:26.966 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f60618930d0>, signals=[15]) 2024-02-20T07:37:26.967 INFO:teuthology.run_tasks:Running task internal.connect... 2024-02-20T07:37:26.997 INFO:teuthology.task.internal:Opening connections... 2024-02-20T07:37:26.997 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi133.front.sepia.ceph.com 2024-02-20T07:37:26.999 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi133.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-20T07:37:27.091 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi183.front.sepia.ceph.com 2024-02-20T07:37:27.092 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi183.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-20T07:37:27.183 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-02-20T07:37:27.189 DEBUG:teuthology.orchestra.run.smithi133:> uname -m 2024-02-20T07:37:27.207 INFO:teuthology.orchestra.run.smithi133.stdout:x86_64 2024-02-20T07:37:27.207 DEBUG:teuthology.orchestra.run.smithi133:> cat /etc/os-release 2024-02-20T07:37:27.265 INFO:teuthology.orchestra.run.smithi133.stdout:NAME="CentOS Stream" 2024-02-20T07:37:27.265 INFO:teuthology.orchestra.run.smithi133.stdout:VERSION="9" 2024-02-20T07:37:27.265 INFO:teuthology.orchestra.run.smithi133.stdout:ID="centos" 2024-02-20T07:37:27.265 INFO:teuthology.orchestra.run.smithi133.stdout:ID_LIKE="rhel fedora" 2024-02-20T07:37:27.265 INFO:teuthology.orchestra.run.smithi133.stdout:VERSION_ID="9" 2024-02-20T07:37:27.265 INFO:teuthology.orchestra.run.smithi133.stdout:PLATFORM_ID="platform:el9" 2024-02-20T07:37:27.265 INFO:teuthology.orchestra.run.smithi133.stdout:PRETTY_NAME="CentOS Stream 9" 2024-02-20T07:37:27.265 INFO:teuthology.orchestra.run.smithi133.stdout:ANSI_COLOR="0;31" 2024-02-20T07:37:27.265 INFO:teuthology.orchestra.run.smithi133.stdout:LOGO="fedora-logo-icon" 2024-02-20T07:37:27.265 INFO:teuthology.orchestra.run.smithi133.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-02-20T07:37:27.265 INFO:teuthology.orchestra.run.smithi133.stdout:HOME_URL="https://centos.org/" 2024-02-20T07:37:27.266 INFO:teuthology.orchestra.run.smithi133.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-02-20T07:37:27.266 INFO:teuthology.orchestra.run.smithi133.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-02-20T07:37:27.266 INFO:teuthology.orchestra.run.smithi133.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-02-20T07:37:27.266 INFO:teuthology.lock.ops:Updating smithi133.front.sepia.ceph.com on lock server 2024-02-20T07:37:27.285 DEBUG:teuthology.orchestra.run.smithi183:> uname -m 2024-02-20T07:37:27.302 INFO:teuthology.orchestra.run.smithi183.stdout:x86_64 2024-02-20T07:37:27.303 DEBUG:teuthology.orchestra.run.smithi183:> cat /etc/os-release 2024-02-20T07:37:27.360 INFO:teuthology.orchestra.run.smithi183.stdout:NAME="CentOS Stream" 2024-02-20T07:37:27.360 INFO:teuthology.orchestra.run.smithi183.stdout:VERSION="9" 2024-02-20T07:37:27.360 INFO:teuthology.orchestra.run.smithi183.stdout:ID="centos" 2024-02-20T07:37:27.360 INFO:teuthology.orchestra.run.smithi183.stdout:ID_LIKE="rhel fedora" 2024-02-20T07:37:27.360 INFO:teuthology.orchestra.run.smithi183.stdout:VERSION_ID="9" 2024-02-20T07:37:27.360 INFO:teuthology.orchestra.run.smithi183.stdout:PLATFORM_ID="platform:el9" 2024-02-20T07:37:27.360 INFO:teuthology.orchestra.run.smithi183.stdout:PRETTY_NAME="CentOS Stream 9" 2024-02-20T07:37:27.360 INFO:teuthology.orchestra.run.smithi183.stdout:ANSI_COLOR="0;31" 2024-02-20T07:37:27.361 INFO:teuthology.orchestra.run.smithi183.stdout:LOGO="fedora-logo-icon" 2024-02-20T07:37:27.361 INFO:teuthology.orchestra.run.smithi183.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-02-20T07:37:27.361 INFO:teuthology.orchestra.run.smithi183.stdout:HOME_URL="https://centos.org/" 2024-02-20T07:37:27.361 INFO:teuthology.orchestra.run.smithi183.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-02-20T07:37:27.361 INFO:teuthology.orchestra.run.smithi183.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-02-20T07:37:27.361 INFO:teuthology.orchestra.run.smithi183.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-02-20T07:37:27.361 INFO:teuthology.lock.ops:Updating smithi183.front.sepia.ceph.com on lock server 2024-02-20T07:37:27.385 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-02-20T07:37:27.395 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-02-20T07:37:27.435 INFO:teuthology.task.internal:Checking for old test directory... 2024-02-20T07:37:27.435 DEBUG:teuthology.orchestra.run.smithi133:> test '!' -e /home/ubuntu/cephtest 2024-02-20T07:37:27.438 DEBUG:teuthology.orchestra.run.smithi183:> test '!' -e /home/ubuntu/cephtest 2024-02-20T07:37:27.454 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-02-20T07:37:27.460 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-02-20T07:37:27.460 DEBUG:teuthology.orchestra.run.smithi133:> test -z $(ls -A /var/lib/ceph) 2024-02-20T07:37:27.495 DEBUG:teuthology.orchestra.run.smithi183:> test -z $(ls -A /var/lib/ceph) 2024-02-20T07:37:27.518 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-02-20T07:37:27.576 INFO:teuthology.run_tasks:Running task kernel... 2024-02-20T07:37:27.592 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-02-20T07:37:27.592 INFO:teuthology.task.kernel:config {'host.a': {'kdb': True, 'sha1': 'distro'}, 'host.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-02-20T07:37:27.593 DEBUG:teuthology.orchestra.run.smithi133:> test -f /run/.containerenv -o -f /.dockerenv 2024-02-20T07:37:27.593 DEBUG:teuthology.orchestra.run.smithi183:> test -f /run/.containerenv -o -f /.dockerenv 2024-02-20T07:37:27.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-20T07:37:27.610 DEBUG:teuthology.orchestra.run.smithi133:> uname -r 2024-02-20T07:37:27.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-20T07:37:27.611 DEBUG:teuthology.orchestra.run.smithi183:> uname -r 2024-02-20T07:37:27.666 INFO:teuthology.orchestra.run.smithi133.stdout:5.14.0-361.el9.x86_64 2024-02-20T07:37:27.666 INFO:teuthology.task.kernel:Running kernel on smithi133: 5.14.0-361.el9.x86_64 2024-02-20T07:37:27.666 DEBUG:teuthology.orchestra.run.smithi133:> sudo yum install -y kernel 2024-02-20T07:37:27.667 INFO:teuthology.orchestra.run.smithi183.stdout:5.14.0-361.el9.x86_64 2024-02-20T07:37:27.668 INFO:teuthology.task.kernel:Running kernel on smithi183: 5.14.0-361.el9.x86_64 2024-02-20T07:37:27.668 DEBUG:teuthology.orchestra.run.smithi183:> sudo yum install -y kernel 2024-02-20T07:37:29.650 INFO:teuthology.orchestra.run.smithi183.stdout:CentOS-9 - AppStream 16 MB/s | 6.7 MB 00:00 2024-02-20T07:37:29.779 INFO:teuthology.orchestra.run.smithi133.stdout:CentOS-9 - AppStream 15 MB/s | 6.7 MB 00:00 2024-02-20T07:37:31.050 INFO:teuthology.orchestra.run.smithi183.stdout:CentOS-9 - BaseOS 7.3 MB/s | 2.0 MB 00:00 2024-02-20T07:37:31.170 INFO:teuthology.orchestra.run.smithi133.stdout:CentOS-9 - BaseOS 7.3 MB/s | 2.0 MB 00:00 2024-02-20T07:37:31.945 INFO:teuthology.orchestra.run.smithi133.stdout:CentOS Stream 9 - BaseOS 18 MB/s | 8.0 MB 00:00 2024-02-20T07:37:34.246 INFO:teuthology.orchestra.run.smithi133.stdout:CentOS Stream 9 - AppStream 16 MB/s | 19 MB 00:01 2024-02-20T07:37:38.904 INFO:teuthology.orchestra.run.smithi133.stdout:CentOS Stream 9 - CRB 5.4 MB/s | 6.1 MB 00:01 2024-02-20T07:37:40.431 INFO:teuthology.orchestra.run.smithi183.stdout:CentOS Stream 9 - BaseOS 944 kB/s | 8.0 MB 00:08 2024-02-20T07:37:40.438 INFO:teuthology.orchestra.run.smithi133.stdout:CentOS Stream 9 - Extras packages 38 kB/s | 15 kB 00:00 2024-02-20T07:37:41.356 INFO:teuthology.orchestra.run.smithi133.stdout:Extra Packages for Enterprise Linux 26 MB/s | 20 MB 00:00 2024-02-20T07:37:42.190 INFO:teuthology.orchestra.run.smithi183.stdout:CentOS Stream 9 - AppStream 29 MB/s | 19 MB 00:00 2024-02-20T07:37:46.298 INFO:teuthology.orchestra.run.smithi133.stdout:lab-extras 12 kB/s | 1.7 kB 00:00 2024-02-20T07:37:48.588 INFO:teuthology.orchestra.run.smithi133.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-02-20T07:37:48.588 INFO:teuthology.orchestra.run.smithi133.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-02-20T07:37:48.599 INFO:teuthology.orchestra.run.smithi183.stdout:CentOS Stream 9 - CRB 2.1 MB/s | 6.1 MB 00:02 2024-02-20T07:37:48.688 INFO:teuthology.orchestra.run.smithi133.stdout:Dependencies resolved. 2024-02-20T07:37:48.697 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2024-02-20T07:37:48.698 INFO:teuthology.orchestra.run.smithi133.stdout: Package Arch Version Repository Size 2024-02-20T07:37:48.698 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2024-02-20T07:37:48.698 INFO:teuthology.orchestra.run.smithi133.stdout:Installing: 2024-02-20T07:37:48.698 INFO:teuthology.orchestra.run.smithi133.stdout: kernel x86_64 5.14.0-421.el9 CentOS-BaseOS 6.1 M 2024-02-20T07:37:48.698 INFO:teuthology.orchestra.run.smithi133.stdout:Installing dependencies: 2024-02-20T07:37:48.698 INFO:teuthology.orchestra.run.smithi133.stdout: kernel-core x86_64 5.14.0-421.el9 CentOS-BaseOS 21 M 2024-02-20T07:37:48.698 INFO:teuthology.orchestra.run.smithi133.stdout: kernel-modules x86_64 5.14.0-421.el9 CentOS-BaseOS 39 M 2024-02-20T07:37:48.698 INFO:teuthology.orchestra.run.smithi133.stdout: kernel-modules-core x86_64 5.14.0-421.el9 CentOS-BaseOS 33 M 2024-02-20T07:37:48.698 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-02-20T07:37:48.698 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction Summary 2024-02-20T07:37:48.698 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2024-02-20T07:37:48.698 INFO:teuthology.orchestra.run.smithi133.stdout:Install 4 Packages 2024-02-20T07:37:48.698 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-02-20T07:37:48.699 INFO:teuthology.orchestra.run.smithi133.stdout:Total download size: 100 M 2024-02-20T07:37:48.699 INFO:teuthology.orchestra.run.smithi133.stdout:Installed size: 123 M 2024-02-20T07:37:48.699 INFO:teuthology.orchestra.run.smithi133.stdout:Downloading Packages: 2024-02-20T07:37:49.032 INFO:teuthology.orchestra.run.smithi133.stdout:(1/4): kernel-5.14.0-421.el9.x86_64.rpm 18 MB/s | 6.1 MB 00:00 2024-02-20T07:37:49.649 INFO:teuthology.orchestra.run.smithi133.stdout:(2/4): kernel-core-5.14.0-421.el9.x86_64.rpm 22 MB/s | 21 MB 00:00 2024-02-20T07:37:50.074 INFO:teuthology.orchestra.run.smithi183.stdout:CentOS Stream 9 - Extras packages 42 kB/s | 15 kB 00:00 2024-02-20T07:37:50.749 INFO:teuthology.orchestra.run.smithi133.stdout:(3/4): kernel-modules-core-5.14.0-421.el9.x86_6 20 MB/s | 33 MB 00:01 2024-02-20T07:37:51.207 INFO:teuthology.orchestra.run.smithi133.stdout:(4/4): kernel-modules-5.14.0-421.el9.x86_64.rpm 16 MB/s | 39 MB 00:02 2024-02-20T07:37:51.211 INFO:teuthology.orchestra.run.smithi133.stdout:-------------------------------------------------------------------------------- 2024-02-20T07:37:51.211 INFO:teuthology.orchestra.run.smithi133.stdout:Total 40 MB/s | 100 MB 00:02 2024-02-20T07:37:51.211 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction check 2024-02-20T07:37:51.602 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction check succeeded. 2024-02-20T07:37:51.602 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction test 2024-02-20T07:37:52.640 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction test succeeded. 2024-02-20T07:37:52.641 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction 2024-02-20T07:37:55.050 INFO:teuthology.orchestra.run.smithi133.stdout: Preparing : 1/1 2024-02-20T07:37:55.262 INFO:teuthology.orchestra.run.smithi183.stdout:Extra Packages for Enterprise Linux 4.0 MB/s | 20 MB 00:05 2024-02-20T07:37:55.725 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : kernel-modules-core-5.14.0-421.el9.x86_64 1/4 2024-02-20T07:37:55.849 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : kernel-core-5.14.0-421.el9.x86_64 2/4 2024-02-20T07:37:58.582 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: kernel-core-5.14.0-421.el9.x86_64 2/4 2024-02-20T07:37:58.758 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : kernel-modules-5.14.0-421.el9.x86_64 3/4 2024-02-20T07:38:00.091 INFO:teuthology.orchestra.run.smithi183.stdout:lab-extras 30 kB/s | 1.7 kB 00:00 2024-02-20T07:38:02.331 INFO:teuthology.orchestra.run.smithi183.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-02-20T07:38:02.332 INFO:teuthology.orchestra.run.smithi183.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-02-20T07:38:02.425 INFO:teuthology.orchestra.run.smithi183.stdout:Dependencies resolved. 2024-02-20T07:38:02.435 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-02-20T07:38:02.435 INFO:teuthology.orchestra.run.smithi183.stdout: Package Arch Version Repository Size 2024-02-20T07:38:02.435 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-02-20T07:38:02.435 INFO:teuthology.orchestra.run.smithi183.stdout:Installing: 2024-02-20T07:38:02.435 INFO:teuthology.orchestra.run.smithi183.stdout: kernel x86_64 5.14.0-421.el9 CentOS-BaseOS 6.1 M 2024-02-20T07:38:02.435 INFO:teuthology.orchestra.run.smithi183.stdout:Installing dependencies: 2024-02-20T07:38:02.436 INFO:teuthology.orchestra.run.smithi183.stdout: kernel-core x86_64 5.14.0-421.el9 CentOS-BaseOS 21 M 2024-02-20T07:38:02.436 INFO:teuthology.orchestra.run.smithi183.stdout: kernel-modules x86_64 5.14.0-421.el9 CentOS-BaseOS 39 M 2024-02-20T07:38:02.436 INFO:teuthology.orchestra.run.smithi183.stdout: kernel-modules-core x86_64 5.14.0-421.el9 CentOS-BaseOS 33 M 2024-02-20T07:38:02.436 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-02-20T07:38:02.436 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction Summary 2024-02-20T07:38:02.436 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-02-20T07:38:02.436 INFO:teuthology.orchestra.run.smithi183.stdout:Install 4 Packages 2024-02-20T07:38:02.436 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-02-20T07:38:02.437 INFO:teuthology.orchestra.run.smithi183.stdout:Total download size: 100 M 2024-02-20T07:38:02.437 INFO:teuthology.orchestra.run.smithi183.stdout:Installed size: 123 M 2024-02-20T07:38:02.437 INFO:teuthology.orchestra.run.smithi183.stdout:Downloading Packages: 2024-02-20T07:38:03.476 INFO:teuthology.orchestra.run.smithi183.stdout:(1/4): kernel-5.14.0-421.el9.x86_64.rpm 5.8 MB/s | 6.1 MB 00:01 2024-02-20T07:38:03.490 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: kernel-modules-5.14.0-421.el9.x86_64 3/4 2024-02-20T07:38:03.704 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : kernel-5.14.0-421.el9.x86_64 4/4 2024-02-20T07:38:04.676 INFO:teuthology.orchestra.run.smithi183.stdout:(2/4): kernel-core-5.14.0-421.el9.x86_64.rpm 9.3 MB/s | 21 MB 00:02 2024-02-20T07:38:05.901 INFO:teuthology.orchestra.run.smithi183.stdout:(3/4): kernel-modules-5.14.0-421.el9.x86_64.rpm 11 MB/s | 39 MB 00:03 2024-02-20T07:38:06.193 INFO:teuthology.orchestra.run.smithi183.stdout:(4/4): kernel-modules-core-5.14.0-421.el9.x86_6 12 MB/s | 33 MB 00:02 2024-02-20T07:38:06.197 INFO:teuthology.orchestra.run.smithi183.stdout:-------------------------------------------------------------------------------- 2024-02-20T07:38:06.197 INFO:teuthology.orchestra.run.smithi183.stdout:Total 26 MB/s | 100 MB 00:03 2024-02-20T07:38:06.197 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction check 2024-02-20T07:38:06.577 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction check succeeded. 2024-02-20T07:38:06.577 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction test 2024-02-20T07:38:07.607 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction test succeeded. 2024-02-20T07:38:07.607 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction 2024-02-20T07:38:08.418 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: kernel-modules-core-5.14.0-421.el9.x86_64 4/4 2024-02-20T07:38:10.028 INFO:teuthology.orchestra.run.smithi183.stdout: Preparing : 1/1 2024-02-20T07:38:10.667 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : kernel-modules-core-5.14.0-421.el9.x86_64 1/4 2024-02-20T07:38:10.801 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : kernel-core-5.14.0-421.el9.x86_64 2/4 2024-02-20T07:38:13.559 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: kernel-core-5.14.0-421.el9.x86_64 2/4 2024-02-20T07:38:13.746 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : kernel-modules-5.14.0-421.el9.x86_64 3/4 2024-02-20T07:38:18.431 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: kernel-modules-5.14.0-421.el9.x86_64 3/4 2024-02-20T07:38:18.532 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : kernel-5.14.0-421.el9.x86_64 4/4 2024-02-20T07:38:23.268 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: kernel-modules-core-5.14.0-421.el9.x86_64 4/4 2024-02-20T07:38:37.854 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: kernel-core-5.14.0-421.el9.x86_64 4/4 2024-02-20T07:38:37.854 INFO:teuthology.orchestra.run.smithi133.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-02-20T07:38:37.855 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-02-20T07:38:37.926 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: kernel-modules-5.14.0-421.el9.x86_64 4/4 2024-02-20T07:38:38.315 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: kernel-5.14.0-421.el9.x86_64 4/4 2024-02-20T07:38:38.315 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : kernel-5.14.0-421.el9.x86_64 1/4 2024-02-20T07:38:38.315 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : kernel-core-5.14.0-421.el9.x86_64 2/4 2024-02-20T07:38:38.316 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : kernel-modules-5.14.0-421.el9.x86_64 3/4 2024-02-20T07:38:38.716 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : kernel-modules-core-5.14.0-421.el9.x86_64 4/4 2024-02-20T07:38:38.716 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-02-20T07:38:38.716 INFO:teuthology.orchestra.run.smithi133.stdout:Installed: 2024-02-20T07:38:38.716 INFO:teuthology.orchestra.run.smithi133.stdout: kernel-5.14.0-421.el9.x86_64 2024-02-20T07:38:38.716 INFO:teuthology.orchestra.run.smithi133.stdout: kernel-core-5.14.0-421.el9.x86_64 2024-02-20T07:38:38.716 INFO:teuthology.orchestra.run.smithi133.stdout: kernel-modules-5.14.0-421.el9.x86_64 2024-02-20T07:38:38.717 INFO:teuthology.orchestra.run.smithi133.stdout: kernel-modules-core-5.14.0-421.el9.x86_64 2024-02-20T07:38:38.717 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-02-20T07:38:38.717 INFO:teuthology.orchestra.run.smithi133.stdout:Complete! 2024-02-20T07:38:39.250 DEBUG:teuthology.orchestra.run.smithi133:> rpm -q kernel | sort -rV | head -n 1 2024-02-20T07:38:39.425 INFO:teuthology.orchestra.run.smithi133.stdout:kernel-5.14.0-421.el9.x86_64 2024-02-20T07:38:39.426 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-421.el9.x86_64 2024-02-20T07:38:39.426 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-421.el9.x86_64 2024-02-20T07:38:39.426 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-02-20T07:38:39.426 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-02-20T07:38:39.426 INFO:teuthology.task.kernel:Installing distro kernel on host.a... 2024-02-20T07:38:39.426 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi133.front.sepia.ceph.com, path=None, version=distro) 2024-02-20T07:38:39.427 DEBUG:teuthology.orchestra.run.smithi133:> rpm -q kernel | sort -rV | head -n 1 2024-02-20T07:38:39.534 INFO:teuthology.orchestra.run.smithi133.stdout:kernel-5.14.0-421.el9.x86_64 2024-02-20T07:38:39.534 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-421.el9.x86_64 2024-02-20T07:38:39.534 DEBUG:teuthology.orchestra.run.smithi133:> sudo rpm -qi grub2-tools 2024-02-20T07:38:39.574 INFO:teuthology.orchestra.run.smithi133.stdout:Name : grub2-tools 2024-02-20T07:38:39.575 INFO:teuthology.orchestra.run.smithi133.stdout:Epoch : 1 2024-02-20T07:38:39.575 INFO:teuthology.orchestra.run.smithi133.stdout:Version : 2.06 2024-02-20T07:38:39.575 INFO:teuthology.orchestra.run.smithi133.stdout:Release : 68.el9 2024-02-20T07:38:39.575 INFO:teuthology.orchestra.run.smithi133.stdout:Architecture: x86_64 2024-02-20T07:38:39.575 INFO:teuthology.orchestra.run.smithi133.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-02-20T07:38:39.575 INFO:teuthology.orchestra.run.smithi133.stdout:Group : Unspecified 2024-02-20T07:38:39.575 INFO:teuthology.orchestra.run.smithi133.stdout:Size : 8264254 2024-02-20T07:38:39.575 INFO:teuthology.orchestra.run.smithi133.stdout:License : GPLv3+ 2024-02-20T07:38:39.575 INFO:teuthology.orchestra.run.smithi133.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-02-20T07:38:39.575 INFO:teuthology.orchestra.run.smithi133.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-02-20T07:38:39.575 INFO:teuthology.orchestra.run.smithi133.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-02-20T07:38:39.575 INFO:teuthology.orchestra.run.smithi133.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-02-20T07:38:39.575 INFO:teuthology.orchestra.run.smithi133.stdout:Packager : builder@centos.org 2024-02-20T07:38:39.575 INFO:teuthology.orchestra.run.smithi133.stdout:Vendor : CentOS 2024-02-20T07:38:39.576 INFO:teuthology.orchestra.run.smithi133.stdout:URL : http://www.gnu.org/software/grub/ 2024-02-20T07:38:39.576 INFO:teuthology.orchestra.run.smithi133.stdout:Summary : Support tools for GRUB. 2024-02-20T07:38:39.576 INFO:teuthology.orchestra.run.smithi133.stdout:Description : 2024-02-20T07:38:39.576 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-02-20T07:38:39.576 INFO:teuthology.orchestra.run.smithi133.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-02-20T07:38:39.576 INFO:teuthology.orchestra.run.smithi133.stdout:customizable bootloader with modular architecture. It supports a rich 2024-02-20T07:38:39.576 INFO:teuthology.orchestra.run.smithi133.stdout:variety of kernel formats, file systems, computer architectures and 2024-02-20T07:38:39.576 INFO:teuthology.orchestra.run.smithi133.stdout:hardware devices. 2024-02-20T07:38:39.576 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-02-20T07:38:39.576 INFO:teuthology.orchestra.run.smithi133.stdout:This subpackage provides tools for support of all platforms. 2024-02-20T07:38:39.577 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-02-20T07:38:39.577 INFO:teuthology.task.kernel:Updating grub on smithi133 to boot 5.14.0-421.el9.x86_64 2024-02-20T07:38:39.577 DEBUG:teuthology.orchestra.run.smithi133:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-02-20T07:38:40.195 INFO:teuthology.orchestra.run.smithi133.stderr:Generating grub configuration file ... 2024-02-20T07:38:41.566 INFO:teuthology.orchestra.run.smithi133.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-02-20T07:38:41.598 INFO:teuthology.orchestra.run.smithi133.stderr:done 2024-02-20T07:38:41.600 DEBUG:teuthology.orchestra.run.smithi133:> mktemp 2024-02-20T07:38:41.615 INFO:teuthology.orchestra.run.smithi133.stdout:/tmp/tmp.Fw0JKx3Hjl 2024-02-20T07:38:41.615 DEBUG:teuthology.orchestra.run.smithi133:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.Fw0JKx3Hjl 2024-02-20T07:38:41.683 DEBUG:teuthology.orchestra.run.smithi133:> sudo chmod 0666 /tmp/tmp.Fw0JKx3Hjl 2024-02-20T07:38:41.820 DEBUG:teuthology.orchestra.remote:smithi133:/tmp/tmp.Fw0JKx3Hjl is 6KB 2024-02-20T07:38:41.871 DEBUG:teuthology.orchestra.run.smithi133:> rm -fr /tmp/tmp.Fw0JKx3Hjl 2024-02-20T07:38:41.885 DEBUG:teuthology.orchestra.run.smithi133:> sudo /bin/ls /boot/loader/entries || true 2024-02-20T07:38:41.950 INFO:teuthology.orchestra.run.smithi133.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-02-20T07:38:41.950 INFO:teuthology.orchestra.run.smithi133.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-02-20T07:38:41.951 INFO:teuthology.orchestra.run.smithi133.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-02-20T07:38:41.951 INFO:teuthology.orchestra.run.smithi133.stdout:15949e3c88704300882482eddfd7baae-5.14.0-421.el9.x86_64.conf 2024-02-20T07:38:41.952 DEBUG:teuthology.orchestra.run.smithi133:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-421.el9.x86_64 2024-02-20T07:38:42.106 DEBUG:teuthology.orchestra.run.smithi133:> sudo shutdown -r now 2024-02-20T07:38:52.517 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: kernel-core-5.14.0-421.el9.x86_64 4/4 2024-02-20T07:38:52.517 INFO:teuthology.orchestra.run.smithi183.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-02-20T07:38:52.517 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-02-20T07:38:52.587 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: kernel-modules-5.14.0-421.el9.x86_64 4/4 2024-02-20T07:38:52.940 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: kernel-5.14.0-421.el9.x86_64 4/4 2024-02-20T07:38:52.940 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : kernel-5.14.0-421.el9.x86_64 1/4 2024-02-20T07:38:52.940 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : kernel-core-5.14.0-421.el9.x86_64 2/4 2024-02-20T07:38:52.940 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : kernel-modules-5.14.0-421.el9.x86_64 3/4 2024-02-20T07:38:53.367 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : kernel-modules-core-5.14.0-421.el9.x86_64 4/4 2024-02-20T07:38:53.368 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-02-20T07:38:53.368 INFO:teuthology.orchestra.run.smithi183.stdout:Installed: 2024-02-20T07:38:53.368 INFO:teuthology.orchestra.run.smithi183.stdout: kernel-5.14.0-421.el9.x86_64 2024-02-20T07:38:53.368 INFO:teuthology.orchestra.run.smithi183.stdout: kernel-core-5.14.0-421.el9.x86_64 2024-02-20T07:38:53.368 INFO:teuthology.orchestra.run.smithi183.stdout: kernel-modules-5.14.0-421.el9.x86_64 2024-02-20T07:38:53.368 INFO:teuthology.orchestra.run.smithi183.stdout: kernel-modules-core-5.14.0-421.el9.x86_64 2024-02-20T07:38:53.368 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-02-20T07:38:53.368 INFO:teuthology.orchestra.run.smithi183.stdout:Complete! 2024-02-20T07:38:53.845 DEBUG:teuthology.orchestra.run.smithi183:> rpm -q kernel | sort -rV | head -n 1 2024-02-20T07:38:53.970 INFO:teuthology.orchestra.run.smithi183.stdout:kernel-5.14.0-421.el9.x86_64 2024-02-20T07:38:53.971 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-421.el9.x86_64 2024-02-20T07:38:53.971 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-421.el9.x86_64 2024-02-20T07:38:53.971 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-02-20T07:38:53.971 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-02-20T07:38:53.972 INFO:teuthology.task.kernel:Installing distro kernel on host.b... 2024-02-20T07:38:53.972 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi183.front.sepia.ceph.com, path=None, version=distro) 2024-02-20T07:38:53.972 DEBUG:teuthology.orchestra.run.smithi183:> rpm -q kernel | sort -rV | head -n 1 2024-02-20T07:38:54.079 INFO:teuthology.orchestra.run.smithi183.stdout:kernel-5.14.0-421.el9.x86_64 2024-02-20T07:38:54.079 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-421.el9.x86_64 2024-02-20T07:38:54.079 DEBUG:teuthology.orchestra.run.smithi183:> sudo rpm -qi grub2-tools 2024-02-20T07:38:54.113 INFO:teuthology.orchestra.run.smithi183.stdout:Name : grub2-tools 2024-02-20T07:38:54.114 INFO:teuthology.orchestra.run.smithi183.stdout:Epoch : 1 2024-02-20T07:38:54.114 INFO:teuthology.orchestra.run.smithi183.stdout:Version : 2.06 2024-02-20T07:38:54.114 INFO:teuthology.orchestra.run.smithi183.stdout:Release : 68.el9 2024-02-20T07:38:54.114 INFO:teuthology.orchestra.run.smithi183.stdout:Architecture: x86_64 2024-02-20T07:38:54.114 INFO:teuthology.orchestra.run.smithi183.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-02-20T07:38:54.114 INFO:teuthology.orchestra.run.smithi183.stdout:Group : Unspecified 2024-02-20T07:38:54.114 INFO:teuthology.orchestra.run.smithi183.stdout:Size : 8264254 2024-02-20T07:38:54.114 INFO:teuthology.orchestra.run.smithi183.stdout:License : GPLv3+ 2024-02-20T07:38:54.114 INFO:teuthology.orchestra.run.smithi183.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-02-20T07:38:54.114 INFO:teuthology.orchestra.run.smithi183.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-02-20T07:38:54.114 INFO:teuthology.orchestra.run.smithi183.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-02-20T07:38:54.114 INFO:teuthology.orchestra.run.smithi183.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-02-20T07:38:54.115 INFO:teuthology.orchestra.run.smithi183.stdout:Packager : builder@centos.org 2024-02-20T07:38:54.115 INFO:teuthology.orchestra.run.smithi183.stdout:Vendor : CentOS 2024-02-20T07:38:54.115 INFO:teuthology.orchestra.run.smithi183.stdout:URL : http://www.gnu.org/software/grub/ 2024-02-20T07:38:54.115 INFO:teuthology.orchestra.run.smithi183.stdout:Summary : Support tools for GRUB. 2024-02-20T07:38:54.115 INFO:teuthology.orchestra.run.smithi183.stdout:Description : 2024-02-20T07:38:54.115 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-02-20T07:38:54.115 INFO:teuthology.orchestra.run.smithi183.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-02-20T07:38:54.115 INFO:teuthology.orchestra.run.smithi183.stdout:customizable bootloader with modular architecture. It supports a rich 2024-02-20T07:38:54.115 INFO:teuthology.orchestra.run.smithi183.stdout:variety of kernel formats, file systems, computer architectures and 2024-02-20T07:38:54.115 INFO:teuthology.orchestra.run.smithi183.stdout:hardware devices. 2024-02-20T07:38:54.115 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-02-20T07:38:54.115 INFO:teuthology.orchestra.run.smithi183.stdout:This subpackage provides tools for support of all platforms. 2024-02-20T07:38:54.116 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-02-20T07:38:54.116 INFO:teuthology.task.kernel:Updating grub on smithi183 to boot 5.14.0-421.el9.x86_64 2024-02-20T07:38:54.117 DEBUG:teuthology.orchestra.run.smithi183:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-02-20T07:38:54.682 INFO:teuthology.orchestra.run.smithi183.stderr:Generating grub configuration file ... 2024-02-20T07:38:56.068 INFO:teuthology.orchestra.run.smithi183.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-02-20T07:38:56.094 INFO:teuthology.orchestra.run.smithi183.stderr:done 2024-02-20T07:38:56.097 DEBUG:teuthology.orchestra.run.smithi183:> mktemp 2024-02-20T07:38:56.114 INFO:teuthology.orchestra.run.smithi183.stdout:/tmp/tmp.70J3m83shv 2024-02-20T07:38:56.115 DEBUG:teuthology.orchestra.run.smithi183:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.70J3m83shv 2024-02-20T07:38:56.183 DEBUG:teuthology.orchestra.run.smithi183:> sudo chmod 0666 /tmp/tmp.70J3m83shv 2024-02-20T07:38:56.327 DEBUG:teuthology.orchestra.remote:smithi183:/tmp/tmp.70J3m83shv is 6KB 2024-02-20T07:38:56.376 DEBUG:teuthology.orchestra.run.smithi183:> rm -fr /tmp/tmp.70J3m83shv 2024-02-20T07:38:56.390 DEBUG:teuthology.orchestra.run.smithi183:> sudo /bin/ls /boot/loader/entries || true 2024-02-20T07:38:56.456 INFO:teuthology.orchestra.run.smithi183.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-02-20T07:38:56.456 INFO:teuthology.orchestra.run.smithi183.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-02-20T07:38:56.456 INFO:teuthology.orchestra.run.smithi183.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-02-20T07:38:56.459 INFO:teuthology.orchestra.run.smithi183.stdout:15949e3c88704300882482eddfd7baae-5.14.0-421.el9.x86_64.conf 2024-02-20T07:38:56.460 DEBUG:teuthology.orchestra.run.smithi183:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-421.el9.x86_64 2024-02-20T07:38:56.616 DEBUG:teuthology.orchestra.run.smithi183:> sudo shutdown -r now 2024-02-20T07:39:12.124 INFO:teuthology.task.kernel:Checking client host.a for new kernel version... 2024-02-20T07:39:12.125 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi133.front.sepia.ceph.com' 2024-02-20T07:39:12.126 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi133.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-20T07:39:26.632 INFO:teuthology.task.kernel:Checking client host.b for new kernel version... 2024-02-20T07:39:26.632 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi183.front.sepia.ceph.com' 2024-02-20T07:39:26.633 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi183.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-20T07:39:30.496 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.133 2024-02-20T07:39:39.504 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi133.front.sepia.ceph.com' 2024-02-20T07:39:39.505 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi133.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-20T07:39:42.556 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.133 2024-02-20T07:39:45.088 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.183 2024-02-20T07:39:54.096 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi183.front.sepia.ceph.com' 2024-02-20T07:39:54.097 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi183.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-20T07:39:54.557 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi133.front.sepia.ceph.com' 2024-02-20T07:39:54.558 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi133.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-20T07:39:57.148 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.183 2024-02-20T07:40:09.160 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi183.front.sepia.ceph.com' 2024-02-20T07:40:09.161 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi183.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-20T07:40:12.988 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.133 2024-02-20T07:40:28.000 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi133.front.sepia.ceph.com' 2024-02-20T07:40:28.001 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi133.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-20T07:40:28.288 DEBUG:teuthology.orchestra.run.smithi133:> true 2024-02-20T07:40:28.654 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi133.front.sepia.ceph.com' 2024-02-20T07:40:28.654 INFO:teuthology.task.kernel:Checking kernel version of host.a, want "5.14.0-421.el9.x86_64"... 2024-02-20T07:40:28.654 DEBUG:teuthology.orchestra.run.smithi133:> uname -r 2024-02-20T07:40:28.668 INFO:teuthology.orchestra.run.smithi133.stdout:5.14.0-421.el9.x86_64 2024-02-20T07:40:28.669 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-421.el9.x86_64 vs 5.14.0-421.el9.x86_64 2024-02-20T07:40:28.669 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-02-20T07:40:28.669 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-02-20T07:40:29.670 INFO:teuthology.task.kernel:Enabling kdb on host.a... 2024-02-20T07:40:29.670 DEBUG:teuthology.orchestra.run.smithi133:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-02-20T07:40:29.736 INFO:teuthology.orchestra.run.smithi133.stdout:ttyS1 2024-02-20T07:40:29.786 DEBUG:teuthology.parallel:result is None 2024-02-20T07:40:41.064 DEBUG:teuthology.orchestra.run.smithi183:> true 2024-02-20T07:40:41.601 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi183.front.sepia.ceph.com' 2024-02-20T07:40:41.602 INFO:teuthology.task.kernel:Checking kernel version of host.b, want "5.14.0-421.el9.x86_64"... 2024-02-20T07:40:41.602 DEBUG:teuthology.orchestra.run.smithi183:> uname -r 2024-02-20T07:40:41.616 INFO:teuthology.orchestra.run.smithi183.stdout:5.14.0-421.el9.x86_64 2024-02-20T07:40:41.616 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-421.el9.x86_64 vs 5.14.0-421.el9.x86_64 2024-02-20T07:40:41.616 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-02-20T07:40:41.616 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-02-20T07:40:42.617 INFO:teuthology.task.kernel:Enabling kdb on host.b... 2024-02-20T07:40:42.618 DEBUG:teuthology.orchestra.run.smithi183:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-02-20T07:40:42.684 INFO:teuthology.orchestra.run.smithi183.stdout:ttyS1 2024-02-20T07:40:42.715 DEBUG:teuthology.parallel:result is None 2024-02-20T07:40:42.715 INFO:teuthology.run_tasks:Running task internal.base... 2024-02-20T07:40:42.722 INFO:teuthology.task.internal:Creating test directory... 2024-02-20T07:40:42.722 DEBUG:teuthology.orchestra.run.smithi133:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-02-20T07:40:42.726 DEBUG:teuthology.orchestra.run.smithi183:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-02-20T07:40:42.741 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-02-20T07:40:42.755 INFO:teuthology.run_tasks:Running task internal.archive... 2024-02-20T07:40:42.807 INFO:teuthology.task.internal:Creating archive directory... 2024-02-20T07:40:42.808 DEBUG:teuthology.orchestra.run.smithi133:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-02-20T07:40:42.810 DEBUG:teuthology.orchestra.run.smithi183:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-02-20T07:40:42.975 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-02-20T07:40:42.982 INFO:teuthology.task.internal:Enabling coredump saving... 2024-02-20T07:40:42.982 DEBUG:teuthology.orchestra.run.smithi133:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-02-20T07:40:42.985 DEBUG:teuthology.orchestra.run.smithi183:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-02-20T07:40:43.021 INFO:teuthology.orchestra.run.smithi133.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-20T07:40:43.035 INFO:teuthology.orchestra.run.smithi133.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-20T07:40:43.127 INFO:teuthology.orchestra.run.smithi183.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-20T07:40:43.141 INFO:teuthology.orchestra.run.smithi183.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-20T07:40:43.142 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-02-20T07:40:43.154 INFO:teuthology.task.internal:Configuring sudo... 2024-02-20T07:40:43.155 DEBUG:teuthology.orchestra.run.smithi133:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-02-20T07:40:43.157 DEBUG:teuthology.orchestra.run.smithi183:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-02-20T07:40:43.185 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-02-20T07:40:43.221 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-02-20T07:40:43.222 DEBUG:teuthology.orchestra.run.smithi133:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-02-20T07:40:43.227 DEBUG:teuthology.orchestra.run.smithi183:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-02-20T07:40:43.242 DEBUG:teuthology.orchestra.run.smithi133:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-02-20T07:40:43.326 DEBUG:teuthology.orchestra.run.smithi133:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-02-20T07:40:43.379 DEBUG:teuthology.orchestra.run.smithi133:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-02-20T07:40:43.461 DEBUG:teuthology.orchestra.run.smithi133:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-02-20T07:40:43.505 DEBUG:teuthology.orchestra.run.smithi133:> set -ex 2024-02-20T07:40:43.505 DEBUG:teuthology.orchestra.run.smithi133:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-02-20T07:40:43.572 DEBUG:teuthology.orchestra.run.smithi183:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-02-20T07:40:43.618 DEBUG:teuthology.orchestra.run.smithi183:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-02-20T07:40:43.666 DEBUG:teuthology.orchestra.run.smithi183:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-02-20T07:40:43.750 DEBUG:teuthology.orchestra.run.smithi183:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-02-20T07:40:43.797 DEBUG:teuthology.orchestra.run.smithi183:> set -ex 2024-02-20T07:40:43.797 DEBUG:teuthology.orchestra.run.smithi183:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-02-20T07:40:43.868 DEBUG:teuthology.orchestra.run.smithi133:> sudo service rsyslog restart 2024-02-20T07:40:43.870 DEBUG:teuthology.orchestra.run.smithi183:> sudo service rsyslog restart 2024-02-20T07:40:43.918 INFO:teuthology.orchestra.run.smithi133.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-02-20T07:40:43.957 INFO:teuthology.orchestra.run.smithi183.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-02-20T07:40:44.371 INFO:teuthology.run_tasks:Running task internal.timer... 2024-02-20T07:40:44.378 INFO:teuthology.task.internal:Starting timer... 2024-02-20T07:40:44.379 INFO:teuthology.run_tasks:Running task pcp... 2024-02-20T07:40:44.421 INFO:teuthology.run_tasks:Running task selinux... 2024-02-20T07:40:44.454 DEBUG:teuthology.task:Applying overrides for task selinux: {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']} 2024-02-20T07:40:44.455 DEBUG:teuthology.orchestra.run.smithi133:> sudo service auditd rotate 2024-02-20T07:40:44.502 INFO:teuthology.orchestra.run.smithi133.stdout:Rotating logs: 2024-02-20T07:40:44.504 DEBUG:teuthology.orchestra.run.smithi183:> sudo service auditd rotate 2024-02-20T07:40:44.550 INFO:teuthology.orchestra.run.smithi183.stdout:Rotating logs: 2024-02-20T07:40:44.552 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-02-20T07:40:44.552 DEBUG:teuthology.orchestra.run.smithi133:> /usr/sbin/getenforce 2024-02-20T07:40:44.586 INFO:teuthology.orchestra.run.smithi133.stdout:Permissive 2024-02-20T07:40:44.586 DEBUG:teuthology.orchestra.run.smithi183:> /usr/sbin/getenforce 2024-02-20T07:40:44.625 INFO:teuthology.orchestra.run.smithi183.stdout:Permissive 2024-02-20T07:40:44.626 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi133.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi183.front.sepia.ceph.com': 'permissive'} 2024-02-20T07:40:44.626 DEBUG:teuthology.orchestra.run.smithi133:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-02-20T07:40:44.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-20T07:40:44.655 DEBUG:teuthology.orchestra.run.smithi183:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-02-20T07:40:44.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-20T07:40:44.697 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-02-20T07:40:44.697 DEBUG:teuthology.orchestra.run.smithi133:> sudo /usr/sbin/setenforce permissive 2024-02-20T07:40:44.729 DEBUG:teuthology.orchestra.run.smithi183:> sudo /usr/sbin/setenforce permissive 2024-02-20T07:40:44.770 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-02-20T07:40:44.780 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-02-20T07:40:44.798 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-02-20T07:40:44.800 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi133.front.sepia.ceph.com,smithi183.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-02-20T07:44:57.629 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi133.front.sepia.ceph.com'), Remote(name='ubuntu@smithi183.front.sepia.ceph.com')] 2024-02-20T07:44:57.630 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi133.front.sepia.ceph.com' 2024-02-20T07:44:57.631 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi133.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-20T07:44:57.720 DEBUG:teuthology.orchestra.run.smithi133:> true 2024-02-20T07:44:57.799 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi133.front.sepia.ceph.com' 2024-02-20T07:44:57.800 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi183.front.sepia.ceph.com' 2024-02-20T07:44:57.800 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi183.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-20T07:44:57.889 DEBUG:teuthology.orchestra.run.smithi183:> true 2024-02-20T07:44:57.968 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi183.front.sepia.ceph.com' 2024-02-20T07:44:57.968 INFO:teuthology.run_tasks:Running task clock... 2024-02-20T07:44:57.980 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-02-20T07:44:57.980 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-02-20T07:44:57.980 DEBUG:teuthology.orchestra.run.smithi133:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-02-20T07:44:57.983 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-02-20T07:44:57.983 DEBUG:teuthology.orchestra.run.smithi183:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-02-20T07:44:58.013 INFO:teuthology.orchestra.run.smithi133.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-02-20T07:44:58.030 INFO:teuthology.orchestra.run.smithi133.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-02-20T07:44:58.040 INFO:teuthology.orchestra.run.smithi183.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-02-20T07:44:58.057 INFO:teuthology.orchestra.run.smithi183.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-02-20T07:44:58.059 INFO:teuthology.orchestra.run.smithi133.stderr:sudo: ntpd: command not found 2024-02-20T07:44:58.072 INFO:teuthology.orchestra.run.smithi133.stdout:506 Cannot talk to daemon 2024-02-20T07:44:58.086 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: ntpd: command not found 2024-02-20T07:44:58.089 INFO:teuthology.orchestra.run.smithi133.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-02-20T07:44:58.100 INFO:teuthology.orchestra.run.smithi183.stdout:506 Cannot talk to daemon 2024-02-20T07:44:58.105 INFO:teuthology.orchestra.run.smithi133.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-02-20T07:44:58.116 INFO:teuthology.orchestra.run.smithi183.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-02-20T07:44:58.133 INFO:teuthology.orchestra.run.smithi183.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-02-20T07:44:58.155 INFO:teuthology.orchestra.run.smithi133.stderr:bash: line 1: ntpq: command not found 2024-02-20T07:44:58.158 INFO:teuthology.orchestra.run.smithi133.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-02-20T07:44:58.158 INFO:teuthology.orchestra.run.smithi133.stdout:=============================================================================== 2024-02-20T07:44:58.158 INFO:teuthology.orchestra.run.smithi133.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-20T07:44:58.158 INFO:teuthology.orchestra.run.smithi133.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-20T07:44:58.158 INFO:teuthology.orchestra.run.smithi133.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-20T07:44:58.158 INFO:teuthology.orchestra.run.smithi133.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-20T07:44:58.181 INFO:teuthology.orchestra.run.smithi183.stderr:bash: line 1: ntpq: command not found 2024-02-20T07:44:58.184 INFO:teuthology.orchestra.run.smithi183.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-02-20T07:44:58.184 INFO:teuthology.orchestra.run.smithi183.stdout:=============================================================================== 2024-02-20T07:44:58.184 INFO:teuthology.orchestra.run.smithi183.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-20T07:44:58.184 INFO:teuthology.orchestra.run.smithi183.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-20T07:44:58.184 INFO:teuthology.orchestra.run.smithi183.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-20T07:44:58.184 INFO:teuthology.orchestra.run.smithi183.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-20T07:44:58.185 INFO:teuthology.run_tasks:Running task install... 2024-02-20T07:44:58.195 DEBUG:teuthology.task.install:project ceph 2024-02-20T07:44:58.195 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '91ff50e903dcc99ee3ecf9523050878eb452da5c'}} 2024-02-20T07:44:58.195 DEBUG:teuthology.task.install:config {'branch': 'quincy', 'exclude_packages': ['ceph-volume'], 'flavor': 'default', 'sha1': '91ff50e903dcc99ee3ecf9523050878eb452da5c'} 2024-02-20T07:44:58.195 INFO:teuthology.task.install:Using flavor: default 2024-02-20T07:44:58.201 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-02-20T07:44:58.201 INFO:teuthology.task.install:extra packages: [] 2024-02-20T07:44:58.201 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': 'quincy', 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': ['ceph-volume'], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '91ff50e903dcc99ee3ecf9523050878eb452da5c', 'tag': None, 'wait_for_package': False} 2024-02-20T07:44:58.201 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-02-20T07:44:58.201 INFO:teuthology.packaging:ref: None 2024-02-20T07:44:58.201 INFO:teuthology.packaging:tag: None 2024-02-20T07:44:58.201 INFO:teuthology.packaging:branch: quincy 2024-02-20T07:44:58.202 INFO:teuthology.packaging:sha1: 91ff50e903dcc99ee3ecf9523050878eb452da5c 2024-02-20T07:44:58.202 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=quincy 2024-02-20T07:44:58.204 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': 'quincy', 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': ['ceph-volume'], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '91ff50e903dcc99ee3ecf9523050878eb452da5c', 'tag': None, 'wait_for_package': False} 2024-02-20T07:44:58.204 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-02-20T07:44:58.204 INFO:teuthology.packaging:ref: None 2024-02-20T07:44:58.204 INFO:teuthology.packaging:tag: None 2024-02-20T07:44:58.204 INFO:teuthology.packaging:branch: quincy 2024-02-20T07:44:58.205 INFO:teuthology.packaging:sha1: 91ff50e903dcc99ee3ecf9523050878eb452da5c 2024-02-20T07:44:58.205 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=quincy 2024-02-20T07:44:58.447 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/quincy/c80e7168938c48b49e5ce2e002f48bf150b5f4a4/centos/9/flavors/default/ 2024-02-20T07:44:58.448 INFO:teuthology.task.install.rpm:Package version is 17.2.7-521.gc80e7168 2024-02-20T07:44:58.512 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/quincy/c80e7168938c48b49e5ce2e002f48bf150b5f4a4/centos/9/flavors/default/ 2024-02-20T07:44:58.512 INFO:teuthology.task.install.rpm:Package version is 17.2.7-521.gc80e7168 2024-02-20T07:44:58.624 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/quincy/c80e7168938c48b49e5ce2e002f48bf150b5f4a4/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/quincy/c80e7168938c48b49e5ce2e002f48bf150b5f4a4/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/quincy/c80e7168938c48b49e5ce2e002f48bf150b5f4a4/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-02-20T07:44:58.625 DEBUG:teuthology.orchestra.run.smithi133:> set -ex 2024-02-20T07:44:58.625 DEBUG:teuthology.orchestra.run.smithi133:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-02-20T07:44:58.635 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/quincy/c80e7168938c48b49e5ce2e002f48bf150b5f4a4/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/quincy/c80e7168938c48b49e5ce2e002f48bf150b5f4a4/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/quincy/c80e7168938c48b49e5ce2e002f48bf150b5f4a4/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-02-20T07:44:58.635 DEBUG:teuthology.orchestra.run.smithi183:> set -ex 2024-02-20T07:44:58.636 DEBUG:teuthology.orchestra.run.smithi183:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-02-20T07:44:58.653 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-02-20T07:44:58.654 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-02-20T07:44:58.654 INFO:teuthology.packaging:ref: None 2024-02-20T07:44:58.654 INFO:teuthology.packaging:tag: None 2024-02-20T07:44:58.654 INFO:teuthology.packaging:branch: quincy 2024-02-20T07:44:58.654 INFO:teuthology.packaging:sha1: 91ff50e903dcc99ee3ecf9523050878eb452da5c 2024-02-20T07:44:58.654 DEBUG:teuthology.orchestra.run.smithi133:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;ref/quincy/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-02-20T07:44:58.665 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-02-20T07:44:58.665 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-02-20T07:44:58.665 INFO:teuthology.packaging:ref: None 2024-02-20T07:44:58.665 INFO:teuthology.packaging:tag: None 2024-02-20T07:44:58.665 INFO:teuthology.packaging:branch: quincy 2024-02-20T07:44:58.665 INFO:teuthology.packaging:sha1: 91ff50e903dcc99ee3ecf9523050878eb452da5c 2024-02-20T07:44:58.665 DEBUG:teuthology.orchestra.run.smithi183:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;ref/quincy/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-02-20T07:44:58.720 DEBUG:teuthology.orchestra.run.smithi133:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-02-20T07:44:58.733 DEBUG:teuthology.orchestra.run.smithi183:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-02-20T07:44:58.800 DEBUG:teuthology.orchestra.run.smithi133:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-02-20T07:44:58.814 DEBUG:teuthology.orchestra.run.smithi183:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-02-20T07:44:58.867 INFO:teuthology.orchestra.run.smithi133.stdout:check_obsoletes = 1 2024-02-20T07:44:58.868 DEBUG:teuthology.orchestra.run.smithi133:> sudo yum clean all 2024-02-20T07:44:58.879 INFO:teuthology.orchestra.run.smithi183.stdout:check_obsoletes = 1 2024-02-20T07:44:58.881 DEBUG:teuthology.orchestra.run.smithi183:> sudo yum clean all 2024-02-20T07:44:59.103 INFO:teuthology.orchestra.run.smithi133.stdout:74 files removed 2024-02-20T07:44:59.116 INFO:teuthology.orchestra.run.smithi183.stdout:74 files removed 2024-02-20T07:44:59.130 DEBUG:teuthology.orchestra.run.smithi133:> sudo yum -y install ceph-radosgw 2024-02-20T07:44:59.143 DEBUG:teuthology.orchestra.run.smithi183:> sudo yum -y install ceph-radosgw 2024-02-20T07:44:59.789 INFO:teuthology.orchestra.run.smithi183.stdout:ceph packages for x86_64 237 kB/s | 77 kB 00:00 2024-02-20T07:44:59.943 INFO:teuthology.orchestra.run.smithi133.stdout:ceph packages for x86_64 156 kB/s | 77 kB 00:00 2024-02-20T07:45:00.256 INFO:teuthology.orchestra.run.smithi183.stdout:ceph noarch packages 22 kB/s | 9.1 kB 00:00 2024-02-20T07:45:00.369 INFO:teuthology.orchestra.run.smithi133.stdout:ceph noarch packages 24 kB/s | 9.1 kB 00:00 2024-02-20T07:45:00.707 INFO:teuthology.orchestra.run.smithi183.stdout:ceph source packages 4.3 kB/s | 1.8 kB 00:00 2024-02-20T07:45:00.978 INFO:teuthology.orchestra.run.smithi133.stdout:ceph source packages 3.1 kB/s | 1.8 kB 00:00 2024-02-20T07:45:02.205 INFO:teuthology.orchestra.run.smithi133.stdout:CentOS-9 - AppStream 5.6 MB/s | 6.7 MB 00:01 2024-02-20T07:45:03.826 INFO:teuthology.orchestra.run.smithi183.stdout:CentOS-9 - AppStream 2.2 MB/s | 6.7 MB 00:03 2024-02-20T07:45:03.889 INFO:teuthology.orchestra.run.smithi133.stdout:CentOS-9 - BaseOS 3.9 MB/s | 2.0 MB 00:00 2024-02-20T07:45:05.409 INFO:teuthology.orchestra.run.smithi183.stdout:CentOS-9 - BaseOS 4.2 MB/s | 2.0 MB 00:00 2024-02-20T07:45:06.229 INFO:teuthology.orchestra.run.smithi133.stdout:CentOS Stream 9 - BaseOS 3.9 MB/s | 8.0 MB 00:02 2024-02-20T07:45:07.425 INFO:teuthology.orchestra.run.smithi183.stdout:CentOS Stream 9 - BaseOS 4.7 MB/s | 8.0 MB 00:01 2024-02-20T07:45:08.055 INFO:teuthology.orchestra.run.smithi133.stdout:CentOS Stream 9 - AppStream 26 MB/s | 19 MB 00:00 2024-02-20T07:45:09.503 INFO:teuthology.orchestra.run.smithi183.stdout:CentOS Stream 9 - AppStream 19 MB/s | 19 MB 00:00 2024-02-20T07:45:13.047 INFO:teuthology.orchestra.run.smithi133.stdout:CentOS Stream 9 - CRB 4.2 MB/s | 6.1 MB 00:01 2024-02-20T07:45:13.526 INFO:teuthology.orchestra.run.smithi183.stdout:CentOS Stream 9 - CRB 12 MB/s | 6.1 MB 00:00 2024-02-20T07:45:14.599 INFO:teuthology.orchestra.run.smithi133.stdout:CentOS Stream 9 - Extras packages 35 kB/s | 15 kB 00:00 2024-02-20T07:45:15.185 INFO:teuthology.orchestra.run.smithi183.stdout:CentOS Stream 9 - Extras packages 28 kB/s | 15 kB 00:00 2024-02-20T07:45:15.783 INFO:teuthology.orchestra.run.smithi133.stdout:Extra Packages for Enterprise Linux 19 MB/s | 20 MB 00:01 2024-02-20T07:45:16.561 INFO:teuthology.orchestra.run.smithi183.stdout:Extra Packages for Enterprise Linux 16 MB/s | 20 MB 00:01 2024-02-20T07:45:20.600 INFO:teuthology.orchestra.run.smithi133.stdout:lab-extras 36 kB/s | 1.7 kB 00:00 2024-02-20T07:45:21.528 INFO:teuthology.orchestra.run.smithi183.stdout:lab-extras 12 kB/s | 1.7 kB 00:00 2024-02-20T07:45:23.203 INFO:teuthology.orchestra.run.smithi133.stdout:Dependencies resolved. 2024-02-20T07:45:23.210 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2024-02-20T07:45:23.211 INFO:teuthology.orchestra.run.smithi133.stdout: Package Arch Version Repository Size 2024-02-20T07:45:23.211 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2024-02-20T07:45:23.211 INFO:teuthology.orchestra.run.smithi133.stdout:Installing: 2024-02-20T07:45:23.211 INFO:teuthology.orchestra.run.smithi133.stdout: ceph-radosgw x86_64 2:17.2.7-521.gc80e7168.el9 ceph 14 M 2024-02-20T07:45:23.211 INFO:teuthology.orchestra.run.smithi133.stdout:Upgrading: 2024-02-20T07:45:23.211 INFO:teuthology.orchestra.run.smithi133.stdout: librados2 x86_64 2:17.2.7-521.gc80e7168.el9 ceph 3.1 M 2024-02-20T07:45:23.211 INFO:teuthology.orchestra.run.smithi133.stdout: librbd1 x86_64 2:17.2.7-521.gc80e7168.el9 ceph 2.9 M 2024-02-20T07:45:23.211 INFO:teuthology.orchestra.run.smithi133.stdout:Installing dependencies: 2024-02-20T07:45:23.211 INFO:teuthology.orchestra.run.smithi133.stdout: ceph-base x86_64 2:17.2.7-521.gc80e7168.el9 ceph 5.0 M 2024-02-20T07:45:23.211 INFO:teuthology.orchestra.run.smithi133.stdout: ceph-common x86_64 2:17.2.7-521.gc80e7168.el9 ceph 23 M 2024-02-20T07:45:23.211 INFO:teuthology.orchestra.run.smithi133.stdout: ceph-selinux x86_64 2:17.2.7-521.gc80e7168.el9 ceph 25 k 2024-02-20T07:45:23.212 INFO:teuthology.orchestra.run.smithi133.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-02-20T07:45:23.212 INFO:teuthology.orchestra.run.smithi133.stdout: libcephfs2 x86_64 2:17.2.7-521.gc80e7168.el9 ceph 635 k 2024-02-20T07:45:23.212 INFO:teuthology.orchestra.run.smithi133.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-02-20T07:45:23.212 INFO:teuthology.orchestra.run.smithi133.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-02-20T07:45:23.212 INFO:teuthology.orchestra.run.smithi133.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-02-20T07:45:23.212 INFO:teuthology.orchestra.run.smithi133.stdout: libradosstriper1 x86_64 2:17.2.7-521.gc80e7168.el9 ceph 457 k 2024-02-20T07:45:23.212 INFO:teuthology.orchestra.run.smithi133.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-02-20T07:45:23.212 INFO:teuthology.orchestra.run.smithi133.stdout: librgw2 x86_64 2:17.2.7-521.gc80e7168.el9 ceph 4.9 M 2024-02-20T07:45:23.212 INFO:teuthology.orchestra.run.smithi133.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-02-20T07:45:23.212 INFO:teuthology.orchestra.run.smithi133.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-02-20T07:45:23.212 INFO:teuthology.orchestra.run.smithi133.stdout: python3-ceph-argparse x86_64 2:17.2.7-521.gc80e7168.el9 ceph 45 k 2024-02-20T07:45:23.212 INFO:teuthology.orchestra.run.smithi133.stdout: python3-ceph-common x86_64 2:17.2.7-521.gc80e7168.el9 ceph 116 k 2024-02-20T07:45:23.212 INFO:teuthology.orchestra.run.smithi133.stdout: python3-cephfs x86_64 2:17.2.7-521.gc80e7168.el9 ceph 156 k 2024-02-20T07:45:23.212 INFO:teuthology.orchestra.run.smithi133.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-02-20T07:45:23.213 INFO:teuthology.orchestra.run.smithi133.stdout: python3-rados x86_64 2:17.2.7-521.gc80e7168.el9 ceph 320 k 2024-02-20T07:45:23.213 INFO:teuthology.orchestra.run.smithi133.stdout: python3-rbd x86_64 2:17.2.7-521.gc80e7168.el9 ceph 293 k 2024-02-20T07:45:23.213 INFO:teuthology.orchestra.run.smithi133.stdout: python3-rgw x86_64 2:17.2.7-521.gc80e7168.el9 ceph 99 k 2024-02-20T07:45:23.213 INFO:teuthology.orchestra.run.smithi133.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-02-20T07:45:23.213 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-02-20T07:45:23.213 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction Summary 2024-02-20T07:45:23.213 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2024-02-20T07:45:23.213 INFO:teuthology.orchestra.run.smithi133.stdout:Install 22 Packages 2024-02-20T07:45:23.213 INFO:teuthology.orchestra.run.smithi133.stdout:Upgrade 2 Packages 2024-02-20T07:45:23.213 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-02-20T07:45:23.214 INFO:teuthology.orchestra.run.smithi133.stdout:Total download size: 58 M 2024-02-20T07:45:23.214 INFO:teuthology.orchestra.run.smithi133.stdout:Downloading Packages: 2024-02-20T07:45:24.142 INFO:teuthology.orchestra.run.smithi183.stdout:Dependencies resolved. 2024-02-20T07:45:24.165 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-02-20T07:45:24.165 INFO:teuthology.orchestra.run.smithi183.stdout: Package Arch Version Repository Size 2024-02-20T07:45:24.165 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-02-20T07:45:24.165 INFO:teuthology.orchestra.run.smithi183.stdout:Installing: 2024-02-20T07:45:24.165 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-radosgw x86_64 2:17.2.7-521.gc80e7168.el9 ceph 14 M 2024-02-20T07:45:24.165 INFO:teuthology.orchestra.run.smithi183.stdout:Upgrading: 2024-02-20T07:45:24.165 INFO:teuthology.orchestra.run.smithi183.stdout: librados2 x86_64 2:17.2.7-521.gc80e7168.el9 ceph 3.1 M 2024-02-20T07:45:24.165 INFO:teuthology.orchestra.run.smithi183.stdout: librbd1 x86_64 2:17.2.7-521.gc80e7168.el9 ceph 2.9 M 2024-02-20T07:45:24.165 INFO:teuthology.orchestra.run.smithi183.stdout:Installing dependencies: 2024-02-20T07:45:24.165 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-base x86_64 2:17.2.7-521.gc80e7168.el9 ceph 5.0 M 2024-02-20T07:45:24.165 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-common x86_64 2:17.2.7-521.gc80e7168.el9 ceph 23 M 2024-02-20T07:45:24.165 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-selinux x86_64 2:17.2.7-521.gc80e7168.el9 ceph 25 k 2024-02-20T07:45:24.166 INFO:teuthology.orchestra.run.smithi183.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-02-20T07:45:24.166 INFO:teuthology.orchestra.run.smithi183.stdout: libcephfs2 x86_64 2:17.2.7-521.gc80e7168.el9 ceph 635 k 2024-02-20T07:45:24.166 INFO:teuthology.orchestra.run.smithi183.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-02-20T07:45:24.166 INFO:teuthology.orchestra.run.smithi183.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-02-20T07:45:24.166 INFO:teuthology.orchestra.run.smithi183.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-02-20T07:45:24.166 INFO:teuthology.orchestra.run.smithi183.stdout: libradosstriper1 x86_64 2:17.2.7-521.gc80e7168.el9 ceph 457 k 2024-02-20T07:45:24.166 INFO:teuthology.orchestra.run.smithi183.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-02-20T07:45:24.166 INFO:teuthology.orchestra.run.smithi183.stdout: librgw2 x86_64 2:17.2.7-521.gc80e7168.el9 ceph 4.9 M 2024-02-20T07:45:24.166 INFO:teuthology.orchestra.run.smithi183.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-02-20T07:45:24.166 INFO:teuthology.orchestra.run.smithi183.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-02-20T07:45:24.166 INFO:teuthology.orchestra.run.smithi183.stdout: python3-ceph-argparse x86_64 2:17.2.7-521.gc80e7168.el9 ceph 45 k 2024-02-20T07:45:24.166 INFO:teuthology.orchestra.run.smithi183.stdout: python3-ceph-common x86_64 2:17.2.7-521.gc80e7168.el9 ceph 116 k 2024-02-20T07:45:24.166 INFO:teuthology.orchestra.run.smithi183.stdout: python3-cephfs x86_64 2:17.2.7-521.gc80e7168.el9 ceph 156 k 2024-02-20T07:45:24.166 INFO:teuthology.orchestra.run.smithi183.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-02-20T07:45:24.166 INFO:teuthology.orchestra.run.smithi183.stdout: python3-rados x86_64 2:17.2.7-521.gc80e7168.el9 ceph 320 k 2024-02-20T07:45:24.167 INFO:teuthology.orchestra.run.smithi183.stdout: python3-rbd x86_64 2:17.2.7-521.gc80e7168.el9 ceph 293 k 2024-02-20T07:45:24.167 INFO:teuthology.orchestra.run.smithi183.stdout: python3-rgw x86_64 2:17.2.7-521.gc80e7168.el9 ceph 99 k 2024-02-20T07:45:24.167 INFO:teuthology.orchestra.run.smithi183.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-02-20T07:45:24.167 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-02-20T07:45:24.167 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction Summary 2024-02-20T07:45:24.167 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-02-20T07:45:24.167 INFO:teuthology.orchestra.run.smithi183.stdout:Install 22 Packages 2024-02-20T07:45:24.167 INFO:teuthology.orchestra.run.smithi183.stdout:Upgrade 2 Packages 2024-02-20T07:45:24.167 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-02-20T07:45:24.168 INFO:teuthology.orchestra.run.smithi183.stdout:Total download size: 58 M 2024-02-20T07:45:24.168 INFO:teuthology.orchestra.run.smithi183.stdout:Downloading Packages: 2024-02-20T07:45:24.708 INFO:teuthology.orchestra.run.smithi133.stdout:(1/24): ceph-radosgw-17.2.7-521.gc80e7168.el9.x 10 MB/s | 14 MB 00:01 2024-02-20T07:45:24.783 INFO:teuthology.orchestra.run.smithi133.stdout:(2/24): ceph-base-17.2.7-521.gc80e7168.el9.x86_ 3.6 MB/s | 5.0 MB 00:01 2024-02-20T07:45:24.809 INFO:teuthology.orchestra.run.smithi133.stdout:(3/24): ceph-selinux-17.2.7-521.gc80e7168.el9.x 251 kB/s | 25 kB 00:00 2024-02-20T07:45:24.876 INFO:teuthology.orchestra.run.smithi133.stdout:(4/24): libradosstriper1-17.2.7-521.gc80e7168.e 6.7 MB/s | 457 kB 00:00 2024-02-20T07:45:24.909 INFO:teuthology.orchestra.run.smithi133.stdout:(5/24): libcephfs2-17.2.7-521.gc80e7168.el9.x86 4.9 MB/s | 635 kB 00:00 2024-02-20T07:45:25.010 INFO:teuthology.orchestra.run.smithi133.stdout:(6/24): python3-ceph-argparse-17.2.7-521.gc80e7 451 kB/s | 45 kB 00:00 2024-02-20T07:45:25.110 INFO:teuthology.orchestra.run.smithi133.stdout:(7/24): python3-ceph-common-17.2.7-521.gc80e716 1.1 MB/s | 116 kB 00:00 2024-02-20T07:45:25.218 INFO:teuthology.orchestra.run.smithi133.stdout:(8/24): python3-cephfs-17.2.7-521.gc80e7168.el9 1.4 MB/s | 156 kB 00:00 2024-02-20T07:45:25.277 INFO:teuthology.orchestra.run.smithi133.stdout:(9/24): librgw2-17.2.7-521.gc80e7168.el9.x86_64 12 MB/s | 4.9 MB 00:00 2024-02-20T07:45:25.302 INFO:teuthology.orchestra.run.smithi133.stdout:(10/24): python3-rados-17.2.7-521.gc80e7168.el9 3.7 MB/s | 320 kB 00:00 2024-02-20T07:45:25.344 INFO:teuthology.orchestra.run.smithi133.stdout:(11/24): python3-rbd-17.2.7-521.gc80e7168.el9.x 4.3 MB/s | 293 kB 00:00 2024-02-20T07:45:25.403 INFO:teuthology.orchestra.run.smithi133.stdout:(12/24): python3-rgw-17.2.7-521.gc80e7168.el9.x 991 kB/s | 99 kB 00:00 2024-02-20T07:45:25.495 INFO:teuthology.orchestra.run.smithi133.stdout:(13/24): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.0 MB/s | 159 kB 00:00 2024-02-20T07:45:25.520 INFO:teuthology.orchestra.run.smithi133.stdout:(14/24): librabbitmq-0.11.0-7.el9.x86_64.rpm 382 kB/s | 44 kB 00:00 2024-02-20T07:45:25.562 INFO:teuthology.orchestra.run.smithi133.stdout:(15/24): librdkafka-1.6.1-102.el9.x86_64.rpm 9.7 MB/s | 661 kB 00:00 2024-02-20T07:45:25.587 INFO:teuthology.orchestra.run.smithi133.stdout:(16/24): lttng-ust-2.12.0-6.el9.x86_64.rpm 4.1 MB/s | 282 kB 00:00 2024-02-20T07:45:25.612 INFO:teuthology.orchestra.run.smithi133.stdout:(17/24): python3-prettytable-0.7.2-27.el9.noarc 826 kB/s | 41 kB 00:00 2024-02-20T07:45:25.680 INFO:teuthology.orchestra.run.smithi133.stdout:(18/24): liboath-2.6.7-2.el9.x86_64.rpm 729 kB/s | 49 kB 00:00 2024-02-20T07:45:25.705 INFO:teuthology.orchestra.run.smithi133.stdout:(19/24): gperftools-libs-2.9.1-2.el9.x86_64.rpm 2.6 MB/s | 309 kB 00:00 2024-02-20T07:45:25.730 INFO:teuthology.orchestra.run.smithi133.stdout:(20/24): libunwind-1.6.2-1.el9.x86_64.rpm 1.3 MB/s | 67 kB 00:00 2024-02-20T07:45:25.830 INFO:teuthology.orchestra.run.smithi133.stdout:(21/24): thrift-0.15.0-2.el9.x86_64.rpm 13 MB/s | 1.6 MB 00:00 2024-02-20T07:45:26.664 INFO:teuthology.orchestra.run.smithi133.stdout:(22/24): librbd1-17.2.7-521.gc80e7168.el9.x86_6 3.5 MB/s | 2.9 MB 00:00 2024-02-20T07:45:26.745 INFO:teuthology.orchestra.run.smithi183.stdout:(1/24): ceph-base-17.2.7-521.gc80e7168.el9.x86_ 2.0 MB/s | 5.0 MB 00:02 2024-02-20T07:45:26.887 INFO:teuthology.orchestra.run.smithi183.stdout:(2/24): ceph-radosgw-17.2.7-521.gc80e7168.el9.x 5.3 MB/s | 14 MB 00:02 2024-02-20T07:45:26.912 INFO:teuthology.orchestra.run.smithi183.stdout:(3/24): ceph-selinux-17.2.7-521.gc80e7168.el9.x 151 kB/s | 25 kB 00:00 2024-02-20T07:45:26.922 INFO:teuthology.orchestra.run.smithi133.stdout:(23/24): ceph-common-17.2.7-521.gc80e7168.el9.x 6.3 MB/s | 23 MB 00:03 2024-02-20T07:45:26.946 INFO:teuthology.orchestra.run.smithi183.stdout:(4/24): libcephfs2-17.2.7-521.gc80e7168.el9.x86 11 MB/s | 635 kB 00:00 2024-02-20T07:45:26.983 INFO:teuthology.orchestra.run.smithi133.stdout:(24/24): librados2-17.2.7-521.gc80e7168.el9.x86 2.5 MB/s | 3.1 MB 00:01 2024-02-20T07:45:26.983 INFO:teuthology.orchestra.run.smithi133.stdout:-------------------------------------------------------------------------------- 2024-02-20T07:45:26.984 INFO:teuthology.orchestra.run.smithi133.stdout:Total 15 MB/s | 58 MB 00:03 2024-02-20T07:45:26.984 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction check 2024-02-20T07:45:27.013 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction check succeeded. 2024-02-20T07:45:27.013 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction test 2024-02-20T07:45:27.030 INFO:teuthology.orchestra.run.smithi183.stdout:(5/24): libradosstriper1-17.2.7-521.gc80e7168.e 3.8 MB/s | 457 kB 00:00 2024-02-20T07:45:27.113 INFO:teuthology.orchestra.run.smithi183.stdout:(6/24): python3-ceph-argparse-17.2.7-521.gc80e7 541 kB/s | 45 kB 00:00 2024-02-20T07:45:27.188 INFO:teuthology.orchestra.run.smithi183.stdout:(7/24): python3-ceph-common-17.2.7-521.gc80e716 1.5 MB/s | 116 kB 00:00 2024-02-20T07:45:27.255 INFO:teuthology.orchestra.run.smithi183.stdout:(8/24): librgw2-17.2.7-521.gc80e7168.el9.x86_64 16 MB/s | 4.9 MB 00:00 2024-02-20T07:45:27.281 INFO:teuthology.orchestra.run.smithi183.stdout:(9/24): python3-cephfs-17.2.7-521.gc80e7168.el9 1.7 MB/s | 156 kB 00:00 2024-02-20T07:45:27.306 INFO:teuthology.orchestra.run.smithi183.stdout:(10/24): python3-rados-17.2.7-521.gc80e7168.el9 6.2 MB/s | 320 kB 00:00 2024-02-20T07:45:27.348 INFO:teuthology.orchestra.run.smithi183.stdout:(11/24): python3-rgw-17.2.7-521.gc80e7168.el9.x 2.3 MB/s | 99 kB 00:00 2024-02-20T07:45:27.373 INFO:teuthology.orchestra.run.smithi183.stdout:(12/24): python3-rbd-17.2.7-521.gc80e7168.el9.x 3.1 MB/s | 293 kB 00:00 2024-02-20T07:45:27.462 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction test succeeded. 2024-02-20T07:45:27.463 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction 2024-02-20T07:45:27.473 INFO:teuthology.orchestra.run.smithi183.stdout:(13/24): librabbitmq-0.11.0-7.el9.x86_64.rpm 442 kB/s | 44 kB 00:00 2024-02-20T07:45:27.532 INFO:teuthology.orchestra.run.smithi183.stdout:(14/24): libpmemobj-1.12.1-1.el9.x86_64.rpm 865 kB/s | 159 kB 00:00 2024-02-20T07:45:27.657 INFO:teuthology.orchestra.run.smithi183.stdout:(15/24): librdkafka-1.6.1-102.el9.x86_64.rpm 3.5 MB/s | 661 kB 00:00 2024-02-20T07:45:27.682 INFO:teuthology.orchestra.run.smithi183.stdout:(16/24): lttng-ust-2.12.0-6.el9.x86_64.rpm 1.8 MB/s | 282 kB 00:00 2024-02-20T07:45:27.708 INFO:teuthology.orchestra.run.smithi183.stdout:(17/24): python3-prettytable-0.7.2-27.el9.noarc 828 kB/s | 41 kB 00:00 2024-02-20T07:45:27.800 INFO:teuthology.orchestra.run.smithi183.stdout:(18/24): liboath-2.6.7-2.el9.x86_64.rpm 530 kB/s | 49 kB 00:00 2024-02-20T07:45:28.033 INFO:teuthology.orchestra.run.smithi183.stdout:(19/24): libunwind-1.6.2-1.el9.x86_64.rpm 289 kB/s | 67 kB 00:00 2024-02-20T07:45:28.058 INFO:teuthology.orchestra.run.smithi183.stdout:(20/24): gperftools-libs-2.9.1-2.el9.x86_64.rpm 822 kB/s | 309 kB 00:00 2024-02-20T07:45:28.396 INFO:teuthology.orchestra.run.smithi183.stdout:(21/24): thrift-0.15.0-2.el9.x86_64.rpm 4.4 MB/s | 1.6 MB 00:00 2024-02-20T07:45:28.634 INFO:teuthology.orchestra.run.smithi183.stdout:(22/24): ceph-common-17.2.7-521.gc80e7168.el9.x 5.1 MB/s | 23 MB 00:04 2024-02-20T07:45:28.675 INFO:teuthology.orchestra.run.smithi133.stdout: Preparing : 1/1 2024-02-20T07:45:28.726 INFO:teuthology.orchestra.run.smithi183.stdout:(23/24): librbd1-17.2.7-521.gc80e7168.el9.x86_6 8.8 MB/s | 2.9 MB 00:00 2024-02-20T07:45:28.776 INFO:teuthology.orchestra.run.smithi183.stdout:(24/24): librados2-17.2.7-521.gc80e7168.el9.x86 4.3 MB/s | 3.1 MB 00:00 2024-02-20T07:45:28.780 INFO:teuthology.orchestra.run.smithi183.stdout:-------------------------------------------------------------------------------- 2024-02-20T07:45:28.780 INFO:teuthology.orchestra.run.smithi183.stdout:Total 13 MB/s | 58 MB 00:04 2024-02-20T07:45:28.781 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction check 2024-02-20T07:45:28.814 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction check succeeded. 2024-02-20T07:45:28.814 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction test 2024-02-20T07:45:29.013 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/26 2024-02-20T07:45:29.085 INFO:teuthology.orchestra.run.smithi133.stdout: Upgrading : librados2-2:17.2.7-521.gc80e7168.el9.x86_64 2/26 2024-02-20T07:45:29.169 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: librados2-2:17.2.7-521.gc80e7168.el9.x86_64 2/26 2024-02-20T07:45:29.246 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-rados-2:17.2.7-521.gc80e7168.el9.x86_64 3/26 2024-02-20T07:45:29.260 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction test succeeded. 2024-02-20T07:45:29.260 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction 2024-02-20T07:45:29.270 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : libcephfs2-2:17.2.7-521.gc80e7168.el9.x86_64 4/26 2024-02-20T07:45:29.351 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: libcephfs2-2:17.2.7-521.gc80e7168.el9.x86_64 4/26 2024-02-20T07:45:29.395 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : thrift-0.15.0-2.el9.x86_64 5/26 2024-02-20T07:45:29.433 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 6/26 2024-02-20T07:45:29.865 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 7/26 2024-02-20T07:45:29.898 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : librgw2-2:17.2.7-521.gc80e7168.el9.x86_64 8/26 2024-02-20T07:45:29.943 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: librgw2-2:17.2.7-521.gc80e7168.el9.x86_64 8/26 2024-02-20T07:45:29.972 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : liboath-2.6.7-2.el9.x86_64 9/26 2024-02-20T07:45:30.015 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-ceph-argparse-2:17.2.7-521.gc80e7168.el9.x 10/26 2024-02-20T07:45:30.055 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-cephfs-2:17.2.7-521.gc80e7168.el9.x86_64 11/26 2024-02-20T07:45:30.123 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-rgw-2:17.2.7-521.gc80e7168.el9.x86_64 12/26 2024-02-20T07:45:30.150 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : libradosstriper1-2:17.2.7-521.gc80e7168.el9.x86_64 13/26 2024-02-20T07:45:30.207 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: libradosstriper1-2:17.2.7-521.gc80e7168.el9.x86_64 13/26 2024-02-20T07:45:30.247 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 14/26 2024-02-20T07:45:30.292 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 15/26 2024-02-20T07:45:30.324 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 16/26 2024-02-20T07:45:30.496 INFO:teuthology.orchestra.run.smithi183.stdout: Preparing : 1/1 2024-02-20T07:45:30.626 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 17/26 2024-02-20T07:45:30.654 INFO:teuthology.orchestra.run.smithi133.stdout: Upgrading : librbd1-2:17.2.7-521.gc80e7168.el9.x86_64 18/26 2024-02-20T07:45:30.726 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: librbd1-2:17.2.7-521.gc80e7168.el9.x86_64 18/26 2024-02-20T07:45:30.792 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-rbd-2:17.2.7-521.gc80e7168.el9.x86_64 19/26 2024-02-20T07:45:30.823 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-ceph-common-2:17.2.7-521.gc80e7168.el9.x86 20/26 2024-02-20T07:45:30.853 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/26 2024-02-20T07:45:30.903 INFO:teuthology.orchestra.run.smithi183.stdout: Upgrading : librados2-2:17.2.7-521.gc80e7168.el9.x86_64 2/26 2024-02-20T07:45:30.987 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: librados2-2:17.2.7-521.gc80e7168.el9.x86_64 2/26 2024-02-20T07:45:31.073 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-rados-2:17.2.7-521.gc80e7168.el9.x86_64 3/26 2024-02-20T07:45:31.104 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : libcephfs2-2:17.2.7-521.gc80e7168.el9.x86_64 4/26 2024-02-20T07:45:31.184 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: libcephfs2-2:17.2.7-521.gc80e7168.el9.x86_64 4/26 2024-02-20T07:45:31.229 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : thrift-0.15.0-2.el9.x86_64 5/26 2024-02-20T07:45:31.267 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 6/26 2024-02-20T07:45:31.702 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 7/26 2024-02-20T07:45:31.732 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : librgw2-2:17.2.7-521.gc80e7168.el9.x86_64 8/26 2024-02-20T07:45:31.794 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: librgw2-2:17.2.7-521.gc80e7168.el9.x86_64 8/26 2024-02-20T07:45:31.823 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : liboath-2.6.7-2.el9.x86_64 9/26 2024-02-20T07:45:31.874 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-ceph-argparse-2:17.2.7-521.gc80e7168.el9.x 10/26 2024-02-20T07:45:31.913 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-cephfs-2:17.2.7-521.gc80e7168.el9.x86_64 11/26 2024-02-20T07:45:31.974 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-rgw-2:17.2.7-521.gc80e7168.el9.x86_64 12/26 2024-02-20T07:45:32.001 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : libradosstriper1-2:17.2.7-521.gc80e7168.el9.x86_64 13/26 2024-02-20T07:45:32.067 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: libradosstriper1-2:17.2.7-521.gc80e7168.el9.x86_64 13/26 2024-02-20T07:45:32.115 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 14/26 2024-02-20T07:45:32.151 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 15/26 2024-02-20T07:45:32.184 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 16/26 2024-02-20T07:45:32.481 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 17/26 2024-02-20T07:45:32.514 INFO:teuthology.orchestra.run.smithi183.stdout: Upgrading : librbd1-2:17.2.7-521.gc80e7168.el9.x86_64 18/26 2024-02-20T07:45:32.602 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: librbd1-2:17.2.7-521.gc80e7168.el9.x86_64 18/26 2024-02-20T07:45:32.676 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-rbd-2:17.2.7-521.gc80e7168.el9.x86_64 19/26 2024-02-20T07:45:32.707 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-ceph-common-2:17.2.7-521.gc80e7168.el9.x86 20/26 2024-02-20T07:45:33.224 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: ceph-common-2:17.2.7-521.gc80e7168.el9.x86_64 21/26 2024-02-20T07:45:33.270 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : ceph-common-2:17.2.7-521.gc80e7168.el9.x86_64 21/26 2024-02-20T07:45:33.811 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: ceph-common-2:17.2.7-521.gc80e7168.el9.x86_64 21/26 2024-02-20T07:45:33.848 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : ceph-base-2:17.2.7-521.gc80e7168.el9.x86_64 22/26 2024-02-20T07:45:33.922 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: ceph-base-2:17.2.7-521.gc80e7168.el9.x86_64 22/26 2024-02-20T07:45:33.922 INFO:teuthology.orchestra.run.smithi133.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-02-20T07:45:33.922 INFO:teuthology.orchestra.run.smithi133.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-02-20T07:45:33.922 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-02-20T07:45:33.957 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : ceph-selinux-2:17.2.7-521.gc80e7168.el9.x86_64 23/26 2024-02-20T07:45:35.171 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: ceph-common-2:17.2.7-521.gc80e7168.el9.x86_64 21/26 2024-02-20T07:45:35.204 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : ceph-common-2:17.2.7-521.gc80e7168.el9.x86_64 21/26 2024-02-20T07:45:35.743 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: ceph-common-2:17.2.7-521.gc80e7168.el9.x86_64 21/26 2024-02-20T07:45:35.781 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : ceph-base-2:17.2.7-521.gc80e7168.el9.x86_64 22/26 2024-02-20T07:45:35.871 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: ceph-base-2:17.2.7-521.gc80e7168.el9.x86_64 22/26 2024-02-20T07:45:35.871 INFO:teuthology.orchestra.run.smithi183.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-02-20T07:45:35.871 INFO:teuthology.orchestra.run.smithi183.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-02-20T07:45:35.872 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-02-20T07:45:35.907 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : ceph-selinux-2:17.2.7-521.gc80e7168.el9.x86_64 23/26 2024-02-20T07:45:43.585 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: ceph-selinux-2:17.2.7-521.gc80e7168.el9.x86_64 23/26 2024-02-20T07:45:43.585 INFO:teuthology.orchestra.run.smithi133.stdout:skipping the directory /sys 2024-02-20T07:45:43.586 INFO:teuthology.orchestra.run.smithi133.stdout:skipping the directory /proc 2024-02-20T07:45:43.586 INFO:teuthology.orchestra.run.smithi133.stdout:skipping the directory /mnt 2024-02-20T07:45:43.586 INFO:teuthology.orchestra.run.smithi133.stdout:skipping the directory /var/tmp 2024-02-20T07:45:43.586 INFO:teuthology.orchestra.run.smithi133.stdout:skipping the directory /home 2024-02-20T07:45:43.586 INFO:teuthology.orchestra.run.smithi133.stdout:skipping the directory /root 2024-02-20T07:45:43.586 INFO:teuthology.orchestra.run.smithi133.stdout:skipping the directory /tmp 2024-02-20T07:45:43.586 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-02-20T07:45:44.841 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : ceph-radosgw-2:17.2.7-521.gc80e7168.el9.x86_64 24/26 2024-02-20T07:45:44.918 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: ceph-radosgw-2:17.2.7-521.gc80e7168.el9.x86_64 24/26 2024-02-20T07:45:44.918 INFO:teuthology.orchestra.run.smithi133.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-02-20T07:45:44.918 INFO:teuthology.orchestra.run.smithi133.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-02-20T07:45:44.918 INFO:teuthology.orchestra.run.smithi133.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-02-20T07:45:44.918 INFO:teuthology.orchestra.run.smithi133.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-02-20T07:45:44.918 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-02-20T07:45:44.920 INFO:teuthology.orchestra.run.smithi133.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 25/26 2024-02-20T07:45:45.013 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 25/26 2024-02-20T07:45:45.013 INFO:teuthology.orchestra.run.smithi133.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 26/26 2024-02-20T07:45:45.571 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: ceph-selinux-2:17.2.7-521.gc80e7168.el9.x86_64 23/26 2024-02-20T07:45:45.571 INFO:teuthology.orchestra.run.smithi183.stdout:skipping the directory /sys 2024-02-20T07:45:45.571 INFO:teuthology.orchestra.run.smithi183.stdout:skipping the directory /proc 2024-02-20T07:45:45.571 INFO:teuthology.orchestra.run.smithi183.stdout:skipping the directory /mnt 2024-02-20T07:45:45.572 INFO:teuthology.orchestra.run.smithi183.stdout:skipping the directory /var/tmp 2024-02-20T07:45:45.572 INFO:teuthology.orchestra.run.smithi183.stdout:skipping the directory /home 2024-02-20T07:45:45.572 INFO:teuthology.orchestra.run.smithi183.stdout:skipping the directory /root 2024-02-20T07:45:45.572 INFO:teuthology.orchestra.run.smithi183.stdout:skipping the directory /tmp 2024-02-20T07:45:45.572 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-02-20T07:45:46.242 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 26/26 2024-02-20T07:45:46.242 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : ceph-base-2:17.2.7-521.gc80e7168.el9.x86_64 1/26 2024-02-20T07:45:46.242 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : ceph-common-2:17.2.7-521.gc80e7168.el9.x86_64 2/26 2024-02-20T07:45:46.242 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : ceph-radosgw-2:17.2.7-521.gc80e7168.el9.x86_64 3/26 2024-02-20T07:45:46.242 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : ceph-selinux-2:17.2.7-521.gc80e7168.el9.x86_64 4/26 2024-02-20T07:45:46.242 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : libcephfs2-2:17.2.7-521.gc80e7168.el9.x86_64 5/26 2024-02-20T07:45:46.243 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : libradosstriper1-2:17.2.7-521.gc80e7168.el9.x86_64 6/26 2024-02-20T07:45:46.243 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : librgw2-2:17.2.7-521.gc80e7168.el9.x86_64 7/26 2024-02-20T07:45:46.243 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-ceph-argparse-2:17.2.7-521.gc80e7168.el9.x 8/26 2024-02-20T07:45:46.243 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-ceph-common-2:17.2.7-521.gc80e7168.el9.x86 9/26 2024-02-20T07:45:46.243 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-cephfs-2:17.2.7-521.gc80e7168.el9.x86_64 10/26 2024-02-20T07:45:46.243 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-rados-2:17.2.7-521.gc80e7168.el9.x86_64 11/26 2024-02-20T07:45:46.243 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-rbd-2:17.2.7-521.gc80e7168.el9.x86_64 12/26 2024-02-20T07:45:46.243 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-rgw-2:17.2.7-521.gc80e7168.el9.x86_64 13/26 2024-02-20T07:45:46.243 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 14/26 2024-02-20T07:45:46.243 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 15/26 2024-02-20T07:45:46.243 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 16/26 2024-02-20T07:45:46.243 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 17/26 2024-02-20T07:45:46.243 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 18/26 2024-02-20T07:45:46.243 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 19/26 2024-02-20T07:45:46.244 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 20/26 2024-02-20T07:45:46.244 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 21/26 2024-02-20T07:45:46.244 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 22/26 2024-02-20T07:45:46.244 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : librados2-2:17.2.7-521.gc80e7168.el9.x86_64 23/26 2024-02-20T07:45:46.244 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 24/26 2024-02-20T07:45:46.244 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : librbd1-2:17.2.7-521.gc80e7168.el9.x86_64 25/26 2024-02-20T07:45:46.861 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : ceph-radosgw-2:17.2.7-521.gc80e7168.el9.x86_64 24/26 2024-02-20T07:45:46.942 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: ceph-radosgw-2:17.2.7-521.gc80e7168.el9.x86_64 24/26 2024-02-20T07:45:46.942 INFO:teuthology.orchestra.run.smithi183.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-02-20T07:45:46.942 INFO:teuthology.orchestra.run.smithi183.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-02-20T07:45:46.942 INFO:teuthology.orchestra.run.smithi183.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-02-20T07:45:46.942 INFO:teuthology.orchestra.run.smithi183.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-02-20T07:45:46.942 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-02-20T07:45:46.944 INFO:teuthology.orchestra.run.smithi183.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 25/26 2024-02-20T07:45:47.049 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 25/26 2024-02-20T07:45:47.049 INFO:teuthology.orchestra.run.smithi183.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 26/26 2024-02-20T07:45:47.194 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 26/26 2024-02-20T07:45:47.194 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-02-20T07:45:47.194 INFO:teuthology.orchestra.run.smithi133.stdout:Upgraded: 2024-02-20T07:45:47.194 INFO:teuthology.orchestra.run.smithi133.stdout: librados2-2:17.2.7-521.gc80e7168.el9.x86_64 2024-02-20T07:45:47.195 INFO:teuthology.orchestra.run.smithi133.stdout: librbd1-2:17.2.7-521.gc80e7168.el9.x86_64 2024-02-20T07:45:47.195 INFO:teuthology.orchestra.run.smithi133.stdout:Installed: 2024-02-20T07:45:47.195 INFO:teuthology.orchestra.run.smithi133.stdout: ceph-base-2:17.2.7-521.gc80e7168.el9.x86_64 2024-02-20T07:45:47.195 INFO:teuthology.orchestra.run.smithi133.stdout: ceph-common-2:17.2.7-521.gc80e7168.el9.x86_64 2024-02-20T07:45:47.195 INFO:teuthology.orchestra.run.smithi133.stdout: ceph-radosgw-2:17.2.7-521.gc80e7168.el9.x86_64 2024-02-20T07:45:47.195 INFO:teuthology.orchestra.run.smithi133.stdout: ceph-selinux-2:17.2.7-521.gc80e7168.el9.x86_64 2024-02-20T07:45:47.196 INFO:teuthology.orchestra.run.smithi133.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-02-20T07:45:47.196 INFO:teuthology.orchestra.run.smithi133.stdout: libcephfs2-2:17.2.7-521.gc80e7168.el9.x86_64 2024-02-20T07:45:47.196 INFO:teuthology.orchestra.run.smithi133.stdout: liboath-2.6.7-2.el9.x86_64 2024-02-20T07:45:47.196 INFO:teuthology.orchestra.run.smithi133.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-02-20T07:45:47.196 INFO:teuthology.orchestra.run.smithi133.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-02-20T07:45:47.197 INFO:teuthology.orchestra.run.smithi133.stdout: libradosstriper1-2:17.2.7-521.gc80e7168.el9.x86_64 2024-02-20T07:45:47.197 INFO:teuthology.orchestra.run.smithi133.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-02-20T07:45:47.197 INFO:teuthology.orchestra.run.smithi133.stdout: librgw2-2:17.2.7-521.gc80e7168.el9.x86_64 2024-02-20T07:45:47.197 INFO:teuthology.orchestra.run.smithi133.stdout: libunwind-1.6.2-1.el9.x86_64 2024-02-20T07:45:47.197 INFO:teuthology.orchestra.run.smithi133.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-02-20T07:45:47.197 INFO:teuthology.orchestra.run.smithi133.stdout: python3-ceph-argparse-2:17.2.7-521.gc80e7168.el9.x86_64 2024-02-20T07:45:47.198 INFO:teuthology.orchestra.run.smithi133.stdout: python3-ceph-common-2:17.2.7-521.gc80e7168.el9.x86_64 2024-02-20T07:45:47.198 INFO:teuthology.orchestra.run.smithi133.stdout: python3-cephfs-2:17.2.7-521.gc80e7168.el9.x86_64 2024-02-20T07:45:47.198 INFO:teuthology.orchestra.run.smithi133.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-02-20T07:45:47.198 INFO:teuthology.orchestra.run.smithi133.stdout: python3-rados-2:17.2.7-521.gc80e7168.el9.x86_64 2024-02-20T07:45:47.198 INFO:teuthology.orchestra.run.smithi133.stdout: python3-rbd-2:17.2.7-521.gc80e7168.el9.x86_64 2024-02-20T07:45:47.198 INFO:teuthology.orchestra.run.smithi133.stdout: python3-rgw-2:17.2.7-521.gc80e7168.el9.x86_64 2024-02-20T07:45:47.198 INFO:teuthology.orchestra.run.smithi133.stdout: thrift-0.15.0-2.el9.x86_64 2024-02-20T07:45:47.199 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-02-20T07:45:47.199 INFO:teuthology.orchestra.run.smithi133.stdout:Complete! 2024-02-20T07:45:47.539 DEBUG:teuthology.orchestra.run.smithi133:> sudo yum -y install ceph-test 2024-02-20T07:45:48.119 INFO:teuthology.orchestra.run.smithi133.stdout:Last metadata expiration check: 0:00:28 ago on Tue 20 Feb 2024 07:45:20 AM UTC. 2024-02-20T07:45:48.283 INFO:teuthology.orchestra.run.smithi133.stdout:Dependencies resolved. 2024-02-20T07:45:48.284 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2024-02-20T07:45:48.284 INFO:teuthology.orchestra.run.smithi133.stdout: Package Arch Version Repository Size 2024-02-20T07:45:48.284 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2024-02-20T07:45:48.284 INFO:teuthology.orchestra.run.smithi133.stdout:Installing: 2024-02-20T07:45:48.284 INFO:teuthology.orchestra.run.smithi133.stdout: ceph-test x86_64 2:17.2.7-521.gc80e7168.el9 ceph 48 M 2024-02-20T07:45:48.284 INFO:teuthology.orchestra.run.smithi133.stdout:Installing dependencies: 2024-02-20T07:45:48.284 INFO:teuthology.orchestra.run.smithi133.stdout: jq x86_64 1.6-16.el9 CentOS-BaseOS 186 k 2024-02-20T07:45:48.284 INFO:teuthology.orchestra.run.smithi133.stdout: libcephsqlite x86_64 2:17.2.7-521.gc80e7168.el9 ceph 163 k 2024-02-20T07:45:48.284 INFO:teuthology.orchestra.run.smithi133.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-BaseOS 217 k 2024-02-20T07:45:48.285 INFO:teuthology.orchestra.run.smithi133.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-02-20T07:45:48.285 INFO:teuthology.orchestra.run.smithi133.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-02-20T07:45:48.285 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-02-20T07:45:48.285 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction Summary 2024-02-20T07:45:48.285 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2024-02-20T07:45:48.285 INFO:teuthology.orchestra.run.smithi133.stdout:Install 6 Packages 2024-02-20T07:45:48.285 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-02-20T07:45:48.285 INFO:teuthology.orchestra.run.smithi133.stdout:Total download size: 49 M 2024-02-20T07:45:48.286 INFO:teuthology.orchestra.run.smithi133.stdout:Installed size: 196 M 2024-02-20T07:45:48.286 INFO:teuthology.orchestra.run.smithi133.stdout:Downloading Packages: 2024-02-20T07:45:48.348 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 26/26 2024-02-20T07:45:48.348 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : ceph-base-2:17.2.7-521.gc80e7168.el9.x86_64 1/26 2024-02-20T07:45:48.348 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : ceph-common-2:17.2.7-521.gc80e7168.el9.x86_64 2/26 2024-02-20T07:45:48.348 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : ceph-radosgw-2:17.2.7-521.gc80e7168.el9.x86_64 3/26 2024-02-20T07:45:48.348 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : ceph-selinux-2:17.2.7-521.gc80e7168.el9.x86_64 4/26 2024-02-20T07:45:48.348 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : libcephfs2-2:17.2.7-521.gc80e7168.el9.x86_64 5/26 2024-02-20T07:45:48.348 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : libradosstriper1-2:17.2.7-521.gc80e7168.el9.x86_64 6/26 2024-02-20T07:45:48.348 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : librgw2-2:17.2.7-521.gc80e7168.el9.x86_64 7/26 2024-02-20T07:45:48.348 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-ceph-argparse-2:17.2.7-521.gc80e7168.el9.x 8/26 2024-02-20T07:45:48.349 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-ceph-common-2:17.2.7-521.gc80e7168.el9.x86 9/26 2024-02-20T07:45:48.349 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-cephfs-2:17.2.7-521.gc80e7168.el9.x86_64 10/26 2024-02-20T07:45:48.349 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-rados-2:17.2.7-521.gc80e7168.el9.x86_64 11/26 2024-02-20T07:45:48.349 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-rbd-2:17.2.7-521.gc80e7168.el9.x86_64 12/26 2024-02-20T07:45:48.349 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-rgw-2:17.2.7-521.gc80e7168.el9.x86_64 13/26 2024-02-20T07:45:48.349 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 14/26 2024-02-20T07:45:48.349 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 15/26 2024-02-20T07:45:48.349 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 16/26 2024-02-20T07:45:48.349 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 17/26 2024-02-20T07:45:48.349 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 18/26 2024-02-20T07:45:48.349 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 19/26 2024-02-20T07:45:48.349 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 20/26 2024-02-20T07:45:48.349 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 21/26 2024-02-20T07:45:48.349 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 22/26 2024-02-20T07:45:48.349 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : librados2-2:17.2.7-521.gc80e7168.el9.x86_64 23/26 2024-02-20T07:45:48.350 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 24/26 2024-02-20T07:45:48.350 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : librbd1-2:17.2.7-521.gc80e7168.el9.x86_64 25/26 2024-02-20T07:45:48.649 INFO:teuthology.orchestra.run.smithi133.stdout:(1/6): socat-1.7.4.1-5.el9.x86_64.rpm 827 kB/s | 300 kB 00:00 2024-02-20T07:45:48.683 INFO:teuthology.orchestra.run.smithi133.stdout:(2/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.8 MB/s | 63 kB 00:00 2024-02-20T07:45:48.758 INFO:teuthology.orchestra.run.smithi133.stdout:(3/6): jq-1.6-16.el9.x86_64.rpm 2.4 MB/s | 186 kB 00:00 2024-02-20T07:45:48.783 INFO:teuthology.orchestra.run.smithi133.stdout:(4/6): libcephsqlite-17.2.7-521.gc80e7168.el9.x 327 kB/s | 163 kB 00:00 2024-02-20T07:45:48.859 INFO:teuthology.orchestra.run.smithi133.stdout:(5/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 2.1 MB/s | 217 kB 00:00 2024-02-20T07:45:49.261 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 26/26 2024-02-20T07:45:49.261 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-02-20T07:45:49.262 INFO:teuthology.orchestra.run.smithi183.stdout:Upgraded: 2024-02-20T07:45:49.262 INFO:teuthology.orchestra.run.smithi183.stdout: librados2-2:17.2.7-521.gc80e7168.el9.x86_64 2024-02-20T07:45:49.262 INFO:teuthology.orchestra.run.smithi183.stdout: librbd1-2:17.2.7-521.gc80e7168.el9.x86_64 2024-02-20T07:45:49.262 INFO:teuthology.orchestra.run.smithi183.stdout:Installed: 2024-02-20T07:45:49.262 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-base-2:17.2.7-521.gc80e7168.el9.x86_64 2024-02-20T07:45:49.262 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-common-2:17.2.7-521.gc80e7168.el9.x86_64 2024-02-20T07:45:49.262 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-radosgw-2:17.2.7-521.gc80e7168.el9.x86_64 2024-02-20T07:45:49.262 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-selinux-2:17.2.7-521.gc80e7168.el9.x86_64 2024-02-20T07:45:49.262 INFO:teuthology.orchestra.run.smithi183.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-02-20T07:45:49.262 INFO:teuthology.orchestra.run.smithi183.stdout: libcephfs2-2:17.2.7-521.gc80e7168.el9.x86_64 2024-02-20T07:45:49.262 INFO:teuthology.orchestra.run.smithi183.stdout: liboath-2.6.7-2.el9.x86_64 2024-02-20T07:45:49.262 INFO:teuthology.orchestra.run.smithi183.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-02-20T07:45:49.263 INFO:teuthology.orchestra.run.smithi183.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-02-20T07:45:49.263 INFO:teuthology.orchestra.run.smithi183.stdout: libradosstriper1-2:17.2.7-521.gc80e7168.el9.x86_64 2024-02-20T07:45:49.263 INFO:teuthology.orchestra.run.smithi183.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-02-20T07:45:49.263 INFO:teuthology.orchestra.run.smithi183.stdout: librgw2-2:17.2.7-521.gc80e7168.el9.x86_64 2024-02-20T07:45:49.263 INFO:teuthology.orchestra.run.smithi183.stdout: libunwind-1.6.2-1.el9.x86_64 2024-02-20T07:45:49.263 INFO:teuthology.orchestra.run.smithi183.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-02-20T07:45:49.263 INFO:teuthology.orchestra.run.smithi183.stdout: python3-ceph-argparse-2:17.2.7-521.gc80e7168.el9.x86_64 2024-02-20T07:45:49.263 INFO:teuthology.orchestra.run.smithi183.stdout: python3-ceph-common-2:17.2.7-521.gc80e7168.el9.x86_64 2024-02-20T07:45:49.263 INFO:teuthology.orchestra.run.smithi183.stdout: python3-cephfs-2:17.2.7-521.gc80e7168.el9.x86_64 2024-02-20T07:45:49.263 INFO:teuthology.orchestra.run.smithi183.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-02-20T07:45:49.264 INFO:teuthology.orchestra.run.smithi183.stdout: python3-rados-2:17.2.7-521.gc80e7168.el9.x86_64 2024-02-20T07:45:49.264 INFO:teuthology.orchestra.run.smithi183.stdout: python3-rbd-2:17.2.7-521.gc80e7168.el9.x86_64 2024-02-20T07:45:49.264 INFO:teuthology.orchestra.run.smithi183.stdout: python3-rgw-2:17.2.7-521.gc80e7168.el9.x86_64 2024-02-20T07:45:49.264 INFO:teuthology.orchestra.run.smithi183.stdout: thrift-0.15.0-2.el9.x86_64 2024-02-20T07:45:49.264 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-02-20T07:45:49.264 INFO:teuthology.orchestra.run.smithi183.stdout:Complete! 2024-02-20T07:45:49.656 DEBUG:teuthology.orchestra.run.smithi183:> sudo yum -y install ceph-test 2024-02-20T07:45:50.253 INFO:teuthology.orchestra.run.smithi183.stdout:Last metadata expiration check: 0:00:29 ago on Tue 20 Feb 2024 07:45:21 AM UTC. 2024-02-20T07:45:50.418 INFO:teuthology.orchestra.run.smithi183.stdout:Dependencies resolved. 2024-02-20T07:45:50.419 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-02-20T07:45:50.419 INFO:teuthology.orchestra.run.smithi183.stdout: Package Arch Version Repository Size 2024-02-20T07:45:50.419 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-02-20T07:45:50.419 INFO:teuthology.orchestra.run.smithi183.stdout:Installing: 2024-02-20T07:45:50.419 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-test x86_64 2:17.2.7-521.gc80e7168.el9 ceph 48 M 2024-02-20T07:45:50.419 INFO:teuthology.orchestra.run.smithi183.stdout:Installing dependencies: 2024-02-20T07:45:50.419 INFO:teuthology.orchestra.run.smithi183.stdout: jq x86_64 1.6-16.el9 CentOS-BaseOS 186 k 2024-02-20T07:45:50.419 INFO:teuthology.orchestra.run.smithi183.stdout: libcephsqlite x86_64 2:17.2.7-521.gc80e7168.el9 ceph 163 k 2024-02-20T07:45:50.419 INFO:teuthology.orchestra.run.smithi183.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-BaseOS 217 k 2024-02-20T07:45:50.419 INFO:teuthology.orchestra.run.smithi183.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-02-20T07:45:50.419 INFO:teuthology.orchestra.run.smithi183.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-02-20T07:45:50.419 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-02-20T07:45:50.420 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction Summary 2024-02-20T07:45:50.420 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-02-20T07:45:50.420 INFO:teuthology.orchestra.run.smithi183.stdout:Install 6 Packages 2024-02-20T07:45:50.420 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-02-20T07:45:50.420 INFO:teuthology.orchestra.run.smithi183.stdout:Total download size: 49 M 2024-02-20T07:45:50.420 INFO:teuthology.orchestra.run.smithi183.stdout:Installed size: 196 M 2024-02-20T07:45:50.421 INFO:teuthology.orchestra.run.smithi183.stdout:Downloading Packages: 2024-02-20T07:45:50.592 INFO:teuthology.orchestra.run.smithi183.stdout:(1/6): socat-1.7.4.1-5.el9.x86_64.rpm 1.7 MB/s | 300 kB 00:00 2024-02-20T07:45:50.618 INFO:teuthology.orchestra.run.smithi183.stdout:(2/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 2.5 MB/s | 63 kB 00:00 2024-02-20T07:45:50.660 INFO:teuthology.orchestra.run.smithi183.stdout:(3/6): jq-1.6-16.el9.x86_64.rpm 4.4 MB/s | 186 kB 00:00 2024-02-20T07:45:50.701 INFO:teuthology.orchestra.run.smithi183.stdout:(4/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 5.1 MB/s | 217 kB 00:00 2024-02-20T07:45:50.793 INFO:teuthology.orchestra.run.smithi183.stdout:(5/6): libcephsqlite-17.2.7-521.gc80e7168.el9.x 437 kB/s | 163 kB 00:00 2024-02-20T07:45:53.370 INFO:teuthology.orchestra.run.smithi183.stdout:(6/6): ceph-test-17.2.7-521.gc80e7168.el9.x86_6 16 MB/s | 48 MB 00:02 2024-02-20T07:45:53.371 INFO:teuthology.orchestra.run.smithi183.stdout:-------------------------------------------------------------------------------- 2024-02-20T07:45:53.371 INFO:teuthology.orchestra.run.smithi183.stdout:Total 17 MB/s | 49 MB 00:02 2024-02-20T07:45:53.372 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction check 2024-02-20T07:45:53.390 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction check succeeded. 2024-02-20T07:45:53.390 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction test 2024-02-20T07:45:53.781 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction test succeeded. 2024-02-20T07:45:53.781 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction 2024-02-20T07:45:54.363 INFO:teuthology.orchestra.run.smithi183.stdout: Preparing : 1/1 2024-02-20T07:45:54.446 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 1/6 2024-02-20T07:45:54.477 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : jq-1.6-16.el9.x86_64 2/6 2024-02-20T07:45:54.528 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 3/6 2024-02-20T07:45:54.571 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 4/6 2024-02-20T07:45:54.597 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : libcephsqlite-2:17.2.7-521.gc80e7168.el9.x86_64 5/6 2024-02-20T07:45:59.154 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: libcephsqlite-2:17.2.7-521.gc80e7168.el9.x86_64 5/6 2024-02-20T07:45:59.237 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : ceph-test-2:17.2.7-521.gc80e7168.el9.x86_64 6/6 2024-02-20T07:46:01.267 INFO:teuthology.orchestra.run.smithi133.stdout:(6/6): ceph-test-17.2.7-521.gc80e7168.el9.x86_6 3.7 MB/s | 48 MB 00:12 2024-02-20T07:46:01.270 INFO:teuthology.orchestra.run.smithi133.stdout:-------------------------------------------------------------------------------- 2024-02-20T07:46:01.270 INFO:teuthology.orchestra.run.smithi133.stdout:Total 3.8 MB/s | 49 MB 00:12 2024-02-20T07:46:01.270 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction check 2024-02-20T07:46:01.289 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction check succeeded. 2024-02-20T07:46:01.289 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction test 2024-02-20T07:46:01.517 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: ceph-test-2:17.2.7-521.gc80e7168.el9.x86_64 6/6 2024-02-20T07:46:01.517 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : ceph-test-2:17.2.7-521.gc80e7168.el9.x86_64 1/6 2024-02-20T07:46:01.517 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : libcephsqlite-2:17.2.7-521.gc80e7168.el9.x86_64 2/6 2024-02-20T07:46:01.517 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 3/6 2024-02-20T07:46:01.517 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/6 2024-02-20T07:46:01.517 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : jq-1.6-16.el9.x86_64 5/6 2024-02-20T07:46:01.682 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction test succeeded. 2024-02-20T07:46:01.682 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction 2024-02-20T07:46:01.904 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 6/6 2024-02-20T07:46:01.904 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-02-20T07:46:01.904 INFO:teuthology.orchestra.run.smithi183.stdout:Installed: 2024-02-20T07:46:01.904 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-test-2:17.2.7-521.gc80e7168.el9.x86_64 2024-02-20T07:46:01.904 INFO:teuthology.orchestra.run.smithi183.stdout: jq-1.6-16.el9.x86_64 2024-02-20T07:46:01.904 INFO:teuthology.orchestra.run.smithi183.stdout: libcephsqlite-2:17.2.7-521.gc80e7168.el9.x86_64 2024-02-20T07:46:01.905 INFO:teuthology.orchestra.run.smithi183.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-02-20T07:46:01.905 INFO:teuthology.orchestra.run.smithi183.stdout: socat-1.7.4.1-5.el9.x86_64 2024-02-20T07:46:01.905 INFO:teuthology.orchestra.run.smithi183.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-02-20T07:46:01.905 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-02-20T07:46:01.905 INFO:teuthology.orchestra.run.smithi183.stdout:Complete! 2024-02-20T07:46:02.196 DEBUG:teuthology.orchestra.run.smithi183:> sudo yum -y install ceph 2024-02-20T07:46:02.276 INFO:teuthology.orchestra.run.smithi133.stdout: Preparing : 1/1 2024-02-20T07:46:02.370 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 1/6 2024-02-20T07:46:02.399 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : jq-1.6-16.el9.x86_64 2/6 2024-02-20T07:46:02.451 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 3/6 2024-02-20T07:46:02.492 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 4/6 2024-02-20T07:46:02.520 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : libcephsqlite-2:17.2.7-521.gc80e7168.el9.x86_64 5/6 2024-02-20T07:46:02.778 INFO:teuthology.orchestra.run.smithi183.stdout:Last metadata expiration check: 0:00:41 ago on Tue 20 Feb 2024 07:45:21 AM UTC. 2024-02-20T07:46:02.950 INFO:teuthology.orchestra.run.smithi183.stdout:Dependencies resolved. 2024-02-20T07:46:02.952 INFO:teuthology.orchestra.run.smithi183.stdout:======================================================================================== 2024-02-20T07:46:02.952 INFO:teuthology.orchestra.run.smithi183.stdout: Package Arch Version Repository Size 2024-02-20T07:46:02.952 INFO:teuthology.orchestra.run.smithi183.stdout:======================================================================================== 2024-02-20T07:46:02.953 INFO:teuthology.orchestra.run.smithi183.stdout:Installing: 2024-02-20T07:46:02.953 INFO:teuthology.orchestra.run.smithi183.stdout: ceph x86_64 2:17.2.7-521.gc80e7168.el9 ceph 6.5 k 2024-02-20T07:46:02.953 INFO:teuthology.orchestra.run.smithi183.stdout:Installing dependencies: 2024-02-20T07:46:02.953 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-mds x86_64 2:17.2.7-521.gc80e7168.el9 ceph 2.1 M 2024-02-20T07:46:02.953 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-mgr x86_64 2:17.2.7-521.gc80e7168.el9 ceph 1.4 M 2024-02-20T07:46:02.953 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-mgr-modules-core noarch 2:17.2.7-521.gc80e7168.el9 ceph-noarch 239 k 2024-02-20T07:46:02.953 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-mon x86_64 2:17.2.7-521.gc80e7168.el9 ceph 3.7 M 2024-02-20T07:46:02.953 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-osd x86_64 2:17.2.7-521.gc80e7168.el9 ceph 15 M 2024-02-20T07:46:02.953 INFO:teuthology.orchestra.run.smithi183.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-02-20T07:46:02.953 INFO:teuthology.orchestra.run.smithi183.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-02-20T07:46:02.953 INFO:teuthology.orchestra.run.smithi183.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-02-20T07:46:02.954 INFO:teuthology.orchestra.run.smithi183.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-02-20T07:46:02.954 INFO:teuthology.orchestra.run.smithi183.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-02-20T07:46:02.954 INFO:teuthology.orchestra.run.smithi183.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-02-20T07:46:02.954 INFO:teuthology.orchestra.run.smithi183.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-02-20T07:46:02.954 INFO:teuthology.orchestra.run.smithi183.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-02-20T07:46:02.954 INFO:teuthology.orchestra.run.smithi183.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-02-20T07:46:02.954 INFO:teuthology.orchestra.run.smithi183.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-02-20T07:46:02.954 INFO:teuthology.orchestra.run.smithi183.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-02-20T07:46:02.954 INFO:teuthology.orchestra.run.smithi183.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-02-20T07:46:02.954 INFO:teuthology.orchestra.run.smithi183.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-02-20T07:46:02.954 INFO:teuthology.orchestra.run.smithi183.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-02-20T07:46:02.954 INFO:teuthology.orchestra.run.smithi183.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-02-20T07:46:02.955 INFO:teuthology.orchestra.run.smithi183.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-02-20T07:46:02.955 INFO:teuthology.orchestra.run.smithi183.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-02-20T07:46:02.955 INFO:teuthology.orchestra.run.smithi183.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-02-20T07:46:02.955 INFO:teuthology.orchestra.run.smithi183.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-02-20T07:46:02.955 INFO:teuthology.orchestra.run.smithi183.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-02-20T07:46:02.955 INFO:teuthology.orchestra.run.smithi183.stdout: python3-requests noarch 2.25.1-8.el9 CentOS-BaseOS 113 k 2024-02-20T07:46:02.955 INFO:teuthology.orchestra.run.smithi183.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-02-20T07:46:02.955 INFO:teuthology.orchestra.run.smithi183.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-02-20T07:46:02.955 INFO:teuthology.orchestra.run.smithi183.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-02-20T07:46:02.955 INFO:teuthology.orchestra.run.smithi183.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-02-20T07:46:02.955 INFO:teuthology.orchestra.run.smithi183.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-02-20T07:46:02.955 INFO:teuthology.orchestra.run.smithi183.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-02-20T07:46:02.955 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-02-20T07:46:02.955 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction Summary 2024-02-20T07:46:02.956 INFO:teuthology.orchestra.run.smithi183.stdout:======================================================================================== 2024-02-20T07:46:02.956 INFO:teuthology.orchestra.run.smithi183.stdout:Install 33 Packages 2024-02-20T07:46:02.956 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-02-20T07:46:02.956 INFO:teuthology.orchestra.run.smithi183.stdout:Total download size: 27 M 2024-02-20T07:46:02.956 INFO:teuthology.orchestra.run.smithi183.stdout:Installed size: 99 M 2024-02-20T07:46:02.956 INFO:teuthology.orchestra.run.smithi183.stdout:Downloading Packages: 2024-02-20T07:46:03.545 INFO:teuthology.orchestra.run.smithi183.stdout:(1/33): ceph-17.2.7-521.gc80e7168.el9.x86_64.rp 28 kB/s | 6.5 kB 00:00 2024-02-20T07:46:03.687 INFO:teuthology.orchestra.run.smithi183.stdout:(2/33): ceph-mds-17.2.7-521.gc80e7168.el9.x86_6 5.5 MB/s | 2.1 MB 00:00 2024-02-20T07:46:03.721 INFO:teuthology.orchestra.run.smithi183.stdout:(3/33): ceph-mgr-17.2.7-521.gc80e7168.el9.x86_6 3.4 MB/s | 1.4 MB 00:00 2024-02-20T07:46:03.821 INFO:teuthology.orchestra.run.smithi183.stdout:(4/33): ceph-mgr-modules-core-17.2.7-521.gc80e7 2.3 MB/s | 239 kB 00:00 2024-02-20T07:46:03.930 INFO:teuthology.orchestra.run.smithi183.stdout:(5/33): python3-mako-1.1.4-6.el9.noarch.rpm 1.3 MB/s | 144 kB 00:00 2024-02-20T07:46:03.963 INFO:teuthology.orchestra.run.smithi183.stdout:(6/33): python3-markupsafe-1.1.1-12.el9.x86_64. 951 kB/s | 32 kB 00:00 2024-02-20T07:46:03.997 INFO:teuthology.orchestra.run.smithi183.stdout:(7/33): python3-pytz-2021.1-5.el9.noarch.rpm 1.4 MB/s | 47 kB 00:00 2024-02-20T07:46:04.031 INFO:teuthology.orchestra.run.smithi183.stdout:(8/33): python3-toml-0.10.2-6.el9.noarch.rpm 1.1 MB/s | 39 kB 00:00 2024-02-20T07:46:04.081 INFO:teuthology.orchestra.run.smithi183.stdout:(9/33): python3-cffi-1.14.5-5.el9.x86_64.rpm 4.7 MB/s | 241 kB 00:00 2024-02-20T07:46:04.148 INFO:teuthology.orchestra.run.smithi183.stdout:(10/33): python3-cryptography-36.0.1-4.el9.x86_ 17 MB/s | 1.1 MB 00:00 2024-02-20T07:46:04.182 INFO:teuthology.orchestra.run.smithi183.stdout:(11/33): python3-ply-3.11-14.el9.noarch.rpm 3.0 MB/s | 103 kB 00:00 2024-02-20T07:46:04.215 INFO:teuthology.orchestra.run.smithi183.stdout:(12/33): python3-pycparser-2.20-6.el9.noarch.rp 3.6 MB/s | 124 kB 00:00 2024-02-20T07:46:04.249 INFO:teuthology.orchestra.run.smithi183.stdout:(13/33): python3-requests-2.25.1-8.el9.noarch.r 3.3 MB/s | 113 kB 00:00 2024-02-20T07:46:04.399 INFO:teuthology.orchestra.run.smithi183.stdout:(14/33): ceph-osd-17.2.7-521.gc80e7168.el9.x86_ 22 MB/s | 15 MB 00:00 2024-02-20T07:46:04.424 INFO:teuthology.orchestra.run.smithi183.stdout:(15/33): python3-urllib3-1.26.5-5.el9.noarch.rp 1.0 MB/s | 187 kB 00:00 2024-02-20T07:46:04.725 INFO:teuthology.orchestra.run.smithi183.stdout:(16/33): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 134 kB/s | 43 kB 00:00 2024-02-20T07:46:04.817 INFO:teuthology.orchestra.run.smithi183.stdout:(17/33): python3-cheroot-8.6.0-4.el9.noarch.rpm 438 kB/s | 172 kB 00:00 2024-02-20T07:46:04.909 INFO:teuthology.orchestra.run.smithi183.stdout:(18/33): python3-jaraco-8.2.1-3.el9.noarch.rpm 117 kB/s | 11 kB 00:00 2024-02-20T07:46:04.959 INFO:teuthology.orchestra.run.smithi183.stdout:(19/33): python3-cherrypy-18.6.1-2.el9.noarch.r 1.5 MB/s | 358 kB 00:00 2024-02-20T07:46:05.001 INFO:teuthology.orchestra.run.smithi183.stdout:(20/33): python3-jaraco-classes-3.2.1-5.el9.noa 192 kB/s | 18 kB 00:00 2024-02-20T07:46:05.051 INFO:teuthology.orchestra.run.smithi183.stdout:(21/33): python3-jaraco-collections-3.0.0-8.el9 251 kB/s | 23 kB 00:00 2024-02-20T07:46:05.093 INFO:teuthology.orchestra.run.smithi183.stdout:(22/33): python3-jaraco-functools-3.5.0-2.el9.n 211 kB/s | 19 kB 00:00 2024-02-20T07:46:05.143 INFO:teuthology.orchestra.run.smithi183.stdout:(23/33): python3-jaraco-text-3.2.0-6.el9.noarch 214 kB/s | 20 kB 00:00 2024-02-20T07:46:05.185 INFO:teuthology.orchestra.run.smithi183.stdout:(24/33): python3-logutils-0.3.5-21.el9.noarch.r 504 kB/s | 46 kB 00:00 2024-02-20T07:46:05.244 INFO:teuthology.orchestra.run.smithi183.stdout:(25/33): ceph-mon-17.2.7-521.gc80e7168.el9.x86_ 2.2 MB/s | 3.7 MB 00:01 2024-02-20T07:46:05.269 INFO:teuthology.orchestra.run.smithi183.stdout:(26/33): python3-more-itertools-8.12.0-2.el9.no 628 kB/s | 79 kB 00:00 2024-02-20T07:46:05.353 INFO:teuthology.orchestra.run.smithi183.stdout:(27/33): python3-pecan-1.4.2-3.el9.noarch.rpm 1.6 MB/s | 272 kB 00:00 2024-02-20T07:46:05.378 INFO:teuthology.orchestra.run.smithi183.stdout:(28/33): python3-pyOpenSSL-21.0.0-1.el9.noarch. 829 kB/s | 90 kB 00:00 2024-02-20T07:46:05.445 INFO:teuthology.orchestra.run.smithi183.stdout:(29/33): python3-tempora-5.0.0-2.el9.noarch.rpm 390 kB/s | 36 kB 00:00 2024-02-20T07:46:05.479 INFO:teuthology.orchestra.run.smithi183.stdout:(30/33): python3-webob-1.8.7-6.el9.noarch.rpm 2.2 MB/s | 230 kB 00:00 2024-02-20T07:46:05.504 INFO:teuthology.orchestra.run.smithi183.stdout:(31/33): python3-portend-3.1.0-2.el9.noarch.rpm 63 kB/s | 16 kB 00:00 2024-02-20T07:46:05.546 INFO:teuthology.orchestra.run.smithi183.stdout:(32/33): python3-werkzeug-2.0.3-3.el9.1.noarch. 4.2 MB/s | 427 kB 00:00 2024-02-20T07:46:05.571 INFO:teuthology.orchestra.run.smithi183.stdout:(33/33): python3-zc-lockfile-2.0-10.el9.noarch. 217 kB/s | 20 kB 00:00 2024-02-20T07:46:05.575 INFO:teuthology.orchestra.run.smithi183.stdout:-------------------------------------------------------------------------------- 2024-02-20T07:46:05.575 INFO:teuthology.orchestra.run.smithi183.stdout:Total 10 MB/s | 27 MB 00:02 2024-02-20T07:46:05.576 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction check 2024-02-20T07:46:05.614 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction check succeeded. 2024-02-20T07:46:05.614 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction test 2024-02-20T07:46:05.961 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction test succeeded. 2024-02-20T07:46:05.961 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction 2024-02-20T07:46:06.926 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: libcephsqlite-2:17.2.7-521.gc80e7168.el9.x86_64 5/6 2024-02-20T07:46:07.010 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : ceph-test-2:17.2.7-521.gc80e7168.el9.x86_64 6/6 2024-02-20T07:46:07.186 INFO:teuthology.orchestra.run.smithi183.stdout: Preparing : 1/1 2024-02-20T07:46:07.257 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/33 2024-02-20T07:46:07.290 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/33 2024-02-20T07:46:07.322 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 3/33 2024-02-20T07:46:07.414 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 4/33 2024-02-20T07:46:07.484 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 5/33 2024-02-20T07:46:07.534 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 6/33 2024-02-20T07:46:07.605 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 7/33 2024-02-20T07:46:07.658 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 8/33 2024-02-20T07:46:07.697 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 9/33 2024-02-20T07:46:07.746 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-ply-3.11-14.el9.noarch 10/33 2024-02-20T07:46:07.802 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 11/33 2024-02-20T07:46:07.991 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 12/33 2024-02-20T07:46:08.051 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 13/33 2024-02-20T07:46:08.081 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 14/33 2024-02-20T07:46:08.116 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 15/33 2024-02-20T07:46:08.146 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 16/33 2024-02-20T07:46:08.222 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 17/33 2024-02-20T07:46:08.256 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 18/33 2024-02-20T07:46:08.290 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 19/33 2024-02-20T07:46:08.330 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 20/33 2024-02-20T07:46:08.365 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 21/33 2024-02-20T07:46:08.399 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 22/33 2024-02-20T07:46:08.488 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 23/33 2024-02-20T07:46:08.535 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 24/33 2024-02-20T07:46:08.610 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 25/33 2024-02-20T07:46:08.746 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 26/33 2024-02-20T07:46:08.879 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 27/33 2024-02-20T07:46:09.045 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : ceph-mgr-modules-core-2:17.2.7-521.gc80e7168.el9.n 28/33 2024-02-20T07:46:09.077 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : ceph-mgr-2:17.2.7-521.gc80e7168.el9.x86_64 29/33 2024-02-20T07:46:09.102 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: ceph-mgr-2:17.2.7-521.gc80e7168.el9.x86_64 29/33 2024-02-20T07:46:09.102 INFO:teuthology.orchestra.run.smithi183.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-02-20T07:46:09.102 INFO:teuthology.orchestra.run.smithi183.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-02-20T07:46:09.102 INFO:teuthology.orchestra.run.smithi183.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-02-20T07:46:09.102 INFO:teuthology.orchestra.run.smithi183.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-02-20T07:46:09.102 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-02-20T07:46:09.189 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: ceph-test-2:17.2.7-521.gc80e7168.el9.x86_64 6/6 2024-02-20T07:46:09.189 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : ceph-test-2:17.2.7-521.gc80e7168.el9.x86_64 1/6 2024-02-20T07:46:09.189 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : libcephsqlite-2:17.2.7-521.gc80e7168.el9.x86_64 2/6 2024-02-20T07:46:09.190 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 3/6 2024-02-20T07:46:09.190 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/6 2024-02-20T07:46:09.190 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : jq-1.6-16.el9.x86_64 5/6 2024-02-20T07:46:09.612 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 6/6 2024-02-20T07:46:09.612 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-02-20T07:46:09.612 INFO:teuthology.orchestra.run.smithi133.stdout:Installed: 2024-02-20T07:46:09.613 INFO:teuthology.orchestra.run.smithi133.stdout: ceph-test-2:17.2.7-521.gc80e7168.el9.x86_64 2024-02-20T07:46:09.613 INFO:teuthology.orchestra.run.smithi133.stdout: jq-1.6-16.el9.x86_64 2024-02-20T07:46:09.613 INFO:teuthology.orchestra.run.smithi133.stdout: libcephsqlite-2:17.2.7-521.gc80e7168.el9.x86_64 2024-02-20T07:46:09.613 INFO:teuthology.orchestra.run.smithi133.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-02-20T07:46:09.613 INFO:teuthology.orchestra.run.smithi133.stdout: socat-1.7.4.1-5.el9.x86_64 2024-02-20T07:46:09.613 INFO:teuthology.orchestra.run.smithi133.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-02-20T07:46:09.613 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-02-20T07:46:09.613 INFO:teuthology.orchestra.run.smithi133.stdout:Complete! 2024-02-20T07:46:09.888 DEBUG:teuthology.orchestra.run.smithi133:> sudo yum -y install ceph 2024-02-20T07:46:10.474 INFO:teuthology.orchestra.run.smithi133.stdout:Last metadata expiration check: 0:00:50 ago on Tue 20 Feb 2024 07:45:20 AM UTC. 2024-02-20T07:46:10.487 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : ceph-osd-2:17.2.7-521.gc80e7168.el9.x86_64 30/33 2024-02-20T07:46:10.515 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: ceph-osd-2:17.2.7-521.gc80e7168.el9.x86_64 30/33 2024-02-20T07:46:10.515 INFO:teuthology.orchestra.run.smithi183.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-02-20T07:46:10.515 INFO:teuthology.orchestra.run.smithi183.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-02-20T07:46:10.515 INFO:teuthology.orchestra.run.smithi183.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-02-20T07:46:10.515 INFO:teuthology.orchestra.run.smithi183.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-02-20T07:46:10.516 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-02-20T07:46:10.648 INFO:teuthology.orchestra.run.smithi133.stdout:Dependencies resolved. 2024-02-20T07:46:10.651 INFO:teuthology.orchestra.run.smithi133.stdout:======================================================================================== 2024-02-20T07:46:10.651 INFO:teuthology.orchestra.run.smithi133.stdout: Package Arch Version Repository Size 2024-02-20T07:46:10.651 INFO:teuthology.orchestra.run.smithi133.stdout:======================================================================================== 2024-02-20T07:46:10.651 INFO:teuthology.orchestra.run.smithi133.stdout:Installing: 2024-02-20T07:46:10.651 INFO:teuthology.orchestra.run.smithi133.stdout: ceph x86_64 2:17.2.7-521.gc80e7168.el9 ceph 6.5 k 2024-02-20T07:46:10.651 INFO:teuthology.orchestra.run.smithi133.stdout:Installing dependencies: 2024-02-20T07:46:10.652 INFO:teuthology.orchestra.run.smithi133.stdout: ceph-mds x86_64 2:17.2.7-521.gc80e7168.el9 ceph 2.1 M 2024-02-20T07:46:10.652 INFO:teuthology.orchestra.run.smithi133.stdout: ceph-mgr x86_64 2:17.2.7-521.gc80e7168.el9 ceph 1.4 M 2024-02-20T07:46:10.652 INFO:teuthology.orchestra.run.smithi133.stdout: ceph-mgr-modules-core noarch 2:17.2.7-521.gc80e7168.el9 ceph-noarch 239 k 2024-02-20T07:46:10.652 INFO:teuthology.orchestra.run.smithi133.stdout: ceph-mon x86_64 2:17.2.7-521.gc80e7168.el9 ceph 3.7 M 2024-02-20T07:46:10.652 INFO:teuthology.orchestra.run.smithi133.stdout: ceph-osd x86_64 2:17.2.7-521.gc80e7168.el9 ceph 15 M 2024-02-20T07:46:10.652 INFO:teuthology.orchestra.run.smithi133.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-02-20T07:46:10.652 INFO:teuthology.orchestra.run.smithi133.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-02-20T07:46:10.652 INFO:teuthology.orchestra.run.smithi133.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-02-20T07:46:10.652 INFO:teuthology.orchestra.run.smithi133.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-02-20T07:46:10.652 INFO:teuthology.orchestra.run.smithi133.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-02-20T07:46:10.652 INFO:teuthology.orchestra.run.smithi133.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-02-20T07:46:10.652 INFO:teuthology.orchestra.run.smithi133.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-02-20T07:46:10.652 INFO:teuthology.orchestra.run.smithi133.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-02-20T07:46:10.652 INFO:teuthology.orchestra.run.smithi133.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-02-20T07:46:10.653 INFO:teuthology.orchestra.run.smithi133.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-02-20T07:46:10.653 INFO:teuthology.orchestra.run.smithi133.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-02-20T07:46:10.653 INFO:teuthology.orchestra.run.smithi133.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-02-20T07:46:10.653 INFO:teuthology.orchestra.run.smithi133.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-02-20T07:46:10.653 INFO:teuthology.orchestra.run.smithi133.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-02-20T07:46:10.653 INFO:teuthology.orchestra.run.smithi133.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-02-20T07:46:10.653 INFO:teuthology.orchestra.run.smithi133.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-02-20T07:46:10.653 INFO:teuthology.orchestra.run.smithi133.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-02-20T07:46:10.653 INFO:teuthology.orchestra.run.smithi133.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-02-20T07:46:10.653 INFO:teuthology.orchestra.run.smithi133.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-02-20T07:46:10.653 INFO:teuthology.orchestra.run.smithi133.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-02-20T07:46:10.653 INFO:teuthology.orchestra.run.smithi133.stdout: python3-requests noarch 2.25.1-8.el9 CentOS-BaseOS 113 k 2024-02-20T07:46:10.653 INFO:teuthology.orchestra.run.smithi133.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-02-20T07:46:10.653 INFO:teuthology.orchestra.run.smithi133.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-02-20T07:46:10.653 INFO:teuthology.orchestra.run.smithi133.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-02-20T07:46:10.654 INFO:teuthology.orchestra.run.smithi133.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-02-20T07:46:10.654 INFO:teuthology.orchestra.run.smithi133.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-02-20T07:46:10.654 INFO:teuthology.orchestra.run.smithi133.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-02-20T07:46:10.654 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-02-20T07:46:10.654 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction Summary 2024-02-20T07:46:10.654 INFO:teuthology.orchestra.run.smithi133.stdout:======================================================================================== 2024-02-20T07:46:10.654 INFO:teuthology.orchestra.run.smithi133.stdout:Install 33 Packages 2024-02-20T07:46:10.654 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-02-20T07:46:10.655 INFO:teuthology.orchestra.run.smithi133.stdout:Total download size: 27 M 2024-02-20T07:46:10.655 INFO:teuthology.orchestra.run.smithi133.stdout:Installed size: 99 M 2024-02-20T07:46:10.655 INFO:teuthology.orchestra.run.smithi133.stdout:Downloading Packages: 2024-02-20T07:46:10.896 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : ceph-mon-2:17.2.7-521.gc80e7168.el9.x86_64 31/33 2024-02-20T07:46:10.919 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: ceph-mon-2:17.2.7-521.gc80e7168.el9.x86_64 31/33 2024-02-20T07:46:10.919 INFO:teuthology.orchestra.run.smithi183.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-02-20T07:46:10.919 INFO:teuthology.orchestra.run.smithi183.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-02-20T07:46:10.919 INFO:teuthology.orchestra.run.smithi183.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-02-20T07:46:10.919 INFO:teuthology.orchestra.run.smithi183.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-02-20T07:46:10.919 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-02-20T07:46:10.960 INFO:teuthology.orchestra.run.smithi133.stdout:(1/33): ceph-17.2.7-521.gc80e7168.el9.x86_64.rp 42 kB/s | 6.5 kB 00:00 2024-02-20T07:46:11.122 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : ceph-mds-2:17.2.7-521.gc80e7168.el9.x86_64 32/33 2024-02-20T07:46:11.147 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: ceph-mds-2:17.2.7-521.gc80e7168.el9.x86_64 32/33 2024-02-20T07:46:11.147 INFO:teuthology.orchestra.run.smithi183.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-02-20T07:46:11.147 INFO:teuthology.orchestra.run.smithi183.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-02-20T07:46:11.147 INFO:teuthology.orchestra.run.smithi183.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-02-20T07:46:11.148 INFO:teuthology.orchestra.run.smithi183.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-02-20T07:46:11.148 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-02-20T07:46:11.218 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : ceph-2:17.2.7-521.gc80e7168.el9.x86_64 33/33 2024-02-20T07:46:11.277 INFO:teuthology.orchestra.run.smithi133.stdout:(2/33): ceph-mds-17.2.7-521.gc80e7168.el9.x86_6 4.4 MB/s | 2.1 MB 00:00 2024-02-20T07:46:11.436 INFO:teuthology.orchestra.run.smithi133.stdout:(3/33): ceph-mgr-17.2.7-521.gc80e7168.el9.x86_6 2.2 MB/s | 1.4 MB 00:00 2024-02-20T07:46:11.519 INFO:teuthology.orchestra.run.smithi133.stdout:(4/33): ceph-mon-17.2.7-521.gc80e7168.el9.x86_6 6.6 MB/s | 3.7 MB 00:00 2024-02-20T07:46:11.545 INFO:teuthology.orchestra.run.smithi133.stdout:(5/33): ceph-mgr-modules-core-17.2.7-521.gc80e7 2.2 MB/s | 239 kB 00:00 2024-02-20T07:46:11.653 INFO:teuthology.orchestra.run.smithi133.stdout:(6/33): python3-markupsafe-1.1.1-12.el9.x86_64. 371 kB/s | 32 kB 00:00 2024-02-20T07:46:11.678 INFO:teuthology.orchestra.run.smithi133.stdout:(7/33): python3-mako-1.1.4-6.el9.noarch.rpm 908 kB/s | 144 kB 00:00 2024-02-20T07:46:11.704 INFO:teuthology.orchestra.run.smithi133.stdout:(8/33): python3-pytz-2021.1-5.el9.noarch.rpm 942 kB/s | 47 kB 00:00 2024-02-20T07:46:11.721 INFO:teuthology.orchestra.run.smithi133.stdout:(9/33): python3-toml-0.10.2-6.el9.noarch.rpm 922 kB/s | 39 kB 00:00 2024-02-20T07:46:11.754 INFO:teuthology.orchestra.run.smithi133.stdout:(10/33): python3-cffi-1.14.5-5.el9.x86_64.rpm 4.7 MB/s | 241 kB 00:00 2024-02-20T07:46:11.813 INFO:teuthology.orchestra.run.smithi133.stdout:(11/33): python3-cryptography-36.0.1-4.el9.x86_ 12 MB/s | 1.1 MB 00:00 2024-02-20T07:46:11.838 INFO:teuthology.orchestra.run.smithi133.stdout:(12/33): python3-ply-3.11-14.el9.noarch.rpm 1.2 MB/s | 103 kB 00:00 2024-02-20T07:46:11.863 INFO:teuthology.orchestra.run.smithi133.stdout:(13/33): python3-pycparser-2.20-6.el9.noarch.rp 2.4 MB/s | 124 kB 00:00 2024-02-20T07:46:11.889 INFO:teuthology.orchestra.run.smithi133.stdout:(14/33): python3-requests-2.25.1-8.el9.noarch.r 2.2 MB/s | 113 kB 00:00 2024-02-20T07:46:11.914 INFO:teuthology.orchestra.run.smithi133.stdout:(15/33): python3-urllib3-1.26.5-5.el9.noarch.rp 3.6 MB/s | 187 kB 00:00 2024-02-20T07:46:12.314 INFO:teuthology.orchestra.run.smithi133.stdout:(16/33): ceph-osd-17.2.7-521.gc80e7168.el9.x86_ 15 MB/s | 15 MB 00:01 2024-02-20T07:46:12.861 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: ceph-2:17.2.7-521.gc80e7168.el9.x86_64 33/33 2024-02-20T07:46:12.861 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : ceph-2:17.2.7-521.gc80e7168.el9.x86_64 1/33 2024-02-20T07:46:12.861 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : ceph-mds-2:17.2.7-521.gc80e7168.el9.x86_64 2/33 2024-02-20T07:46:12.861 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : ceph-mgr-2:17.2.7-521.gc80e7168.el9.x86_64 3/33 2024-02-20T07:46:12.861 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : ceph-mon-2:17.2.7-521.gc80e7168.el9.x86_64 4/33 2024-02-20T07:46:12.861 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : ceph-osd-2:17.2.7-521.gc80e7168.el9.x86_64 5/33 2024-02-20T07:46:12.861 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : ceph-mgr-modules-core-2:17.2.7-521.gc80e7168.el9.n 6/33 2024-02-20T07:46:12.861 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 7/33 2024-02-20T07:46:12.861 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 8/33 2024-02-20T07:46:12.862 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 9/33 2024-02-20T07:46:12.862 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 10/33 2024-02-20T07:46:12.862 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 11/33 2024-02-20T07:46:12.862 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 12/33 2024-02-20T07:46:12.862 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-ply-3.11-14.el9.noarch 13/33 2024-02-20T07:46:12.862 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 14/33 2024-02-20T07:46:12.862 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 15/33 2024-02-20T07:46:12.862 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 16/33 2024-02-20T07:46:12.862 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 17/33 2024-02-20T07:46:12.862 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 18/33 2024-02-20T07:46:12.862 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 19/33 2024-02-20T07:46:12.862 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 20/33 2024-02-20T07:46:12.862 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 21/33 2024-02-20T07:46:12.862 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 22/33 2024-02-20T07:46:12.862 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 23/33 2024-02-20T07:46:12.863 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 24/33 2024-02-20T07:46:12.863 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 25/33 2024-02-20T07:46:12.864 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 26/33 2024-02-20T07:46:12.864 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 27/33 2024-02-20T07:46:12.864 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 28/33 2024-02-20T07:46:12.864 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 29/33 2024-02-20T07:46:12.864 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 30/33 2024-02-20T07:46:12.864 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 31/33 2024-02-20T07:46:12.864 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 32/33 2024-02-20T07:46:12.906 INFO:teuthology.orchestra.run.smithi133.stdout:(17/33): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 43 kB/s | 43 kB 00:01 2024-02-20T07:46:13.165 INFO:teuthology.orchestra.run.smithi133.stdout:(18/33): python3-jaraco-8.2.1-3.el9.noarch.rpm 41 kB/s | 11 kB 00:00 2024-02-20T07:46:13.365 INFO:teuthology.orchestra.run.smithi133.stdout:(19/33): python3-cheroot-8.6.0-4.el9.noarch.rpm 118 kB/s | 172 kB 00:01 2024-02-20T07:46:13.440 INFO:teuthology.orchestra.run.smithi133.stdout:(20/33): python3-jaraco-classes-3.2.1-5.el9.noa 64 kB/s | 18 kB 00:00 2024-02-20T07:46:13.665 INFO:teuthology.orchestra.run.smithi133.stdout:(21/33): python3-jaraco-collections-3.0.0-8.el9 77 kB/s | 23 kB 00:00 2024-02-20T07:46:13.914 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 33/33 2024-02-20T07:46:13.915 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-02-20T07:46:13.915 INFO:teuthology.orchestra.run.smithi183.stdout:Installed: 2024-02-20T07:46:13.915 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-2:17.2.7-521.gc80e7168.el9.x86_64 2024-02-20T07:46:13.915 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-mds-2:17.2.7-521.gc80e7168.el9.x86_64 2024-02-20T07:46:13.916 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-mgr-2:17.2.7-521.gc80e7168.el9.x86_64 2024-02-20T07:46:13.916 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-mgr-modules-core-2:17.2.7-521.gc80e7168.el9.noarch 2024-02-20T07:46:13.916 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-mon-2:17.2.7-521.gc80e7168.el9.x86_64 2024-02-20T07:46:13.917 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-osd-2:17.2.7-521.gc80e7168.el9.x86_64 2024-02-20T07:46:13.917 INFO:teuthology.orchestra.run.smithi183.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-02-20T07:46:13.917 INFO:teuthology.orchestra.run.smithi183.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-02-20T07:46:13.917 INFO:teuthology.orchestra.run.smithi183.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-02-20T07:46:13.917 INFO:teuthology.orchestra.run.smithi183.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-02-20T07:46:13.918 INFO:teuthology.orchestra.run.smithi183.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-02-20T07:46:13.918 INFO:teuthology.orchestra.run.smithi183.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-02-20T07:46:13.918 INFO:teuthology.orchestra.run.smithi183.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-02-20T07:46:13.918 INFO:teuthology.orchestra.run.smithi183.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-02-20T07:46:13.918 INFO:teuthology.orchestra.run.smithi183.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-02-20T07:46:13.918 INFO:teuthology.orchestra.run.smithi183.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-02-20T07:46:13.918 INFO:teuthology.orchestra.run.smithi183.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-02-20T07:46:13.918 INFO:teuthology.orchestra.run.smithi183.stdout: python3-mako-1.1.4-6.el9.noarch 2024-02-20T07:46:13.918 INFO:teuthology.orchestra.run.smithi183.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-02-20T07:46:13.918 INFO:teuthology.orchestra.run.smithi183.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-02-20T07:46:13.918 INFO:teuthology.orchestra.run.smithi183.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-02-20T07:46:13.918 INFO:teuthology.orchestra.run.smithi183.stdout: python3-ply-3.11-14.el9.noarch 2024-02-20T07:46:13.918 INFO:teuthology.orchestra.run.smithi183.stdout: python3-portend-3.1.0-2.el9.noarch 2024-02-20T07:46:13.919 INFO:teuthology.orchestra.run.smithi183.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-02-20T07:46:13.919 INFO:teuthology.orchestra.run.smithi183.stdout: python3-pycparser-2.20-6.el9.noarch 2024-02-20T07:46:13.919 INFO:teuthology.orchestra.run.smithi183.stdout: python3-pytz-2021.1-5.el9.noarch 2024-02-20T07:46:13.919 INFO:teuthology.orchestra.run.smithi183.stdout: python3-requests-2.25.1-8.el9.noarch 2024-02-20T07:46:13.919 INFO:teuthology.orchestra.run.smithi183.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-02-20T07:46:13.919 INFO:teuthology.orchestra.run.smithi183.stdout: python3-toml-0.10.2-6.el9.noarch 2024-02-20T07:46:13.919 INFO:teuthology.orchestra.run.smithi183.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-02-20T07:46:13.919 INFO:teuthology.orchestra.run.smithi183.stdout: python3-webob-1.8.7-6.el9.noarch 2024-02-20T07:46:13.919 INFO:teuthology.orchestra.run.smithi183.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-02-20T07:46:13.919 INFO:teuthology.orchestra.run.smithi183.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-02-20T07:46:13.919 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-02-20T07:46:13.919 INFO:teuthology.orchestra.run.smithi183.stdout:Complete! 2024-02-20T07:46:13.920 INFO:teuthology.orchestra.run.smithi133.stdout:(22/33): python3-jaraco-functools-3.5.0-2.el9.n 67 kB/s | 19 kB 00:00 2024-02-20T07:46:13.942 INFO:teuthology.orchestra.run.smithi133.stdout:(23/33): python3-jaraco-text-3.2.0-6.el9.noarch 71 kB/s | 20 kB 00:00 2024-02-20T07:46:14.066 INFO:teuthology.orchestra.run.smithi133.stdout:(24/33): python3-logutils-0.3.5-21.el9.noarch.r 139 kB/s | 46 kB 00:00 2024-02-20T07:46:14.117 INFO:teuthology.orchestra.run.smithi133.stdout:(25/33): python3-cherrypy-18.6.1-2.el9.noarch.r 199 kB/s | 358 kB 00:01 2024-02-20T07:46:14.219 DEBUG:teuthology.orchestra.run.smithi183:> sudo yum -y install ceph-base 2024-02-20T07:46:14.317 INFO:teuthology.orchestra.run.smithi133.stdout:(26/33): python3-more-itertools-8.12.0-2.el9.no 210 kB/s | 79 kB 00:00 2024-02-20T07:46:14.634 INFO:teuthology.orchestra.run.smithi133.stdout:(27/33): python3-pecan-1.4.2-3.el9.noarch.rpm 480 kB/s | 272 kB 00:00 2024-02-20T07:46:14.813 INFO:teuthology.orchestra.run.smithi183.stdout:Last metadata expiration check: 0:00:53 ago on Tue 20 Feb 2024 07:45:21 AM UTC. 2024-02-20T07:46:14.892 INFO:teuthology.orchestra.run.smithi133.stdout:(28/33): python3-portend-3.1.0-2.el9.noarch.rpm 21 kB/s | 16 kB 00:00 2024-02-20T07:46:14.926 INFO:teuthology.orchestra.run.smithi133.stdout:(29/33): python3-pyOpenSSL-21.0.0-1.el9.noarch. 148 kB/s | 90 kB 00:00 2024-02-20T07:46:14.927 INFO:teuthology.orchestra.run.smithi183.stdout:Package ceph-base-2:17.2.7-521.gc80e7168.el9.x86_64 is already installed. 2024-02-20T07:46:14.951 INFO:teuthology.orchestra.run.smithi133.stdout:(30/33): python3-tempora-5.0.0-2.el9.noarch.rpm 113 kB/s | 36 kB 00:00 2024-02-20T07:46:14.978 INFO:teuthology.orchestra.run.smithi183.stdout:Dependencies resolved. 2024-02-20T07:46:14.979 INFO:teuthology.orchestra.run.smithi183.stdout:Nothing to do. 2024-02-20T07:46:14.979 INFO:teuthology.orchestra.run.smithi183.stdout:Complete! 2024-02-20T07:46:15.044 DEBUG:teuthology.orchestra.run.smithi183:> sudo yum -y install cephadm 2024-02-20T07:46:15.176 INFO:teuthology.orchestra.run.smithi133.stdout:(31/33): python3-webob-1.8.7-6.el9.noarch.rpm 810 kB/s | 230 kB 00:00 2024-02-20T07:46:15.343 INFO:teuthology.orchestra.run.smithi133.stdout:(32/33): python3-werkzeug-2.0.3-3.el9.1.noarch. 1.1 MB/s | 427 kB 00:00 2024-02-20T07:46:15.368 INFO:teuthology.orchestra.run.smithi133.stdout:(33/33): python3-zc-lockfile-2.0-10.el9.noarch. 48 kB/s | 20 kB 00:00 2024-02-20T07:46:15.376 INFO:teuthology.orchestra.run.smithi133.stdout:-------------------------------------------------------------------------------- 2024-02-20T07:46:15.376 INFO:teuthology.orchestra.run.smithi133.stdout:Total 5.7 MB/s | 27 MB 00:04 2024-02-20T07:46:15.377 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction check 2024-02-20T07:46:15.415 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction check succeeded. 2024-02-20T07:46:15.415 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction test 2024-02-20T07:46:15.622 INFO:teuthology.orchestra.run.smithi183.stdout:Last metadata expiration check: 0:00:54 ago on Tue 20 Feb 2024 07:45:21 AM UTC. 2024-02-20T07:46:15.764 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction test succeeded. 2024-02-20T07:46:15.764 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction 2024-02-20T07:46:15.789 INFO:teuthology.orchestra.run.smithi183.stdout:Dependencies resolved. 2024-02-20T07:46:15.790 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-02-20T07:46:15.791 INFO:teuthology.orchestra.run.smithi183.stdout: Package Arch Version Repository Size 2024-02-20T07:46:15.791 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-02-20T07:46:15.791 INFO:teuthology.orchestra.run.smithi183.stdout:Installing: 2024-02-20T07:46:15.791 INFO:teuthology.orchestra.run.smithi183.stdout: cephadm noarch 2:17.2.7-521.gc80e7168.el9 ceph-noarch 80 k 2024-02-20T07:46:15.791 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-02-20T07:46:15.791 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction Summary 2024-02-20T07:46:15.791 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-02-20T07:46:15.791 INFO:teuthology.orchestra.run.smithi183.stdout:Install 1 Package 2024-02-20T07:46:15.792 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-02-20T07:46:15.792 INFO:teuthology.orchestra.run.smithi183.stdout:Total download size: 80 k 2024-02-20T07:46:15.792 INFO:teuthology.orchestra.run.smithi183.stdout:Installed size: 364 k 2024-02-20T07:46:15.792 INFO:teuthology.orchestra.run.smithi183.stdout:Downloading Packages: 2024-02-20T07:46:15.995 INFO:teuthology.orchestra.run.smithi183.stdout:cephadm-17.2.7-521.gc80e7168.el9.noarch.rpm 395 kB/s | 80 kB 00:00 2024-02-20T07:46:15.996 INFO:teuthology.orchestra.run.smithi183.stdout:-------------------------------------------------------------------------------- 2024-02-20T07:46:15.996 INFO:teuthology.orchestra.run.smithi183.stdout:Total 392 kB/s | 80 kB 00:00 2024-02-20T07:46:15.996 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction check 2024-02-20T07:46:16.001 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction check succeeded. 2024-02-20T07:46:16.001 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction test 2024-02-20T07:46:16.003 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction test succeeded. 2024-02-20T07:46:16.003 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction 2024-02-20T07:46:16.075 INFO:teuthology.orchestra.run.smithi183.stdout: Preparing : 1/1 2024-02-20T07:46:16.534 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: cephadm-2:17.2.7-521.gc80e7168.el9.noarch 1/1 2024-02-20T07:46:16.633 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : cephadm-2:17.2.7-521.gc80e7168.el9.noarch 1/1 2024-02-20T07:46:16.970 INFO:teuthology.orchestra.run.smithi133.stdout: Preparing : 1/1 2024-02-20T07:46:16.987 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: cephadm-2:17.2.7-521.gc80e7168.el9.noarch 1/1 2024-02-20T07:46:17.030 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/33 2024-02-20T07:46:17.061 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/33 2024-02-20T07:46:17.094 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 3/33 2024-02-20T07:46:17.186 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 4/33 2024-02-20T07:46:17.264 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 5/33 2024-02-20T07:46:17.285 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : cephadm-2:17.2.7-521.gc80e7168.el9.noarch 1/1 2024-02-20T07:46:17.285 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-02-20T07:46:17.285 INFO:teuthology.orchestra.run.smithi183.stdout:Installed: 2024-02-20T07:46:17.285 INFO:teuthology.orchestra.run.smithi183.stdout: cephadm-2:17.2.7-521.gc80e7168.el9.noarch 2024-02-20T07:46:17.285 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-02-20T07:46:17.285 INFO:teuthology.orchestra.run.smithi183.stdout:Complete! 2024-02-20T07:46:17.305 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 6/33 2024-02-20T07:46:17.377 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 7/33 2024-02-20T07:46:17.430 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 8/33 2024-02-20T07:46:17.474 DEBUG:teuthology.orchestra.run.smithi183:> sudo yum -y install ceph-immutable-object-cache 2024-02-20T07:46:17.480 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 9/33 2024-02-20T07:46:17.526 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-ply-3.11-14.el9.noarch 10/33 2024-02-20T07:46:17.581 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 11/33 2024-02-20T07:46:17.766 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 12/33 2024-02-20T07:46:17.823 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 13/33 2024-02-20T07:46:17.854 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 14/33 2024-02-20T07:46:17.888 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 15/33 2024-02-20T07:46:17.918 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 16/33 2024-02-20T07:46:17.994 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 17/33 2024-02-20T07:46:18.046 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 18/33 2024-02-20T07:46:18.057 INFO:teuthology.orchestra.run.smithi183.stdout:Last metadata expiration check: 0:00:57 ago on Tue 20 Feb 2024 07:45:21 AM UTC. 2024-02-20T07:46:18.079 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 19/33 2024-02-20T07:46:18.120 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 20/33 2024-02-20T07:46:18.154 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 21/33 2024-02-20T07:46:18.189 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 22/33 2024-02-20T07:46:18.225 INFO:teuthology.orchestra.run.smithi183.stdout:Dependencies resolved. 2024-02-20T07:46:18.225 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-02-20T07:46:18.226 INFO:teuthology.orchestra.run.smithi183.stdout: Package Arch Version Repo Size 2024-02-20T07:46:18.226 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-02-20T07:46:18.226 INFO:teuthology.orchestra.run.smithi183.stdout:Installing: 2024-02-20T07:46:18.226 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-immutable-object-cache x86_64 2:17.2.7-521.gc80e7168.el9 ceph 137 k 2024-02-20T07:46:18.226 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-02-20T07:46:18.226 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction Summary 2024-02-20T07:46:18.226 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-02-20T07:46:18.226 INFO:teuthology.orchestra.run.smithi183.stdout:Install 1 Package 2024-02-20T07:46:18.226 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-02-20T07:46:18.227 INFO:teuthology.orchestra.run.smithi183.stdout:Total download size: 137 k 2024-02-20T07:46:18.227 INFO:teuthology.orchestra.run.smithi183.stdout:Installed size: 379 k 2024-02-20T07:46:18.227 INFO:teuthology.orchestra.run.smithi183.stdout:Downloading Packages: 2024-02-20T07:46:18.278 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 23/33 2024-02-20T07:46:18.316 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 24/33 2024-02-20T07:46:18.392 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 25/33 2024-02-20T07:46:18.459 INFO:teuthology.orchestra.run.smithi183.stdout:ceph-immutable-object-cache-17.2.7-521.gc80e716 591 kB/s | 137 kB 00:00 2024-02-20T07:46:18.460 INFO:teuthology.orchestra.run.smithi183.stdout:-------------------------------------------------------------------------------- 2024-02-20T07:46:18.460 INFO:teuthology.orchestra.run.smithi183.stdout:Total 587 kB/s | 137 kB 00:00 2024-02-20T07:46:18.460 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction check 2024-02-20T07:46:18.467 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction check succeeded. 2024-02-20T07:46:18.467 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction test 2024-02-20T07:46:18.518 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction test succeeded. 2024-02-20T07:46:18.518 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction 2024-02-20T07:46:18.528 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 26/33 2024-02-20T07:46:18.662 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 27/33 2024-02-20T07:46:18.663 INFO:teuthology.orchestra.run.smithi183.stdout: Preparing : 1/1 2024-02-20T07:46:18.720 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : ceph-immutable-object-cache-2:17.2.7-521.gc80e7168.e 1/1 2024-02-20T07:46:18.746 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: ceph-immutable-object-cache-2:17.2.7-521.gc80e7168.e 1/1 2024-02-20T07:46:18.746 INFO:teuthology.orchestra.run.smithi183.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-02-20T07:46:18.746 INFO:teuthology.orchestra.run.smithi183.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-02-20T07:46:18.746 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-02-20T07:46:18.813 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : ceph-mgr-modules-core-2:17.2.7-521.gc80e7168.el9.n 28/33 2024-02-20T07:46:18.844 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : ceph-mgr-2:17.2.7-521.gc80e7168.el9.x86_64 29/33 2024-02-20T07:46:18.870 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: ceph-mgr-2:17.2.7-521.gc80e7168.el9.x86_64 29/33 2024-02-20T07:46:18.870 INFO:teuthology.orchestra.run.smithi133.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-02-20T07:46:18.870 INFO:teuthology.orchestra.run.smithi133.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-02-20T07:46:18.870 INFO:teuthology.orchestra.run.smithi133.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-02-20T07:46:18.870 INFO:teuthology.orchestra.run.smithi133.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-02-20T07:46:18.870 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-02-20T07:46:19.778 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : ceph-immutable-object-cache-2:17.2.7-521.gc80e7168.e 1/1 2024-02-20T07:46:19.778 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-02-20T07:46:19.778 INFO:teuthology.orchestra.run.smithi183.stdout:Installed: 2024-02-20T07:46:19.778 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-immutable-object-cache-2:17.2.7-521.gc80e7168.el9.x86_64 2024-02-20T07:46:19.778 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-02-20T07:46:19.778 INFO:teuthology.orchestra.run.smithi183.stdout:Complete! 2024-02-20T07:46:20.024 DEBUG:teuthology.orchestra.run.smithi183:> sudo yum -y install ceph-mgr 2024-02-20T07:46:20.194 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : ceph-osd-2:17.2.7-521.gc80e7168.el9.x86_64 30/33 2024-02-20T07:46:20.221 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: ceph-osd-2:17.2.7-521.gc80e7168.el9.x86_64 30/33 2024-02-20T07:46:20.221 INFO:teuthology.orchestra.run.smithi133.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-02-20T07:46:20.221 INFO:teuthology.orchestra.run.smithi133.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-02-20T07:46:20.221 INFO:teuthology.orchestra.run.smithi133.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-02-20T07:46:20.221 INFO:teuthology.orchestra.run.smithi133.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-02-20T07:46:20.221 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-02-20T07:46:20.595 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : ceph-mon-2:17.2.7-521.gc80e7168.el9.x86_64 31/33 2024-02-20T07:46:20.618 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: ceph-mon-2:17.2.7-521.gc80e7168.el9.x86_64 31/33 2024-02-20T07:46:20.618 INFO:teuthology.orchestra.run.smithi133.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-02-20T07:46:20.618 INFO:teuthology.orchestra.run.smithi133.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-02-20T07:46:20.619 INFO:teuthology.orchestra.run.smithi133.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-02-20T07:46:20.619 INFO:teuthology.orchestra.run.smithi133.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-02-20T07:46:20.619 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-02-20T07:46:20.620 INFO:teuthology.orchestra.run.smithi183.stdout:Last metadata expiration check: 0:00:59 ago on Tue 20 Feb 2024 07:45:21 AM UTC. 2024-02-20T07:46:20.736 INFO:teuthology.orchestra.run.smithi183.stdout:Package ceph-mgr-2:17.2.7-521.gc80e7168.el9.x86_64 is already installed. 2024-02-20T07:46:20.790 INFO:teuthology.orchestra.run.smithi183.stdout:Dependencies resolved. 2024-02-20T07:46:20.791 INFO:teuthology.orchestra.run.smithi183.stdout:Nothing to do. 2024-02-20T07:46:20.791 INFO:teuthology.orchestra.run.smithi183.stdout:Complete! 2024-02-20T07:46:20.820 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : ceph-mds-2:17.2.7-521.gc80e7168.el9.x86_64 32/33 2024-02-20T07:46:20.844 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: ceph-mds-2:17.2.7-521.gc80e7168.el9.x86_64 32/33 2024-02-20T07:46:20.844 INFO:teuthology.orchestra.run.smithi133.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-02-20T07:46:20.844 INFO:teuthology.orchestra.run.smithi133.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-02-20T07:46:20.844 INFO:teuthology.orchestra.run.smithi133.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-02-20T07:46:20.844 INFO:teuthology.orchestra.run.smithi133.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-02-20T07:46:20.844 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-02-20T07:46:20.851 DEBUG:teuthology.orchestra.run.smithi183:> sudo yum -y install ceph-mgr-dashboard 2024-02-20T07:46:20.909 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : ceph-2:17.2.7-521.gc80e7168.el9.x86_64 33/33 2024-02-20T07:46:21.424 INFO:teuthology.orchestra.run.smithi183.stdout:Last metadata expiration check: 0:01:00 ago on Tue 20 Feb 2024 07:45:21 AM UTC. 2024-02-20T07:46:21.590 INFO:teuthology.orchestra.run.smithi183.stdout:Dependencies resolved. 2024-02-20T07:46:21.590 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-02-20T07:46:21.591 INFO:teuthology.orchestra.run.smithi183.stdout: Package Arch Version Repository Size 2024-02-20T07:46:21.591 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-02-20T07:46:21.591 INFO:teuthology.orchestra.run.smithi183.stdout:Installing: 2024-02-20T07:46:21.591 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-mgr-dashboard noarch 2:17.2.7-521.gc80e7168.el9 ceph-noarch 1.7 M 2024-02-20T07:46:21.591 INFO:teuthology.orchestra.run.smithi183.stdout:Installing dependencies: 2024-02-20T07:46:21.591 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-grafana-dashboards noarch 2:17.2.7-521.gc80e7168.el9 ceph-noarch 23 k 2024-02-20T07:46:21.591 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-prometheus-alerts noarch 2:17.2.7-521.gc80e7168.el9 ceph-noarch 15 k 2024-02-20T07:46:21.591 INFO:teuthology.orchestra.run.smithi183.stdout: python3-jwt noarch 2.4.0-1.el9 epel 41 k 2024-02-20T07:46:21.591 INFO:teuthology.orchestra.run.smithi183.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-02-20T07:46:21.591 INFO:teuthology.orchestra.run.smithi183.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-02-20T07:46:21.591 INFO:teuthology.orchestra.run.smithi183.stdout:Installing weak dependencies: 2024-02-20T07:46:21.591 INFO:teuthology.orchestra.run.smithi183.stdout: python3-jwt+crypto noarch 2.4.0-1.el9 epel 9.0 k 2024-02-20T07:46:21.591 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-02-20T07:46:21.591 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction Summary 2024-02-20T07:46:21.591 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-02-20T07:46:21.592 INFO:teuthology.orchestra.run.smithi183.stdout:Install 7 Packages 2024-02-20T07:46:21.592 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-02-20T07:46:21.592 INFO:teuthology.orchestra.run.smithi183.stdout:Total download size: 2.0 M 2024-02-20T07:46:21.592 INFO:teuthology.orchestra.run.smithi183.stdout:Installed size: 8.1 M 2024-02-20T07:46:21.592 INFO:teuthology.orchestra.run.smithi183.stdout:Downloading Packages: 2024-02-20T07:46:21.914 INFO:teuthology.orchestra.run.smithi183.stdout:(1/7): ceph-prometheus-alerts-17.2.7-521.gc80e7 57 kB/s | 15 kB 00:00 2024-02-20T07:46:21.940 INFO:teuthology.orchestra.run.smithi183.stdout:(2/7): ceph-grafana-dashboards-17.2.7-521.gc80e 83 kB/s | 23 kB 00:00 2024-02-20T07:46:22.057 INFO:teuthology.orchestra.run.smithi183.stdout:(3/7): python3-jwt+crypto-2.4.0-1.el9.noarch.rp 64 kB/s | 9.0 kB 00:00 2024-02-20T07:46:22.123 INFO:teuthology.orchestra.run.smithi183.stdout:(4/7): ceph-mgr-dashboard-17.2.7-521.gc80e7168. 3.6 MB/s | 1.7 MB 00:00 2024-02-20T07:46:22.149 INFO:teuthology.orchestra.run.smithi183.stdout:(5/7): python3-jwt-2.4.0-1.el9.noarch.rpm 195 kB/s | 41 kB 00:00 2024-02-20T07:46:22.176 INFO:teuthology.orchestra.run.smithi183.stdout:(6/7): python3-repoze-lru-0.7-16.el9.noarch.rpm 264 kB/s | 31 kB 00:00 2024-02-20T07:46:22.208 INFO:teuthology.orchestra.run.smithi183.stdout:(7/7): python3-routes-2.5.1-5.el9.noarch.rpm 2.2 MB/s | 188 kB 00:00 2024-02-20T07:46:22.211 INFO:teuthology.orchestra.run.smithi183.stdout:-------------------------------------------------------------------------------- 2024-02-20T07:46:22.212 INFO:teuthology.orchestra.run.smithi183.stdout:Total 3.2 MB/s | 2.0 MB 00:00 2024-02-20T07:46:22.212 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction check 2024-02-20T07:46:22.219 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction check succeeded. 2024-02-20T07:46:22.220 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction test 2024-02-20T07:46:22.288 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction test succeeded. 2024-02-20T07:46:22.289 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction 2024-02-20T07:46:22.459 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: ceph-2:17.2.7-521.gc80e7168.el9.x86_64 33/33 2024-02-20T07:46:22.459 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : ceph-2:17.2.7-521.gc80e7168.el9.x86_64 1/33 2024-02-20T07:46:22.459 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : ceph-mds-2:17.2.7-521.gc80e7168.el9.x86_64 2/33 2024-02-20T07:46:22.459 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : ceph-mgr-2:17.2.7-521.gc80e7168.el9.x86_64 3/33 2024-02-20T07:46:22.459 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : ceph-mon-2:17.2.7-521.gc80e7168.el9.x86_64 4/33 2024-02-20T07:46:22.459 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : ceph-osd-2:17.2.7-521.gc80e7168.el9.x86_64 5/33 2024-02-20T07:46:22.459 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : ceph-mgr-modules-core-2:17.2.7-521.gc80e7168.el9.n 6/33 2024-02-20T07:46:22.460 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 7/33 2024-02-20T07:46:22.460 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 8/33 2024-02-20T07:46:22.460 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 9/33 2024-02-20T07:46:22.460 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 10/33 2024-02-20T07:46:22.460 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 11/33 2024-02-20T07:46:22.460 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 12/33 2024-02-20T07:46:22.460 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-ply-3.11-14.el9.noarch 13/33 2024-02-20T07:46:22.460 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 14/33 2024-02-20T07:46:22.460 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 15/33 2024-02-20T07:46:22.460 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 16/33 2024-02-20T07:46:22.460 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 17/33 2024-02-20T07:46:22.460 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 18/33 2024-02-20T07:46:22.460 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 19/33 2024-02-20T07:46:22.460 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 20/33 2024-02-20T07:46:22.461 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 21/33 2024-02-20T07:46:22.461 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 22/33 2024-02-20T07:46:22.461 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 23/33 2024-02-20T07:46:22.461 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 24/33 2024-02-20T07:46:22.461 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 25/33 2024-02-20T07:46:22.461 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 26/33 2024-02-20T07:46:22.461 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 27/33 2024-02-20T07:46:22.461 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 28/33 2024-02-20T07:46:22.461 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 29/33 2024-02-20T07:46:22.461 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 30/33 2024-02-20T07:46:22.461 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 31/33 2024-02-20T07:46:22.461 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 32/33 2024-02-20T07:46:22.599 INFO:teuthology.orchestra.run.smithi183.stdout: Preparing : 1/1 2024-02-20T07:46:22.651 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-jwt-2.4.0-1.el9.noarch 1/7 2024-02-20T07:46:22.677 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-jwt+crypto-2.4.0-1.el9.noarch 2/7 2024-02-20T07:46:22.731 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/7 2024-02-20T07:46:22.772 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/7 2024-02-20T07:46:22.800 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : ceph-prometheus-alerts-2:17.2.7-521.gc80e7168.el9.no 5/7 2024-02-20T07:46:23.030 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : ceph-grafana-dashboards-2:17.2.7-521.gc80e7168.el9.n 6/7 2024-02-20T07:46:23.062 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : ceph-mgr-dashboard-2:17.2.7-521.gc80e7168.el9.noarch 7/7 2024-02-20T07:46:23.408 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 33/33 2024-02-20T07:46:23.408 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-02-20T07:46:23.408 INFO:teuthology.orchestra.run.smithi133.stdout:Installed: 2024-02-20T07:46:23.408 INFO:teuthology.orchestra.run.smithi133.stdout: ceph-2:17.2.7-521.gc80e7168.el9.x86_64 2024-02-20T07:46:23.408 INFO:teuthology.orchestra.run.smithi133.stdout: ceph-mds-2:17.2.7-521.gc80e7168.el9.x86_64 2024-02-20T07:46:23.409 INFO:teuthology.orchestra.run.smithi133.stdout: ceph-mgr-2:17.2.7-521.gc80e7168.el9.x86_64 2024-02-20T07:46:23.409 INFO:teuthology.orchestra.run.smithi133.stdout: ceph-mgr-modules-core-2:17.2.7-521.gc80e7168.el9.noarch 2024-02-20T07:46:23.409 INFO:teuthology.orchestra.run.smithi133.stdout: ceph-mon-2:17.2.7-521.gc80e7168.el9.x86_64 2024-02-20T07:46:23.409 INFO:teuthology.orchestra.run.smithi133.stdout: ceph-osd-2:17.2.7-521.gc80e7168.el9.x86_64 2024-02-20T07:46:23.409 INFO:teuthology.orchestra.run.smithi133.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-02-20T07:46:23.409 INFO:teuthology.orchestra.run.smithi133.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-02-20T07:46:23.409 INFO:teuthology.orchestra.run.smithi133.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-02-20T07:46:23.409 INFO:teuthology.orchestra.run.smithi133.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-02-20T07:46:23.409 INFO:teuthology.orchestra.run.smithi133.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-02-20T07:46:23.409 INFO:teuthology.orchestra.run.smithi133.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-02-20T07:46:23.409 INFO:teuthology.orchestra.run.smithi133.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-02-20T07:46:23.409 INFO:teuthology.orchestra.run.smithi133.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-02-20T07:46:23.409 INFO:teuthology.orchestra.run.smithi133.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-02-20T07:46:23.409 INFO:teuthology.orchestra.run.smithi133.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-02-20T07:46:23.409 INFO:teuthology.orchestra.run.smithi133.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-02-20T07:46:23.410 INFO:teuthology.orchestra.run.smithi133.stdout: python3-mako-1.1.4-6.el9.noarch 2024-02-20T07:46:23.410 INFO:teuthology.orchestra.run.smithi133.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-02-20T07:46:23.410 INFO:teuthology.orchestra.run.smithi133.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-02-20T07:46:23.410 INFO:teuthology.orchestra.run.smithi133.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-02-20T07:46:23.410 INFO:teuthology.orchestra.run.smithi133.stdout: python3-ply-3.11-14.el9.noarch 2024-02-20T07:46:23.410 INFO:teuthology.orchestra.run.smithi133.stdout: python3-portend-3.1.0-2.el9.noarch 2024-02-20T07:46:23.410 INFO:teuthology.orchestra.run.smithi133.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-02-20T07:46:23.410 INFO:teuthology.orchestra.run.smithi133.stdout: python3-pycparser-2.20-6.el9.noarch 2024-02-20T07:46:23.410 INFO:teuthology.orchestra.run.smithi133.stdout: python3-pytz-2021.1-5.el9.noarch 2024-02-20T07:46:23.410 INFO:teuthology.orchestra.run.smithi133.stdout: python3-requests-2.25.1-8.el9.noarch 2024-02-20T07:46:23.410 INFO:teuthology.orchestra.run.smithi133.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-02-20T07:46:23.410 INFO:teuthology.orchestra.run.smithi133.stdout: python3-toml-0.10.2-6.el9.noarch 2024-02-20T07:46:23.410 INFO:teuthology.orchestra.run.smithi133.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-02-20T07:46:23.410 INFO:teuthology.orchestra.run.smithi133.stdout: python3-webob-1.8.7-6.el9.noarch 2024-02-20T07:46:23.410 INFO:teuthology.orchestra.run.smithi133.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-02-20T07:46:23.411 INFO:teuthology.orchestra.run.smithi133.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-02-20T07:46:23.411 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-02-20T07:46:23.411 INFO:teuthology.orchestra.run.smithi133.stdout:Complete! 2024-02-20T07:46:23.567 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: ceph-mgr-dashboard-2:17.2.7-521.gc80e7168.el9.noarch 7/7 2024-02-20T07:46:23.567 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : ceph-grafana-dashboards-2:17.2.7-521.gc80e7168.el9.n 1/7 2024-02-20T07:46:23.568 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : ceph-mgr-dashboard-2:17.2.7-521.gc80e7168.el9.noarch 2/7 2024-02-20T07:46:23.568 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : ceph-prometheus-alerts-2:17.2.7-521.gc80e7168.el9.no 3/7 2024-02-20T07:46:23.568 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-jwt+crypto-2.4.0-1.el9.noarch 4/7 2024-02-20T07:46:23.568 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-jwt-2.4.0-1.el9.noarch 5/7 2024-02-20T07:46:23.568 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 6/7 2024-02-20T07:46:23.760 DEBUG:teuthology.orchestra.run.smithi133:> sudo yum -y install ceph-base 2024-02-20T07:46:23.876 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 7/7 2024-02-20T07:46:23.876 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-02-20T07:46:23.876 INFO:teuthology.orchestra.run.smithi183.stdout:Installed: 2024-02-20T07:46:23.876 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-grafana-dashboards-2:17.2.7-521.gc80e7168.el9.noarch 2024-02-20T07:46:23.876 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-mgr-dashboard-2:17.2.7-521.gc80e7168.el9.noarch 2024-02-20T07:46:23.876 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-prometheus-alerts-2:17.2.7-521.gc80e7168.el9.noarch 2024-02-20T07:46:23.876 INFO:teuthology.orchestra.run.smithi183.stdout: python3-jwt-2.4.0-1.el9.noarch 2024-02-20T07:46:23.876 INFO:teuthology.orchestra.run.smithi183.stdout: python3-jwt+crypto-2.4.0-1.el9.noarch 2024-02-20T07:46:23.876 INFO:teuthology.orchestra.run.smithi183.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-02-20T07:46:23.876 INFO:teuthology.orchestra.run.smithi183.stdout: python3-routes-2.5.1-5.el9.noarch 2024-02-20T07:46:23.876 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-02-20T07:46:23.876 INFO:teuthology.orchestra.run.smithi183.stdout:Complete! 2024-02-20T07:46:24.068 DEBUG:teuthology.orchestra.run.smithi183:> sudo yum -y install ceph-mgr-diskprediction-local 2024-02-20T07:46:24.349 INFO:teuthology.orchestra.run.smithi133.stdout:Last metadata expiration check: 0:01:04 ago on Tue 20 Feb 2024 07:45:20 AM UTC. 2024-02-20T07:46:24.462 INFO:teuthology.orchestra.run.smithi133.stdout:Package ceph-base-2:17.2.7-521.gc80e7168.el9.x86_64 is already installed. 2024-02-20T07:46:24.512 INFO:teuthology.orchestra.run.smithi133.stdout:Dependencies resolved. 2024-02-20T07:46:24.513 INFO:teuthology.orchestra.run.smithi133.stdout:Nothing to do. 2024-02-20T07:46:24.513 INFO:teuthology.orchestra.run.smithi133.stdout:Complete! 2024-02-20T07:46:24.582 DEBUG:teuthology.orchestra.run.smithi133:> sudo yum -y install cephadm 2024-02-20T07:46:24.642 INFO:teuthology.orchestra.run.smithi183.stdout:Last metadata expiration check: 0:01:03 ago on Tue 20 Feb 2024 07:45:21 AM UTC. 2024-02-20T07:46:24.810 INFO:teuthology.orchestra.run.smithi183.stdout:Dependencies resolved. 2024-02-20T07:46:24.811 INFO:teuthology.orchestra.run.smithi183.stdout:=========================================================================================== 2024-02-20T07:46:24.811 INFO:teuthology.orchestra.run.smithi183.stdout: Package Arch Version Repository Size 2024-02-20T07:46:24.811 INFO:teuthology.orchestra.run.smithi183.stdout:=========================================================================================== 2024-02-20T07:46:24.811 INFO:teuthology.orchestra.run.smithi183.stdout:Installing: 2024-02-20T07:46:24.811 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-mgr-diskprediction-local noarch 2:17.2.7-521.gc80e7168.el9 ceph-noarch 7.4 M 2024-02-20T07:46:24.811 INFO:teuthology.orchestra.run.smithi183.stdout:Installing dependencies: 2024-02-20T07:46:24.812 INFO:teuthology.orchestra.run.smithi183.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-02-20T07:46:24.812 INFO:teuthology.orchestra.run.smithi183.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-02-20T07:46:24.812 INFO:teuthology.orchestra.run.smithi183.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-02-20T07:46:24.812 INFO:teuthology.orchestra.run.smithi183.stdout: libgfortran x86_64 11.4.1-3.el9 CentOS-BaseOS 802 k 2024-02-20T07:46:24.812 INFO:teuthology.orchestra.run.smithi183.stdout: libquadmath x86_64 11.4.1-3.el9 CentOS-BaseOS 194 k 2024-02-20T07:46:24.812 INFO:teuthology.orchestra.run.smithi183.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-02-20T07:46:24.812 INFO:teuthology.orchestra.run.smithi183.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-02-20T07:46:24.812 INFO:teuthology.orchestra.run.smithi183.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-02-20T07:46:24.812 INFO:teuthology.orchestra.run.smithi183.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-02-20T07:46:24.812 INFO:teuthology.orchestra.run.smithi183.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-02-20T07:46:24.812 INFO:teuthology.orchestra.run.smithi183.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-02-20T07:46:24.812 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-02-20T07:46:24.812 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction Summary 2024-02-20T07:46:24.813 INFO:teuthology.orchestra.run.smithi183.stdout:=========================================================================================== 2024-02-20T07:46:24.813 INFO:teuthology.orchestra.run.smithi183.stdout:Install 12 Packages 2024-02-20T07:46:24.813 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-02-20T07:46:24.813 INFO:teuthology.orchestra.run.smithi183.stdout:Total download size: 37 M 2024-02-20T07:46:24.813 INFO:teuthology.orchestra.run.smithi183.stdout:Installed size: 209 M 2024-02-20T07:46:24.813 INFO:teuthology.orchestra.run.smithi183.stdout:Downloading Packages: 2024-02-20T07:46:25.015 INFO:teuthology.orchestra.run.smithi183.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 225 kB/s | 32 kB 00:00 2024-02-20T07:46:25.048 INFO:teuthology.orchestra.run.smithi183.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 537 kB/s | 17 kB 00:00 2024-02-20T07:46:25.099 INFO:teuthology.orchestra.run.smithi183.stdout:(3/12): openblas-0.3.21-2.el9.x86_64.rpm 696 kB/s | 34 kB 00:00 2024-02-20T07:46:25.149 INFO:teuthology.orchestra.run.smithi183.stdout:(4/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 11 MB/s | 3.0 MB 00:00 2024-02-20T07:46:25.160 INFO:teuthology.orchestra.run.smithi133.stdout:Last metadata expiration check: 0:01:05 ago on Tue 20 Feb 2024 07:45:20 AM UTC. 2024-02-20T07:46:25.324 INFO:teuthology.orchestra.run.smithi183.stdout:(5/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 21 MB/s | 4.7 MB 00:00 2024-02-20T07:46:25.326 INFO:teuthology.orchestra.run.smithi133.stdout:Dependencies resolved. 2024-02-20T07:46:25.326 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2024-02-20T07:46:25.327 INFO:teuthology.orchestra.run.smithi133.stdout: Package Arch Version Repository Size 2024-02-20T07:46:25.327 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2024-02-20T07:46:25.327 INFO:teuthology.orchestra.run.smithi133.stdout:Installing: 2024-02-20T07:46:25.327 INFO:teuthology.orchestra.run.smithi133.stdout: cephadm noarch 2:17.2.7-521.gc80e7168.el9 ceph-noarch 80 k 2024-02-20T07:46:25.327 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-02-20T07:46:25.327 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction Summary 2024-02-20T07:46:25.327 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2024-02-20T07:46:25.327 INFO:teuthology.orchestra.run.smithi133.stdout:Install 1 Package 2024-02-20T07:46:25.327 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-02-20T07:46:25.328 INFO:teuthology.orchestra.run.smithi133.stdout:Total download size: 80 k 2024-02-20T07:46:25.328 INFO:teuthology.orchestra.run.smithi133.stdout:Installed size: 364 k 2024-02-20T07:46:25.328 INFO:teuthology.orchestra.run.smithi133.stdout:Downloading Packages: 2024-02-20T07:46:25.391 INFO:teuthology.orchestra.run.smithi183.stdout:(6/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 20 MB/s | 4.8 MB 00:00 2024-02-20T07:46:25.417 INFO:teuthology.orchestra.run.smithi183.stdout:(7/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.4 MB/s | 217 kB 00:00 2024-02-20T07:46:25.559 INFO:teuthology.orchestra.run.smithi183.stdout:(8/12): libgfortran-11.4.1-3.el9.x86_64.rpm 5.6 MB/s | 802 kB 00:00 2024-02-20T07:46:25.677 INFO:teuthology.orchestra.run.smithi133.stdout:cephadm-17.2.7-521.gc80e7168.el9.noarch.rpm 231 kB/s | 80 kB 00:00 2024-02-20T07:46:25.678 INFO:teuthology.orchestra.run.smithi133.stdout:-------------------------------------------------------------------------------- 2024-02-20T07:46:25.678 INFO:teuthology.orchestra.run.smithi133.stdout:Total 230 kB/s | 80 kB 00:00 2024-02-20T07:46:25.678 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction check 2024-02-20T07:46:25.683 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction check succeeded. 2024-02-20T07:46:25.683 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction test 2024-02-20T07:46:25.685 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction test succeeded. 2024-02-20T07:46:25.685 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction 2024-02-20T07:46:25.725 INFO:teuthology.orchestra.run.smithi183.stdout:(9/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 47 MB/s | 16 MB 00:00 2024-02-20T07:46:25.751 INFO:teuthology.orchestra.run.smithi183.stdout:(10/12): libquadmath-11.4.1-3.el9.x86_64.rpm 1.0 MB/s | 194 kB 00:00 2024-02-20T07:46:25.765 INFO:teuthology.orchestra.run.smithi133.stdout: Preparing : 1/1 2024-02-20T07:46:25.976 INFO:teuthology.orchestra.run.smithi183.stdout:(11/12): python3-devel-3.9.17-2.el9.x86_64.rpm 983 kB/s | 245 kB 00:00 2024-02-20T07:46:26.175 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: cephadm-2:17.2.7-521.gc80e7168.el9.noarch 1/1 2024-02-20T07:46:26.256 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : cephadm-2:17.2.7-521.gc80e7168.el9.noarch 1/1 2024-02-20T07:46:26.326 INFO:teuthology.orchestra.run.smithi183.stdout:(12/12): ceph-mgr-diskprediction-local-17.2.7-5 5.1 MB/s | 7.4 MB 00:01 2024-02-20T07:46:26.329 INFO:teuthology.orchestra.run.smithi183.stdout:-------------------------------------------------------------------------------- 2024-02-20T07:46:26.329 INFO:teuthology.orchestra.run.smithi183.stdout:Total 24 MB/s | 37 MB 00:01 2024-02-20T07:46:26.349 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction check 2024-02-20T07:46:26.373 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction check succeeded. 2024-02-20T07:46:26.373 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction test 2024-02-20T07:46:26.649 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: cephadm-2:17.2.7-521.gc80e7168.el9.noarch 1/1 2024-02-20T07:46:26.805 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction test succeeded. 2024-02-20T07:46:26.805 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction 2024-02-20T07:46:26.986 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : cephadm-2:17.2.7-521.gc80e7168.el9.noarch 1/1 2024-02-20T07:46:26.986 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-02-20T07:46:26.986 INFO:teuthology.orchestra.run.smithi133.stdout:Installed: 2024-02-20T07:46:26.986 INFO:teuthology.orchestra.run.smithi133.stdout: cephadm-2:17.2.7-521.gc80e7168.el9.noarch 2024-02-20T07:46:26.986 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-02-20T07:46:26.986 INFO:teuthology.orchestra.run.smithi133.stdout:Complete! 2024-02-20T07:46:27.166 DEBUG:teuthology.orchestra.run.smithi133:> sudo yum -y install ceph-immutable-object-cache 2024-02-20T07:46:27.501 INFO:teuthology.orchestra.run.smithi183.stdout: Preparing : 1/1 2024-02-20T07:46:27.589 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-02-20T07:46:27.622 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-02-20T07:46:27.725 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-02-20T07:46:27.754 INFO:teuthology.orchestra.run.smithi133.stdout:Last metadata expiration check: 0:01:07 ago on Tue 20 Feb 2024 07:45:20 AM UTC. 2024-02-20T07:46:27.758 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-02-20T07:46:27.926 INFO:teuthology.orchestra.run.smithi133.stdout:Dependencies resolved. 2024-02-20T07:46:27.927 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2024-02-20T07:46:27.927 INFO:teuthology.orchestra.run.smithi133.stdout: Package Arch Version Repo Size 2024-02-20T07:46:27.927 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2024-02-20T07:46:27.927 INFO:teuthology.orchestra.run.smithi133.stdout:Installing: 2024-02-20T07:46:27.927 INFO:teuthology.orchestra.run.smithi133.stdout: ceph-immutable-object-cache x86_64 2:17.2.7-521.gc80e7168.el9 ceph 137 k 2024-02-20T07:46:27.927 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-02-20T07:46:27.928 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction Summary 2024-02-20T07:46:27.928 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2024-02-20T07:46:27.928 INFO:teuthology.orchestra.run.smithi133.stdout:Install 1 Package 2024-02-20T07:46:27.928 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-02-20T07:46:27.928 INFO:teuthology.orchestra.run.smithi133.stdout:Total download size: 137 k 2024-02-20T07:46:27.928 INFO:teuthology.orchestra.run.smithi133.stdout:Installed size: 379 k 2024-02-20T07:46:27.928 INFO:teuthology.orchestra.run.smithi133.stdout:Downloading Packages: 2024-02-20T07:46:27.929 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-02-20T07:46:27.957 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-02-20T07:46:28.048 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-02-20T07:46:28.248 INFO:teuthology.orchestra.run.smithi133.stdout:ceph-immutable-object-cache-17.2.7-521.gc80e716 428 kB/s | 137 kB 00:00 2024-02-20T07:46:28.249 INFO:teuthology.orchestra.run.smithi133.stdout:-------------------------------------------------------------------------------- 2024-02-20T07:46:28.249 INFO:teuthology.orchestra.run.smithi133.stdout:Total 426 kB/s | 137 kB 00:00 2024-02-20T07:46:28.249 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction check 2024-02-20T07:46:28.256 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction check succeeded. 2024-02-20T07:46:28.256 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction test 2024-02-20T07:46:28.307 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction test succeeded. 2024-02-20T07:46:28.308 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction 2024-02-20T07:46:28.453 INFO:teuthology.orchestra.run.smithi133.stdout: Preparing : 1/1 2024-02-20T07:46:28.508 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : ceph-immutable-object-cache-2:17.2.7-521.gc80e7168.e 1/1 2024-02-20T07:46:28.532 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: ceph-immutable-object-cache-2:17.2.7-521.gc80e7168.e 1/1 2024-02-20T07:46:28.532 INFO:teuthology.orchestra.run.smithi133.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-02-20T07:46:28.532 INFO:teuthology.orchestra.run.smithi133.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-02-20T07:46:28.532 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-02-20T07:46:28.704 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-02-20T07:46:28.820 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-02-20T07:46:29.550 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : ceph-immutable-object-cache-2:17.2.7-521.gc80e7168.e 1/1 2024-02-20T07:46:29.551 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-02-20T07:46:29.551 INFO:teuthology.orchestra.run.smithi133.stdout:Installed: 2024-02-20T07:46:29.551 INFO:teuthology.orchestra.run.smithi133.stdout: ceph-immutable-object-cache-2:17.2.7-521.gc80e7168.el9.x86_64 2024-02-20T07:46:29.551 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-02-20T07:46:29.551 INFO:teuthology.orchestra.run.smithi133.stdout:Complete! 2024-02-20T07:46:29.782 DEBUG:teuthology.orchestra.run.smithi133:> sudo yum -y install ceph-mgr 2024-02-20T07:46:30.074 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-02-20T07:46:30.377 INFO:teuthology.orchestra.run.smithi133.stdout:Last metadata expiration check: 0:01:10 ago on Tue 20 Feb 2024 07:45:20 AM UTC. 2024-02-20T07:46:30.491 INFO:teuthology.orchestra.run.smithi133.stdout:Package ceph-mgr-2:17.2.7-521.gc80e7168.el9.x86_64 is already installed. 2024-02-20T07:46:30.542 INFO:teuthology.orchestra.run.smithi133.stdout:Dependencies resolved. 2024-02-20T07:46:30.543 INFO:teuthology.orchestra.run.smithi133.stdout:Nothing to do. 2024-02-20T07:46:30.543 INFO:teuthology.orchestra.run.smithi133.stdout:Complete! 2024-02-20T07:46:30.608 DEBUG:teuthology.orchestra.run.smithi133:> sudo yum -y install ceph-mgr-dashboard 2024-02-20T07:46:31.108 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-02-20T07:46:31.132 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : ceph-mgr-diskprediction-local-2:17.2.7-521.gc80e71 12/12 2024-02-20T07:46:31.174 INFO:teuthology.orchestra.run.smithi133.stdout:Last metadata expiration check: 0:01:11 ago on Tue 20 Feb 2024 07:45:20 AM UTC. 2024-02-20T07:46:31.341 INFO:teuthology.orchestra.run.smithi133.stdout:Dependencies resolved. 2024-02-20T07:46:31.342 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2024-02-20T07:46:31.342 INFO:teuthology.orchestra.run.smithi133.stdout: Package Arch Version Repository Size 2024-02-20T07:46:31.342 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2024-02-20T07:46:31.342 INFO:teuthology.orchestra.run.smithi133.stdout:Installing: 2024-02-20T07:46:31.342 INFO:teuthology.orchestra.run.smithi133.stdout: ceph-mgr-dashboard noarch 2:17.2.7-521.gc80e7168.el9 ceph-noarch 1.7 M 2024-02-20T07:46:31.342 INFO:teuthology.orchestra.run.smithi133.stdout:Installing dependencies: 2024-02-20T07:46:31.342 INFO:teuthology.orchestra.run.smithi133.stdout: ceph-grafana-dashboards noarch 2:17.2.7-521.gc80e7168.el9 ceph-noarch 23 k 2024-02-20T07:46:31.342 INFO:teuthology.orchestra.run.smithi133.stdout: ceph-prometheus-alerts noarch 2:17.2.7-521.gc80e7168.el9 ceph-noarch 15 k 2024-02-20T07:46:31.343 INFO:teuthology.orchestra.run.smithi133.stdout: python3-jwt noarch 2.4.0-1.el9 epel 41 k 2024-02-20T07:46:31.343 INFO:teuthology.orchestra.run.smithi133.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-02-20T07:46:31.343 INFO:teuthology.orchestra.run.smithi133.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-02-20T07:46:31.343 INFO:teuthology.orchestra.run.smithi133.stdout:Installing weak dependencies: 2024-02-20T07:46:31.343 INFO:teuthology.orchestra.run.smithi133.stdout: python3-jwt+crypto noarch 2.4.0-1.el9 epel 9.0 k 2024-02-20T07:46:31.343 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-02-20T07:46:31.343 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction Summary 2024-02-20T07:46:31.343 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2024-02-20T07:46:31.343 INFO:teuthology.orchestra.run.smithi133.stdout:Install 7 Packages 2024-02-20T07:46:31.343 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-02-20T07:46:31.344 INFO:teuthology.orchestra.run.smithi133.stdout:Total download size: 2.0 M 2024-02-20T07:46:31.344 INFO:teuthology.orchestra.run.smithi133.stdout:Installed size: 8.1 M 2024-02-20T07:46:31.344 INFO:teuthology.orchestra.run.smithi133.stdout:Downloading Packages: 2024-02-20T07:46:31.584 INFO:teuthology.orchestra.run.smithi133.stdout:(1/7): ceph-grafana-dashboards-17.2.7-521.gc80e 118 kB/s | 23 kB 00:00 2024-02-20T07:46:31.685 INFO:teuthology.orchestra.run.smithi133.stdout:(2/7): ceph-prometheus-alerts-17.2.7-521.gc80e7 49 kB/s | 15 kB 00:00 2024-02-20T07:46:31.785 INFO:teuthology.orchestra.run.smithi133.stdout:(3/7): python3-jwt+crypto-2.4.0-1.el9.noarch.rp 45 kB/s | 9.0 kB 00:00 2024-02-20T07:46:31.961 INFO:teuthology.orchestra.run.smithi133.stdout:(4/7): ceph-mgr-dashboard-17.2.7-521.gc80e7168. 3.0 MB/s | 1.7 MB 00:00 2024-02-20T07:46:31.986 INFO:teuthology.orchestra.run.smithi133.stdout:(5/7): python3-repoze-lru-0.7-16.el9.noarch.rpm 153 kB/s | 31 kB 00:00 2024-02-20T07:46:32.028 INFO:teuthology.orchestra.run.smithi133.stdout:(6/7): python3-jwt-2.4.0-1.el9.noarch.rpm 119 kB/s | 41 kB 00:00 2024-02-20T07:46:32.228 INFO:teuthology.orchestra.run.smithi133.stdout:(7/7): python3-routes-2.5.1-5.el9.noarch.rpm 706 kB/s | 188 kB 00:00 2024-02-20T07:46:32.229 INFO:teuthology.orchestra.run.smithi133.stdout:-------------------------------------------------------------------------------- 2024-02-20T07:46:32.230 INFO:teuthology.orchestra.run.smithi133.stdout:Total 2.2 MB/s | 2.0 MB 00:00 2024-02-20T07:46:32.230 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction check 2024-02-20T07:46:32.238 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction check succeeded. 2024-02-20T07:46:32.238 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction test 2024-02-20T07:46:32.300 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction test succeeded. 2024-02-20T07:46:32.300 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction 2024-02-20T07:46:32.598 INFO:teuthology.orchestra.run.smithi133.stdout: Preparing : 1/1 2024-02-20T07:46:32.662 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-jwt-2.4.0-1.el9.noarch 1/7 2024-02-20T07:46:32.696 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-jwt+crypto-2.4.0-1.el9.noarch 2/7 2024-02-20T07:46:32.742 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/7 2024-02-20T07:46:32.775 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/7 2024-02-20T07:46:32.803 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : ceph-prometheus-alerts-2:17.2.7-521.gc80e7168.el9.no 5/7 2024-02-20T07:46:33.031 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : ceph-grafana-dashboards-2:17.2.7-521.gc80e7168.el9.n 6/7 2024-02-20T07:46:33.064 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : ceph-mgr-dashboard-2:17.2.7-521.gc80e7168.el9.noarch 7/7 2024-02-20T07:46:33.637 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: ceph-mgr-dashboard-2:17.2.7-521.gc80e7168.el9.noarch 7/7 2024-02-20T07:46:33.638 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : ceph-grafana-dashboards-2:17.2.7-521.gc80e7168.el9.n 1/7 2024-02-20T07:46:33.638 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : ceph-mgr-dashboard-2:17.2.7-521.gc80e7168.el9.noarch 2/7 2024-02-20T07:46:33.638 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : ceph-prometheus-alerts-2:17.2.7-521.gc80e7168.el9.no 3/7 2024-02-20T07:46:33.638 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-jwt+crypto-2.4.0-1.el9.noarch 4/7 2024-02-20T07:46:33.638 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-jwt-2.4.0-1.el9.noarch 5/7 2024-02-20T07:46:33.638 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 6/7 2024-02-20T07:46:33.744 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:17.2.7-521.gc80e71 12/12 2024-02-20T07:46:33.744 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : ceph-mgr-diskprediction-local-2:17.2.7-521.gc80e71 1/12 2024-02-20T07:46:33.745 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-02-20T07:46:33.745 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-02-20T07:46:33.745 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-02-20T07:46:33.745 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-02-20T07:46:33.745 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-02-20T07:46:33.745 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-02-20T07:46:33.745 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-02-20T07:46:33.745 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-02-20T07:46:33.745 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-02-20T07:46:33.745 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-02-20T07:46:33.966 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 7/7 2024-02-20T07:46:33.966 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-02-20T07:46:33.966 INFO:teuthology.orchestra.run.smithi133.stdout:Installed: 2024-02-20T07:46:33.967 INFO:teuthology.orchestra.run.smithi133.stdout: ceph-grafana-dashboards-2:17.2.7-521.gc80e7168.el9.noarch 2024-02-20T07:46:33.967 INFO:teuthology.orchestra.run.smithi133.stdout: ceph-mgr-dashboard-2:17.2.7-521.gc80e7168.el9.noarch 2024-02-20T07:46:33.967 INFO:teuthology.orchestra.run.smithi133.stdout: ceph-prometheus-alerts-2:17.2.7-521.gc80e7168.el9.noarch 2024-02-20T07:46:33.967 INFO:teuthology.orchestra.run.smithi133.stdout: python3-jwt-2.4.0-1.el9.noarch 2024-02-20T07:46:33.967 INFO:teuthology.orchestra.run.smithi133.stdout: python3-jwt+crypto-2.4.0-1.el9.noarch 2024-02-20T07:46:33.967 INFO:teuthology.orchestra.run.smithi133.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-02-20T07:46:33.967 INFO:teuthology.orchestra.run.smithi133.stdout: python3-routes-2.5.1-5.el9.noarch 2024-02-20T07:46:33.967 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-02-20T07:46:33.967 INFO:teuthology.orchestra.run.smithi133.stdout:Complete! 2024-02-20T07:46:34.177 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-02-20T07:46:34.177 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-02-20T07:46:34.177 INFO:teuthology.orchestra.run.smithi183.stdout:Installed: 2024-02-20T07:46:34.178 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-mgr-diskprediction-local-2:17.2.7-521.gc80e7168.el9.noarch 2024-02-20T07:46:34.178 INFO:teuthology.orchestra.run.smithi183.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-02-20T07:46:34.178 INFO:teuthology.orchestra.run.smithi183.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-02-20T07:46:34.178 INFO:teuthology.orchestra.run.smithi183.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-02-20T07:46:34.178 INFO:teuthology.orchestra.run.smithi183.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-02-20T07:46:34.178 INFO:teuthology.orchestra.run.smithi183.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-02-20T07:46:34.178 INFO:teuthology.orchestra.run.smithi183.stdout: openblas-0.3.21-2.el9.x86_64 2024-02-20T07:46:34.178 INFO:teuthology.orchestra.run.smithi183.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-02-20T07:46:34.178 INFO:teuthology.orchestra.run.smithi183.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-02-20T07:46:34.178 INFO:teuthology.orchestra.run.smithi183.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-02-20T07:46:34.178 INFO:teuthology.orchestra.run.smithi183.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-02-20T07:46:34.178 INFO:teuthology.orchestra.run.smithi183.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-02-20T07:46:34.178 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-02-20T07:46:34.178 INFO:teuthology.orchestra.run.smithi183.stdout:Complete! 2024-02-20T07:46:34.184 DEBUG:teuthology.orchestra.run.smithi133:> sudo yum -y install ceph-mgr-diskprediction-local 2024-02-20T07:46:34.564 DEBUG:teuthology.orchestra.run.smithi183:> sudo yum -y install ceph-mgr-rook 2024-02-20T07:46:34.757 INFO:teuthology.orchestra.run.smithi133.stdout:Last metadata expiration check: 0:01:14 ago on Tue 20 Feb 2024 07:45:20 AM UTC. 2024-02-20T07:46:34.925 INFO:teuthology.orchestra.run.smithi133.stdout:Dependencies resolved. 2024-02-20T07:46:34.926 INFO:teuthology.orchestra.run.smithi133.stdout:=========================================================================================== 2024-02-20T07:46:34.926 INFO:teuthology.orchestra.run.smithi133.stdout: Package Arch Version Repository Size 2024-02-20T07:46:34.926 INFO:teuthology.orchestra.run.smithi133.stdout:=========================================================================================== 2024-02-20T07:46:34.926 INFO:teuthology.orchestra.run.smithi133.stdout:Installing: 2024-02-20T07:46:34.927 INFO:teuthology.orchestra.run.smithi133.stdout: ceph-mgr-diskprediction-local noarch 2:17.2.7-521.gc80e7168.el9 ceph-noarch 7.4 M 2024-02-20T07:46:34.927 INFO:teuthology.orchestra.run.smithi133.stdout:Installing dependencies: 2024-02-20T07:46:34.927 INFO:teuthology.orchestra.run.smithi133.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-02-20T07:46:34.927 INFO:teuthology.orchestra.run.smithi133.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-02-20T07:46:34.927 INFO:teuthology.orchestra.run.smithi133.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-02-20T07:46:34.927 INFO:teuthology.orchestra.run.smithi133.stdout: libgfortran x86_64 11.4.1-3.el9 CentOS-BaseOS 802 k 2024-02-20T07:46:34.927 INFO:teuthology.orchestra.run.smithi133.stdout: libquadmath x86_64 11.4.1-3.el9 CentOS-BaseOS 194 k 2024-02-20T07:46:34.927 INFO:teuthology.orchestra.run.smithi133.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-02-20T07:46:34.927 INFO:teuthology.orchestra.run.smithi133.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-02-20T07:46:34.927 INFO:teuthology.orchestra.run.smithi133.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-02-20T07:46:34.927 INFO:teuthology.orchestra.run.smithi133.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-02-20T07:46:34.927 INFO:teuthology.orchestra.run.smithi133.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-02-20T07:46:34.927 INFO:teuthology.orchestra.run.smithi133.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-02-20T07:46:34.927 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-02-20T07:46:34.927 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction Summary 2024-02-20T07:46:34.928 INFO:teuthology.orchestra.run.smithi133.stdout:=========================================================================================== 2024-02-20T07:46:34.928 INFO:teuthology.orchestra.run.smithi133.stdout:Install 12 Packages 2024-02-20T07:46:34.928 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-02-20T07:46:34.928 INFO:teuthology.orchestra.run.smithi133.stdout:Total download size: 37 M 2024-02-20T07:46:34.928 INFO:teuthology.orchestra.run.smithi133.stdout:Installed size: 209 M 2024-02-20T07:46:34.928 INFO:teuthology.orchestra.run.smithi133.stdout:Downloading Packages: 2024-02-20T07:46:35.145 INFO:teuthology.orchestra.run.smithi183.stdout:Last metadata expiration check: 0:01:14 ago on Tue 20 Feb 2024 07:45:21 AM UTC. 2024-02-20T07:46:35.310 INFO:teuthology.orchestra.run.smithi183.stdout:Dependencies resolved. 2024-02-20T07:46:35.311 INFO:teuthology.orchestra.run.smithi183.stdout:======================================================================================= 2024-02-20T07:46:35.312 INFO:teuthology.orchestra.run.smithi183.stdout: Package Arch Version Repository Size 2024-02-20T07:46:35.312 INFO:teuthology.orchestra.run.smithi183.stdout:======================================================================================= 2024-02-20T07:46:35.312 INFO:teuthology.orchestra.run.smithi183.stdout:Installing: 2024-02-20T07:46:35.312 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-mgr-rook noarch 2:17.2.7-521.gc80e7168.el9 ceph-noarch 47 k 2024-02-20T07:46:35.312 INFO:teuthology.orchestra.run.smithi183.stdout:Installing dependencies: 2024-02-20T07:46:35.312 INFO:teuthology.orchestra.run.smithi183.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-02-20T07:46:35.312 INFO:teuthology.orchestra.run.smithi183.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-02-20T07:46:35.312 INFO:teuthology.orchestra.run.smithi183.stdout: python3-google-auth noarch 1:2.27.0-1.el9 epel 203 k 2024-02-20T07:46:35.312 INFO:teuthology.orchestra.run.smithi183.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-02-20T07:46:35.312 INFO:teuthology.orchestra.run.smithi183.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-02-20T07:46:35.313 INFO:teuthology.orchestra.run.smithi183.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-02-20T07:46:35.313 INFO:teuthology.orchestra.run.smithi183.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-02-20T07:46:35.313 INFO:teuthology.orchestra.run.smithi183.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-02-20T07:46:35.313 INFO:teuthology.orchestra.run.smithi183.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-02-20T07:46:35.313 INFO:teuthology.orchestra.run.smithi183.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-02-20T07:46:35.313 INFO:teuthology.orchestra.run.smithi183.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-02-20T07:46:35.313 INFO:teuthology.orchestra.run.smithi183.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-02-20T07:46:35.313 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-02-20T07:46:35.313 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction Summary 2024-02-20T07:46:35.313 INFO:teuthology.orchestra.run.smithi183.stdout:======================================================================================= 2024-02-20T07:46:35.313 INFO:teuthology.orchestra.run.smithi183.stdout:Install 13 Packages 2024-02-20T07:46:35.314 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-02-20T07:46:35.314 INFO:teuthology.orchestra.run.smithi183.stdout:Total download size: 2.0 M 2024-02-20T07:46:35.314 INFO:teuthology.orchestra.run.smithi183.stdout:Installed size: 26 M 2024-02-20T07:46:35.314 INFO:teuthology.orchestra.run.smithi183.stdout:Downloading Packages: 2024-02-20T07:46:35.343 INFO:teuthology.orchestra.run.smithi133.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 232 kB/s | 32 kB 00:00 2024-02-20T07:46:35.385 INFO:teuthology.orchestra.run.smithi133.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 423 kB/s | 17 kB 00:00 2024-02-20T07:46:35.460 INFO:teuthology.orchestra.run.smithi133.stdout:(3/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 12 MB/s | 3.0 MB 00:00 2024-02-20T07:46:35.485 INFO:teuthology.orchestra.run.smithi133.stdout:(4/12): openblas-0.3.21-2.el9.x86_64.rpm 349 kB/s | 34 kB 00:00 2024-02-20T07:46:35.607 INFO:teuthology.orchestra.run.smithi183.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 170 kB/s | 24 kB 00:00 2024-02-20T07:46:35.632 INFO:teuthology.orchestra.run.smithi183.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 107 kB/s | 18 kB 00:00 2024-02-20T07:46:35.652 INFO:teuthology.orchestra.run.smithi133.stdout:(5/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 25 MB/s | 4.7 MB 00:00 2024-02-20T07:46:35.682 INFO:teuthology.orchestra.run.smithi183.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.3 MB/s | 174 kB 00:00 2024-02-20T07:46:35.707 INFO:teuthology.orchestra.run.smithi183.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.7 MB/s | 132 kB 00:00 2024-02-20T07:46:35.711 INFO:teuthology.orchestra.run.smithi133.stdout:(6/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 21 MB/s | 4.8 MB 00:00 2024-02-20T07:46:35.733 INFO:teuthology.orchestra.run.smithi183.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 4.1 MB/s | 211 kB 00:00 2024-02-20T07:46:35.736 INFO:teuthology.orchestra.run.smithi133.stdout:(7/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.6 MB/s | 217 kB 00:00 2024-02-20T07:46:35.758 INFO:teuthology.orchestra.run.smithi183.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 859 kB/s | 43 kB 00:00 2024-02-20T07:46:35.791 INFO:teuthology.orchestra.run.smithi183.stdout:(7/13): ceph-mgr-rook-17.2.7-521.gc80e7168.el9. 145 kB/s | 47 kB 00:00 2024-02-20T07:46:35.875 INFO:teuthology.orchestra.run.smithi183.stdout:(8/13): python3-certifi-2021.10.8-2.el9.noarch. 128 kB/s | 15 kB 00:00 2024-02-20T07:46:35.901 INFO:teuthology.orchestra.run.smithi183.stdout:(9/13): python3-cachetools-4.2.4-1.el9.noarch.r 192 kB/s | 32 kB 00:00 2024-02-20T07:46:35.903 INFO:teuthology.orchestra.run.smithi133.stdout:(8/12): libgfortran-11.4.1-3.el9.x86_64.rpm 4.8 MB/s | 802 kB 00:00 2024-02-20T07:46:36.001 INFO:teuthology.orchestra.run.smithi183.stdout:(10/13): python3-google-auth-2.27.0-1.el9.noarc 970 kB/s | 203 kB 00:00 2024-02-20T07:46:36.056 INFO:teuthology.orchestra.run.smithi133.stdout:(9/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 46 MB/s | 16 MB 00:00 2024-02-20T07:46:36.076 INFO:teuthology.orchestra.run.smithi183.stdout:(11/13): python3-rsa-4.9-2.el9.noarch.rpm 337 kB/s | 59 kB 00:00 2024-02-20T07:46:36.078 INFO:teuthology.orchestra.run.smithi133.stdout:(10/12): libquadmath-11.4.1-3.el9.x86_64.rpm 1.1 MB/s | 194 kB 00:00 2024-02-20T07:46:36.176 INFO:teuthology.orchestra.run.smithi183.stdout:(12/13): python3-kubernetes-26.1.0-2.el9.noarch 3.4 MB/s | 1.0 MB 00:00 2024-02-20T07:46:36.187 INFO:teuthology.orchestra.run.smithi133.stdout:(11/12): ceph-mgr-diskprediction-local-17.2.7-5 7.5 MB/s | 7.4 MB 00:00 2024-02-20T07:46:36.210 INFO:teuthology.orchestra.run.smithi183.stdout:(13/13): python3-websocket-client-1.2.3-2.el9.n 430 kB/s | 90 kB 00:00 2024-02-20T07:46:36.212 INFO:teuthology.orchestra.run.smithi183.stdout:-------------------------------------------------------------------------------- 2024-02-20T07:46:36.212 INFO:teuthology.orchestra.run.smithi183.stdout:Total 2.3 MB/s | 2.0 MB 00:00 2024-02-20T07:46:36.212 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction check 2024-02-20T07:46:36.225 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction check succeeded. 2024-02-20T07:46:36.225 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction test 2024-02-20T07:46:36.322 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction test succeeded. 2024-02-20T07:46:36.323 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction 2024-02-20T07:46:36.354 INFO:teuthology.orchestra.run.smithi133.stdout:(12/12): python3-devel-3.9.17-2.el9.x86_64.rpm 819 kB/s | 245 kB 00:00 2024-02-20T07:46:36.357 INFO:teuthology.orchestra.run.smithi133.stdout:-------------------------------------------------------------------------------- 2024-02-20T07:46:36.357 INFO:teuthology.orchestra.run.smithi133.stdout:Total 26 MB/s | 37 MB 00:01 2024-02-20T07:46:36.377 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction check 2024-02-20T07:46:36.403 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction check succeeded. 2024-02-20T07:46:36.403 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction test 2024-02-20T07:46:36.832 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction test succeeded. 2024-02-20T07:46:36.832 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction 2024-02-20T07:46:36.844 INFO:teuthology.orchestra.run.smithi183.stdout: Preparing : 1/1 2024-02-20T07:46:37.007 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-02-20T07:46:37.070 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-02-20T07:46:37.132 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-02-20T07:46:37.170 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-02-20T07:46:37.212 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-02-20T07:46:37.305 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-02-20T07:46:37.407 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-google-auth-1:2.27.0-1.el9.noarch 7/13 2024-02-20T07:46:37.463 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-02-20T07:46:37.531 INFO:teuthology.orchestra.run.smithi133.stdout: Preparing : 1/1 2024-02-20T07:46:37.614 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-02-20T07:46:37.641 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-02-20T07:46:37.734 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-02-20T07:46:37.778 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-02-20T07:46:37.956 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-02-20T07:46:37.985 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-02-20T07:46:38.069 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-02-20T07:46:38.183 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-02-20T07:46:38.246 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-02-20T07:46:38.277 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-02-20T07:46:38.323 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-02-20T07:46:38.347 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : ceph-mgr-rook-2:17.2.7-521.gc80e7168.el9.noarch 13/13 2024-02-20T07:46:38.742 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-02-20T07:46:38.856 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-02-20T07:46:39.151 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: ceph-mgr-rook-2:17.2.7-521.gc80e7168.el9.noarch 13/13 2024-02-20T07:46:39.151 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : ceph-mgr-rook-2:17.2.7-521.gc80e7168.el9.noarch 1/13 2024-02-20T07:46:39.151 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-02-20T07:46:39.151 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-02-20T07:46:39.151 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-02-20T07:46:39.151 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-02-20T07:46:39.151 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-02-20T07:46:39.151 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-02-20T07:46:39.151 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-02-20T07:46:39.152 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-02-20T07:46:39.152 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-google-auth-1:2.27.0-1.el9.noarch 10/13 2024-02-20T07:46:39.152 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-02-20T07:46:39.152 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-02-20T07:46:39.532 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-02-20T07:46:39.533 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-02-20T07:46:39.533 INFO:teuthology.orchestra.run.smithi183.stdout:Installed: 2024-02-20T07:46:39.533 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-mgr-rook-2:17.2.7-521.gc80e7168.el9.noarch 2024-02-20T07:46:39.533 INFO:teuthology.orchestra.run.smithi183.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-02-20T07:46:39.533 INFO:teuthology.orchestra.run.smithi183.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-02-20T07:46:39.533 INFO:teuthology.orchestra.run.smithi183.stdout: python3-google-auth-1:2.27.0-1.el9.noarch 2024-02-20T07:46:39.534 INFO:teuthology.orchestra.run.smithi183.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-02-20T07:46:39.534 INFO:teuthology.orchestra.run.smithi183.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-02-20T07:46:39.534 INFO:teuthology.orchestra.run.smithi183.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-02-20T07:46:39.534 INFO:teuthology.orchestra.run.smithi183.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-02-20T07:46:39.534 INFO:teuthology.orchestra.run.smithi183.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-02-20T07:46:39.534 INFO:teuthology.orchestra.run.smithi183.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-02-20T07:46:39.535 INFO:teuthology.orchestra.run.smithi183.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-02-20T07:46:39.535 INFO:teuthology.orchestra.run.smithi183.stdout: python3-rsa-4.9-2.el9.noarch 2024-02-20T07:46:39.535 INFO:teuthology.orchestra.run.smithi183.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-02-20T07:46:39.535 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-02-20T07:46:39.535 INFO:teuthology.orchestra.run.smithi183.stdout:Complete! 2024-02-20T07:46:39.764 DEBUG:teuthology.orchestra.run.smithi183:> sudo yum -y install ceph-mgr-cephadm 2024-02-20T07:46:40.101 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-02-20T07:46:40.371 INFO:teuthology.orchestra.run.smithi183.stdout:Last metadata expiration check: 0:01:19 ago on Tue 20 Feb 2024 07:45:21 AM UTC. 2024-02-20T07:46:40.548 INFO:teuthology.orchestra.run.smithi183.stdout:Dependencies resolved. 2024-02-20T07:46:40.548 INFO:teuthology.orchestra.run.smithi183.stdout:======================================================================================= 2024-02-20T07:46:40.549 INFO:teuthology.orchestra.run.smithi183.stdout: Package Arch Version Repository Size 2024-02-20T07:46:40.549 INFO:teuthology.orchestra.run.smithi183.stdout:======================================================================================= 2024-02-20T07:46:40.549 INFO:teuthology.orchestra.run.smithi183.stdout:Installing: 2024-02-20T07:46:40.549 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-mgr-cephadm noarch 2:17.2.7-521.gc80e7168.el9 ceph-noarch 121 k 2024-02-20T07:46:40.549 INFO:teuthology.orchestra.run.smithi183.stdout:Installing dependencies: 2024-02-20T07:46:40.549 INFO:teuthology.orchestra.run.smithi183.stdout: python3-asyncssh noarch 2.13.2-4.el9 epel 548 k 2024-02-20T07:46:40.549 INFO:teuthology.orchestra.run.smithi183.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-02-20T07:46:40.549 INFO:teuthology.orchestra.run.smithi183.stdout: python3-jinja2 noarch 2.11.3-5.el9 CentOS-AppStream 227 k 2024-02-20T07:46:40.549 INFO:teuthology.orchestra.run.smithi183.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-02-20T07:46:40.549 INFO:teuthology.orchestra.run.smithi183.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-02-20T07:46:40.549 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-02-20T07:46:40.549 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction Summary 2024-02-20T07:46:40.549 INFO:teuthology.orchestra.run.smithi183.stdout:======================================================================================= 2024-02-20T07:46:40.549 INFO:teuthology.orchestra.run.smithi183.stdout:Install 6 Packages 2024-02-20T07:46:40.550 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-02-20T07:46:40.550 INFO:teuthology.orchestra.run.smithi183.stdout:Total download size: 6.8 M 2024-02-20T07:46:40.550 INFO:teuthology.orchestra.run.smithi183.stdout:Installed size: 33 M 2024-02-20T07:46:40.550 INFO:teuthology.orchestra.run.smithi183.stdout:Downloading Packages: 2024-02-20T07:46:40.942 INFO:teuthology.orchestra.run.smithi183.stdout:(1/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 917 kB/s | 227 kB 00:00 2024-02-20T07:46:41.127 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-02-20T07:46:41.151 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : ceph-mgr-diskprediction-local-2:17.2.7-521.gc80e71 12/12 2024-02-20T07:46:41.212 INFO:teuthology.orchestra.run.smithi183.stdout:(2/6): ceph-mgr-cephadm-17.2.7-521.gc80e7168.el 234 kB/s | 121 kB 00:00 2024-02-20T07:46:41.329 INFO:teuthology.orchestra.run.smithi183.stdout:(3/6): python3-babel-2.9.1-2.el9.noarch.rpm 9.1 MB/s | 5.8 MB 00:00 2024-02-20T07:46:41.538 INFO:teuthology.orchestra.run.smithi183.stdout:(4/6): python3-asyncssh-2.13.2-4.el9.noarch.rpm 921 kB/s | 548 kB 00:00 2024-02-20T07:46:41.596 INFO:teuthology.orchestra.run.smithi183.stdout:(5/6): python3-natsort-7.1.1-5.el9.noarch.rpm 151 kB/s | 58 kB 00:00 2024-02-20T07:46:41.672 INFO:teuthology.orchestra.run.smithi183.stdout:(6/6): python3-typing-extensions-4.4.0-2.el9.no 149 kB/s | 51 kB 00:00 2024-02-20T07:46:41.673 INFO:teuthology.orchestra.run.smithi183.stdout:-------------------------------------------------------------------------------- 2024-02-20T07:46:41.674 INFO:teuthology.orchestra.run.smithi183.stdout:Total 6.0 MB/s | 6.8 MB 00:01 2024-02-20T07:46:41.674 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction check 2024-02-20T07:46:41.683 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction check succeeded. 2024-02-20T07:46:41.683 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction test 2024-02-20T07:46:41.772 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction test succeeded. 2024-02-20T07:46:41.773 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction 2024-02-20T07:46:42.074 INFO:teuthology.orchestra.run.smithi183.stdout: Preparing : 1/1 2024-02-20T07:46:42.234 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-02-20T07:46:42.289 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-asyncssh-2.13.2-4.el9.noarch 2/6 2024-02-20T07:46:42.806 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-02-20T07:46:42.882 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-02-20T07:46:42.946 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-02-20T07:46:42.968 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : ceph-mgr-cephadm-2:17.2.7-521.gc80e7168.el9.noarch 6/6 2024-02-20T07:46:43.678 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:17.2.7-521.gc80e71 12/12 2024-02-20T07:46:43.678 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : ceph-mgr-diskprediction-local-2:17.2.7-521.gc80e71 1/12 2024-02-20T07:46:43.678 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-02-20T07:46:43.678 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-02-20T07:46:43.679 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-02-20T07:46:43.679 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-02-20T07:46:43.679 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-02-20T07:46:43.679 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-02-20T07:46:43.679 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-02-20T07:46:43.679 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-02-20T07:46:43.679 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-02-20T07:46:43.679 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-02-20T07:46:43.824 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: ceph-mgr-cephadm-2:17.2.7-521.gc80e7168.el9.noarch 6/6 2024-02-20T07:46:43.824 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : ceph-mgr-cephadm-2:17.2.7-521.gc80e7168.el9.noarch 1/6 2024-02-20T07:46:43.824 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-02-20T07:46:43.825 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-02-20T07:46:43.825 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-asyncssh-2.13.2-4.el9.noarch 4/6 2024-02-20T07:46:43.825 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-02-20T07:46:44.121 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-02-20T07:46:44.121 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-02-20T07:46:44.121 INFO:teuthology.orchestra.run.smithi183.stdout:Installed: 2024-02-20T07:46:44.121 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-mgr-cephadm-2:17.2.7-521.gc80e7168.el9.noarch 2024-02-20T07:46:44.121 INFO:teuthology.orchestra.run.smithi183.stdout: python3-asyncssh-2.13.2-4.el9.noarch 2024-02-20T07:46:44.121 INFO:teuthology.orchestra.run.smithi183.stdout: python3-babel-2.9.1-2.el9.noarch 2024-02-20T07:46:44.121 INFO:teuthology.orchestra.run.smithi183.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-02-20T07:46:44.121 INFO:teuthology.orchestra.run.smithi183.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-02-20T07:46:44.121 INFO:teuthology.orchestra.run.smithi183.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-02-20T07:46:44.121 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-02-20T07:46:44.122 INFO:teuthology.orchestra.run.smithi183.stdout:Complete! 2024-02-20T07:46:44.134 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-02-20T07:46:44.134 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-02-20T07:46:44.134 INFO:teuthology.orchestra.run.smithi133.stdout:Installed: 2024-02-20T07:46:44.134 INFO:teuthology.orchestra.run.smithi133.stdout: ceph-mgr-diskprediction-local-2:17.2.7-521.gc80e7168.el9.noarch 2024-02-20T07:46:44.134 INFO:teuthology.orchestra.run.smithi133.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-02-20T07:46:44.134 INFO:teuthology.orchestra.run.smithi133.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-02-20T07:46:44.134 INFO:teuthology.orchestra.run.smithi133.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-02-20T07:46:44.134 INFO:teuthology.orchestra.run.smithi133.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-02-20T07:46:44.134 INFO:teuthology.orchestra.run.smithi133.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-02-20T07:46:44.135 INFO:teuthology.orchestra.run.smithi133.stdout: openblas-0.3.21-2.el9.x86_64 2024-02-20T07:46:44.135 INFO:teuthology.orchestra.run.smithi133.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-02-20T07:46:44.135 INFO:teuthology.orchestra.run.smithi133.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-02-20T07:46:44.135 INFO:teuthology.orchestra.run.smithi133.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-02-20T07:46:44.135 INFO:teuthology.orchestra.run.smithi133.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-02-20T07:46:44.135 INFO:teuthology.orchestra.run.smithi133.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-02-20T07:46:44.135 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-02-20T07:46:44.135 INFO:teuthology.orchestra.run.smithi133.stdout:Complete! 2024-02-20T07:46:44.373 DEBUG:teuthology.orchestra.run.smithi183:> sudo yum -y install ceph-fuse 2024-02-20T07:46:44.528 DEBUG:teuthology.orchestra.run.smithi133:> sudo yum -y install ceph-mgr-rook 2024-02-20T07:46:44.950 INFO:teuthology.orchestra.run.smithi183.stdout:Last metadata expiration check: 0:01:23 ago on Tue 20 Feb 2024 07:45:21 AM UTC. 2024-02-20T07:46:45.115 INFO:teuthology.orchestra.run.smithi133.stdout:Last metadata expiration check: 0:01:25 ago on Tue 20 Feb 2024 07:45:20 AM UTC. 2024-02-20T07:46:45.118 INFO:teuthology.orchestra.run.smithi183.stdout:Dependencies resolved. 2024-02-20T07:46:45.119 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-02-20T07:46:45.120 INFO:teuthology.orchestra.run.smithi183.stdout: Package Arch Version Repository Size 2024-02-20T07:46:45.120 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-02-20T07:46:45.120 INFO:teuthology.orchestra.run.smithi183.stdout:Installing: 2024-02-20T07:46:45.120 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-fuse x86_64 2:17.2.7-521.gc80e7168.el9 ceph 812 k 2024-02-20T07:46:45.120 INFO:teuthology.orchestra.run.smithi183.stdout:Installing dependencies: 2024-02-20T07:46:45.120 INFO:teuthology.orchestra.run.smithi183.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-02-20T07:46:45.121 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-02-20T07:46:45.121 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction Summary 2024-02-20T07:46:45.121 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-02-20T07:46:45.121 INFO:teuthology.orchestra.run.smithi183.stdout:Install 2 Packages 2024-02-20T07:46:45.121 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-02-20T07:46:45.121 INFO:teuthology.orchestra.run.smithi183.stdout:Total download size: 891 k 2024-02-20T07:46:45.121 INFO:teuthology.orchestra.run.smithi183.stdout:Installed size: 2.6 M 2024-02-20T07:46:45.121 INFO:teuthology.orchestra.run.smithi183.stdout:Downloading Packages: 2024-02-20T07:46:45.283 INFO:teuthology.orchestra.run.smithi133.stdout:Dependencies resolved. 2024-02-20T07:46:45.284 INFO:teuthology.orchestra.run.smithi133.stdout:======================================================================================= 2024-02-20T07:46:45.284 INFO:teuthology.orchestra.run.smithi133.stdout: Package Arch Version Repository Size 2024-02-20T07:46:45.284 INFO:teuthology.orchestra.run.smithi133.stdout:======================================================================================= 2024-02-20T07:46:45.284 INFO:teuthology.orchestra.run.smithi133.stdout:Installing: 2024-02-20T07:46:45.284 INFO:teuthology.orchestra.run.smithi133.stdout: ceph-mgr-rook noarch 2:17.2.7-521.gc80e7168.el9 ceph-noarch 47 k 2024-02-20T07:46:45.284 INFO:teuthology.orchestra.run.smithi133.stdout:Installing dependencies: 2024-02-20T07:46:45.284 INFO:teuthology.orchestra.run.smithi133.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-02-20T07:46:45.284 INFO:teuthology.orchestra.run.smithi133.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-02-20T07:46:45.285 INFO:teuthology.orchestra.run.smithi133.stdout: python3-google-auth noarch 1:2.27.0-1.el9 epel 203 k 2024-02-20T07:46:45.285 INFO:teuthology.orchestra.run.smithi133.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-02-20T07:46:45.285 INFO:teuthology.orchestra.run.smithi133.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-02-20T07:46:45.285 INFO:teuthology.orchestra.run.smithi133.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-02-20T07:46:45.285 INFO:teuthology.orchestra.run.smithi133.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-02-20T07:46:45.285 INFO:teuthology.orchestra.run.smithi133.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-02-20T07:46:45.285 INFO:teuthology.orchestra.run.smithi133.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-02-20T07:46:45.285 INFO:teuthology.orchestra.run.smithi133.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-02-20T07:46:45.285 INFO:teuthology.orchestra.run.smithi133.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-02-20T07:46:45.285 INFO:teuthology.orchestra.run.smithi133.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-02-20T07:46:45.285 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-02-20T07:46:45.285 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction Summary 2024-02-20T07:46:45.285 INFO:teuthology.orchestra.run.smithi133.stdout:======================================================================================= 2024-02-20T07:46:45.285 INFO:teuthology.orchestra.run.smithi133.stdout:Install 13 Packages 2024-02-20T07:46:45.285 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-02-20T07:46:45.286 INFO:teuthology.orchestra.run.smithi133.stdout:Total download size: 2.0 M 2024-02-20T07:46:45.286 INFO:teuthology.orchestra.run.smithi133.stdout:Installed size: 26 M 2024-02-20T07:46:45.286 INFO:teuthology.orchestra.run.smithi133.stdout:Downloading Packages: 2024-02-20T07:46:45.307 INFO:teuthology.orchestra.run.smithi183.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 422 kB/s | 79 kB 00:00 2024-02-20T07:46:45.615 INFO:teuthology.orchestra.run.smithi183.stdout:(2/2): ceph-fuse-17.2.7-521.gc80e7168.el9.x86_6 1.6 MB/s | 812 kB 00:00 2024-02-20T07:46:45.616 INFO:teuthology.orchestra.run.smithi183.stdout:-------------------------------------------------------------------------------- 2024-02-20T07:46:45.617 INFO:teuthology.orchestra.run.smithi183.stdout:Total 1.8 MB/s | 891 kB 00:00 2024-02-20T07:46:45.617 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction check 2024-02-20T07:46:45.638 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction check succeeded. 2024-02-20T07:46:45.638 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction test 2024-02-20T07:46:45.714 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction test succeeded. 2024-02-20T07:46:45.715 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction 2024-02-20T07:46:45.792 INFO:teuthology.orchestra.run.smithi133.stdout:(1/13): python3-jsonpointer-2.0-4.el9.noarch.rp 133 kB/s | 18 kB 00:00 2024-02-20T07:46:45.818 INFO:teuthology.orchestra.run.smithi133.stdout:(2/13): python3-jsonpatch-1.21-16.el9.noarch.rp 150 kB/s | 24 kB 00:00 2024-02-20T07:46:45.868 INFO:teuthology.orchestra.run.smithi133.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.3 MB/s | 174 kB 00:00 2024-02-20T07:46:45.893 INFO:teuthology.orchestra.run.smithi133.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.7 MB/s | 132 kB 00:00 2024-02-20T07:46:45.897 INFO:teuthology.orchestra.run.smithi183.stdout: Preparing : 1/1 2024-02-20T07:46:45.919 INFO:teuthology.orchestra.run.smithi133.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 4.1 MB/s | 211 kB 00:00 2024-02-20T07:46:45.936 INFO:teuthology.orchestra.run.smithi133.stdout:(6/13): ceph-mgr-rook-17.2.7-521.gc80e7168.el9. 170 kB/s | 47 kB 00:00 2024-02-20T07:46:45.953 INFO:teuthology.orchestra.run.smithi133.stdout:(7/13): python3-requests-oauthlib-1.3.0-12.el9. 729 kB/s | 43 kB 00:00 2024-02-20T07:46:46.035 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-02-20T07:46:46.098 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : ceph-fuse-2:17.2.7-521.gc80e7168.el9.x86_64 2/2 2024-02-20T07:46:46.211 INFO:teuthology.orchestra.run.smithi133.stdout:(8/13): python3-certifi-2021.10.8-2.el9.noarch. 58 kB/s | 15 kB 00:00 2024-02-20T07:46:46.228 INFO:teuthology.orchestra.run.smithi133.stdout:(9/13): python3-cachetools-4.2.4-1.el9.noarch.r 104 kB/s | 32 kB 00:00 2024-02-20T07:46:46.413 INFO:teuthology.orchestra.run.smithi133.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 322 kB/s | 59 kB 00:00 2024-02-20T07:46:46.445 INFO:teuthology.orchestra.run.smithi133.stdout:(11/13): python3-google-auth-2.27.0-1.el9.noarc 412 kB/s | 203 kB 00:00 2024-02-20T07:46:46.512 INFO:teuthology.orchestra.run.smithi133.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 895 kB/s | 90 kB 00:00 2024-02-20T07:46:46.784 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: ceph-fuse-2:17.2.7-521.gc80e7168.el9.x86_64 2/2 2024-02-20T07:46:46.784 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : ceph-fuse-2:17.2.7-521.gc80e7168.el9.x86_64 1/2 2024-02-20T07:46:46.795 INFO:teuthology.orchestra.run.smithi133.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 1.7 MB/s | 1.0 MB 00:00 2024-02-20T07:46:46.797 INFO:teuthology.orchestra.run.smithi133.stdout:-------------------------------------------------------------------------------- 2024-02-20T07:46:46.798 INFO:teuthology.orchestra.run.smithi133.stdout:Total 1.4 MB/s | 2.0 MB 00:01 2024-02-20T07:46:46.798 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction check 2024-02-20T07:46:46.813 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction check succeeded. 2024-02-20T07:46:46.813 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction test 2024-02-20T07:46:46.919 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction test succeeded. 2024-02-20T07:46:46.933 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction 2024-02-20T07:46:47.114 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-02-20T07:46:47.114 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-02-20T07:46:47.114 INFO:teuthology.orchestra.run.smithi183.stdout:Installed: 2024-02-20T07:46:47.114 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-fuse-2:17.2.7-521.gc80e7168.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-02-20T07:46:47.114 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-02-20T07:46:47.114 INFO:teuthology.orchestra.run.smithi183.stdout:Complete! 2024-02-20T07:46:47.398 DEBUG:teuthology.orchestra.run.smithi183:> sudo yum -y install librados-devel 2024-02-20T07:46:47.463 INFO:teuthology.orchestra.run.smithi133.stdout: Preparing : 1/1 2024-02-20T07:46:47.626 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-02-20T07:46:47.680 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-02-20T07:46:47.735 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-02-20T07:46:47.773 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-02-20T07:46:47.813 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-02-20T07:46:47.915 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-02-20T07:46:47.997 INFO:teuthology.orchestra.run.smithi183.stdout:Last metadata expiration check: 0:01:26 ago on Tue 20 Feb 2024 07:45:21 AM UTC. 2024-02-20T07:46:48.016 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-google-auth-1:2.27.0-1.el9.noarch 7/13 2024-02-20T07:46:48.073 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-02-20T07:46:48.163 INFO:teuthology.orchestra.run.smithi183.stdout:Dependencies resolved. 2024-02-20T07:46:48.164 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-02-20T07:46:48.164 INFO:teuthology.orchestra.run.smithi183.stdout: Package Arch Version Repo Size 2024-02-20T07:46:48.164 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-02-20T07:46:48.164 INFO:teuthology.orchestra.run.smithi183.stdout:Installing: 2024-02-20T07:46:48.164 INFO:teuthology.orchestra.run.smithi183.stdout: librados-devel x86_64 2:17.2.7-521.gc80e7168.el9 ceph 126 k 2024-02-20T07:46:48.164 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-02-20T07:46:48.164 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction Summary 2024-02-20T07:46:48.164 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-02-20T07:46:48.164 INFO:teuthology.orchestra.run.smithi183.stdout:Install 1 Package 2024-02-20T07:46:48.165 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-02-20T07:46:48.165 INFO:teuthology.orchestra.run.smithi183.stdout:Total download size: 126 k 2024-02-20T07:46:48.165 INFO:teuthology.orchestra.run.smithi183.stdout:Installed size: 448 k 2024-02-20T07:46:48.165 INFO:teuthology.orchestra.run.smithi183.stdout:Downloading Packages: 2024-02-20T07:46:48.511 INFO:teuthology.orchestra.run.smithi183.stdout:librados-devel-17.2.7-521.gc80e7168.el9.x86_64. 365 kB/s | 126 kB 00:00 2024-02-20T07:46:48.511 INFO:teuthology.orchestra.run.smithi183.stdout:-------------------------------------------------------------------------------- 2024-02-20T07:46:48.511 INFO:teuthology.orchestra.run.smithi183.stdout:Total 364 kB/s | 126 kB 00:00 2024-02-20T07:46:48.511 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction check 2024-02-20T07:46:48.522 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction check succeeded. 2024-02-20T07:46:48.522 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction test 2024-02-20T07:46:48.572 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction test succeeded. 2024-02-20T07:46:48.572 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction 2024-02-20T07:46:48.728 INFO:teuthology.orchestra.run.smithi183.stdout: Preparing : 1/1 2024-02-20T07:46:48.796 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-02-20T07:46:48.825 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : librados-devel-2:17.2.7-521.gc80e7168.el9.x86_64 1/1 2024-02-20T07:46:48.845 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-02-20T07:46:48.879 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-02-20T07:46:48.928 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-02-20T07:46:48.940 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : ceph-mgr-rook-2:17.2.7-521.gc80e7168.el9.noarch 13/13 2024-02-20T07:46:49.279 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: librados-devel-2:17.2.7-521.gc80e7168.el9.x86_64 1/1 2024-02-20T07:46:49.626 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : librados-devel-2:17.2.7-521.gc80e7168.el9.x86_64 1/1 2024-02-20T07:46:49.626 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-02-20T07:46:49.626 INFO:teuthology.orchestra.run.smithi183.stdout:Installed: 2024-02-20T07:46:49.627 INFO:teuthology.orchestra.run.smithi183.stdout: librados-devel-2:17.2.7-521.gc80e7168.el9.x86_64 2024-02-20T07:46:49.627 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-02-20T07:46:49.627 INFO:teuthology.orchestra.run.smithi183.stdout:Complete! 2024-02-20T07:46:49.766 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: ceph-mgr-rook-2:17.2.7-521.gc80e7168.el9.noarch 13/13 2024-02-20T07:46:49.766 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : ceph-mgr-rook-2:17.2.7-521.gc80e7168.el9.noarch 1/13 2024-02-20T07:46:49.766 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-02-20T07:46:49.766 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-02-20T07:46:49.766 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-02-20T07:46:49.766 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-02-20T07:46:49.766 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-02-20T07:46:49.766 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-02-20T07:46:49.766 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-02-20T07:46:49.766 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-02-20T07:46:49.767 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-google-auth-1:2.27.0-1.el9.noarch 10/13 2024-02-20T07:46:49.767 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-02-20T07:46:49.767 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-02-20T07:46:49.867 DEBUG:teuthology.orchestra.run.smithi183:> sudo yum -y install libcephfs2 2024-02-20T07:46:50.146 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-02-20T07:46:50.147 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-02-20T07:46:50.147 INFO:teuthology.orchestra.run.smithi133.stdout:Installed: 2024-02-20T07:46:50.147 INFO:teuthology.orchestra.run.smithi133.stdout: ceph-mgr-rook-2:17.2.7-521.gc80e7168.el9.noarch 2024-02-20T07:46:50.147 INFO:teuthology.orchestra.run.smithi133.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-02-20T07:46:50.147 INFO:teuthology.orchestra.run.smithi133.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-02-20T07:46:50.147 INFO:teuthology.orchestra.run.smithi133.stdout: python3-google-auth-1:2.27.0-1.el9.noarch 2024-02-20T07:46:50.148 INFO:teuthology.orchestra.run.smithi133.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-02-20T07:46:50.148 INFO:teuthology.orchestra.run.smithi133.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-02-20T07:46:50.148 INFO:teuthology.orchestra.run.smithi133.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-02-20T07:46:50.148 INFO:teuthology.orchestra.run.smithi133.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-02-20T07:46:50.148 INFO:teuthology.orchestra.run.smithi133.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-02-20T07:46:50.148 INFO:teuthology.orchestra.run.smithi133.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-02-20T07:46:50.148 INFO:teuthology.orchestra.run.smithi133.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-02-20T07:46:50.148 INFO:teuthology.orchestra.run.smithi133.stdout: python3-rsa-4.9-2.el9.noarch 2024-02-20T07:46:50.149 INFO:teuthology.orchestra.run.smithi133.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-02-20T07:46:50.149 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-02-20T07:46:50.149 INFO:teuthology.orchestra.run.smithi133.stdout:Complete! 2024-02-20T07:46:50.424 DEBUG:teuthology.orchestra.run.smithi133:> sudo yum -y install ceph-mgr-cephadm 2024-02-20T07:46:50.462 INFO:teuthology.orchestra.run.smithi183.stdout:Last metadata expiration check: 0:01:29 ago on Tue 20 Feb 2024 07:45:21 AM UTC. 2024-02-20T07:46:50.577 INFO:teuthology.orchestra.run.smithi183.stdout:Package libcephfs2-2:17.2.7-521.gc80e7168.el9.x86_64 is already installed. 2024-02-20T07:46:50.631 INFO:teuthology.orchestra.run.smithi183.stdout:Dependencies resolved. 2024-02-20T07:46:50.632 INFO:teuthology.orchestra.run.smithi183.stdout:Nothing to do. 2024-02-20T07:46:50.632 INFO:teuthology.orchestra.run.smithi183.stdout:Complete! 2024-02-20T07:46:50.697 DEBUG:teuthology.orchestra.run.smithi183:> sudo yum -y install libcephfs-devel 2024-02-20T07:46:50.999 INFO:teuthology.orchestra.run.smithi133.stdout:Last metadata expiration check: 0:01:30 ago on Tue 20 Feb 2024 07:45:20 AM UTC. 2024-02-20T07:46:51.167 INFO:teuthology.orchestra.run.smithi133.stdout:Dependencies resolved. 2024-02-20T07:46:51.168 INFO:teuthology.orchestra.run.smithi133.stdout:======================================================================================= 2024-02-20T07:46:51.168 INFO:teuthology.orchestra.run.smithi133.stdout: Package Arch Version Repository Size 2024-02-20T07:46:51.168 INFO:teuthology.orchestra.run.smithi133.stdout:======================================================================================= 2024-02-20T07:46:51.171 INFO:teuthology.orchestra.run.smithi133.stdout:Installing: 2024-02-20T07:46:51.171 INFO:teuthology.orchestra.run.smithi133.stdout: ceph-mgr-cephadm noarch 2:17.2.7-521.gc80e7168.el9 ceph-noarch 121 k 2024-02-20T07:46:51.171 INFO:teuthology.orchestra.run.smithi133.stdout:Installing dependencies: 2024-02-20T07:46:51.171 INFO:teuthology.orchestra.run.smithi133.stdout: python3-asyncssh noarch 2.13.2-4.el9 epel 548 k 2024-02-20T07:46:51.171 INFO:teuthology.orchestra.run.smithi133.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-02-20T07:46:51.171 INFO:teuthology.orchestra.run.smithi133.stdout: python3-jinja2 noarch 2.11.3-5.el9 CentOS-AppStream 227 k 2024-02-20T07:46:51.171 INFO:teuthology.orchestra.run.smithi133.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-02-20T07:46:51.172 INFO:teuthology.orchestra.run.smithi133.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-02-20T07:46:51.172 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-02-20T07:46:51.172 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction Summary 2024-02-20T07:46:51.172 INFO:teuthology.orchestra.run.smithi133.stdout:======================================================================================= 2024-02-20T07:46:51.172 INFO:teuthology.orchestra.run.smithi133.stdout:Install 6 Packages 2024-02-20T07:46:51.172 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-02-20T07:46:51.173 INFO:teuthology.orchestra.run.smithi133.stdout:Total download size: 6.8 M 2024-02-20T07:46:51.173 INFO:teuthology.orchestra.run.smithi133.stdout:Installed size: 33 M 2024-02-20T07:46:51.173 INFO:teuthology.orchestra.run.smithi133.stdout:Downloading Packages: 2024-02-20T07:46:51.279 INFO:teuthology.orchestra.run.smithi183.stdout:Last metadata expiration check: 0:01:30 ago on Tue 20 Feb 2024 07:45:21 AM UTC. 2024-02-20T07:46:51.447 INFO:teuthology.orchestra.run.smithi183.stdout:Dependencies resolved. 2024-02-20T07:46:51.448 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-02-20T07:46:51.448 INFO:teuthology.orchestra.run.smithi183.stdout: Package Arch Version Repo Size 2024-02-20T07:46:51.448 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-02-20T07:46:51.448 INFO:teuthology.orchestra.run.smithi183.stdout:Installing: 2024-02-20T07:46:51.448 INFO:teuthology.orchestra.run.smithi183.stdout: libcephfs-devel x86_64 2:17.2.7-521.gc80e7168.el9 ceph 25 k 2024-02-20T07:46:51.448 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-02-20T07:46:51.448 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction Summary 2024-02-20T07:46:51.448 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-02-20T07:46:51.448 INFO:teuthology.orchestra.run.smithi183.stdout:Install 1 Package 2024-02-20T07:46:51.448 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-02-20T07:46:51.448 INFO:teuthology.orchestra.run.smithi183.stdout:Total download size: 25 k 2024-02-20T07:46:51.449 INFO:teuthology.orchestra.run.smithi183.stdout:Installed size: 110 k 2024-02-20T07:46:51.449 INFO:teuthology.orchestra.run.smithi183.stdout:Downloading Packages: 2024-02-20T07:46:51.671 INFO:teuthology.orchestra.run.smithi133.stdout:(1/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 1.4 MB/s | 227 kB 00:00 2024-02-20T07:46:51.700 INFO:teuthology.orchestra.run.smithi183.stdout:libcephfs-devel-17.2.7-521.gc80e7168.el9.x86_64 99 kB/s | 25 kB 00:00 2024-02-20T07:46:51.700 INFO:teuthology.orchestra.run.smithi183.stdout:-------------------------------------------------------------------------------- 2024-02-20T07:46:51.700 INFO:teuthology.orchestra.run.smithi183.stdout:Total 99 kB/s | 25 kB 00:00 2024-02-20T07:46:51.701 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction check 2024-02-20T07:46:51.704 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction check succeeded. 2024-02-20T07:46:51.704 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction test 2024-02-20T07:46:51.705 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction test succeeded. 2024-02-20T07:46:51.706 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction 2024-02-20T07:46:51.746 INFO:teuthology.orchestra.run.smithi133.stdout:(2/6): ceph-mgr-cephadm-17.2.7-521.gc80e7168.el 521 kB/s | 121 kB 00:00 2024-02-20T07:46:51.805 INFO:teuthology.orchestra.run.smithi183.stdout: Preparing : 1/1 2024-02-20T07:46:51.821 INFO:teuthology.orchestra.run.smithi133.stdout:(3/6): python3-babel-2.9.1-2.el9.noarch.rpm 19 MB/s | 5.8 MB 00:00 2024-02-20T07:46:51.922 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : libcephfs-devel-2:17.2.7-521.gc80e7168.el9.x86_64 1/1 2024-02-20T07:46:52.097 INFO:teuthology.orchestra.run.smithi133.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 166 kB/s | 58 kB 00:00 2024-02-20T07:46:52.172 INFO:teuthology.orchestra.run.smithi133.stdout:(5/6): python3-typing-extensions-4.4.0-2.el9.no 146 kB/s | 51 kB 00:00 2024-02-20T07:46:52.295 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: libcephfs-devel-2:17.2.7-521.gc80e7168.el9.x86_64 1/1 2024-02-20T07:46:52.330 INFO:teuthology.orchestra.run.smithi133.stdout:(6/6): python3-asyncssh-2.13.2-4.el9.noarch.rpm 832 kB/s | 548 kB 00:00 2024-02-20T07:46:52.332 INFO:teuthology.orchestra.run.smithi133.stdout:-------------------------------------------------------------------------------- 2024-02-20T07:46:52.332 INFO:teuthology.orchestra.run.smithi133.stdout:Total 5.8 MB/s | 6.8 MB 00:01 2024-02-20T07:46:52.333 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction check 2024-02-20T07:46:52.342 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction check succeeded. 2024-02-20T07:46:52.342 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction test 2024-02-20T07:46:52.431 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction test succeeded. 2024-02-20T07:46:52.431 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction 2024-02-20T07:46:52.559 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : libcephfs-devel-2:17.2.7-521.gc80e7168.el9.x86_64 1/1 2024-02-20T07:46:52.560 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-02-20T07:46:52.560 INFO:teuthology.orchestra.run.smithi183.stdout:Installed: 2024-02-20T07:46:52.560 INFO:teuthology.orchestra.run.smithi183.stdout: libcephfs-devel-2:17.2.7-521.gc80e7168.el9.x86_64 2024-02-20T07:46:52.560 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-02-20T07:46:52.560 INFO:teuthology.orchestra.run.smithi183.stdout:Complete! 2024-02-20T07:46:52.723 INFO:teuthology.orchestra.run.smithi133.stdout: Preparing : 1/1 2024-02-20T07:46:52.724 DEBUG:teuthology.orchestra.run.smithi183:> sudo yum -y install librados2 2024-02-20T07:46:52.884 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-02-20T07:46:52.940 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-asyncssh-2.13.2-4.el9.noarch 2/6 2024-02-20T07:46:53.302 INFO:teuthology.orchestra.run.smithi183.stdout:Last metadata expiration check: 0:01:32 ago on Tue 20 Feb 2024 07:45:21 AM UTC. 2024-02-20T07:46:53.416 INFO:teuthology.orchestra.run.smithi183.stdout:Package librados2-2:17.2.7-521.gc80e7168.el9.x86_64 is already installed. 2024-02-20T07:46:53.468 INFO:teuthology.orchestra.run.smithi183.stdout:Dependencies resolved. 2024-02-20T07:46:53.469 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-02-20T07:46:53.469 INFO:teuthology.orchestra.run.smithi183.stdout:Nothing to do. 2024-02-20T07:46:53.469 INFO:teuthology.orchestra.run.smithi183.stdout:Complete! 2024-02-20T07:46:53.532 DEBUG:teuthology.orchestra.run.smithi183:> sudo yum -y install librbd1 2024-02-20T07:46:53.552 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-02-20T07:46:53.606 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-02-20T07:46:53.636 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : ceph-mgr-cephadm-2:17.2.7-521.gc80e7168.el9.noarch 6/6 2024-02-20T07:46:54.112 INFO:teuthology.orchestra.run.smithi183.stdout:Last metadata expiration check: 0:01:33 ago on Tue 20 Feb 2024 07:45:21 AM UTC. 2024-02-20T07:46:54.226 INFO:teuthology.orchestra.run.smithi183.stdout:Package librbd1-2:17.2.7-521.gc80e7168.el9.x86_64 is already installed. 2024-02-20T07:46:54.278 INFO:teuthology.orchestra.run.smithi183.stdout:Dependencies resolved. 2024-02-20T07:46:54.279 INFO:teuthology.orchestra.run.smithi183.stdout:Nothing to do. 2024-02-20T07:46:54.279 INFO:teuthology.orchestra.run.smithi183.stdout:Complete! 2024-02-20T07:46:54.344 DEBUG:teuthology.orchestra.run.smithi183:> sudo yum -y install python3-rados 2024-02-20T07:46:54.460 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: ceph-mgr-cephadm-2:17.2.7-521.gc80e7168.el9.noarch 6/6 2024-02-20T07:46:54.460 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : ceph-mgr-cephadm-2:17.2.7-521.gc80e7168.el9.noarch 1/6 2024-02-20T07:46:54.460 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-02-20T07:46:54.460 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-02-20T07:46:54.460 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-asyncssh-2.13.2-4.el9.noarch 4/6 2024-02-20T07:46:54.460 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-02-20T07:46:54.771 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-02-20T07:46:54.771 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-02-20T07:46:54.771 INFO:teuthology.orchestra.run.smithi133.stdout:Installed: 2024-02-20T07:46:54.771 INFO:teuthology.orchestra.run.smithi133.stdout: ceph-mgr-cephadm-2:17.2.7-521.gc80e7168.el9.noarch 2024-02-20T07:46:54.771 INFO:teuthology.orchestra.run.smithi133.stdout: python3-asyncssh-2.13.2-4.el9.noarch 2024-02-20T07:46:54.771 INFO:teuthology.orchestra.run.smithi133.stdout: python3-babel-2.9.1-2.el9.noarch 2024-02-20T07:46:54.771 INFO:teuthology.orchestra.run.smithi133.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-02-20T07:46:54.771 INFO:teuthology.orchestra.run.smithi133.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-02-20T07:46:54.772 INFO:teuthology.orchestra.run.smithi133.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-02-20T07:46:54.772 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-02-20T07:46:54.772 INFO:teuthology.orchestra.run.smithi133.stdout:Complete! 2024-02-20T07:46:54.920 INFO:teuthology.orchestra.run.smithi183.stdout:Last metadata expiration check: 0:01:33 ago on Tue 20 Feb 2024 07:45:21 AM UTC. 2024-02-20T07:46:55.036 INFO:teuthology.orchestra.run.smithi183.stdout:Package python3-rados-2:17.2.7-521.gc80e7168.el9.x86_64 is already installed. 2024-02-20T07:46:55.053 DEBUG:teuthology.orchestra.run.smithi133:> sudo yum -y install ceph-fuse 2024-02-20T07:46:55.088 INFO:teuthology.orchestra.run.smithi183.stdout:Dependencies resolved. 2024-02-20T07:46:55.089 INFO:teuthology.orchestra.run.smithi183.stdout:Nothing to do. 2024-02-20T07:46:55.089 INFO:teuthology.orchestra.run.smithi183.stdout:Complete! 2024-02-20T07:46:55.152 DEBUG:teuthology.orchestra.run.smithi183:> sudo yum -y install python3-rgw 2024-02-20T07:46:55.632 INFO:teuthology.orchestra.run.smithi133.stdout:Last metadata expiration check: 0:01:35 ago on Tue 20 Feb 2024 07:45:20 AM UTC. 2024-02-20T07:46:55.727 INFO:teuthology.orchestra.run.smithi183.stdout:Last metadata expiration check: 0:01:34 ago on Tue 20 Feb 2024 07:45:21 AM UTC. 2024-02-20T07:46:55.800 INFO:teuthology.orchestra.run.smithi133.stdout:Dependencies resolved. 2024-02-20T07:46:55.800 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2024-02-20T07:46:55.800 INFO:teuthology.orchestra.run.smithi133.stdout: Package Arch Version Repository Size 2024-02-20T07:46:55.801 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2024-02-20T07:46:55.801 INFO:teuthology.orchestra.run.smithi133.stdout:Installing: 2024-02-20T07:46:55.801 INFO:teuthology.orchestra.run.smithi133.stdout: ceph-fuse x86_64 2:17.2.7-521.gc80e7168.el9 ceph 812 k 2024-02-20T07:46:55.801 INFO:teuthology.orchestra.run.smithi133.stdout:Installing dependencies: 2024-02-20T07:46:55.801 INFO:teuthology.orchestra.run.smithi133.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-02-20T07:46:55.801 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-02-20T07:46:55.801 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction Summary 2024-02-20T07:46:55.801 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2024-02-20T07:46:55.801 INFO:teuthology.orchestra.run.smithi133.stdout:Install 2 Packages 2024-02-20T07:46:55.801 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-02-20T07:46:55.802 INFO:teuthology.orchestra.run.smithi133.stdout:Total download size: 891 k 2024-02-20T07:46:55.802 INFO:teuthology.orchestra.run.smithi133.stdout:Installed size: 2.6 M 2024-02-20T07:46:55.802 INFO:teuthology.orchestra.run.smithi133.stdout:Downloading Packages: 2024-02-20T07:46:55.841 INFO:teuthology.orchestra.run.smithi183.stdout:Package python3-rgw-2:17.2.7-521.gc80e7168.el9.x86_64 is already installed. 2024-02-20T07:46:55.893 INFO:teuthology.orchestra.run.smithi183.stdout:Dependencies resolved. 2024-02-20T07:46:55.893 INFO:teuthology.orchestra.run.smithi183.stdout:Nothing to do. 2024-02-20T07:46:55.894 INFO:teuthology.orchestra.run.smithi183.stdout:Complete! 2024-02-20T07:46:55.940 INFO:teuthology.orchestra.run.smithi133.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 571 kB/s | 79 kB 00:00 2024-02-20T07:46:55.952 DEBUG:teuthology.orchestra.run.smithi183:> sudo yum -y install python3-cephfs 2024-02-20T07:46:56.140 INFO:teuthology.orchestra.run.smithi133.stdout:(2/2): ceph-fuse-17.2.7-521.gc80e7168.el9.x86_6 2.3 MB/s | 812 kB 00:00 2024-02-20T07:46:56.141 INFO:teuthology.orchestra.run.smithi133.stdout:-------------------------------------------------------------------------------- 2024-02-20T07:46:56.142 INFO:teuthology.orchestra.run.smithi133.stdout:Total 2.6 MB/s | 891 kB 00:00 2024-02-20T07:46:56.142 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction check 2024-02-20T07:46:56.163 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction check succeeded. 2024-02-20T07:46:56.163 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction test 2024-02-20T07:46:56.239 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction test succeeded. 2024-02-20T07:46:56.239 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction 2024-02-20T07:46:56.421 INFO:teuthology.orchestra.run.smithi133.stdout: Preparing : 1/1 2024-02-20T07:46:56.531 INFO:teuthology.orchestra.run.smithi183.stdout:Last metadata expiration check: 0:01:35 ago on Tue 20 Feb 2024 07:45:21 AM UTC. 2024-02-20T07:46:56.560 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-02-20T07:46:56.623 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : ceph-fuse-2:17.2.7-521.gc80e7168.el9.x86_64 2/2 2024-02-20T07:46:56.646 INFO:teuthology.orchestra.run.smithi183.stdout:Package python3-cephfs-2:17.2.7-521.gc80e7168.el9.x86_64 is already installed. 2024-02-20T07:46:56.698 INFO:teuthology.orchestra.run.smithi183.stdout:Dependencies resolved. 2024-02-20T07:46:56.699 INFO:teuthology.orchestra.run.smithi183.stdout:Nothing to do. 2024-02-20T07:46:56.699 INFO:teuthology.orchestra.run.smithi183.stdout:Complete! 2024-02-20T07:46:56.760 DEBUG:teuthology.orchestra.run.smithi183:> sudo yum -y install python3-rbd 2024-02-20T07:46:57.338 INFO:teuthology.orchestra.run.smithi183.stdout:Last metadata expiration check: 0:01:36 ago on Tue 20 Feb 2024 07:45:21 AM UTC. 2024-02-20T07:46:57.389 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: ceph-fuse-2:17.2.7-521.gc80e7168.el9.x86_64 2/2 2024-02-20T07:46:57.389 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : ceph-fuse-2:17.2.7-521.gc80e7168.el9.x86_64 1/2 2024-02-20T07:46:57.452 INFO:teuthology.orchestra.run.smithi183.stdout:Package python3-rbd-2:17.2.7-521.gc80e7168.el9.x86_64 is already installed. 2024-02-20T07:46:57.508 INFO:teuthology.orchestra.run.smithi183.stdout:Dependencies resolved. 2024-02-20T07:46:57.508 INFO:teuthology.orchestra.run.smithi183.stdout:Nothing to do. 2024-02-20T07:46:57.508 INFO:teuthology.orchestra.run.smithi183.stdout:Complete! 2024-02-20T07:46:57.568 DEBUG:teuthology.orchestra.run.smithi183:> sudo yum -y install rbd-fuse 2024-02-20T07:46:57.708 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-02-20T07:46:57.708 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-02-20T07:46:57.708 INFO:teuthology.orchestra.run.smithi133.stdout:Installed: 2024-02-20T07:46:57.708 INFO:teuthology.orchestra.run.smithi133.stdout: ceph-fuse-2:17.2.7-521.gc80e7168.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-02-20T07:46:57.708 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-02-20T07:46:57.709 INFO:teuthology.orchestra.run.smithi133.stdout:Complete! 2024-02-20T07:46:57.943 DEBUG:teuthology.orchestra.run.smithi133:> sudo yum -y install librados-devel 2024-02-20T07:46:58.150 INFO:teuthology.orchestra.run.smithi183.stdout:Last metadata expiration check: 0:01:37 ago on Tue 20 Feb 2024 07:45:21 AM UTC. 2024-02-20T07:46:58.318 INFO:teuthology.orchestra.run.smithi183.stdout:Dependencies resolved. 2024-02-20T07:46:58.319 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-02-20T07:46:58.319 INFO:teuthology.orchestra.run.smithi183.stdout: Package Architecture Version Repository Size 2024-02-20T07:46:58.319 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-02-20T07:46:58.319 INFO:teuthology.orchestra.run.smithi183.stdout:Installing: 2024-02-20T07:46:58.319 INFO:teuthology.orchestra.run.smithi183.stdout: rbd-fuse x86_64 2:17.2.7-521.gc80e7168.el9 ceph 81 k 2024-02-20T07:46:58.319 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-02-20T07:46:58.319 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction Summary 2024-02-20T07:46:58.320 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-02-20T07:46:58.320 INFO:teuthology.orchestra.run.smithi183.stdout:Install 1 Package 2024-02-20T07:46:58.320 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-02-20T07:46:58.320 INFO:teuthology.orchestra.run.smithi183.stdout:Total download size: 81 k 2024-02-20T07:46:58.320 INFO:teuthology.orchestra.run.smithi183.stdout:Installed size: 222 k 2024-02-20T07:46:58.320 INFO:teuthology.orchestra.run.smithi183.stdout:Downloading Packages: 2024-02-20T07:46:58.532 INFO:teuthology.orchestra.run.smithi133.stdout:Last metadata expiration check: 0:01:38 ago on Tue 20 Feb 2024 07:45:20 AM UTC. 2024-02-20T07:46:58.696 INFO:teuthology.orchestra.run.smithi133.stdout:Dependencies resolved. 2024-02-20T07:46:58.697 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2024-02-20T07:46:58.697 INFO:teuthology.orchestra.run.smithi133.stdout: Package Arch Version Repo Size 2024-02-20T07:46:58.697 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2024-02-20T07:46:58.697 INFO:teuthology.orchestra.run.smithi133.stdout:Installing: 2024-02-20T07:46:58.697 INFO:teuthology.orchestra.run.smithi133.stdout: librados-devel x86_64 2:17.2.7-521.gc80e7168.el9 ceph 126 k 2024-02-20T07:46:58.697 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-02-20T07:46:58.697 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction Summary 2024-02-20T07:46:58.697 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2024-02-20T07:46:58.697 INFO:teuthology.orchestra.run.smithi133.stdout:Install 1 Package 2024-02-20T07:46:58.698 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-02-20T07:46:58.698 INFO:teuthology.orchestra.run.smithi133.stdout:Total download size: 126 k 2024-02-20T07:46:58.698 INFO:teuthology.orchestra.run.smithi133.stdout:Installed size: 448 k 2024-02-20T07:46:58.698 INFO:teuthology.orchestra.run.smithi133.stdout:Downloading Packages: 2024-02-20T07:46:58.927 INFO:teuthology.orchestra.run.smithi183.stdout:rbd-fuse-17.2.7-521.gc80e7168.el9.x86_64.rpm 134 kB/s | 81 kB 00:00 2024-02-20T07:46:58.928 INFO:teuthology.orchestra.run.smithi183.stdout:-------------------------------------------------------------------------------- 2024-02-20T07:46:58.928 INFO:teuthology.orchestra.run.smithi183.stdout:Total 134 kB/s | 81 kB 00:00 2024-02-20T07:46:58.929 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction check 2024-02-20T07:46:58.936 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction check succeeded. 2024-02-20T07:46:58.936 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction test 2024-02-20T07:46:58.991 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction test succeeded. 2024-02-20T07:46:58.992 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction 2024-02-20T07:46:59.131 INFO:teuthology.orchestra.run.smithi183.stdout: Preparing : 1/1 2024-02-20T07:46:59.202 INFO:teuthology.orchestra.run.smithi133.stdout:librados-devel-17.2.7-521.gc80e7168.el9.x86_64. 251 kB/s | 126 kB 00:00 2024-02-20T07:46:59.203 INFO:teuthology.orchestra.run.smithi133.stdout:-------------------------------------------------------------------------------- 2024-02-20T07:46:59.203 INFO:teuthology.orchestra.run.smithi133.stdout:Total 250 kB/s | 126 kB 00:00 2024-02-20T07:46:59.203 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction check 2024-02-20T07:46:59.209 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction check succeeded. 2024-02-20T07:46:59.209 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction test 2024-02-20T07:46:59.233 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : rbd-fuse-2:17.2.7-521.gc80e7168.el9.x86_64 1/1 2024-02-20T07:46:59.265 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction test succeeded. 2024-02-20T07:46:59.265 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction 2024-02-20T07:46:59.416 INFO:teuthology.orchestra.run.smithi133.stdout: Preparing : 1/1 2024-02-20T07:46:59.531 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : librados-devel-2:17.2.7-521.gc80e7168.el9.x86_64 1/1 2024-02-20T07:46:59.636 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: rbd-fuse-2:17.2.7-521.gc80e7168.el9.x86_64 1/1 2024-02-20T07:46:59.958 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : rbd-fuse-2:17.2.7-521.gc80e7168.el9.x86_64 1/1 2024-02-20T07:46:59.959 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-02-20T07:46:59.959 INFO:teuthology.orchestra.run.smithi183.stdout:Installed: 2024-02-20T07:46:59.959 INFO:teuthology.orchestra.run.smithi183.stdout: rbd-fuse-2:17.2.7-521.gc80e7168.el9.x86_64 2024-02-20T07:46:59.959 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-02-20T07:46:59.959 INFO:teuthology.orchestra.run.smithi183.stdout:Complete! 2024-02-20T07:46:59.982 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: librados-devel-2:17.2.7-521.gc80e7168.el9.x86_64 1/1 2024-02-20T07:47:00.197 DEBUG:teuthology.orchestra.run.smithi183:> sudo yum -y install rbd-mirror 2024-02-20T07:47:00.284 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : librados-devel-2:17.2.7-521.gc80e7168.el9.x86_64 1/1 2024-02-20T07:47:00.285 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-02-20T07:47:00.285 INFO:teuthology.orchestra.run.smithi133.stdout:Installed: 2024-02-20T07:47:00.285 INFO:teuthology.orchestra.run.smithi133.stdout: librados-devel-2:17.2.7-521.gc80e7168.el9.x86_64 2024-02-20T07:47:00.285 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-02-20T07:47:00.285 INFO:teuthology.orchestra.run.smithi133.stdout:Complete! 2024-02-20T07:47:00.548 DEBUG:teuthology.orchestra.run.smithi133:> sudo yum -y install libcephfs2 2024-02-20T07:47:00.790 INFO:teuthology.orchestra.run.smithi183.stdout:Last metadata expiration check: 0:01:39 ago on Tue 20 Feb 2024 07:45:21 AM UTC. 2024-02-20T07:47:00.961 INFO:teuthology.orchestra.run.smithi183.stdout:Dependencies resolved. 2024-02-20T07:47:00.962 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-02-20T07:47:00.962 INFO:teuthology.orchestra.run.smithi183.stdout: Package Arch Version Repo Size 2024-02-20T07:47:00.962 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-02-20T07:47:00.962 INFO:teuthology.orchestra.run.smithi183.stdout:Installing: 2024-02-20T07:47:00.962 INFO:teuthology.orchestra.run.smithi183.stdout: rbd-mirror x86_64 2:17.2.7-521.gc80e7168.el9 ceph 2.9 M 2024-02-20T07:47:00.962 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-02-20T07:47:00.962 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction Summary 2024-02-20T07:47:00.963 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-02-20T07:47:00.963 INFO:teuthology.orchestra.run.smithi183.stdout:Install 1 Package 2024-02-20T07:47:00.963 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-02-20T07:47:00.963 INFO:teuthology.orchestra.run.smithi183.stdout:Total download size: 2.9 M 2024-02-20T07:47:00.963 INFO:teuthology.orchestra.run.smithi183.stdout:Installed size: 12 M 2024-02-20T07:47:00.963 INFO:teuthology.orchestra.run.smithi183.stdout:Downloading Packages: 2024-02-20T07:47:01.137 INFO:teuthology.orchestra.run.smithi133.stdout:Last metadata expiration check: 0:01:41 ago on Tue 20 Feb 2024 07:45:20 AM UTC. 2024-02-20T07:47:01.255 INFO:teuthology.orchestra.run.smithi133.stdout:Package libcephfs2-2:17.2.7-521.gc80e7168.el9.x86_64 is already installed. 2024-02-20T07:47:01.308 INFO:teuthology.orchestra.run.smithi133.stdout:Dependencies resolved. 2024-02-20T07:47:01.309 INFO:teuthology.orchestra.run.smithi133.stdout:Nothing to do. 2024-02-20T07:47:01.309 INFO:teuthology.orchestra.run.smithi133.stdout:Complete! 2024-02-20T07:47:01.347 INFO:teuthology.orchestra.run.smithi183.stdout:rbd-mirror-17.2.7-521.gc80e7168.el9.x86_64.rpm 7.7 MB/s | 2.9 MB 00:00 2024-02-20T07:47:01.347 INFO:teuthology.orchestra.run.smithi183.stdout:-------------------------------------------------------------------------------- 2024-02-20T07:47:01.347 INFO:teuthology.orchestra.run.smithi183.stdout:Total 7.6 MB/s | 2.9 MB 00:00 2024-02-20T07:47:01.348 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction check 2024-02-20T07:47:01.357 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction check succeeded. 2024-02-20T07:47:01.357 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction test 2024-02-20T07:47:01.379 DEBUG:teuthology.orchestra.run.smithi133:> sudo yum -y install libcephfs-devel 2024-02-20T07:47:01.429 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction test succeeded. 2024-02-20T07:47:01.429 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction 2024-02-20T07:47:01.837 INFO:teuthology.orchestra.run.smithi183.stdout: Preparing : 1/1 2024-02-20T07:47:01.939 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : rbd-mirror-2:17.2.7-521.gc80e7168.el9.x86_64 1/1 2024-02-20T07:47:01.955 INFO:teuthology.orchestra.run.smithi133.stdout:Last metadata expiration check: 0:01:41 ago on Tue 20 Feb 2024 07:45:20 AM UTC. 2024-02-20T07:47:01.962 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: rbd-mirror-2:17.2.7-521.gc80e7168.el9.x86_64 1/1 2024-02-20T07:47:01.963 INFO:teuthology.orchestra.run.smithi183.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-02-20T07:47:01.963 INFO:teuthology.orchestra.run.smithi183.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-02-20T07:47:01.963 INFO:teuthology.orchestra.run.smithi183.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-02-20T07:47:01.963 INFO:teuthology.orchestra.run.smithi183.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-02-20T07:47:01.963 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-02-20T07:47:02.121 INFO:teuthology.orchestra.run.smithi133.stdout:Dependencies resolved. 2024-02-20T07:47:02.122 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2024-02-20T07:47:02.122 INFO:teuthology.orchestra.run.smithi133.stdout: Package Arch Version Repo Size 2024-02-20T07:47:02.122 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2024-02-20T07:47:02.122 INFO:teuthology.orchestra.run.smithi133.stdout:Installing: 2024-02-20T07:47:02.122 INFO:teuthology.orchestra.run.smithi133.stdout: libcephfs-devel x86_64 2:17.2.7-521.gc80e7168.el9 ceph 25 k 2024-02-20T07:47:02.122 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-02-20T07:47:02.122 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction Summary 2024-02-20T07:47:02.122 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2024-02-20T07:47:02.122 INFO:teuthology.orchestra.run.smithi133.stdout:Install 1 Package 2024-02-20T07:47:02.122 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-02-20T07:47:02.122 INFO:teuthology.orchestra.run.smithi133.stdout:Total download size: 25 k 2024-02-20T07:47:02.123 INFO:teuthology.orchestra.run.smithi133.stdout:Installed size: 110 k 2024-02-20T07:47:02.123 INFO:teuthology.orchestra.run.smithi133.stdout:Downloading Packages: 2024-02-20T07:47:02.306 INFO:teuthology.orchestra.run.smithi133.stdout:libcephfs-devel-17.2.7-521.gc80e7168.el9.x86_64 136 kB/s | 25 kB 00:00 2024-02-20T07:47:02.306 INFO:teuthology.orchestra.run.smithi133.stdout:-------------------------------------------------------------------------------- 2024-02-20T07:47:02.306 INFO:teuthology.orchestra.run.smithi133.stdout:Total 135 kB/s | 25 kB 00:00 2024-02-20T07:47:02.306 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction check 2024-02-20T07:47:02.310 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction check succeeded. 2024-02-20T07:47:02.310 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction test 2024-02-20T07:47:02.311 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction test succeeded. 2024-02-20T07:47:02.312 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction 2024-02-20T07:47:02.395 INFO:teuthology.orchestra.run.smithi133.stdout: Preparing : 1/1 2024-02-20T07:47:02.504 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : libcephfs-devel-2:17.2.7-521.gc80e7168.el9.x86_64 1/1 2024-02-20T07:47:02.883 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: libcephfs-devel-2:17.2.7-521.gc80e7168.el9.x86_64 1/1 2024-02-20T07:47:03.140 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : rbd-mirror-2:17.2.7-521.gc80e7168.el9.x86_64 1/1 2024-02-20T07:47:03.140 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-02-20T07:47:03.140 INFO:teuthology.orchestra.run.smithi183.stdout:Installed: 2024-02-20T07:47:03.140 INFO:teuthology.orchestra.run.smithi183.stdout: rbd-mirror-2:17.2.7-521.gc80e7168.el9.x86_64 2024-02-20T07:47:03.140 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-02-20T07:47:03.141 INFO:teuthology.orchestra.run.smithi183.stdout:Complete! 2024-02-20T07:47:03.159 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : libcephfs-devel-2:17.2.7-521.gc80e7168.el9.x86_64 1/1 2024-02-20T07:47:03.160 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-02-20T07:47:03.160 INFO:teuthology.orchestra.run.smithi133.stdout:Installed: 2024-02-20T07:47:03.160 INFO:teuthology.orchestra.run.smithi133.stdout: libcephfs-devel-2:17.2.7-521.gc80e7168.el9.x86_64 2024-02-20T07:47:03.160 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-02-20T07:47:03.160 INFO:teuthology.orchestra.run.smithi133.stdout:Complete! 2024-02-20T07:47:03.345 DEBUG:teuthology.orchestra.run.smithi133:> sudo yum -y install librados2 2024-02-20T07:47:03.359 DEBUG:teuthology.orchestra.run.smithi183:> sudo yum -y install rbd-nbd 2024-02-20T07:47:03.921 INFO:teuthology.orchestra.run.smithi133.stdout:Last metadata expiration check: 0:01:43 ago on Tue 20 Feb 2024 07:45:20 AM UTC. 2024-02-20T07:47:03.951 INFO:teuthology.orchestra.run.smithi183.stdout:Last metadata expiration check: 0:01:42 ago on Tue 20 Feb 2024 07:45:21 AM UTC. 2024-02-20T07:47:04.035 INFO:teuthology.orchestra.run.smithi133.stdout:Package librados2-2:17.2.7-521.gc80e7168.el9.x86_64 is already installed. 2024-02-20T07:47:04.087 INFO:teuthology.orchestra.run.smithi133.stdout:Dependencies resolved. 2024-02-20T07:47:04.088 INFO:teuthology.orchestra.run.smithi133.stdout:Nothing to do. 2024-02-20T07:47:04.088 INFO:teuthology.orchestra.run.smithi133.stdout:Complete! 2024-02-20T07:47:04.118 INFO:teuthology.orchestra.run.smithi183.stdout:Dependencies resolved. 2024-02-20T07:47:04.119 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-02-20T07:47:04.119 INFO:teuthology.orchestra.run.smithi183.stdout: Package Architecture Version Repository Size 2024-02-20T07:47:04.119 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-02-20T07:47:04.119 INFO:teuthology.orchestra.run.smithi183.stdout:Installing: 2024-02-20T07:47:04.119 INFO:teuthology.orchestra.run.smithi183.stdout: rbd-nbd x86_64 2:17.2.7-521.gc80e7168.el9 ceph 165 k 2024-02-20T07:47:04.119 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-02-20T07:47:04.119 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction Summary 2024-02-20T07:47:04.119 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-02-20T07:47:04.119 INFO:teuthology.orchestra.run.smithi183.stdout:Install 1 Package 2024-02-20T07:47:04.119 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-02-20T07:47:04.120 INFO:teuthology.orchestra.run.smithi183.stdout:Total download size: 165 k 2024-02-20T07:47:04.120 INFO:teuthology.orchestra.run.smithi183.stdout:Installed size: 477 k 2024-02-20T07:47:04.120 INFO:teuthology.orchestra.run.smithi183.stdout:Downloading Packages: 2024-02-20T07:47:04.152 DEBUG:teuthology.orchestra.run.smithi133:> sudo yum -y install librbd1 2024-02-20T07:47:04.350 INFO:teuthology.orchestra.run.smithi183.stdout:rbd-nbd-17.2.7-521.gc80e7168.el9.x86_64.rpm 715 kB/s | 165 kB 00:00 2024-02-20T07:47:04.351 INFO:teuthology.orchestra.run.smithi183.stdout:-------------------------------------------------------------------------------- 2024-02-20T07:47:04.351 INFO:teuthology.orchestra.run.smithi183.stdout:Total 711 kB/s | 165 kB 00:00 2024-02-20T07:47:04.351 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction check 2024-02-20T07:47:04.359 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction check succeeded. 2024-02-20T07:47:04.359 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction test 2024-02-20T07:47:04.414 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction test succeeded. 2024-02-20T07:47:04.414 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction 2024-02-20T07:47:04.560 INFO:teuthology.orchestra.run.smithi183.stdout: Preparing : 1/1 2024-02-20T07:47:04.664 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : rbd-nbd-2:17.2.7-521.gc80e7168.el9.x86_64 1/1 2024-02-20T07:47:04.725 INFO:teuthology.orchestra.run.smithi133.stdout:Last metadata expiration check: 0:01:44 ago on Tue 20 Feb 2024 07:45:20 AM UTC. 2024-02-20T07:47:04.838 INFO:teuthology.orchestra.run.smithi133.stdout:Package librbd1-2:17.2.7-521.gc80e7168.el9.x86_64 is already installed. 2024-02-20T07:47:04.890 INFO:teuthology.orchestra.run.smithi133.stdout:Dependencies resolved. 2024-02-20T07:47:04.891 INFO:teuthology.orchestra.run.smithi133.stdout:Nothing to do. 2024-02-20T07:47:04.891 INFO:teuthology.orchestra.run.smithi133.stdout:Complete! 2024-02-20T07:47:04.955 DEBUG:teuthology.orchestra.run.smithi133:> sudo yum -y install python3-rados 2024-02-20T07:47:05.129 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: rbd-nbd-2:17.2.7-521.gc80e7168.el9.x86_64 1/1 2024-02-20T07:47:05.442 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : rbd-nbd-2:17.2.7-521.gc80e7168.el9.x86_64 1/1 2024-02-20T07:47:05.443 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-02-20T07:47:05.443 INFO:teuthology.orchestra.run.smithi183.stdout:Installed: 2024-02-20T07:47:05.443 INFO:teuthology.orchestra.run.smithi183.stdout: rbd-nbd-2:17.2.7-521.gc80e7168.el9.x86_64 2024-02-20T07:47:05.443 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-02-20T07:47:05.443 INFO:teuthology.orchestra.run.smithi183.stdout:Complete! 2024-02-20T07:47:05.529 INFO:teuthology.orchestra.run.smithi133.stdout:Last metadata expiration check: 0:01:45 ago on Tue 20 Feb 2024 07:45:20 AM UTC. 2024-02-20T07:47:05.642 INFO:teuthology.orchestra.run.smithi133.stdout:Package python3-rados-2:17.2.7-521.gc80e7168.el9.x86_64 is already installed. 2024-02-20T07:47:05.694 INFO:teuthology.orchestra.run.smithi133.stdout:Dependencies resolved. 2024-02-20T07:47:05.696 INFO:teuthology.orchestra.run.smithi133.stdout:Nothing to do. 2024-02-20T07:47:05.696 INFO:teuthology.orchestra.run.smithi133.stdout:Complete! 2024-02-20T07:47:05.701 DEBUG:teuthology.parallel:result is None 2024-02-20T07:47:05.763 DEBUG:teuthology.orchestra.run.smithi133:> sudo yum -y install python3-rgw 2024-02-20T07:47:06.340 INFO:teuthology.orchestra.run.smithi133.stdout:Last metadata expiration check: 0:01:46 ago on Tue 20 Feb 2024 07:45:20 AM UTC. 2024-02-20T07:47:06.453 INFO:teuthology.orchestra.run.smithi133.stdout:Package python3-rgw-2:17.2.7-521.gc80e7168.el9.x86_64 is already installed. 2024-02-20T07:47:06.505 INFO:teuthology.orchestra.run.smithi133.stdout:Dependencies resolved. 2024-02-20T07:47:06.506 INFO:teuthology.orchestra.run.smithi133.stdout:Nothing to do. 2024-02-20T07:47:06.506 INFO:teuthology.orchestra.run.smithi133.stdout:Complete! 2024-02-20T07:47:06.571 DEBUG:teuthology.orchestra.run.smithi133:> sudo yum -y install python3-cephfs 2024-02-20T07:47:07.147 INFO:teuthology.orchestra.run.smithi133.stdout:Last metadata expiration check: 0:01:47 ago on Tue 20 Feb 2024 07:45:20 AM UTC. 2024-02-20T07:47:07.261 INFO:teuthology.orchestra.run.smithi133.stdout:Package python3-cephfs-2:17.2.7-521.gc80e7168.el9.x86_64 is already installed. 2024-02-20T07:47:07.313 INFO:teuthology.orchestra.run.smithi133.stdout:Dependencies resolved. 2024-02-20T07:47:07.313 INFO:teuthology.orchestra.run.smithi133.stdout:Nothing to do. 2024-02-20T07:47:07.314 INFO:teuthology.orchestra.run.smithi133.stdout:Complete! 2024-02-20T07:47:07.379 DEBUG:teuthology.orchestra.run.smithi133:> sudo yum -y install python3-rbd 2024-02-20T07:47:07.959 INFO:teuthology.orchestra.run.smithi133.stdout:Last metadata expiration check: 0:01:47 ago on Tue 20 Feb 2024 07:45:20 AM UTC. 2024-02-20T07:47:08.073 INFO:teuthology.orchestra.run.smithi133.stdout:Package python3-rbd-2:17.2.7-521.gc80e7168.el9.x86_64 is already installed. 2024-02-20T07:47:08.125 INFO:teuthology.orchestra.run.smithi133.stdout:Dependencies resolved. 2024-02-20T07:47:08.126 INFO:teuthology.orchestra.run.smithi133.stdout:Nothing to do. 2024-02-20T07:47:08.126 INFO:teuthology.orchestra.run.smithi133.stdout:Complete! 2024-02-20T07:47:08.187 DEBUG:teuthology.orchestra.run.smithi133:> sudo yum -y install rbd-fuse 2024-02-20T07:47:08.766 INFO:teuthology.orchestra.run.smithi133.stdout:Last metadata expiration check: 0:01:48 ago on Tue 20 Feb 2024 07:45:20 AM UTC. 2024-02-20T07:47:08.931 INFO:teuthology.orchestra.run.smithi133.stdout:Dependencies resolved. 2024-02-20T07:47:08.932 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2024-02-20T07:47:08.932 INFO:teuthology.orchestra.run.smithi133.stdout: Package Architecture Version Repository Size 2024-02-20T07:47:08.932 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2024-02-20T07:47:08.932 INFO:teuthology.orchestra.run.smithi133.stdout:Installing: 2024-02-20T07:47:08.932 INFO:teuthology.orchestra.run.smithi133.stdout: rbd-fuse x86_64 2:17.2.7-521.gc80e7168.el9 ceph 81 k 2024-02-20T07:47:08.932 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-02-20T07:47:08.932 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction Summary 2024-02-20T07:47:08.932 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2024-02-20T07:47:08.932 INFO:teuthology.orchestra.run.smithi133.stdout:Install 1 Package 2024-02-20T07:47:08.932 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-02-20T07:47:08.932 INFO:teuthology.orchestra.run.smithi133.stdout:Total download size: 81 k 2024-02-20T07:47:08.932 INFO:teuthology.orchestra.run.smithi133.stdout:Installed size: 222 k 2024-02-20T07:47:08.933 INFO:teuthology.orchestra.run.smithi133.stdout:Downloading Packages: 2024-02-20T07:47:09.150 INFO:teuthology.orchestra.run.smithi133.stdout:rbd-fuse-17.2.7-521.gc80e7168.el9.x86_64.rpm 374 kB/s | 81 kB 00:00 2024-02-20T07:47:09.151 INFO:teuthology.orchestra.run.smithi133.stdout:-------------------------------------------------------------------------------- 2024-02-20T07:47:09.151 INFO:teuthology.orchestra.run.smithi133.stdout:Total 372 kB/s | 81 kB 00:00 2024-02-20T07:47:09.151 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction check 2024-02-20T07:47:09.158 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction check succeeded. 2024-02-20T07:47:09.158 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction test 2024-02-20T07:47:09.213 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction test succeeded. 2024-02-20T07:47:09.213 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction 2024-02-20T07:47:09.351 INFO:teuthology.orchestra.run.smithi133.stdout: Preparing : 1/1 2024-02-20T07:47:09.448 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : rbd-fuse-2:17.2.7-521.gc80e7168.el9.x86_64 1/1 2024-02-20T07:47:09.866 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: rbd-fuse-2:17.2.7-521.gc80e7168.el9.x86_64 1/1 2024-02-20T07:47:10.149 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : rbd-fuse-2:17.2.7-521.gc80e7168.el9.x86_64 1/1 2024-02-20T07:47:10.149 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-02-20T07:47:10.149 INFO:teuthology.orchestra.run.smithi133.stdout:Installed: 2024-02-20T07:47:10.149 INFO:teuthology.orchestra.run.smithi133.stdout: rbd-fuse-2:17.2.7-521.gc80e7168.el9.x86_64 2024-02-20T07:47:10.149 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-02-20T07:47:10.149 INFO:teuthology.orchestra.run.smithi133.stdout:Complete! 2024-02-20T07:47:10.385 DEBUG:teuthology.orchestra.run.smithi133:> sudo yum -y install rbd-mirror 2024-02-20T07:47:10.969 INFO:teuthology.orchestra.run.smithi133.stdout:Last metadata expiration check: 0:01:50 ago on Tue 20 Feb 2024 07:45:20 AM UTC. 2024-02-20T07:47:11.140 INFO:teuthology.orchestra.run.smithi133.stdout:Dependencies resolved. 2024-02-20T07:47:11.142 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2024-02-20T07:47:11.142 INFO:teuthology.orchestra.run.smithi133.stdout: Package Arch Version Repo Size 2024-02-20T07:47:11.142 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2024-02-20T07:47:11.142 INFO:teuthology.orchestra.run.smithi133.stdout:Installing: 2024-02-20T07:47:11.142 INFO:teuthology.orchestra.run.smithi133.stdout: rbd-mirror x86_64 2:17.2.7-521.gc80e7168.el9 ceph 2.9 M 2024-02-20T07:47:11.142 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-02-20T07:47:11.143 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction Summary 2024-02-20T07:47:11.143 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2024-02-20T07:47:11.143 INFO:teuthology.orchestra.run.smithi133.stdout:Install 1 Package 2024-02-20T07:47:11.143 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-02-20T07:47:11.143 INFO:teuthology.orchestra.run.smithi133.stdout:Total download size: 2.9 M 2024-02-20T07:47:11.143 INFO:teuthology.orchestra.run.smithi133.stdout:Installed size: 12 M 2024-02-20T07:47:11.144 INFO:teuthology.orchestra.run.smithi133.stdout:Downloading Packages: 2024-02-20T07:47:11.522 INFO:teuthology.orchestra.run.smithi133.stdout:rbd-mirror-17.2.7-521.gc80e7168.el9.x86_64.rpm 7.8 MB/s | 2.9 MB 00:00 2024-02-20T07:47:11.522 INFO:teuthology.orchestra.run.smithi133.stdout:-------------------------------------------------------------------------------- 2024-02-20T07:47:11.522 INFO:teuthology.orchestra.run.smithi133.stdout:Total 7.8 MB/s | 2.9 MB 00:00 2024-02-20T07:47:11.522 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction check 2024-02-20T07:47:11.530 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction check succeeded. 2024-02-20T07:47:11.530 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction test 2024-02-20T07:47:11.605 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction test succeeded. 2024-02-20T07:47:11.605 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction 2024-02-20T07:47:12.028 INFO:teuthology.orchestra.run.smithi133.stdout: Preparing : 1/1 2024-02-20T07:47:12.079 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : rbd-mirror-2:17.2.7-521.gc80e7168.el9.x86_64 1/1 2024-02-20T07:47:12.104 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: rbd-mirror-2:17.2.7-521.gc80e7168.el9.x86_64 1/1 2024-02-20T07:47:12.104 INFO:teuthology.orchestra.run.smithi133.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-02-20T07:47:12.104 INFO:teuthology.orchestra.run.smithi133.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-02-20T07:47:12.104 INFO:teuthology.orchestra.run.smithi133.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-02-20T07:47:12.105 INFO:teuthology.orchestra.run.smithi133.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-02-20T07:47:12.105 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-02-20T07:47:13.316 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : rbd-mirror-2:17.2.7-521.gc80e7168.el9.x86_64 1/1 2024-02-20T07:47:13.316 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-02-20T07:47:13.316 INFO:teuthology.orchestra.run.smithi133.stdout:Installed: 2024-02-20T07:47:13.317 INFO:teuthology.orchestra.run.smithi133.stdout: rbd-mirror-2:17.2.7-521.gc80e7168.el9.x86_64 2024-02-20T07:47:13.317 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-02-20T07:47:13.317 INFO:teuthology.orchestra.run.smithi133.stdout:Complete! 2024-02-20T07:47:13.552 DEBUG:teuthology.orchestra.run.smithi133:> sudo yum -y install rbd-nbd 2024-02-20T07:47:14.152 INFO:teuthology.orchestra.run.smithi133.stdout:Last metadata expiration check: 0:01:54 ago on Tue 20 Feb 2024 07:45:20 AM UTC. 2024-02-20T07:47:14.321 INFO:teuthology.orchestra.run.smithi133.stdout:Dependencies resolved. 2024-02-20T07:47:14.322 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2024-02-20T07:47:14.322 INFO:teuthology.orchestra.run.smithi133.stdout: Package Architecture Version Repository Size 2024-02-20T07:47:14.322 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2024-02-20T07:47:14.322 INFO:teuthology.orchestra.run.smithi133.stdout:Installing: 2024-02-20T07:47:14.322 INFO:teuthology.orchestra.run.smithi133.stdout: rbd-nbd x86_64 2:17.2.7-521.gc80e7168.el9 ceph 165 k 2024-02-20T07:47:14.322 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-02-20T07:47:14.323 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction Summary 2024-02-20T07:47:14.323 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2024-02-20T07:47:14.323 INFO:teuthology.orchestra.run.smithi133.stdout:Install 1 Package 2024-02-20T07:47:14.323 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-02-20T07:47:14.323 INFO:teuthology.orchestra.run.smithi133.stdout:Total download size: 165 k 2024-02-20T07:47:14.323 INFO:teuthology.orchestra.run.smithi133.stdout:Installed size: 477 k 2024-02-20T07:47:14.323 INFO:teuthology.orchestra.run.smithi133.stdout:Downloading Packages: 2024-02-20T07:47:14.856 INFO:teuthology.orchestra.run.smithi133.stdout:rbd-nbd-17.2.7-521.gc80e7168.el9.x86_64.rpm 310 kB/s | 165 kB 00:00 2024-02-20T07:47:14.856 INFO:teuthology.orchestra.run.smithi133.stdout:-------------------------------------------------------------------------------- 2024-02-20T07:47:14.857 INFO:teuthology.orchestra.run.smithi133.stdout:Total 309 kB/s | 165 kB 00:00 2024-02-20T07:47:14.857 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction check 2024-02-20T07:47:14.864 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction check succeeded. 2024-02-20T07:47:14.864 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction test 2024-02-20T07:47:14.921 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction test succeeded. 2024-02-20T07:47:14.921 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction 2024-02-20T07:47:15.068 INFO:teuthology.orchestra.run.smithi133.stdout: Preparing : 1/1 2024-02-20T07:47:15.171 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : rbd-nbd-2:17.2.7-521.gc80e7168.el9.x86_64 1/1 2024-02-20T07:47:15.614 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: rbd-nbd-2:17.2.7-521.gc80e7168.el9.x86_64 1/1 2024-02-20T07:47:15.916 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : rbd-nbd-2:17.2.7-521.gc80e7168.el9.x86_64 1/1 2024-02-20T07:47:15.916 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-02-20T07:47:15.916 INFO:teuthology.orchestra.run.smithi133.stdout:Installed: 2024-02-20T07:47:15.917 INFO:teuthology.orchestra.run.smithi133.stdout: rbd-nbd-2:17.2.7-521.gc80e7168.el9.x86_64 2024-02-20T07:47:15.917 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-02-20T07:47:15.917 INFO:teuthology.orchestra.run.smithi133.stdout:Complete! 2024-02-20T07:47:16.140 DEBUG:teuthology.parallel:result is None 2024-02-20T07:47:16.141 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-02-20T07:47:16.141 INFO:teuthology.packaging:ref: None 2024-02-20T07:47:16.141 INFO:teuthology.packaging:tag: None 2024-02-20T07:47:16.141 INFO:teuthology.packaging:branch: quincy 2024-02-20T07:47:16.141 INFO:teuthology.packaging:sha1: 91ff50e903dcc99ee3ecf9523050878eb452da5c 2024-02-20T07:47:16.142 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=quincy 2024-02-20T07:47:16.363 DEBUG:teuthology.orchestra.run.smithi133:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-02-20T07:47:16.385 INFO:teuthology.orchestra.run.smithi133.stdout:17.2.7-521.gc80e7168.el9 2024-02-20T07:47:16.385 INFO:teuthology.packaging:The installed version of ceph is 17.2.7-521.gc80e7168.el9 2024-02-20T07:47:16.385 INFO:teuthology.task.install:The correct ceph version 17.2.7-521.gc80e7168 is installed. 2024-02-20T07:47:16.388 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-02-20T07:47:16.388 INFO:teuthology.packaging:ref: None 2024-02-20T07:47:16.388 INFO:teuthology.packaging:tag: None 2024-02-20T07:47:16.388 INFO:teuthology.packaging:branch: quincy 2024-02-20T07:47:16.388 INFO:teuthology.packaging:sha1: 91ff50e903dcc99ee3ecf9523050878eb452da5c 2024-02-20T07:47:16.388 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=quincy 2024-02-20T07:47:16.845 DEBUG:teuthology.orchestra.run.smithi183:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-02-20T07:47:16.870 INFO:teuthology.orchestra.run.smithi183.stdout:17.2.7-521.gc80e7168.el9 2024-02-20T07:47:16.870 INFO:teuthology.packaging:The installed version of ceph is 17.2.7-521.gc80e7168.el9 2024-02-20T07:47:16.870 INFO:teuthology.task.install:The correct ceph version 17.2.7-521.gc80e7168 is installed. 2024-02-20T07:47:16.872 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-02-20T07:47:16.872 DEBUG:teuthology.orchestra.run.smithi133:> set -ex 2024-02-20T07:47:16.872 DEBUG:teuthology.orchestra.run.smithi133:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-02-20T07:47:16.901 DEBUG:teuthology.orchestra.run.smithi183:> set -ex 2024-02-20T07:47:16.902 DEBUG:teuthology.orchestra.run.smithi183:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-02-20T07:47:16.940 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-02-20T07:47:16.941 DEBUG:teuthology.orchestra.run.smithi133:> set -ex 2024-02-20T07:47:16.941 DEBUG:teuthology.orchestra.run.smithi133:> sudo dd of=/usr/bin/daemon-helper 2024-02-20T07:47:16.969 DEBUG:teuthology.orchestra.run.smithi133:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-02-20T07:47:17.037 DEBUG:teuthology.orchestra.run.smithi183:> set -ex 2024-02-20T07:47:17.037 DEBUG:teuthology.orchestra.run.smithi183:> sudo dd of=/usr/bin/daemon-helper 2024-02-20T07:47:17.066 DEBUG:teuthology.orchestra.run.smithi183:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-02-20T07:47:17.135 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-02-20T07:47:17.136 DEBUG:teuthology.orchestra.run.smithi133:> set -ex 2024-02-20T07:47:17.136 DEBUG:teuthology.orchestra.run.smithi133:> sudo dd of=/usr/bin/adjust-ulimits 2024-02-20T07:47:17.170 DEBUG:teuthology.orchestra.run.smithi133:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-02-20T07:47:17.237 DEBUG:teuthology.orchestra.run.smithi183:> set -ex 2024-02-20T07:47:17.237 DEBUG:teuthology.orchestra.run.smithi183:> sudo dd of=/usr/bin/adjust-ulimits 2024-02-20T07:47:17.267 DEBUG:teuthology.orchestra.run.smithi183:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-02-20T07:47:17.336 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-02-20T07:47:17.336 DEBUG:teuthology.orchestra.run.smithi133:> set -ex 2024-02-20T07:47:17.337 DEBUG:teuthology.orchestra.run.smithi133:> sudo dd of=/usr/bin/stdin-killer 2024-02-20T07:47:17.363 DEBUG:teuthology.orchestra.run.smithi133:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-02-20T07:47:17.430 DEBUG:teuthology.orchestra.run.smithi183:> set -ex 2024-02-20T07:47:17.431 DEBUG:teuthology.orchestra.run.smithi183:> sudo dd of=/usr/bin/stdin-killer 2024-02-20T07:47:17.457 DEBUG:teuthology.orchestra.run.smithi183:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-02-20T07:47:17.525 INFO:teuthology.run_tasks:Running task print... 2024-02-20T07:47:17.535 INFO:teuthology.task.print:**** done install task... 2024-02-20T07:47:17.535 INFO:teuthology.run_tasks:Running task cephadm... 2024-02-20T07:47:17.684 INFO:tasks.cephadm:Config: {'cephadm_branch': 'quincy', 'cephadm_git_url': 'https://github.com/ceph/ceph', 'conf': {'osd': {'osd_class_default_list': '*', 'osd_class_load_list': '*', 'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}, 'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'mon pg warn min per osd': 0}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}}, 'image': 'quay.ceph.io/ceph-ci/ceph:quincy', 'roleless': True, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'POOL_APP_NOT_ENABLED', 'overall HEALTH_', 'Replacing daemon', 'OSD_DOWN', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding'], 'sha1': '91ff50e903dcc99ee3ecf9523050878eb452da5c'} 2024-02-20T07:47:17.684 INFO:tasks.cephadm:Cluster image is quay.ceph.io/ceph-ci/ceph:quincy 2024-02-20T07:47:17.685 INFO:tasks.cephadm:Cluster fsid is 4537327c-cfc4-11ee-95bb-87774f69a715 2024-02-20T07:47:17.685 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-02-20T07:47:17.685 INFO:tasks.cephadm:No mon roles; fabricating mons 2024-02-20T07:47:17.685 INFO:tasks.cephadm:Monitor IPs: {'mon.smithi133': '172.21.15.133', 'mon.smithi183': '172.21.15.183'} 2024-02-20T07:47:17.685 INFO:tasks.cephadm:Normalizing hostnames... 2024-02-20T07:47:17.685 DEBUG:teuthology.orchestra.run.smithi133:> sudo hostname $(hostname -s) 2024-02-20T07:47:17.712 DEBUG:teuthology.orchestra.run.smithi183:> sudo hostname $(hostname -s) 2024-02-20T07:47:17.740 INFO:tasks.cephadm:Downloading cephadm (repo https://github.com/ceph/ceph ref quincy)... 2024-02-20T07:47:17.740 DEBUG:teuthology.orchestra.run.smithi133:> curl --silent https://raw.githubusercontent.com/ceph/ceph/quincy/src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-02-20T07:47:18.091 INFO:teuthology.orchestra.run.smithi133.stdout:-rw-r--r--. 1 ubuntu ubuntu 367151 Feb 20 07:47 /home/ubuntu/cephtest/cephadm 2024-02-20T07:47:18.092 DEBUG:teuthology.orchestra.run.smithi183:> curl --silent https://raw.githubusercontent.com/ceph/ceph/quincy/src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-02-20T07:47:18.284 INFO:teuthology.orchestra.run.smithi183.stdout:-rw-r--r--. 1 ubuntu ubuntu 367151 Feb 20 07:47 /home/ubuntu/cephtest/cephadm 2024-02-20T07:47:18.285 DEBUG:teuthology.orchestra.run.smithi133:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-02-20T07:47:18.302 DEBUG:teuthology.orchestra.run.smithi183:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-02-20T07:47:18.328 INFO:tasks.cephadm:Pulling image quay.ceph.io/ceph-ci/ceph:quincy on all hosts... 2024-02-20T07:47:18.328 DEBUG:teuthology.orchestra.run.smithi133:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy pull 2024-02-20T07:47:18.345 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy pull 2024-02-20T07:47:19.182 INFO:teuthology.orchestra.run.smithi133.stderr:Pulling container image quay.ceph.io/ceph-ci/ceph:quincy... 2024-02-20T07:47:19.243 INFO:teuthology.orchestra.run.smithi183.stderr:Pulling container image quay.ceph.io/ceph-ci/ceph:quincy... 2024-02-20T07:48:04.411 INFO:teuthology.orchestra.run.smithi183.stdout:{ 2024-02-20T07:48:04.411 INFO:teuthology.orchestra.run.smithi183.stdout: "ceph_version": "ceph version 17.2.7-521-gc80e7168 (c80e7168938c48b49e5ce2e002f48bf150b5f4a4) quincy (stable)", 2024-02-20T07:48:04.411 INFO:teuthology.orchestra.run.smithi183.stdout: "image_id": "10b0c032c913004d1f4f6c0f2496f5bfddfde4aeddb489f4129f49e49277ea3e", 2024-02-20T07:48:04.412 INFO:teuthology.orchestra.run.smithi183.stdout: "repo_digests": [ 2024-02-20T07:48:04.412 INFO:teuthology.orchestra.run.smithi183.stdout: "quay.ceph.io/ceph-ci/ceph@sha256:479114d4af9dd6c9b2771de62c79e08a4e4ba3cecb535f7d44dca6fc106bb04b" 2024-02-20T07:48:04.412 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-02-20T07:48:04.412 INFO:teuthology.orchestra.run.smithi183.stdout:} 2024-02-20T07:48:04.867 INFO:teuthology.orchestra.run.smithi133.stdout:{ 2024-02-20T07:48:04.868 INFO:teuthology.orchestra.run.smithi133.stdout: "ceph_version": "ceph version 17.2.7-521-gc80e7168 (c80e7168938c48b49e5ce2e002f48bf150b5f4a4) quincy (stable)", 2024-02-20T07:48:04.868 INFO:teuthology.orchestra.run.smithi133.stdout: "image_id": "10b0c032c913004d1f4f6c0f2496f5bfddfde4aeddb489f4129f49e49277ea3e", 2024-02-20T07:48:04.868 INFO:teuthology.orchestra.run.smithi133.stdout: "repo_digests": [ 2024-02-20T07:48:04.868 INFO:teuthology.orchestra.run.smithi133.stdout: "quay.ceph.io/ceph-ci/ceph@sha256:479114d4af9dd6c9b2771de62c79e08a4e4ba3cecb535f7d44dca6fc106bb04b" 2024-02-20T07:48:04.868 INFO:teuthology.orchestra.run.smithi133.stdout: ] 2024-02-20T07:48:04.868 INFO:teuthology.orchestra.run.smithi133.stdout:} 2024-02-20T07:48:04.888 DEBUG:teuthology.orchestra.run.smithi133:> sudo mkdir -p /etc/ceph 2024-02-20T07:48:04.933 DEBUG:teuthology.orchestra.run.smithi183:> sudo mkdir -p /etc/ceph 2024-02-20T07:48:04.965 DEBUG:teuthology.orchestra.run.smithi133:> sudo chmod 777 /etc/ceph 2024-02-20T07:48:04.992 DEBUG:teuthology.orchestra.run.smithi183:> sudo chmod 777 /etc/ceph 2024-02-20T07:48:05.034 INFO:tasks.cephadm:Writing seed config... 2024-02-20T07:48:05.035 INFO:tasks.cephadm: override: [osd] osd_class_default_list = * 2024-02-20T07:48:05.035 INFO:tasks.cephadm: override: [osd] osd_class_load_list = * 2024-02-20T07:48:05.036 INFO:tasks.cephadm: override: [osd] bdev async discard = True 2024-02-20T07:48:05.036 INFO:tasks.cephadm: override: [osd] bdev enable discard = True 2024-02-20T07:48:05.036 INFO:tasks.cephadm: override: [osd] bluestore allocator = bitmap 2024-02-20T07:48:05.036 INFO:tasks.cephadm: override: [osd] bluestore block size = 96636764160 2024-02-20T07:48:05.036 INFO:tasks.cephadm: override: [osd] bluestore fsck on mount = True 2024-02-20T07:48:05.036 INFO:tasks.cephadm: override: [osd] debug bluefs = 1/20 2024-02-20T07:48:05.036 INFO:tasks.cephadm: override: [osd] debug bluestore = 1/20 2024-02-20T07:48:05.036 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-02-20T07:48:05.036 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-02-20T07:48:05.036 INFO:tasks.cephadm: override: [osd] debug rocksdb = 4/10 2024-02-20T07:48:05.036 INFO:tasks.cephadm: override: [osd] mon osd backfillfull_ratio = 0.85 2024-02-20T07:48:05.036 INFO:tasks.cephadm: override: [osd] mon osd full ratio = 0.9 2024-02-20T07:48:05.036 INFO:tasks.cephadm: override: [osd] mon osd nearfull ratio = 0.8 2024-02-20T07:48:05.036 INFO:tasks.cephadm: override: [osd] osd failsafe full ratio = 0.95 2024-02-20T07:48:05.036 INFO:tasks.cephadm: override: [osd] osd objectstore = bluestore 2024-02-20T07:48:05.037 INFO:tasks.cephadm: override: [osd] osd op complaint time = 180 2024-02-20T07:48:05.037 INFO:tasks.cephadm: override: [client] client mount timeout = 600 2024-02-20T07:48:05.037 INFO:tasks.cephadm: override: [client] debug client = 20 2024-02-20T07:48:05.037 INFO:tasks.cephadm: override: [client] debug ms = 1 2024-02-20T07:48:05.037 INFO:tasks.cephadm: override: [client] rados mon op timeout = 900 2024-02-20T07:48:05.037 INFO:tasks.cephadm: override: [client] rados osd op timeout = 900 2024-02-20T07:48:05.037 INFO:tasks.cephadm: override: [global] mon pg warn min per osd = 0 2024-02-20T07:48:05.037 INFO:tasks.cephadm: override: [mds] debug mds = 20 2024-02-20T07:48:05.037 INFO:tasks.cephadm: override: [mds] debug mds balancer = 20 2024-02-20T07:48:05.037 INFO:tasks.cephadm: override: [mds] debug ms = 1 2024-02-20T07:48:05.037 INFO:tasks.cephadm: override: [mds] mds debug frag = True 2024-02-20T07:48:05.037 INFO:tasks.cephadm: override: [mds] mds debug scatterstat = True 2024-02-20T07:48:05.037 INFO:tasks.cephadm: override: [mds] mds op complaint time = 180 2024-02-20T07:48:05.037 INFO:tasks.cephadm: override: [mds] mds verify scatter = True 2024-02-20T07:48:05.037 INFO:tasks.cephadm: override: [mds] osd op complaint time = 180 2024-02-20T07:48:05.038 INFO:tasks.cephadm: override: [mds] rados mon op timeout = 900 2024-02-20T07:48:05.038 INFO:tasks.cephadm: override: [mds] rados osd op timeout = 900 2024-02-20T07:48:05.038 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-02-20T07:48:05.038 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-02-20T07:48:05.038 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-02-20T07:48:05.038 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-02-20T07:48:05.038 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-02-20T07:48:05.038 INFO:tasks.cephadm: override: [mon] mon op complaint time = 120 2024-02-20T07:48:05.039 DEBUG:teuthology.orchestra.run.smithi133:> set -ex 2024-02-20T07:48:05.039 DEBUG:teuthology.orchestra.run.smithi133:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-02-20T07:48:05.054 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = 4537327c-cfc4-11ee-95bb-87774f69a715 mon pg warn min per osd = 0 [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = true bdev debug aio = true osd sloppy crc = true osd_class_default_list = * osd_class_load_list = * bdev async discard = True bdev enable discard = True bluestore allocator = bitmap bluestore block size = 96636764160 bluestore fsck on mount = True debug bluefs = 1/20 debug bluestore = 1/20 debug ms = 1 debug osd = 20 debug rocksdb = 4/10 mon osd backfillfull_ratio = 0.85 mon osd full ratio = 0.9 mon osd nearfull ratio = 0.8 osd failsafe full ratio = 0.95 osd objectstore = bluestore osd op complaint time = 180 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 mon op complaint time = 120 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true [client] client mount timeout = 600 debug client = 20 debug ms = 1 rados mon op timeout = 900 rados osd op timeout = 900 [mds] debug mds = 20 debug mds balancer = 20 debug ms = 1 mds debug frag = True mds debug scatterstat = True mds op complaint time = 180 mds verify scatter = True osd op complaint time = 180 rados mon op timeout = 900 rados osd op timeout = 900 2024-02-20T07:48:05.054 DEBUG:teuthology.orchestra.run.smithi133:mon.smithi133> sudo journalctl -f -n 0 -u ceph-4537327c-cfc4-11ee-95bb-87774f69a715@mon.smithi133.service 2024-02-20T07:48:05.096 INFO:tasks.cephadm:Bootstrapping... 2024-02-20T07:48:05.097 DEBUG:teuthology.orchestra.run.smithi133:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy -v bootstrap --fsid 4537327c-cfc4-11ee-95bb-87774f69a715 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-ip 172.21.15.133 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-02-20T07:48:05.298 INFO:teuthology.orchestra.run.smithi133.stdout:-------------------------------------------------------------------------------- 2024-02-20T07:48:05.298 INFO:teuthology.orchestra.run.smithi133.stdout:cephadm ['--image', 'quay.ceph.io/ceph-ci/ceph:quincy', '-v', 'bootstrap', '--fsid', '4537327c-cfc4-11ee-95bb-87774f69a715', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-ip', '172.21.15.133', '--skip-admin-label'] 2024-02-20T07:48:05.323 INFO:teuthology.orchestra.run.smithi133.stdout:/bin/podman: stdout 4.9.3-dev 2024-02-20T07:48:05.324 INFO:teuthology.orchestra.run.smithi133.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2024-02-20T07:48:05.324 INFO:teuthology.orchestra.run.smithi133.stdout:Verifying podman|docker is present... 2024-02-20T07:48:05.347 INFO:teuthology.orchestra.run.smithi133.stdout:/bin/podman: stdout 4.9.3-dev 2024-02-20T07:48:05.347 INFO:teuthology.orchestra.run.smithi133.stdout:Verifying lvm2 is present... 2024-02-20T07:48:05.347 INFO:teuthology.orchestra.run.smithi133.stdout:Verifying time synchronization is in place... 2024-02-20T07:48:05.355 INFO:teuthology.orchestra.run.smithi133.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-02-20T07:48:05.355 INFO:teuthology.orchestra.run.smithi133.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-02-20T07:48:05.361 INFO:teuthology.orchestra.run.smithi133.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-02-20T07:48:05.361 INFO:teuthology.orchestra.run.smithi133.stdout:systemctl: stdout inactive 2024-02-20T07:48:05.367 INFO:teuthology.orchestra.run.smithi133.stdout:systemctl: stdout enabled 2024-02-20T07:48:05.374 INFO:teuthology.orchestra.run.smithi133.stdout:systemctl: stdout active 2024-02-20T07:48:05.374 INFO:teuthology.orchestra.run.smithi133.stdout:Unit chronyd.service is enabled and running 2024-02-20T07:48:05.374 INFO:teuthology.orchestra.run.smithi133.stdout:Repeating the final host check... 2024-02-20T07:48:05.397 INFO:teuthology.orchestra.run.smithi133.stdout:/bin/podman: stdout 4.9.3-dev 2024-02-20T07:48:05.397 INFO:teuthology.orchestra.run.smithi133.stdout:podman (/bin/podman) version 4.9.3 is present 2024-02-20T07:48:05.397 INFO:teuthology.orchestra.run.smithi133.stdout:systemctl is present 2024-02-20T07:48:05.397 INFO:teuthology.orchestra.run.smithi133.stdout:lvcreate is present 2024-02-20T07:48:05.404 INFO:teuthology.orchestra.run.smithi133.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-02-20T07:48:05.404 INFO:teuthology.orchestra.run.smithi133.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-02-20T07:48:05.410 INFO:teuthology.orchestra.run.smithi133.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-02-20T07:48:05.410 INFO:teuthology.orchestra.run.smithi133.stdout:systemctl: stdout inactive 2024-02-20T07:48:05.416 INFO:teuthology.orchestra.run.smithi133.stdout:systemctl: stdout enabled 2024-02-20T07:48:05.423 INFO:teuthology.orchestra.run.smithi133.stdout:systemctl: stdout active 2024-02-20T07:48:05.423 INFO:teuthology.orchestra.run.smithi133.stdout:Unit chronyd.service is enabled and running 2024-02-20T07:48:05.423 INFO:teuthology.orchestra.run.smithi133.stdout:Host looks OK 2024-02-20T07:48:05.423 INFO:teuthology.orchestra.run.smithi133.stdout:Cluster fsid: 4537327c-cfc4-11ee-95bb-87774f69a715 2024-02-20T07:48:05.423 INFO:teuthology.orchestra.run.smithi133.stdout:Acquiring lock 140687208643072 on /run/cephadm/4537327c-cfc4-11ee-95bb-87774f69a715.lock 2024-02-20T07:48:05.423 INFO:teuthology.orchestra.run.smithi133.stdout:Lock 140687208643072 acquired on /run/cephadm/4537327c-cfc4-11ee-95bb-87774f69a715.lock 2024-02-20T07:48:05.423 INFO:teuthology.orchestra.run.smithi133.stdout:Verifying IP 172.21.15.133 port 3300 ... 2024-02-20T07:48:05.424 INFO:teuthology.orchestra.run.smithi133.stdout:Verifying IP 172.21.15.133 port 6789 ... 2024-02-20T07:48:05.424 INFO:teuthology.orchestra.run.smithi133.stdout:Base mon IP(s) is [172.21.15.133:3300, 172.21.15.133:6789], mon addrv is [v2:172.21.15.133:3300,v1:172.21.15.133:6789] 2024-02-20T07:48:05.427 INFO:teuthology.orchestra.run.smithi133.stdout:/sbin/ip: stdout default via 172.21.15.254 dev enp3s0f1 proto dhcp src 172.21.15.133 metric 100 2024-02-20T07:48:05.427 INFO:teuthology.orchestra.run.smithi133.stdout:/sbin/ip: stdout 172.21.0.0/20 dev enp3s0f1 proto kernel scope link src 172.21.15.133 metric 100 2024-02-20T07:48:05.430 INFO:teuthology.orchestra.run.smithi133.stdout:/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2024-02-20T07:48:05.430 INFO:teuthology.orchestra.run.smithi133.stdout:/sbin/ip: stdout fe80::/64 dev enp3s0f1 proto kernel metric 256 pref medium 2024-02-20T07:48:05.430 INFO:teuthology.orchestra.run.smithi133.stdout:/sbin/ip: stdout default via fe80::327c:5e00:6487:71e0 dev enp3s0f1 proto ra metric 1024 expires 1754sec hoplimit 64 pref medium 2024-02-20T07:48:05.433 INFO:teuthology.orchestra.run.smithi133.stdout:/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-02-20T07:48:05.433 INFO:teuthology.orchestra.run.smithi133.stdout:/sbin/ip: stdout inet6 ::1/128 scope host 2024-02-20T07:48:05.433 INFO:teuthology.orchestra.run.smithi133.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-02-20T07:48:05.433 INFO:teuthology.orchestra.run.smithi133.stdout:/sbin/ip: stdout 5: enp3s0f1: mtu 1500 state UP qlen 1000 2024-02-20T07:48:05.433 INFO:teuthology.orchestra.run.smithi133.stdout:/sbin/ip: stdout inet6 fe80::ec4:7aff:fe88:8087/64 scope link 2024-02-20T07:48:05.433 INFO:teuthology.orchestra.run.smithi133.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-02-20T07:48:05.434 INFO:teuthology.orchestra.run.smithi133.stdout:Mon IP `172.21.15.133` is in CIDR network `172.21.0.0/20` 2024-02-20T07:48:05.434 INFO:teuthology.orchestra.run.smithi133.stdout:Mon IP `172.21.15.133` is in CIDR network `172.21.0.0/20` 2024-02-20T07:48:05.434 INFO:teuthology.orchestra.run.smithi133.stdout:Inferred mon public CIDR from local network configuration ['172.21.0.0/20', '172.21.0.0/20'] 2024-02-20T07:48:05.434 INFO:teuthology.orchestra.run.smithi133.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-02-20T07:48:05.435 INFO:teuthology.orchestra.run.smithi133.stdout:Pulling container image quay.ceph.io/ceph-ci/ceph:quincy... 2024-02-20T07:48:05.925 INFO:teuthology.orchestra.run.smithi133.stdout:/bin/podman: stdout 10b0c032c913004d1f4f6c0f2496f5bfddfde4aeddb489f4129f49e49277ea3e 2024-02-20T07:48:05.926 INFO:teuthology.orchestra.run.smithi133.stdout:/bin/podman: stderr Trying to pull quay.ceph.io/ceph-ci/ceph:quincy... 2024-02-20T07:48:05.926 INFO:teuthology.orchestra.run.smithi133.stdout:/bin/podman: stderr Getting image source signatures 2024-02-20T07:48:05.926 INFO:teuthology.orchestra.run.smithi133.stdout:/bin/podman: stderr Copying blob sha256:f8275265412e57f06c462f3ab36d3ba9cb1e55305ee9294fe928d41e45938854 2024-02-20T07:48:05.926 INFO:teuthology.orchestra.run.smithi133.stdout:/bin/podman: stderr Copying blob sha256:0f642e61a11afb42c0ccb5b05f5809250ad643ed143780d6ff49a5a4c492fc38 2024-02-20T07:48:05.926 INFO:teuthology.orchestra.run.smithi133.stdout:/bin/podman: stderr Copying config sha256:10b0c032c913004d1f4f6c0f2496f5bfddfde4aeddb489f4129f49e49277ea3e 2024-02-20T07:48:05.926 INFO:teuthology.orchestra.run.smithi133.stdout:/bin/podman: stderr Writing manifest to image destination 2024-02-20T07:48:06.813 INFO:teuthology.orchestra.run.smithi133.stdout:ceph: stdout ceph version 17.2.7-521-gc80e7168 (c80e7168938c48b49e5ce2e002f48bf150b5f4a4) quincy (stable) 2024-02-20T07:48:06.813 INFO:teuthology.orchestra.run.smithi133.stdout:Ceph version: ceph version 17.2.7-521-gc80e7168 (c80e7168938c48b49e5ce2e002f48bf150b5f4a4) quincy (stable) 2024-02-20T07:48:06.813 INFO:teuthology.orchestra.run.smithi133.stdout:Extracting ceph user uid/gid from container image... 2024-02-20T07:48:07.760 INFO:teuthology.orchestra.run.smithi133.stdout:stat: stdout 167 167 2024-02-20T07:48:07.760 INFO:teuthology.orchestra.run.smithi133.stdout:Creating initial keys... 2024-02-20T07:48:08.624 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph-authtool: stdout AQA4WdRlJSZtAhAABA5pElcJhzbKSVew9cQ5vA== 2024-02-20T07:48:09.479 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph-authtool: stdout AQA4WdRlLL1nORAAa9jIWRr6yPTWw3f9h8HAsg== 2024-02-20T07:48:10.410 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph-authtool: stdout AQA5WdRl/WnIMRAAtpFUqItmfYyhgh2na0cOxQ== 2024-02-20T07:48:10.411 INFO:teuthology.orchestra.run.smithi133.stdout:Creating initial monmap... 2024-02-20T07:48:11.249 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2024-02-20T07:48:11.249 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = octopus 2024-02-20T07:48:11.250 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to 4537327c-cfc4-11ee-95bb-87774f69a715 2024-02-20T07:48:11.250 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-02-20T07:48:11.250 INFO:teuthology.orchestra.run.smithi133.stdout:monmaptool for smithi133 [v2:172.21.15.133:3300,v1:172.21.15.133:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-02-20T07:48:11.250 INFO:teuthology.orchestra.run.smithi133.stdout:setting min_mon_release = octopus 2024-02-20T07:48:11.250 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/monmaptool: set fsid to 4537327c-cfc4-11ee-95bb-87774f69a715 2024-02-20T07:48:11.250 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-02-20T07:48:11.250 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-02-20T07:48:11.250 INFO:teuthology.orchestra.run.smithi133.stdout:Creating mon... 2024-02-20T07:48:12.171 INFO:teuthology.orchestra.run.smithi133.stdout:create mon.smithi133 on 2024-02-20T07:48:12.363 INFO:teuthology.orchestra.run.smithi133.stdout:systemctl: stderr Removed "/etc/systemd/system/multi-user.target.wants/ceph.target". 2024-02-20T07:48:12.512 INFO:teuthology.orchestra.run.smithi133.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-02-20T07:48:12.653 INFO:teuthology.orchestra.run.smithi133.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-4537327c-cfc4-11ee-95bb-87774f69a715.target → /etc/systemd/system/ceph-4537327c-cfc4-11ee-95bb-87774f69a715.target. 2024-02-20T07:48:12.654 INFO:teuthology.orchestra.run.smithi133.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-4537327c-cfc4-11ee-95bb-87774f69a715.target → /etc/systemd/system/ceph-4537327c-cfc4-11ee-95bb-87774f69a715.target. 2024-02-20T07:48:12.834 INFO:teuthology.orchestra.run.smithi133.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-4537327c-cfc4-11ee-95bb-87774f69a715@mon.smithi133 2024-02-20T07:48:12.834 INFO:teuthology.orchestra.run.smithi133.stdout:systemctl: stderr Failed to reset failed state of unit ceph-4537327c-cfc4-11ee-95bb-87774f69a715@mon.smithi133.service: Unit ceph-4537327c-cfc4-11ee-95bb-87774f69a715@mon.smithi133.service not loaded. 2024-02-20T07:48:12.993 INFO:teuthology.orchestra.run.smithi133.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-4537327c-cfc4-11ee-95bb-87774f69a715.target.wants/ceph-4537327c-cfc4-11ee-95bb-87774f69a715@mon.smithi133.service → /etc/systemd/system/ceph-4537327c-cfc4-11ee-95bb-87774f69a715@.service. 2024-02-20T07:48:13.604 INFO:teuthology.orchestra.run.smithi133.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-02-20T07:48:13.604 INFO:teuthology.orchestra.run.smithi133.stdout:systemctl: stdout disabled 2024-02-20T07:48:13.612 INFO:teuthology.orchestra.run.smithi133.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-02-20T07:48:13.612 INFO:teuthology.orchestra.run.smithi133.stdout:systemctl: stdout inactive 2024-02-20T07:48:13.612 INFO:teuthology.orchestra.run.smithi133.stdout:firewalld.service is not enabled 2024-02-20T07:48:13.612 INFO:teuthology.orchestra.run.smithi133.stdout:Not possible to enable service . firewalld.service is not available 2024-02-20T07:48:13.613 INFO:teuthology.orchestra.run.smithi133.stdout:Waiting for mon to start... 2024-02-20T07:48:13.613 INFO:teuthology.orchestra.run.smithi133.stdout:Waiting for mon... 2024-02-20T07:48:14.992 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:48:14 smithi133 ceph-mon[28744]: mgrmap e1: no daemons active 2024-02-20T07:48:16.492 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:48:16 smithi133 ceph-mon[28744]: from='client.? 172.21.15.133:0/3522033678' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-02-20T07:48:22.025 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stdout cluster: 2024-02-20T07:48:22.032 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stdout id: 4537327c-cfc4-11ee-95bb-87774f69a715 2024-02-20T07:48:22.032 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2024-02-20T07:48:22.032 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stdout 2024-02-20T07:48:22.032 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stdout services: 2024-02-20T07:48:22.032 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum smithi133 (age 2s) 2024-02-20T07:48:22.032 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2024-02-20T07:48:22.032 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2024-02-20T07:48:22.032 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stdout 2024-02-20T07:48:22.033 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stdout data: 2024-02-20T07:48:22.033 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2024-02-20T07:48:22.033 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2024-02-20T07:48:22.033 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2024-02-20T07:48:22.033 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stdout pgs: 2024-02-20T07:48:22.033 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stdout 2024-02-20T07:48:22.033 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:16.216+0000 7f09ba872700 1 Processor -- start 2024-02-20T07:48:22.033 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:16.216+0000 7f09ba872700 1 -- start start 2024-02-20T07:48:22.033 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:16.216+0000 7f09ba872700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f09b4100b80 0x7f09b4100fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:48:22.033 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:16.216+0000 7f09ba872700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09b41014f0 con 0x7f09b4100b80 2024-02-20T07:48:22.033 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:16.217+0000 7f09b3fff700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f09b4100b80 0x7f09b4100fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:48:22.033 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:16.217+0000 7f09b3fff700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f09b4100b80 0x7f09b4100fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.133:42262/0 (socket says 172.21.15.133:42262) 2024-02-20T07:48:22.033 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:16.217+0000 7f09b3fff700 1 -- 172.21.15.133:0/2560529565 learned_addr learned my addr 172.21.15.133:0/2560529565 (peer_addr_for_me v2:172.21.15.133:0/0) 2024-02-20T07:48:22.033 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:16.217+0000 7f09b3fff700 1 -- 172.21.15.133:0/2560529565 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09b4101630 con 0x7f09b4100b80 2024-02-20T07:48:22.033 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:16.217+0000 7f09b3fff700 1 --2- 172.21.15.133:0/2560529565 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f09b4100b80 0x7f09b4100fb0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f09a40098e0 tx=0x7f09a4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5c1b1dbc800f7310 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:48:22.034 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:16.217+0000 7f09b2ffd700 1 -- 172.21.15.133:0/2560529565 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f09a400b860 con 0x7f09b4100b80 2024-02-20T07:48:22.034 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:16.217+0000 7f09b2ffd700 1 -- 172.21.15.133:0/2560529565 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f09a400b9c0 con 0x7f09b4100b80 2024-02-20T07:48:22.034 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:16.217+0000 7f09b2ffd700 1 -- 172.21.15.133:0/2560529565 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f09a400bc90 con 0x7f09b4100b80 2024-02-20T07:48:22.034 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:16.218+0000 7f09ba872700 1 -- 172.21.15.133:0/2560529565 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f09b4100b80 msgr2=0x7f09b4100fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:48:22.034 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:16.218+0000 7f09ba872700 1 --2- 172.21.15.133:0/2560529565 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f09b4100b80 0x7f09b4100fb0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f09a40098e0 tx=0x7f09a4009250 comp rx=0 tx=0).stop 2024-02-20T07:48:22.034 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:16.218+0000 7f09ba872700 1 -- 172.21.15.133:0/2560529565 shutdown_connections 2024-02-20T07:48:22.034 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:16.218+0000 7f09ba872700 1 --2- 172.21.15.133:0/2560529565 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f09b4100b80 0x7f09b4100fb0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:48:22.034 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:16.218+0000 7f09ba872700 1 -- 172.21.15.133:0/2560529565 >> 172.21.15.133:0/2560529565 conn(0x7f09b40fc580 msgr2=0x7f09b40fe990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:48:22.034 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:16.218+0000 7f09ba872700 1 -- 172.21.15.133:0/2560529565 shutdown_connections 2024-02-20T07:48:22.034 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:16.218+0000 7f09ba872700 1 -- 172.21.15.133:0/2560529565 wait complete. 2024-02-20T07:48:22.034 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:16.219+0000 7f09ba872700 1 Processor -- start 2024-02-20T07:48:22.034 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:16.219+0000 7f09ba872700 1 -- start start 2024-02-20T07:48:22.034 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:16.219+0000 7f09ba872700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f09b4100b80 0x7f09b4111080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:48:22.034 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:16.219+0000 7f09ba872700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09b41115c0 con 0x7f09b4100b80 2024-02-20T07:48:22.034 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:16.219+0000 7f09b3fff700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f09b4100b80 0x7f09b4111080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:48:22.035 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:16.219+0000 7f09b3fff700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f09b4100b80 0x7f09b4111080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.133:42270/0 (socket says 172.21.15.133:42270) 2024-02-20T07:48:22.035 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:16.219+0000 7f09b3fff700 1 -- 172.21.15.133:0/3522033678 learned_addr learned my addr 172.21.15.133:0/3522033678 (peer_addr_for_me v2:172.21.15.133:0/0) 2024-02-20T07:48:22.035 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:16.219+0000 7f09b3fff700 1 -- 172.21.15.133:0/3522033678 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09a40094f0 con 0x7f09b4100b80 2024-02-20T07:48:22.035 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:16.220+0000 7f09b3fff700 1 --2- 172.21.15.133:0/3522033678 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f09b4100b80 0x7f09b4111080 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f09a4009290 tx=0x7f09a4016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:48:22.035 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:16.220+0000 7f09b0ff9700 1 -- 172.21.15.133:0/3522033678 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f09a401c5c0 con 0x7f09b4100b80 2024-02-20T07:48:22.035 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:16.220+0000 7f09b0ff9700 1 -- 172.21.15.133:0/3522033678 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f09a401c740 con 0x7f09b4100b80 2024-02-20T07:48:22.035 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:16.220+0000 7f09ba872700 1 -- 172.21.15.133:0/3522033678 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f09b41117c0 con 0x7f09b4100b80 2024-02-20T07:48:22.035 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:16.220+0000 7f09b0ff9700 1 -- 172.21.15.133:0/3522033678 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f09a401ca50 con 0x7f09b4100b80 2024-02-20T07:48:22.035 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:16.220+0000 7f09ba872700 1 -- 172.21.15.133:0/3522033678 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f09b4112460 con 0x7f09b4100b80 2024-02-20T07:48:22.035 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:16.220+0000 7f09b0ff9700 1 -- 172.21.15.133:0/3522033678 <== mon.0 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7f09a401a450 con 0x7f09b4100b80 2024-02-20T07:48:22.035 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:16.220+0000 7f09b0ff9700 1 -- 172.21.15.133:0/3522033678 <== mon.0 v2:172.21.15.133:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7f09a402e790 con 0x7f09b4100b80 2024-02-20T07:48:22.035 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:16.221+0000 7f09ba872700 1 -- 172.21.15.133:0/3522033678 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f09b405f5c0 con 0x7f09b4100b80 2024-02-20T07:48:22.035 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:16.223+0000 7f09b0ff9700 1 -- 172.21.15.133:0/3522033678 <== mon.0 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71349 (secure 0 0 0) 0x7f09a401ce30 con 0x7f09b4100b80 2024-02-20T07:48:22.035 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:16.293+0000 7f09ba872700 1 -- 172.21.15.133:0/3522033678 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "status"} v 0) v1 -- 0x7f09b4064ad0 con 0x7f09b4100b80 2024-02-20T07:48:22.036 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:16.293+0000 7f09b0ff9700 1 -- 172.21.15.133:0/3522033678 <== mon.0 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "status"}]=0 v0) v1 ==== 54+0+318 (secure 0 0 0) 0x7f09a401a840 con 0x7f09b4100b80 2024-02-20T07:48:22.036 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:16.294+0000 7f09ba872700 1 -- 172.21.15.133:0/3522033678 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f09b4100b80 msgr2=0x7f09b4111080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:48:22.036 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:16.294+0000 7f09ba872700 1 --2- 172.21.15.133:0/3522033678 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f09b4100b80 0x7f09b4111080 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f09a4009290 tx=0x7f09a4016000 comp rx=0 tx=0).stop 2024-02-20T07:48:22.036 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:16.294+0000 7f09ba872700 1 -- 172.21.15.133:0/3522033678 shutdown_connections 2024-02-20T07:48:22.036 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:16.294+0000 7f09ba872700 1 --2- 172.21.15.133:0/3522033678 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f09b4100b80 0x7f09b4111080 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:48:22.036 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:16.294+0000 7f09ba872700 1 -- 172.21.15.133:0/3522033678 >> 172.21.15.133:0/3522033678 conn(0x7f09b40fc580 msgr2=0x7f09b40fdfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:48:22.036 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:16.294+0000 7f09ba872700 1 -- 172.21.15.133:0/3522033678 shutdown_connections 2024-02-20T07:48:22.036 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:16.295+0000 7f09ba872700 1 -- 172.21.15.133:0/3522033678 wait complete. 2024-02-20T07:48:22.036 INFO:teuthology.orchestra.run.smithi133.stdout:mon is available 2024-02-20T07:48:22.036 INFO:teuthology.orchestra.run.smithi133.stdout:Assimilating anything we can from ceph.conf... 2024-02-20T07:48:23.492 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:48:23 smithi133 ceph-mon[28744]: from='client.? 172.21.15.133:0/3755556848' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-02-20T07:48:23.493 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:48:23 smithi133 ceph-mon[28744]: from='client.? 172.21.15.133:0/3755556848' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-02-20T07:48:23.890 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stdout 2024-02-20T07:48:23.890 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stdout [global] 2024-02-20T07:48:23.890 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stdout fsid = 4537327c-cfc4-11ee-95bb-87774f69a715 2024-02-20T07:48:23.890 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.133:3300,v1:172.21.15.133:6789] 2024-02-20T07:48:23.890 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-02-20T07:48:23.890 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-02-20T07:48:23.890 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-02-20T07:48:23.890 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-02-20T07:48:23.890 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stdout 2024-02-20T07:48:23.890 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stdout [mgr] 2024-02-20T07:48:23.890 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-02-20T07:48:23.890 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stdout 2024-02-20T07:48:23.890 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stdout [osd] 2024-02-20T07:48:23.891 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-02-20T07:48:23.891 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-02-20T07:48:23.891 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-02-20T07:48:23.891 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-02-20T07:48:23.891 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-02-20T07:48:23.891 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:23.218+0000 7f1e335b7700 1 Processor -- start 2024-02-20T07:48:23.891 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:23.218+0000 7f1e335b7700 1 -- start start 2024-02-20T07:48:23.891 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:23.218+0000 7f1e335b7700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f1e2c101670 0x7f1e2c101a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:48:23.891 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:23.218+0000 7f1e335b7700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e2c101fc0 con 0x7f1e2c101670 2024-02-20T07:48:23.891 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:23.218+0000 7f1e31353700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f1e2c101670 0x7f1e2c101a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:48:23.891 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:23.218+0000 7f1e31353700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f1e2c101670 0x7f1e2c101a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.133:56112/0 (socket says 172.21.15.133:56112) 2024-02-20T07:48:23.891 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:23.218+0000 7f1e31353700 1 -- 172.21.15.133:0/2179270546 learned_addr learned my addr 172.21.15.133:0/2179270546 (peer_addr_for_me v2:172.21.15.133:0/0) 2024-02-20T07:48:23.891 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:23.219+0000 7f1e31353700 1 -- 172.21.15.133:0/2179270546 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e2c102100 con 0x7f1e2c101670 2024-02-20T07:48:23.891 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:23.219+0000 7f1e31353700 1 --2- 172.21.15.133:0/2179270546 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f1e2c101670 0x7f1e2c101a80 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f1e280098e0 tx=0x7f1e28009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e96a419ad84cc7b8 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:48:23.892 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:23.219+0000 7f1e23fff700 1 -- 172.21.15.133:0/2179270546 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1e2800b860 con 0x7f1e2c101670 2024-02-20T07:48:23.892 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:23.219+0000 7f1e23fff700 1 -- 172.21.15.133:0/2179270546 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e2800b9c0 con 0x7f1e2c101670 2024-02-20T07:48:23.892 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:23.219+0000 7f1e23fff700 1 -- 172.21.15.133:0/2179270546 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1e2800bc90 con 0x7f1e2c101670 2024-02-20T07:48:23.892 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:23.219+0000 7f1e335b7700 1 -- 172.21.15.133:0/2179270546 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f1e2c101670 msgr2=0x7f1e2c101a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:48:23.892 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:23.220+0000 7f1e335b7700 1 --2- 172.21.15.133:0/2179270546 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f1e2c101670 0x7f1e2c101a80 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f1e280098e0 tx=0x7f1e28009250 comp rx=0 tx=0).stop 2024-02-20T07:48:23.892 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:23.220+0000 7f1e335b7700 1 -- 172.21.15.133:0/2179270546 shutdown_connections 2024-02-20T07:48:23.892 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:23.220+0000 7f1e335b7700 1 --2- 172.21.15.133:0/2179270546 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f1e2c101670 0x7f1e2c101a80 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:48:23.892 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:23.220+0000 7f1e335b7700 1 -- 172.21.15.133:0/2179270546 >> 172.21.15.133:0/2179270546 conn(0x7f1e2c0fcdd0 msgr2=0x7f1e2c0ff200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:48:23.892 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:23.220+0000 7f1e335b7700 1 -- 172.21.15.133:0/2179270546 shutdown_connections 2024-02-20T07:48:23.892 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:23.220+0000 7f1e335b7700 1 -- 172.21.15.133:0/2179270546 wait complete. 2024-02-20T07:48:23.892 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:23.220+0000 7f1e335b7700 1 Processor -- start 2024-02-20T07:48:23.892 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:23.220+0000 7f1e335b7700 1 -- start start 2024-02-20T07:48:23.892 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:23.220+0000 7f1e335b7700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f1e2c101670 0x7f1e2c113d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:48:23.892 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:23.220+0000 7f1e335b7700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e2c076bf0 con 0x7f1e2c101670 2024-02-20T07:48:23.892 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:23.220+0000 7f1e31353700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f1e2c101670 0x7f1e2c113d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:48:23.893 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:23.220+0000 7f1e31353700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f1e2c101670 0x7f1e2c113d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.133:56128/0 (socket says 172.21.15.133:56128) 2024-02-20T07:48:23.893 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:23.220+0000 7f1e31353700 1 -- 172.21.15.133:0/3755556848 learned_addr learned my addr 172.21.15.133:0/3755556848 (peer_addr_for_me v2:172.21.15.133:0/0) 2024-02-20T07:48:23.893 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:23.221+0000 7f1e31353700 1 -- 172.21.15.133:0/3755556848 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e280094f0 con 0x7f1e2c101670 2024-02-20T07:48:23.893 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:23.221+0000 7f1e31353700 1 --2- 172.21.15.133:0/3755556848 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f1e2c101670 0x7f1e2c113d90 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f1e28003080 tx=0x7f1e28016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:48:23.893 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:23.221+0000 7f1e21ffb700 1 -- 172.21.15.133:0/3755556848 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1e2801c5c0 con 0x7f1e2c101670 2024-02-20T07:48:23.893 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:23.221+0000 7f1e21ffb700 1 -- 172.21.15.133:0/3755556848 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e2801c720 con 0x7f1e2c101670 2024-02-20T07:48:23.893 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:23.221+0000 7f1e335b7700 1 -- 172.21.15.133:0/3755556848 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1e2c076df0 con 0x7f1e2c101670 2024-02-20T07:48:23.893 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:23.221+0000 7f1e21ffb700 1 -- 172.21.15.133:0/3755556848 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1e2801ca30 con 0x7f1e2c101670 2024-02-20T07:48:23.893 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:23.221+0000 7f1e335b7700 1 -- 172.21.15.133:0/3755556848 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1e2c114490 con 0x7f1e2c101670 2024-02-20T07:48:23.893 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:23.221+0000 7f1e21ffb700 1 -- 172.21.15.133:0/3755556848 <== mon.0 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7f1e2801a450 con 0x7f1e2c101670 2024-02-20T07:48:23.893 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:23.221+0000 7f1e21ffb700 1 -- 172.21.15.133:0/3755556848 <== mon.0 v2:172.21.15.133:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7f1e2802e790 con 0x7f1e2c101670 2024-02-20T07:48:23.893 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:23.222+0000 7f1e335b7700 1 -- 172.21.15.133:0/3755556848 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1e2c109ba0 con 0x7f1e2c101670 2024-02-20T07:48:23.893 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:23.224+0000 7f1e21ffb700 1 -- 172.21.15.133:0/3755556848 <== mon.0 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71349 (secure 0 0 0) 0x7f1e2801a890 con 0x7f1e2c101670 2024-02-20T07:48:23.893 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:23.290+0000 7f1e335b7700 1 -- 172.21.15.133:0/3755556848 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) v1 -- 0x7f1e2c064b00 con 0x7f1e2c101670 2024-02-20T07:48:23.894 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:23.294+0000 7f1e21ffb700 1 -- 172.21.15.133:0/3755556848 <== mon.0 v2:172.21.15.133:3300/0 7 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f1e2802ee80 con 0x7f1e2c101670 2024-02-20T07:48:23.894 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:23.294+0000 7f1e21ffb700 1 -- 172.21.15.133:0/3755556848 <== mon.0 v2:172.21.15.133:3300/0 8 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v2) v1 ==== 70+0+431 (secure 0 0 0) 0x7f1e28037590 con 0x7f1e2c101670 2024-02-20T07:48:23.894 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:23.295+0000 7f1e335b7700 1 -- 172.21.15.133:0/3755556848 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f1e2c101670 msgr2=0x7f1e2c113d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:48:23.894 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:23.295+0000 7f1e335b7700 1 --2- 172.21.15.133:0/3755556848 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f1e2c101670 0x7f1e2c113d90 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f1e28003080 tx=0x7f1e28016000 comp rx=0 tx=0).stop 2024-02-20T07:48:23.894 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:23.295+0000 7f1e335b7700 1 -- 172.21.15.133:0/3755556848 shutdown_connections 2024-02-20T07:48:23.894 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:23.295+0000 7f1e335b7700 1 --2- 172.21.15.133:0/3755556848 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f1e2c101670 0x7f1e2c113d90 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:48:23.894 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:23.295+0000 7f1e335b7700 1 -- 172.21.15.133:0/3755556848 >> 172.21.15.133:0/3755556848 conn(0x7f1e2c0fcdd0 msgr2=0x7f1e2c106d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:48:23.894 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:23.295+0000 7f1e335b7700 1 -- 172.21.15.133:0/3755556848 shutdown_connections 2024-02-20T07:48:23.894 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:23.295+0000 7f1e335b7700 1 -- 172.21.15.133:0/3755556848 wait complete. 2024-02-20T07:48:23.894 INFO:teuthology.orchestra.run.smithi133.stdout:Generating new minimal ceph.conf... 2024-02-20T07:48:24.742 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:48:24 smithi133 ceph-mon[28744]: from='client.? 172.21.15.133:0/125964559' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-20T07:48:25.020 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:24.317+0000 7f4fd1d54700 1 Processor -- start 2024-02-20T07:48:25.020 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:24.317+0000 7f4fd1d54700 1 -- start start 2024-02-20T07:48:25.020 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:24.317+0000 7f4fd1d54700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f4fcc102db0 0x7f4fcc1031c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:48:25.020 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:24.317+0000 7f4fd1d54700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4fcc103700 con 0x7f4fcc102db0 2024-02-20T07:48:25.020 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:24.317+0000 7f4fcb7fe700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f4fcc102db0 0x7f4fcc1031c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:48:25.020 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:24.317+0000 7f4fcb7fe700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f4fcc102db0 0x7f4fcc1031c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.133:56130/0 (socket says 172.21.15.133:56130) 2024-02-20T07:48:25.020 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:24.317+0000 7f4fcb7fe700 1 -- 172.21.15.133:0/330817626 learned_addr learned my addr 172.21.15.133:0/330817626 (peer_addr_for_me v2:172.21.15.133:0/0) 2024-02-20T07:48:25.020 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:24.317+0000 7f4fcb7fe700 1 -- 172.21.15.133:0/330817626 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4fcc103840 con 0x7f4fcc102db0 2024-02-20T07:48:25.020 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:24.317+0000 7f4fcb7fe700 1 --2- 172.21.15.133:0/330817626 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f4fcc102db0 0x7f4fcc1031c0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f4fbc0098e0 tx=0x7f4fbc009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=663c6069f2cbcdc1 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:48:25.020 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:24.318+0000 7f4fca7fc700 1 -- 172.21.15.133:0/330817626 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4fbc00b860 con 0x7f4fcc102db0 2024-02-20T07:48:25.020 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:24.318+0000 7f4fca7fc700 1 -- 172.21.15.133:0/330817626 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f4fbc00be50 con 0x7f4fcc102db0 2024-02-20T07:48:25.020 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:24.318+0000 7f4fca7fc700 1 -- 172.21.15.133:0/330817626 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4fbc010380 con 0x7f4fcc102db0 2024-02-20T07:48:25.020 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:24.318+0000 7f4fd1d54700 1 -- 172.21.15.133:0/330817626 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f4fcc102db0 msgr2=0x7f4fcc1031c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:48:25.021 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:24.318+0000 7f4fd1d54700 1 --2- 172.21.15.133:0/330817626 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f4fcc102db0 0x7f4fcc1031c0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f4fbc0098e0 tx=0x7f4fbc009250 comp rx=0 tx=0).stop 2024-02-20T07:48:25.021 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:24.318+0000 7f4fd1d54700 1 -- 172.21.15.133:0/330817626 shutdown_connections 2024-02-20T07:48:25.021 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:24.318+0000 7f4fd1d54700 1 --2- 172.21.15.133:0/330817626 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f4fcc102db0 0x7f4fcc1031c0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:48:25.021 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:24.318+0000 7f4fd1d54700 1 -- 172.21.15.133:0/330817626 >> 172.21.15.133:0/330817626 conn(0x7f4fcc0fc580 msgr2=0x7f4fcc0fe990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:48:25.021 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:24.318+0000 7f4fd1d54700 1 -- 172.21.15.133:0/330817626 shutdown_connections 2024-02-20T07:48:25.021 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:24.318+0000 7f4fd1d54700 1 -- 172.21.15.133:0/330817626 wait complete. 2024-02-20T07:48:25.021 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:24.319+0000 7f4fd1d54700 1 Processor -- start 2024-02-20T07:48:25.021 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:24.319+0000 7f4fd1d54700 1 -- start start 2024-02-20T07:48:25.021 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:24.319+0000 7f4fd1d54700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f4fcc102db0 0x7f4fcc113a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:48:25.021 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:24.319+0000 7f4fd1d54700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4fcc113f80 con 0x7f4fcc102db0 2024-02-20T07:48:25.021 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:24.319+0000 7f4fcb7fe700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f4fcc102db0 0x7f4fcc113a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:48:25.021 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:24.319+0000 7f4fcb7fe700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f4fcc102db0 0x7f4fcc113a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.133:56146/0 (socket says 172.21.15.133:56146) 2024-02-20T07:48:25.021 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:24.319+0000 7f4fcb7fe700 1 -- 172.21.15.133:0/125964559 learned_addr learned my addr 172.21.15.133:0/125964559 (peer_addr_for_me v2:172.21.15.133:0/0) 2024-02-20T07:48:25.021 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:24.319+0000 7f4fcb7fe700 1 -- 172.21.15.133:0/125964559 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4fbc0094f0 con 0x7f4fcc102db0 2024-02-20T07:48:25.022 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:24.320+0000 7f4fcb7fe700 1 --2- 172.21.15.133:0/125964559 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f4fcc102db0 0x7f4fcc113a40 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f4fbc00b440 tx=0x7f4fbc009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:48:25.022 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:24.320+0000 7f4fb3fff700 1 -- 172.21.15.133:0/125964559 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4fbc0107c0 con 0x7f4fcc102db0 2024-02-20T07:48:25.022 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:24.320+0000 7f4fb3fff700 1 -- 172.21.15.133:0/125964559 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f4fbc010db0 con 0x7f4fcc102db0 2024-02-20T07:48:25.022 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:24.320+0000 7f4fb3fff700 1 -- 172.21.15.133:0/125964559 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4fbc01ada0 con 0x7f4fcc102db0 2024-02-20T07:48:25.022 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:24.320+0000 7f4fd1d54700 1 -- 172.21.15.133:0/125964559 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4fcc114180 con 0x7f4fcc102db0 2024-02-20T07:48:25.022 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:24.320+0000 7f4fd1d54700 1 -- 172.21.15.133:0/125964559 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4fcc114e20 con 0x7f4fcc102db0 2024-02-20T07:48:25.022 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:24.320+0000 7f4fb3fff700 1 -- 172.21.15.133:0/125964559 <== mon.0 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7f4fbc02e6c0 con 0x7f4fcc102db0 2024-02-20T07:48:25.022 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:24.320+0000 7f4fb3fff700 1 -- 172.21.15.133:0/125964559 <== mon.0 v2:172.21.15.133:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7f4fbc024950 con 0x7f4fcc102db0 2024-02-20T07:48:25.022 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:24.321+0000 7f4fd1d54700 1 -- 172.21.15.133:0/125964559 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4fcc05f5c0 con 0x7f4fcc102db0 2024-02-20T07:48:25.022 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:24.323+0000 7f4fb3fff700 1 -- 172.21.15.133:0/125964559 <== mon.0 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71349 (secure 0 0 0) 0x7f4fbc024bb0 con 0x7f4fcc102db0 2024-02-20T07:48:25.022 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:24.389+0000 7f4fd1d54700 1 -- 172.21.15.133:0/125964559 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) v1 -- 0x7f4fcc064ad0 con 0x7f4fcc102db0 2024-02-20T07:48:25.022 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:24.389+0000 7f4fb3fff700 1 -- 172.21.15.133:0/125964559 <== mon.0 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v2) v1 ==== 76+0+177 (secure 0 0 0) 0x7f4fbc024d50 con 0x7f4fcc102db0 2024-02-20T07:48:25.022 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:24.390+0000 7f4fd1d54700 1 -- 172.21.15.133:0/125964559 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f4fcc102db0 msgr2=0x7f4fcc113a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:48:25.022 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:24.390+0000 7f4fd1d54700 1 --2- 172.21.15.133:0/125964559 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f4fcc102db0 0x7f4fcc113a40 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f4fbc00b440 tx=0x7f4fbc009250 comp rx=0 tx=0).stop 2024-02-20T07:48:25.022 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:24.390+0000 7f4fd1d54700 1 -- 172.21.15.133:0/125964559 shutdown_connections 2024-02-20T07:48:25.023 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:24.390+0000 7f4fd1d54700 1 --2- 172.21.15.133:0/125964559 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f4fcc102db0 0x7f4fcc113a40 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:48:25.023 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:24.390+0000 7f4fd1d54700 1 -- 172.21.15.133:0/125964559 >> 172.21.15.133:0/125964559 conn(0x7f4fcc0fc580 msgr2=0x7f4fcc0fd4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:48:25.023 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:24.390+0000 7f4fd1d54700 1 -- 172.21.15.133:0/125964559 shutdown_connections 2024-02-20T07:48:25.023 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:24.390+0000 7f4fd1d54700 1 -- 172.21.15.133:0/125964559 wait complete. 2024-02-20T07:48:25.023 INFO:teuthology.orchestra.run.smithi133.stdout:Restarting the monitor... 2024-02-20T07:48:25.243 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:48:25 smithi133 systemd[1]: Stopping Ceph mon.smithi133 for 4537327c-cfc4-11ee-95bb-87774f69a715... 2024-02-20T07:48:25.692 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:48:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[28740]: 2024-02-20T07:48:25.246+0000 7f755dc82700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi133 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-02-20T07:48:25.692 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:48:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[28740]: 2024-02-20T07:48:25.246+0000 7f755dc82700 -1 mon.smithi133@0(leader) e1 *** Got Signal Terminated *** 2024-02-20T07:48:25.692 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:48:25 smithi133 podman[28991]: 2024-02-20 07:48:25.411159627 +0000 UTC m=+0.234685610 container died 04b1646e540534eaa7c4e2147a6528c1d4c5ac7ea059d4d4e5f6a59235de0ba4 (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.build-date=20240214, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, ceph=True, RELEASE=quincy-c80e716, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, maintainer=Guillaume Abrioux ) 2024-02-20T07:48:25.992 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:48:25 smithi133 podman[28991]: 2024-02-20 07:48:25.797705834 +0000 UTC m=+0.621231862 container cleanup 04b1646e540534eaa7c4e2147a6528c1d4c5ac7ea059d4d4e5f6a59235de0ba4 (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=quincy-c80e716, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.build-date=20240214, io.buildah.version=1.33.5) 2024-02-20T07:48:25.993 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:48:25 smithi133 bash[28991]: ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133 2024-02-20T07:48:26.358 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:48:26 smithi133 podman[29003]: 2024-02-20 07:48:26.023472774 +0000 UTC m=+0.608632263 container remove 04b1646e540534eaa7c4e2147a6528c1d4c5ac7ea059d4d4e5f6a59235de0ba4 (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133, CEPH_POINT_RELEASE=, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240214, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, RELEASE=quincy-c80e716, ceph=True) 2024-02-20T07:48:26.359 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:48:26 smithi133 systemd[1]: ceph-4537327c-cfc4-11ee-95bb-87774f69a715@mon.smithi133.service: Deactivated successfully. 2024-02-20T07:48:26.359 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:48:26 smithi133 systemd[1]: Stopped Ceph mon.smithi133 for 4537327c-cfc4-11ee-95bb-87774f69a715. 2024-02-20T07:48:26.359 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:48:26 smithi133 systemd[1]: Starting Ceph mon.smithi133 for 4537327c-cfc4-11ee-95bb-87774f69a715... 2024-02-20T07:48:26.628 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:48:26 smithi133 podman[29097]: 2024-02-20 07:48:26.357517118 +0000 UTC m=+0.095731606 container create fe7046a20c8704d4c5e4b4c19b43a3423bc6705e3d978aa30e3796944d81371e (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133, ceph=True, org.label-schema.build-date=20240214, RELEASE=quincy-c80e716, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_CLEAN=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_BRANCH=HEAD) 2024-02-20T07:48:26.628 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:48:26 smithi133 podman[29097]: 2024-02-20 07:48:26.280123479 +0000 UTC m=+0.018337976 image pull 10b0c032c913004d1f4f6c0f2496f5bfddfde4aeddb489f4129f49e49277ea3e quay.ceph.io/ceph-ci/ceph:quincy 2024-02-20T07:48:26.628 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:48:26 smithi133 podman[29097]: 2024-02-20 07:48:26.543138118 +0000 UTC m=+0.281352606 container init fe7046a20c8704d4c5e4b4c19b43a3423bc6705e3d978aa30e3796944d81371e (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.build-date=20240214, GIT_CLEAN=True, RELEASE=quincy-c80e716, maintainer=Guillaume Abrioux , ceph=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd) 2024-02-20T07:48:26.628 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:48:26 smithi133 podman[29097]: 2024-02-20 07:48:26.546362373 +0000 UTC m=+0.284576863 container start fe7046a20c8704d4c5e4b4c19b43a3423bc6705e3d978aa30e3796944d81371e (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133, GIT_BRANCH=HEAD, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, RELEASE=quincy-c80e716, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240214, org.label-schema.vendor=CentOS, ceph=True) 2024-02-20T07:48:26.628 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:48:26 smithi133 ceph-mon[29117]: set uid:gid to 167:167 (ceph:ceph) 2024-02-20T07:48:26.629 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:48:26 smithi133 ceph-mon[29117]: ceph version 17.2.7-521-gc80e7168 (c80e7168938c48b49e5ce2e002f48bf150b5f4a4) quincy (stable), process ceph-mon, pid 2 2024-02-20T07:48:26.629 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:48:26 smithi133 ceph-mon[29117]: pidfile_write: ignore empty --pid-file 2024-02-20T07:48:26.629 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:48:26 smithi133 ceph-mon[29117]: load: jerasure load: lrc 2024-02-20T07:48:26.629 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:48:26 smithi133 ceph-mon[29117]: rocksdb: RocksDB version: 6.15.5 2024-02-20T07:48:26.629 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:48:26 smithi133 ceph-mon[29117]: rocksdb: Git sha rocksdb_build_git_sha:@0@ 2024-02-20T07:48:26.629 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:48:26 smithi133 ceph-mon[29117]: rocksdb: Compile date Feb 19 2024 2024-02-20T07:48:26.629 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:48:26 smithi133 ceph-mon[29117]: rocksdb: DB SUMMARY 2024-02-20T07:48:26.629 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:48:26 smithi133 ceph-mon[29117]: rocksdb: DB Session ID: CFNEJ5UKSRIO4NBX31SQ 2024-02-20T07:48:26.629 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:48:26 smithi133 ceph-mon[29117]: rocksdb: CURRENT file: CURRENT 2024-02-20T07:48:26.629 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:48:26 smithi133 ceph-mon[29117]: rocksdb: IDENTITY file: IDENTITY 2024-02-20T07:48:26.629 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:48:26 smithi133 ceph-mon[29117]: rocksdb: MANIFEST file: MANIFEST-000009 size: 131 Bytes 2024-02-20T07:48:26.629 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:48:26 smithi133 ceph-mon[29117]: rocksdb: SST files in /var/lib/ceph/mon/ceph-smithi133/store.db dir, Total Num: 1, files: 000008.sst 2024-02-20T07:48:26.629 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:48:26 smithi133 ceph-mon[29117]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-smithi133/store.db: 000010.log size: 98610 ; 2024-02-20T07:48:26.630 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:48:26 smithi133 ceph-mon[29117]: rocksdb: Options.error_if_exists: 0 2024-02-20T07:48:26.630 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:48:26 smithi133 ceph-mon[29117]: rocksdb: Options.create_if_missing: 0 2024-02-20T07:48:26.630 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:48:26 smithi133 ceph-mon[29117]: rocksdb: Options.paranoid_checks: 1 2024-02-20T07:48:26.630 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:48:26 smithi133 ceph-mon[29117]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-02-20T07:48:26.630 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:48:26 smithi133 ceph-mon[29117]: rocksdb: Options.env: 0x5593f410ca80 2024-02-20T07:48:26.630 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:48:26 smithi133 ceph-mon[29117]: rocksdb: Options.fs: Posix File System 2024-02-20T07:48:26.630 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:48:26 smithi133 ceph-mon[29117]: rocksdb: Options.info_log: 0x5593f52efcc0 2024-02-20T07:48:26.630 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:48:26 smithi133 ceph-mon[29117]: rocksdb: Options.max_file_opening_threads: 16 2024-02-20T07:48:26.630 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:48:26 smithi133 ceph-mon[29117]: rocksdb: Options.statistics: (nil) 2024-02-20T07:48:26.630 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:48:26 smithi133 ceph-mon[29117]: rocksdb: Options.use_fsync: 0 2024-02-20T07:48:26.630 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:48:26 smithi133 ceph-mon[29117]: rocksdb: Options.max_log_file_size: 0 2024-02-20T07:48:26.630 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:48:26 smithi133 ceph-mon[29117]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-02-20T07:48:26.630 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:48:26 smithi133 ceph-mon[29117]: rocksdb: Options.log_file_time_to_roll: 0 2024-02-20T07:48:26.630 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:48:26 smithi133 ceph-mon[29117]: rocksdb: Options.keep_log_file_num: 1000 2024-02-20T07:48:26.630 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:48:26 smithi133 ceph-mon[29117]: rocksdb: Options.recycle_log_file_num: 0 2024-02-20T07:48:26.630 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:48:26 smithi133 ceph-mon[29117]: rocksdb: Options.allow_fallocate: 1 2024-02-20T07:48:26.631 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:48:26 smithi133 ceph-mon[29117]: rocksdb: Options.allow_mmap_reads: 0 2024-02-20T07:48:26.631 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:48:26 smithi133 ceph-mon[29117]: rocksdb: Options.allow_mmap_writes: 0 2024-02-20T07:48:26.631 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:48:26 smithi133 ceph-mon[29117]: rocksdb: Options.use_direct_reads: 0 2024-02-20T07:48:26.631 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:48:26 smithi133 ceph-mon[29117]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-02-20T07:48:26.631 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:48:26 smithi133 ceph-mon[29117]: rocksdb: Options.create_missing_column_families: 0 2024-02-20T07:48:26.631 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:48:26 smithi133 ceph-mon[29117]: rocksdb: Options.db_log_dir: 2024-02-20T07:48:26.631 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:48:26 smithi133 ceph-mon[29117]: rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-smithi133/store.db 2024-02-20T07:48:26.631 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:48:26 smithi133 ceph-mon[29117]: rocksdb: Options.table_cache_numshardbits: 6 2024-02-20T07:48:26.631 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:48:26 smithi133 ceph-mon[29117]: rocksdb: Options.WAL_ttl_seconds: 0 2024-02-20T07:48:26.631 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:48:26 smithi133 ceph-mon[29117]: rocksdb: Options.WAL_size_limit_MB: 0 2024-02-20T07:48:26.631 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:48:26 smithi133 ceph-mon[29117]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-02-20T07:48:26.631 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:48:26 smithi133 ceph-mon[29117]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-02-20T07:48:26.631 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:48:26 smithi133 ceph-mon[29117]: rocksdb: Options.is_fd_close_on_exec: 1 2024-02-20T07:48:26.631 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:48:26 smithi133 ceph-mon[29117]: rocksdb: Options.advise_random_on_open: 1 2024-02-20T07:48:26.631 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:48:26 smithi133 ceph-mon[29117]: rocksdb: Options.db_write_buffer_size: 0 2024-02-20T07:48:26.632 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:48:26 smithi133 ceph-mon[29117]: rocksdb: Options.write_buffer_manager: 0x5593f6064ae0 2024-02-20T07:48:26.632 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:48:26 smithi133 ceph-mon[29117]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-02-20T07:48:26.632 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:48:26 smithi133 ceph-mon[29117]: rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2024-02-20T07:48:26.632 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:48:26 smithi133 ceph-mon[29117]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-02-20T07:48:26.632 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:48:26 smithi133 ceph-mon[29117]: rocksdb: Options.use_adaptive_mutex: 0 2024-02-20T07:48:26.632 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:48:26 smithi133 ceph-mon[29117]: rocksdb: Options.rate_limiter: (nil) 2024-02-20T07:48:26.632 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:48:26 smithi133 ceph-mon[29117]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-02-20T07:48:26.632 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:48:26 smithi133 ceph-mon[29117]: rocksdb: Options.wal_recovery_mode: 2 2024-02-20T07:48:26.632 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:48:26 smithi133 ceph-mon[29117]: rocksdb: Options.enable_thread_tracking: 0 2024-02-20T07:48:26.632 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:48:26 smithi133 ceph-mon[29117]: rocksdb: Options.enable_pipelined_write: 0 2024-02-20T07:48:26.634 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:48:26 smithi133 ceph-mon[29117]: rocksdb: Options.unordered_write: 0 2024-02-20T07:48:26.634 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:48:26 smithi133 ceph-mon[29117]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-02-20T07:48:26.634 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:48:26 smithi133 ceph-mon[29117]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-02-20T07:48:26.634 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:48:26 smithi133 ceph-mon[29117]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-02-20T07:48:26.634 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:48:26 smithi133 ceph-mon[29117]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-02-20T07:48:26.634 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:48:26 smithi133 ceph-mon[29117]: rocksdb: Options.row_cache: None 2024-02-20T07:48:26.634 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:48:26 smithi133 ceph-mon[29117]: rocksdb: Options.wal_filter: None 2024-02-20T07:48:26.634 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:48:26 smithi133 ceph-mon[29117]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-02-20T07:48:26.634 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:48:26 smithi133 ceph-mon[29117]: rocksdb: Options.allow_ingest_behind: 0 2024-02-20T07:48:26.634 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:48:26 smithi133 ceph-mon[29117]: rocksdb: Options.preserve_deletes: 0 2024-02-20T07:48:26.634 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:48:26 smithi133 ceph-mon[29117]: rocksdb: Options.two_write_queues: 0 2024-02-20T07:48:26.635 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:48:26 smithi133 ceph-mon[29117]: rocksdb: Options.manual_wal_flush: 0 2024-02-20T07:48:26.635 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:48:26 smithi133 ceph-mon[29117]: rocksdb: Options.atomic_flush: 0 2024-02-20T07:48:26.635 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:48:26 smithi133 ceph-mon[29117]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-02-20T07:48:26.635 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:48:26 smithi133 ceph-mon[29117]: rocksdb: Options.persist_stats_to_disk: 0 2024-02-20T07:48:26.635 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:48:26 smithi133 ceph-mon[29117]: rocksdb: Options.write_dbid_to_manifest: 0 2024-02-20T07:48:26.635 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:48:26 smithi133 ceph-mon[29117]: rocksdb: Options.log_readahead_size: 0 2024-02-20T07:48:26.635 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:48:26 smithi133 ceph-mon[29117]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-02-20T07:48:26.635 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:48:26 smithi133 ceph-mon[29117]: rocksdb: Options.best_efforts_recovery: 0 2024-02-20T07:48:26.635 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:48:26 smithi133 ceph-mon[29117]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-02-20T07:48:26.635 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:48:26 smithi133 ceph-mon[29117]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-02-20T07:48:26.635 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:48:26 smithi133 ceph-mon[29117]: rocksdb: Options.allow_data_in_errors: 0 2024-02-20T07:48:26.635 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:48:26 smithi133 ceph-mon[29117]: rocksdb: Options.db_host_id: __hostname__ 2024-02-20T07:48:26.635 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:48:26 smithi133 ceph-mon[29117]: rocksdb: Options.max_background_jobs: 2 2024-02-20T07:48:26.635 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:48:26 smithi133 ceph-mon[29117]: rocksdb: Options.max_background_compactions: -1 2024-02-20T07:48:26.635 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:48:26 smithi133 ceph-mon[29117]: rocksdb: Options.max_subcompactions: 1 2024-02-20T07:48:26.635 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:48:26 smithi133 ceph-mon[29117]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-02-20T07:48:26.636 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:48:26 smithi133 ceph-mon[29117]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-02-20T07:48:26.636 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:48:26 smithi133 ceph-mon[29117]: rocksdb: Options.delayed_write_rate : 16777216 2024-02-20T07:48:26.636 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:48:26 smithi133 ceph-mon[29117]: rocksdb: Options.max_total_wal_size: 0 2024-02-20T07:48:26.636 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:48:26 smithi133 ceph-mon[29117]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-02-20T07:48:26.636 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:48:26 smithi133 ceph-mon[29117]: rocksdb: Options.stats_dump_period_sec: 600 2024-02-20T07:48:26.636 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:48:26 smithi133 ceph-mon[29117]: rocksdb: Options.stats_persist_period_sec: 600 2024-02-20T07:48:26.636 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:48:26 smithi133 ceph-mon[29117]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-02-20T07:48:26.636 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:48:26 smithi133 ceph-mon[29117]: rocksdb: Options.max_open_files: -1 2024-02-20T07:48:26.636 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:48:26 smithi133 ceph-mon[29117]: rocksdb: Options.bytes_per_sync: 0 2024-02-20T07:48:26.636 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:48:26 smithi133 ceph-mon[29117]: rocksdb: Options.wal_bytes_per_sync: 0 2024-02-20T07:48:26.636 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:48:26 smithi133 ceph-mon[29117]: rocksdb: Options.strict_bytes_per_sync: 0 2024-02-20T07:48:26.636 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:48:26 smithi133 ceph-mon[29117]: rocksdb: Options.compaction_readahead_size: 0 2024-02-20T07:48:26.636 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:48:26 smithi133 ceph-mon[29117]: rocksdb: Options.max_background_flushes: -1 2024-02-20T07:48:26.636 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:48:26 smithi133 ceph-mon[29117]: rocksdb: Compression algorithms supported: 2024-02-20T07:48:26.636 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:48:26 smithi133 ceph-mon[29117]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-02-20T07:48:26.637 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:48:26 smithi133 ceph-mon[29117]: rocksdb: kZSTD supported: 0 2024-02-20T07:48:26.637 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:48:26 smithi133 ceph-mon[29117]: rocksdb: kXpressCompression supported: 0 2024-02-20T07:48:26.637 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:48:26 smithi133 ceph-mon[29117]: rocksdb: kLZ4HCCompression supported: 1 2024-02-20T07:48:26.637 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:48:26 smithi133 ceph-mon[29117]: rocksdb: kZlibCompression supported: 1 2024-02-20T07:48:26.637 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:48:26 smithi133 ceph-mon[29117]: rocksdb: kSnappyCompression supported: 1 2024-02-20T07:48:26.637 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:48:26 smithi133 ceph-mon[29117]: rocksdb: kLZ4Compression supported: 1 2024-02-20T07:48:26.637 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:48:26 smithi133 ceph-mon[29117]: rocksdb: kBZip2Compression supported: 0 2024-02-20T07:48:26.637 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:48:26 smithi133 ceph-mon[29117]: rocksdb: Fast CRC32 supported: Supported on x86 2024-02-20T07:48:26.637 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:48:26 smithi133 ceph-mon[29117]: rocksdb: [db/version_set.cc:4724] Recovering from manifest file: /var/lib/ceph/mon/ceph-smithi133/store.db/MANIFEST-000009 2024-02-20T07:48:26.637 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:48:26 smithi133 ceph-mon[29117]: rocksdb: [db/column_family.cc:595] --------------- Options for column family [default]: 2024-02-20T07:48:26.637 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:48:26 smithi133 ceph-mon[29117]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-02-20T07:48:26.637 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:48:26 smithi133 ceph-mon[29117]: rocksdb: Options.merge_operator: 2024-02-20T07:48:26.637 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:48:26 smithi133 ceph-mon[29117]: rocksdb: Options.compaction_filter: None 2024-02-20T07:48:26.637 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:48:26 smithi133 ceph-mon[29117]: rocksdb: Options.compaction_filter_factory: None 2024-02-20T07:48:26.637 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:48:26 smithi133 ceph-mon[29117]: rocksdb: Options.sst_partitioner_factory: None 2024-02-20T07:48:26.638 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:48:26 smithi133 ceph-mon[29117]: rocksdb: Options.memtable_factory: SkipListFactory 2024-02-20T07:48:26.638 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:48:26 smithi133 ceph-mon[29117]: rocksdb: Options.table_factory: BlockBasedTable 2024-02-20T07:48:26.638 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:48:26 smithi133 ceph-mon[29117]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x5593f5344f58) 2024-02-20T07:48:26.638 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout: cache_index_and_filter_blocks: 1 2024-02-20T07:48:26.638 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-02-20T07:48:26.638 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-02-20T07:48:26.638 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout: pin_top_level_index_and_filter: 1 2024-02-20T07:48:26.638 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout: index_type: 0 2024-02-20T07:48:26.638 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout: data_block_index_type: 0 2024-02-20T07:48:26.638 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout: index_shortening: 1 2024-02-20T07:48:26.638 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout: data_block_hash_table_util_ratio: 0.750000 2024-02-20T07:48:26.638 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout: hash_index_allow_collision: 1 2024-02-20T07:48:26.638 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout: checksum: 1 2024-02-20T07:48:26.638 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout: no_block_cache: 0 2024-02-20T07:48:26.638 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout: block_cache: 0x5593f5377090 2024-02-20T07:48:26.639 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout: block_cache_name: BinnedLRUCache 2024-02-20T07:48:26.639 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout: block_cache_options: 2024-02-20T07:48:26.639 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout: capacity : 536870912 2024-02-20T07:48:26.639 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout: num_shard_bits : 4 2024-02-20T07:48:26.639 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout: strict_capacity_limit : 0 2024-02-20T07:48:26.639 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout: high_pri_pool_ratio: 0.000 2024-02-20T07:48:26.639 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout: block_cache_compressed: (nil) 2024-02-20T07:48:26.639 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout: persistent_cache: (nil) 2024-02-20T07:48:26.639 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout: block_size: 4096 2024-02-20T07:48:26.639 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout: block_size_deviation: 10 2024-02-20T07:48:26.639 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout: block_restart_interval: 16 2024-02-20T07:48:26.639 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout: index_block_restart_interval: 1 2024-02-20T07:48:26.639 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout: metadata_block_size: 4096 2024-02-20T07:48:26.639 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout: partition_filters: 0 2024-02-20T07:48:26.639 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout: use_delta_encoding: 1 2024-02-20T07:48:26.639 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout: filter_policy: rocksdb.BuiltinBloomFilter 2024-02-20T07:48:26.640 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout: whole_key_filtering: 1 2024-02-20T07:48:26.640 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout: verify_compression: 0 2024-02-20T07:48:26.640 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout: read_amp_bytes_per_bit: 0 2024-02-20T07:48:26.640 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout: format_version: 4 2024-02-20T07:48:26.640 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout: enable_index_compression: 1 2024-02-20T07:48:26.640 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout: block_align: 0 2024-02-20T07:48:26.640 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:48:26 smithi133 ceph-mon[29117]: rocksdb: Options.write_buffer_size: 33554432 2024-02-20T07:48:26.640 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:48:26 smithi133 ceph-mon[29117]: rocksdb: Options.max_write_buffer_number: 2 2024-02-20T07:48:26.640 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:48:26 smithi133 ceph-mon[29117]: rocksdb: Options.compression: NoCompression 2024-02-20T07:48:26.640 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:48:26 smithi133 ceph-mon[29117]: rocksdb: Options.bottommost_compression: Disabled 2024-02-20T07:48:26.640 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:48:26 smithi133 ceph-mon[29117]: rocksdb: Options.prefix_extractor: nullptr 2024-02-20T07:48:26.640 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:48:26 smithi133 ceph-mon[29117]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-02-20T07:48:26.640 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:48:26 smithi133 ceph-mon[29117]: rocksdb: Options.num_levels: 7 2024-02-20T07:48:26.641 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:48:26 smithi133 ceph-mon[29117]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-02-20T07:48:26.641 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:48:26 smithi133 ceph-mon[29117]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-02-20T07:48:26.641 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:48:26 smithi133 ceph-mon[29117]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-02-20T07:48:26.641 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:48:26 smithi133 ceph-mon[29117]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-02-20T07:48:26.641 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:48:26 smithi133 ceph-mon[29117]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-02-20T07:48:26.641 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:48:26 smithi133 ceph-mon[29117]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-02-20T07:48:26.641 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:48:26 smithi133 ceph-mon[29117]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-02-20T07:48:26.641 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:48:26 smithi133 ceph-mon[29117]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-02-20T07:48:26.641 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:48:26 smithi133 ceph-mon[29117]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-02-20T07:48:26.641 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:48:26 smithi133 ceph-mon[29117]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-02-20T07:48:26.641 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:48:26 smithi133 ceph-mon[29117]: rocksdb: Options.compression_opts.window_bits: -14 2024-02-20T07:48:26.641 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:48:26 smithi133 ceph-mon[29117]: rocksdb: Options.compression_opts.level: 32767 2024-02-20T07:48:26.641 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:48:26 smithi133 ceph-mon[29117]: rocksdb: Options.compression_opts.strategy: 0 2024-02-20T07:48:26.641 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:48:26 smithi133 ceph-mon[29117]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-02-20T07:48:26.641 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:48:26 smithi133 ceph-mon[29117]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-02-20T07:48:26.642 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:48:26 smithi133 ceph-mon[29117]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-02-20T07:48:26.642 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:48:26 smithi133 ceph-mon[29117]: rocksdb: Options.compression_opts.enabled: false 2024-02-20T07:48:26.642 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:48:26 smithi133 ceph-mon[29117]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-02-20T07:48:26.642 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:48:26 smithi133 ceph-mon[29117]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-02-20T07:48:26.642 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:48:26 smithi133 ceph-mon[29117]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-02-20T07:48:26.642 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:48:26 smithi133 ceph-mon[29117]: rocksdb: Options.target_file_size_base: 67108864 2024-02-20T07:48:26.642 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:48:26 smithi133 ceph-mon[29117]: rocksdb: Options.target_file_size_multiplier: 1 2024-02-20T07:48:26.642 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:48:26 smithi133 ceph-mon[29117]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-02-20T07:48:26.642 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:48:26 smithi133 ceph-mon[29117]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-02-20T07:48:26.642 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:48:26 smithi133 ceph-mon[29117]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-02-20T07:48:26.642 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:48:26 smithi133 ceph-mon[29117]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-02-20T07:48:26.642 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:48:26 smithi133 ceph-mon[29117]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-02-20T07:48:26.642 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:48:26 smithi133 ceph-mon[29117]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-02-20T07:48:26.642 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:48:26 smithi133 ceph-mon[29117]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-02-20T07:48:26.642 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:48:26 smithi133 ceph-mon[29117]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-02-20T07:48:26.643 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:48:26 smithi133 ceph-mon[29117]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-02-20T07:48:26.643 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:48:26 smithi133 ceph-mon[29117]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-02-20T07:48:26.643 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:48:26 smithi133 ceph-mon[29117]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-02-20T07:48:26.643 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:48:26 smithi133 ceph-mon[29117]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-02-20T07:48:26.643 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:48:26 smithi133 ceph-mon[29117]: rocksdb: Options.arena_block_size: 4194304 2024-02-20T07:48:26.643 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:48:26 smithi133 ceph-mon[29117]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-02-20T07:48:26.643 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:48:26 smithi133 ceph-mon[29117]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-02-20T07:48:26.643 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:48:26 smithi133 ceph-mon[29117]: rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2024-02-20T07:48:26.643 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:48:26 smithi133 ceph-mon[29117]: rocksdb: Options.disable_auto_compactions: 0 2024-02-20T07:48:26.643 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:48:26 smithi133 ceph-mon[29117]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-02-20T07:48:26.643 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:48:26 smithi133 ceph-mon[29117]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-02-20T07:48:26.643 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:48:26 smithi133 ceph-mon[29117]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-02-20T07:48:26.643 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:48:26 smithi133 ceph-mon[29117]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-02-20T07:48:26.643 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:48:26 smithi133 ceph-mon[29117]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-02-20T07:48:26.643 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:48:26 smithi133 ceph-mon[29117]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-02-20T07:48:26.643 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:48:26 smithi133 ceph-mon[29117]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-02-20T07:48:26.644 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:48:26 smithi133 ceph-mon[29117]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-02-20T07:48:26.644 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:48:26 smithi133 ceph-mon[29117]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-02-20T07:48:26.644 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:48:26 smithi133 ceph-mon[29117]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-02-20T07:48:26.644 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:48:26 smithi133 ceph-mon[29117]: rocksdb: Options.table_properties_collectors: 2024-02-20T07:48:26.644 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:48:26 smithi133 ceph-mon[29117]: rocksdb: Options.inplace_update_support: 0 2024-02-20T07:48:26.644 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:48:26 smithi133 ceph-mon[29117]: rocksdb: Options.inplace_update_num_locks: 10000 2024-02-20T07:48:26.644 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:48:26 smithi133 ceph-mon[29117]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-02-20T07:48:26.644 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:48:26 smithi133 ceph-mon[29117]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-02-20T07:48:26.644 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:48:26 smithi133 ceph-mon[29117]: rocksdb: Options.memtable_huge_page_size: 0 2024-02-20T07:48:26.644 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:48:26 smithi133 ceph-mon[29117]: rocksdb: Options.bloom_locality: 0 2024-02-20T07:48:26.644 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:48:26 smithi133 ceph-mon[29117]: rocksdb: Options.max_successive_merges: 0 2024-02-20T07:48:26.644 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:48:26 smithi133 ceph-mon[29117]: rocksdb: Options.optimize_filters_for_hits: 0 2024-02-20T07:48:26.644 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:48:26 smithi133 ceph-mon[29117]: rocksdb: Options.paranoid_file_checks: 0 2024-02-20T07:48:26.644 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:48:26 smithi133 ceph-mon[29117]: rocksdb: Options.force_consistency_checks: 1 2024-02-20T07:48:26.644 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:48:26 smithi133 ceph-mon[29117]: rocksdb: Options.report_bg_io_stats: 0 2024-02-20T07:48:26.645 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:48:26 smithi133 ceph-mon[29117]: rocksdb: Options.ttl: 2592000 2024-02-20T07:48:26.645 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:48:26 smithi133 ceph-mon[29117]: rocksdb: Options.periodic_compaction_seconds: 0 2024-02-20T07:48:26.645 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:48:26 smithi133 ceph-mon[29117]: rocksdb: Options.enable_blob_files: false 2024-02-20T07:48:26.645 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:48:26 smithi133 ceph-mon[29117]: rocksdb: Options.min_blob_size: 0 2024-02-20T07:48:26.645 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:48:26 smithi133 ceph-mon[29117]: rocksdb: Options.blob_file_size: 268435456 2024-02-20T07:48:26.645 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:48:26 smithi133 ceph-mon[29117]: rocksdb: Options.blob_compression_type: NoCompression 2024-02-20T07:48:26.645 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:48:26 smithi133 ceph-mon[29117]: rocksdb: Options.enable_blob_garbage_collection: false 2024-02-20T07:48:26.645 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:48:26 smithi133 ceph-mon[29117]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-02-20T07:48:26.645 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:48:26 smithi133 ceph-mon[29117]: rocksdb: [db/version_set.cc:4764] Recovered from manifest file:/var/lib/ceph/mon/ceph-smithi133/store.db/MANIFEST-000009 succeeded,manifest_file_number is 9, next_file_number is 11, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-02-20T07:48:26.645 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:48:26 smithi133 ceph-mon[29117]: rocksdb: [db/version_set.cc:4779] Column family [default] (ID 0), log number is 5 2024-02-20T07:48:26.645 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:48:26 smithi133 ceph-mon[29117]: rocksdb: [db/version_set.cc:4082] Creating manifest 13 2024-02-20T07:48:26.645 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:48:26 smithi133 ceph-mon[29117]: rocksdb: EVENT_LOG_v1 {"time_micros": 1708415306572196, "job": 1, "event": "recovery_started", "wal_files": [10]} 2024-02-20T07:48:26.645 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:48:26 smithi133 ceph-mon[29117]: rocksdb: [db/db_impl/db_impl_open.cc:845] Recovering log #10 mode 2 2024-02-20T07:48:26.646 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:48:26 smithi133 ceph-mon[29117]: rocksdb: [table/block_based/filter_policy.cc:991] Using legacy Bloom filter with high (20) bits/key. Dramatic filter space and/or accuracy improvement is available with format_version>=5. 2024-02-20T07:48:26.646 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:48:26 smithi133 ceph-mon[29117]: rocksdb: EVENT_LOG_v1 {"time_micros": 1708415306573061, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 14, "file_size": 93240, "file_checksum": "", "file_checksum_func_name": "Unknown", "table_properties": {"data_size": 91335, "index_size": 265, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 709, "raw_key_size": 13418, "raw_average_key_size": 49, "raw_value_size": 83997, "raw_average_value_size": 307, "num_data_blocks": 11, "num_entries": 273, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1708415306, "oldest_key_time": 0, "file_creation_time": 0, "db_id": "76c6ec12-9cd2-48f6-8458-b7757a3ede2d", "db_session_id": "CFNEJ5UKSRIO4NBX31SQ"}} 2024-02-20T07:48:26.646 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:48:26 smithi133 ceph-mon[29117]: rocksdb: [db/version_set.cc:4082] Creating manifest 15 2024-02-20T07:48:26.646 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:48:26 smithi133 ceph-mon[29117]: rocksdb: EVENT_LOG_v1 {"time_micros": 1708415306573373, "job": 1, "event": "recovery_finished"} 2024-02-20T07:48:26.646 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:48:26 smithi133 ceph-mon[29117]: rocksdb: [file/delete_scheduler.cc:69] Deleted file /var/lib/ceph/mon/ceph-smithi133/store.db/000010.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-02-20T07:48:26.646 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:48:26 smithi133 ceph-mon[29117]: rocksdb: [db/db_impl/db_impl_open.cc:1700] SstFileManager instance 0x5593f530b180 2024-02-20T07:48:26.646 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:48:26 smithi133 ceph-mon[29117]: rocksdb: DB pointer 0x5593f53e0000 2024-02-20T07:48:26.646 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:48:26 smithi133 ceph-mon[29117]: rocksdb: [db/db_impl/db_impl.cc:901] ------- DUMPING STATS ------- 2024-02-20T07:48:26.646 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:48:26 smithi133 ceph-mon[29117]: rocksdb: [db/db_impl/db_impl.cc:903] 2024-02-20T07:48:26.646 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout: ** DB Stats ** 2024-02-20T07:48:26.646 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-02-20T07:48:26.646 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-02-20T07:48:26.646 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-02-20T07:48:26.646 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-02-20T07:48:26.647 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-02-20T07:48:26.647 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2024-02-20T07:48:26.647 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-02-20T07:48:26.647 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout: 2024-02-20T07:48:26.647 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout: ** Compaction Stats [default] ** 2024-02-20T07:48:26.647 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-02-20T07:48:26.647 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-20T07:48:26.647 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout: L0 2/0 92.85 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 149.7 0.00 0.00 1 0.001 0 0 2024-02-20T07:48:26.647 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout: Sum 2/0 92.85 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 149.7 0.00 0.00 1 0.001 0 0 2024-02-20T07:48:26.647 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 149.7 0.00 0.00 1 0.001 0 0 2024-02-20T07:48:26.647 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout: 2024-02-20T07:48:26.647 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout: ** Compaction Stats [default] ** 2024-02-20T07:48:26.647 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-02-20T07:48:26.647 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-20T07:48:26.648 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 149.7 0.00 0.00 1 0.001 0 0 2024-02-20T07:48:26.648 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-02-20T07:48:26.648 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-02-20T07:48:26.648 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-02-20T07:48:26.648 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-02-20T07:48:26.648 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-02-20T07:48:26.648 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout: AddFile(Keys): cumulative 0, interval 0 2024-02-20T07:48:26.648 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout: Cumulative compaction: 0.00 GB write, 29.23 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-20T07:48:26.648 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout: Interval compaction: 0.00 GB write, 29.23 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-20T07:48:26.648 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-02-20T07:48:26.648 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout: 2024-02-20T07:48:26.648 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout: ** File Read Latency Histogram By Level [default] ** 2024-02-20T07:48:26.648 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout: 2024-02-20T07:48:26.648 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout: ** Compaction Stats [default] ** 2024-02-20T07:48:26.649 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-02-20T07:48:26.649 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-20T07:48:26.649 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout: L0 2/0 92.85 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 149.7 0.00 0.00 1 0.001 0 0 2024-02-20T07:48:26.649 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout: Sum 2/0 92.85 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 149.7 0.00 0.00 1 0.001 0 0 2024-02-20T07:48:26.649 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-02-20T07:48:26.649 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout: 2024-02-20T07:48:26.649 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout: ** Compaction Stats [default] ** 2024-02-20T07:48:26.649 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-02-20T07:48:26.649 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-20T07:48:26.649 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 149.7 0.00 0.00 1 0.001 0 0 2024-02-20T07:48:26.649 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-02-20T07:48:26.649 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-02-20T07:48:26.649 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-02-20T07:48:26.649 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-02-20T07:48:26.649 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-02-20T07:48:26.650 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout: AddFile(Keys): cumulative 0, interval 0 2024-02-20T07:48:26.650 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout: Cumulative compaction: 0.00 GB write, 28.90 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-20T07:48:26.650 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-20T07:48:26.650 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-02-20T07:48:26.650 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout: 2024-02-20T07:48:26.650 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout: ** File Read Latency Histogram By Level [default] ** 2024-02-20T07:48:26.650 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:48:26 smithi133 ceph-mon[29117]: starting mon.smithi133 rank 0 at public addrs [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] at bind addrs [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] mon_data /var/lib/ceph/mon/ceph-smithi133 fsid 4537327c-cfc4-11ee-95bb-87774f69a715 2024-02-20T07:48:26.650 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:48:26 smithi133 ceph-mon[29117]: mon.smithi133@-1(???) e1 preinit fsid 4537327c-cfc4-11ee-95bb-87774f69a715 2024-02-20T07:48:26.650 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:48:26 smithi133 ceph-mon[29117]: mon.smithi133@-1(???).mds e1 new map 2024-02-20T07:48:26.650 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:48:26 smithi133 ceph-mon[29117]: mon.smithi133@-1(???).mds e1 print_map 2024-02-20T07:48:26.650 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout: e1 2024-02-20T07:48:26.650 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-02-20T07:48:26.650 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-20T07:48:26.650 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout: legacy client fscid: -1 2024-02-20T07:48:26.650 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout: 2024-02-20T07:48:26.651 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout: No filesystems configured 2024-02-20T07:48:26.651 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:48:26 smithi133 ceph-mon[29117]: mon.smithi133@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-02-20T07:48:26.651 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:48:26 smithi133 ceph-mon[29117]: mon.smithi133@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-02-20T07:48:26.651 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:48:26 smithi133 ceph-mon[29117]: mon.smithi133@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-02-20T07:48:26.651 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:48:26 smithi133 ceph-mon[29117]: mon.smithi133@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-02-20T07:48:26.651 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:48:26 smithi133 ceph-mon[29117]: mon.smithi133@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-02-20T07:48:26.651 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:48:26 smithi133 bash[29097]: fe7046a20c8704d4c5e4b4c19b43a3423bc6705e3d978aa30e3796944d81371e 2024-02-20T07:48:26.651 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:48:26 smithi133 systemd[1]: Started Ceph mon.smithi133 for 4537327c-cfc4-11ee-95bb-87774f69a715. 2024-02-20T07:48:26.651 INFO:teuthology.orchestra.run.smithi133.stdout:Setting mon public_network to 172.21.0.0/20 2024-02-20T07:48:26.930 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:48:26 smithi133 ceph-mon[29117]: mon.smithi133 is new leader, mons smithi133 in quorum (ranks 0) 2024-02-20T07:48:26.930 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:48:26 smithi133 ceph-mon[29117]: monmap e1: 1 mons at {smithi133=[v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-02-20T07:48:26.931 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:48:26 smithi133 ceph-mon[29117]: fsmap 2024-02-20T07:48:26.931 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:48:26 smithi133 ceph-mon[29117]: osdmap e1: 0 total, 0 up, 0 in 2024-02-20T07:48:26.931 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:48:26 smithi133 ceph-mon[29117]: mgrmap e1: no daemons active 2024-02-20T07:48:27.684 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:27.026+0000 7fc8b1924700 1 Processor -- start 2024-02-20T07:48:27.685 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:27.026+0000 7fc8b1924700 1 -- start start 2024-02-20T07:48:27.685 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:27.026+0000 7fc8b1924700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fc8ac076c60 0x7fc8ac105340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:48:27.685 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:27.026+0000 7fc8b1924700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8ac105880 con 0x7fc8ac076c60 2024-02-20T07:48:27.685 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:27.026+0000 7fc8aaffd700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fc8ac076c60 0x7fc8ac105340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:48:27.685 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:27.026+0000 7fc8aaffd700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fc8ac076c60 0x7fc8ac105340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.133:48138/0 (socket says 172.21.15.133:48138) 2024-02-20T07:48:27.685 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:27.026+0000 7fc8aaffd700 1 -- 172.21.15.133:0/998658262 learned_addr learned my addr 172.21.15.133:0/998658262 (peer_addr_for_me v2:172.21.15.133:0/0) 2024-02-20T07:48:27.685 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:27.027+0000 7fc8aaffd700 1 -- 172.21.15.133:0/998658262 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8ac1059c0 con 0x7fc8ac076c60 2024-02-20T07:48:27.685 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:27.027+0000 7fc8aaffd700 1 --2- 172.21.15.133:0/998658262 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fc8ac076c60 0x7fc8ac105340 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fc89c0098e0 tx=0x7fc89c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f86fcae9052a785 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:48:27.685 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:27.027+0000 7fc8a9ffb700 1 -- 172.21.15.133:0/998658262 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc89c00b860 con 0x7fc8ac076c60 2024-02-20T07:48:27.685 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:27.027+0000 7fc8a9ffb700 1 -- 172.21.15.133:0/998658262 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7fc89c00be50 con 0x7fc8ac076c60 2024-02-20T07:48:27.685 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:27.027+0000 7fc8a9ffb700 1 -- 172.21.15.133:0/998658262 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc89c010380 con 0x7fc8ac076c60 2024-02-20T07:48:27.685 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:27.027+0000 7fc8b1924700 1 -- 172.21.15.133:0/998658262 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fc8ac076c60 msgr2=0x7fc8ac105340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:48:27.685 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:27.027+0000 7fc8b1924700 1 --2- 172.21.15.133:0/998658262 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fc8ac076c60 0x7fc8ac105340 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fc89c0098e0 tx=0x7fc89c009250 comp rx=0 tx=0).stop 2024-02-20T07:48:27.685 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:27.027+0000 7fc8b1924700 1 -- 172.21.15.133:0/998658262 shutdown_connections 2024-02-20T07:48:27.686 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:27.027+0000 7fc8b1924700 1 --2- 172.21.15.133:0/998658262 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fc8ac076c60 0x7fc8ac105340 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:48:27.686 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:27.027+0000 7fc8b1924700 1 -- 172.21.15.133:0/998658262 >> 172.21.15.133:0/998658262 conn(0x7fc8ac0fcdd0 msgr2=0x7fc8ac0ff200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:48:27.686 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:27.027+0000 7fc8b1924700 1 -- 172.21.15.133:0/998658262 shutdown_connections 2024-02-20T07:48:27.686 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:27.027+0000 7fc8b1924700 1 -- 172.21.15.133:0/998658262 wait complete. 2024-02-20T07:48:27.686 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:27.028+0000 7fc8b1924700 1 Processor -- start 2024-02-20T07:48:27.686 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:27.028+0000 7fc8b1924700 1 -- start start 2024-02-20T07:48:27.686 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:27.028+0000 7fc8b1924700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fc8ac076c60 0x7fc8ac111560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:48:27.686 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:27.028+0000 7fc8b1924700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8ac111aa0 con 0x7fc8ac076c60 2024-02-20T07:48:27.686 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:27.028+0000 7fc8aaffd700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fc8ac076c60 0x7fc8ac111560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:48:27.686 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:27.028+0000 7fc8aaffd700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fc8ac076c60 0x7fc8ac111560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.133:48142/0 (socket says 172.21.15.133:48142) 2024-02-20T07:48:27.686 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:27.028+0000 7fc8aaffd700 1 -- 172.21.15.133:0/3241805598 learned_addr learned my addr 172.21.15.133:0/3241805598 (peer_addr_for_me v2:172.21.15.133:0/0) 2024-02-20T07:48:27.686 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:27.029+0000 7fc8aaffd700 1 -- 172.21.15.133:0/3241805598 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc89c0094f0 con 0x7fc8ac076c60 2024-02-20T07:48:27.686 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:27.029+0000 7fc8aaffd700 1 --2- 172.21.15.133:0/3241805598 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fc8ac076c60 0x7fc8ac111560 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fc89c003050 tx=0x7fc89c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:48:27.686 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:27.029+0000 7fc8937fe700 1 -- 172.21.15.133:0/3241805598 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc89c010760 con 0x7fc8ac076c60 2024-02-20T07:48:27.686 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:27.029+0000 7fc8937fe700 1 -- 172.21.15.133:0/3241805598 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7fc89c010d50 con 0x7fc8ac076c60 2024-02-20T07:48:27.687 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:27.029+0000 7fc8b1924700 1 -- 172.21.15.133:0/3241805598 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc8ac111ca0 con 0x7fc8ac076c60 2024-02-20T07:48:27.687 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:27.029+0000 7fc8937fe700 1 -- 172.21.15.133:0/3241805598 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc89c01ae20 con 0x7fc8ac076c60 2024-02-20T07:48:27.687 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:27.029+0000 7fc8b1924700 1 -- 172.21.15.133:0/3241805598 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc8ac112940 con 0x7fc8ac076c60 2024-02-20T07:48:27.687 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:27.029+0000 7fc8937fe700 1 -- 172.21.15.133:0/3241805598 <== mon.0 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7fc89c02e6c0 con 0x7fc8ac076c60 2024-02-20T07:48:27.687 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:27.029+0000 7fc8937fe700 1 -- 172.21.15.133:0/3241805598 <== mon.0 v2:172.21.15.133:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7fc89c024900 con 0x7fc8ac076c60 2024-02-20T07:48:27.687 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:27.030+0000 7fc8b1924700 1 -- 172.21.15.133:0/3241805598 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc8ac05f5c0 con 0x7fc8ac076c60 2024-02-20T07:48:27.687 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:27.032+0000 7fc8937fe700 1 -- 172.21.15.133:0/3241805598 <== mon.0 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71349 (secure 0 0 0) 0x7fc89c029070 con 0x7fc8ac076c60 2024-02-20T07:48:27.687 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:27.097+0000 7fc8b1924700 1 -- 172.21.15.133:0/3241805598 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command([{prefix=config set, name=public_network}] v 0) v1 -- 0x7fc8ac106800 con 0x7fc8ac076c60 2024-02-20T07:48:27.687 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:27.100+0000 7fc8937fe700 1 -- 172.21.15.133:0/3241805598 <== mon.0 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{prefix=config set, name=public_network}]=0 v3)=0 v3) v1 ==== 124+0+0 (secure 0 0 0) 0x7fc89c01a6a0 con 0x7fc8ac076c60 2024-02-20T07:48:27.687 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:27.100+0000 7fc8b1924700 1 -- 172.21.15.133:0/3241805598 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fc8ac076c60 msgr2=0x7fc8ac111560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:48:27.687 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:27.100+0000 7fc8b1924700 1 --2- 172.21.15.133:0/3241805598 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fc8ac076c60 0x7fc8ac111560 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fc89c003050 tx=0x7fc89c009250 comp rx=0 tx=0).stop 2024-02-20T07:48:27.687 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:27.100+0000 7fc8b1924700 1 -- 172.21.15.133:0/3241805598 shutdown_connections 2024-02-20T07:48:27.687 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:27.100+0000 7fc8b1924700 1 --2- 172.21.15.133:0/3241805598 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fc8ac076c60 0x7fc8ac111560 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:48:27.687 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:27.100+0000 7fc8b1924700 1 -- 172.21.15.133:0/3241805598 >> 172.21.15.133:0/3241805598 conn(0x7fc8ac0fcdd0 msgr2=0x7fc8ac0fdbd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:48:27.688 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:27.100+0000 7fc8b1924700 1 -- 172.21.15.133:0/3241805598 shutdown_connections 2024-02-20T07:48:27.688 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:27.100+0000 7fc8b1924700 1 -- 172.21.15.133:0/3241805598 wait complete. 2024-02-20T07:48:27.688 INFO:teuthology.orchestra.run.smithi133.stdout:Wrote config to /etc/ceph/ceph.conf 2024-02-20T07:48:27.688 INFO:teuthology.orchestra.run.smithi133.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-02-20T07:48:27.688 INFO:teuthology.orchestra.run.smithi133.stdout:Creating mgr... 2024-02-20T07:48:27.688 INFO:teuthology.orchestra.run.smithi133.stdout:Verifying port 9283 ... 2024-02-20T07:48:27.849 INFO:teuthology.orchestra.run.smithi133.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-4537327c-cfc4-11ee-95bb-87774f69a715@mgr.smithi133.mrvcbp 2024-02-20T07:48:27.849 INFO:teuthology.orchestra.run.smithi133.stdout:systemctl: stderr Failed to reset failed state of unit ceph-4537327c-cfc4-11ee-95bb-87774f69a715@mgr.smithi133.mrvcbp.service: Unit ceph-4537327c-cfc4-11ee-95bb-87774f69a715@mgr.smithi133.mrvcbp.service not loaded. 2024-02-20T07:48:27.992 INFO:teuthology.orchestra.run.smithi133.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-4537327c-cfc4-11ee-95bb-87774f69a715.target.wants/ceph-4537327c-cfc4-11ee-95bb-87774f69a715@mgr.smithi133.mrvcbp.service → /etc/systemd/system/ceph-4537327c-cfc4-11ee-95bb-87774f69a715@.service. 2024-02-20T07:48:28.231 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:48:28 smithi133 ceph-mon[29117]: from='client.? 172.21.15.133:0/3241805598' entity='client.admin' 2024-02-20T07:48:29.935 INFO:teuthology.orchestra.run.smithi133.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-02-20T07:48:29.935 INFO:teuthology.orchestra.run.smithi133.stdout:systemctl: stdout disabled 2024-02-20T07:48:29.942 INFO:teuthology.orchestra.run.smithi133.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-02-20T07:48:29.943 INFO:teuthology.orchestra.run.smithi133.stdout:systemctl: stdout inactive 2024-02-20T07:48:29.943 INFO:teuthology.orchestra.run.smithi133.stdout:firewalld.service is not enabled 2024-02-20T07:48:29.943 INFO:teuthology.orchestra.run.smithi133.stdout:Not possible to enable service . firewalld.service is not available 2024-02-20T07:48:29.951 INFO:teuthology.orchestra.run.smithi133.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-02-20T07:48:29.951 INFO:teuthology.orchestra.run.smithi133.stdout:systemctl: stdout disabled 2024-02-20T07:48:29.959 INFO:teuthology.orchestra.run.smithi133.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-02-20T07:48:29.959 INFO:teuthology.orchestra.run.smithi133.stdout:systemctl: stdout inactive 2024-02-20T07:48:29.959 INFO:teuthology.orchestra.run.smithi133.stdout:firewalld.service is not enabled 2024-02-20T07:48:29.959 INFO:teuthology.orchestra.run.smithi133.stdout:Not possible to open ports <[9283]>. firewalld.service is not available 2024-02-20T07:48:29.959 INFO:teuthology.orchestra.run.smithi133.stdout:Waiting for mgr to start... 2024-02-20T07:48:29.959 INFO:teuthology.orchestra.run.smithi133.stdout:Waiting for mgr... 2024-02-20T07:48:30.736 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:48:30 smithi133 ceph-mon[29117]: from='client.? 172.21.15.133:0/3068475003' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-02-20T07:48:31.050 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stdout 2024-02-20T07:48:31.050 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stdout { 2024-02-20T07:48:31.051 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stdout "fsid": "4537327c-cfc4-11ee-95bb-87774f69a715", 2024-02-20T07:48:31.051 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stdout "health": { 2024-02-20T07:48:31.051 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-02-20T07:48:31.051 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-02-20T07:48:31.051 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-02-20T07:48:31.051 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stdout }, 2024-02-20T07:48:31.051 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-02-20T07:48:31.051 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-02-20T07:48:31.051 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stdout 0 2024-02-20T07:48:31.051 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stdout ], 2024-02-20T07:48:31.051 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-02-20T07:48:31.051 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stdout "smithi133" 2024-02-20T07:48:31.051 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stdout ], 2024-02-20T07:48:31.051 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stdout "quorum_age": 3, 2024-02-20T07:48:31.051 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stdout "monmap": { 2024-02-20T07:48:31.051 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-02-20T07:48:31.052 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "quincy", 2024-02-20T07:48:31.052 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-02-20T07:48:31.052 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stdout }, 2024-02-20T07:48:31.052 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-02-20T07:48:31.052 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-02-20T07:48:31.052 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-02-20T07:48:31.052 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-02-20T07:48:31.052 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-02-20T07:48:31.059 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-02-20T07:48:31.060 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-02-20T07:48:31.060 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-02-20T07:48:31.060 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stdout }, 2024-02-20T07:48:31.060 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-02-20T07:48:31.060 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-02-20T07:48:31.060 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-02-20T07:48:31.060 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-02-20T07:48:31.060 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-02-20T07:48:31.060 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-02-20T07:48:31.060 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-02-20T07:48:31.060 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-02-20T07:48:31.060 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-02-20T07:48:31.060 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stdout }, 2024-02-20T07:48:31.060 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-02-20T07:48:31.061 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-02-20T07:48:31.061 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-02-20T07:48:31.061 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-02-20T07:48:31.061 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stdout }, 2024-02-20T07:48:31.061 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-02-20T07:48:31.061 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stdout "available": false, 2024-02-20T07:48:31.061 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-02-20T07:48:31.061 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stdout "modules": [ 2024-02-20T07:48:31.061 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stdout "iostat", 2024-02-20T07:48:31.061 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stdout "nfs", 2024-02-20T07:48:31.061 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stdout "restful" 2024-02-20T07:48:31.061 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stdout ], 2024-02-20T07:48:31.061 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stdout "services": {} 2024-02-20T07:48:31.061 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stdout }, 2024-02-20T07:48:31.061 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-02-20T07:48:31.061 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-02-20T07:48:31.062 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stdout "modified": "2024-02-20T07:48:13.550997+0000", 2024-02-20T07:48:31.062 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stdout "services": {} 2024-02-20T07:48:31.062 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stdout }, 2024-02-20T07:48:31.062 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-02-20T07:48:31.062 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stdout } 2024-02-20T07:48:31.062 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:30.347+0000 7f1a1f0c8700 1 Processor -- start 2024-02-20T07:48:31.062 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:30.348+0000 7f1a1f0c8700 1 -- start start 2024-02-20T07:48:31.062 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:30.348+0000 7f1a1f0c8700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f1a18078d20 0x7f1a18077230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:48:31.062 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:30.348+0000 7f1a1f0c8700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a18079130 con 0x7f1a18078d20 2024-02-20T07:48:31.062 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:30.348+0000 7f1a1ce64700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f1a18078d20 0x7f1a18077230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:48:31.062 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:30.348+0000 7f1a1ce64700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f1a18078d20 0x7f1a18077230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.133:48170/0 (socket says 172.21.15.133:48170) 2024-02-20T07:48:31.062 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:30.348+0000 7f1a1ce64700 1 -- 172.21.15.133:0/2403014630 learned_addr learned my addr 172.21.15.133:0/2403014630 (peer_addr_for_me v2:172.21.15.133:0/0) 2024-02-20T07:48:31.062 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:30.348+0000 7f1a1ce64700 1 -- 172.21.15.133:0/2403014630 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a18077770 con 0x7f1a18078d20 2024-02-20T07:48:31.062 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:30.348+0000 7f1a1ce64700 1 --2- 172.21.15.133:0/2403014630 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f1a18078d20 0x7f1a18077230 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f1a0c0098e0 tx=0x7f1a0c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fb07d716935883a2 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:48:31.062 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:30.348+0000 7f1a177fe700 1 -- 172.21.15.133:0/2403014630 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1a0c00b860 con 0x7f1a18078d20 2024-02-20T07:48:31.062 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:30.348+0000 7f1a177fe700 1 -- 172.21.15.133:0/2403014630 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f1a0c00be50 con 0x7f1a18078d20 2024-02-20T07:48:31.063 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:30.348+0000 7f1a177fe700 1 -- 172.21.15.133:0/2403014630 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1a0c010380 con 0x7f1a18078d20 2024-02-20T07:48:31.063 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:30.349+0000 7f1a1f0c8700 1 -- 172.21.15.133:0/2403014630 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f1a18078d20 msgr2=0x7f1a18077230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:48:31.063 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:30.349+0000 7f1a1f0c8700 1 --2- 172.21.15.133:0/2403014630 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f1a18078d20 0x7f1a18077230 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f1a0c0098e0 tx=0x7f1a0c009250 comp rx=0 tx=0).stop 2024-02-20T07:48:31.063 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:30.349+0000 7f1a1f0c8700 1 -- 172.21.15.133:0/2403014630 shutdown_connections 2024-02-20T07:48:31.063 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:30.349+0000 7f1a1f0c8700 1 --2- 172.21.15.133:0/2403014630 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f1a18078d20 0x7f1a18077230 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:48:31.063 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:30.349+0000 7f1a1f0c8700 1 -- 172.21.15.133:0/2403014630 >> 172.21.15.133:0/2403014630 conn(0x7f1a180fded0 msgr2=0x7f1a18100300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:48:31.063 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:30.349+0000 7f1a1f0c8700 1 -- 172.21.15.133:0/2403014630 shutdown_connections 2024-02-20T07:48:31.063 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:30.349+0000 7f1a1f0c8700 1 -- 172.21.15.133:0/2403014630 wait complete. 2024-02-20T07:48:31.063 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:30.350+0000 7f1a1f0c8700 1 Processor -- start 2024-02-20T07:48:31.063 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:30.350+0000 7f1a1f0c8700 1 -- start start 2024-02-20T07:48:31.063 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:30.350+0000 7f1a1f0c8700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f1a18078d20 0x7f1a18113790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:48:31.063 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:30.350+0000 7f1a1f0c8700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a18113cd0 con 0x7f1a18078d20 2024-02-20T07:48:31.063 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:30.350+0000 7f1a1ce64700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f1a18078d20 0x7f1a18113790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:48:31.063 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:30.350+0000 7f1a1ce64700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f1a18078d20 0x7f1a18113790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.133:48184/0 (socket says 172.21.15.133:48184) 2024-02-20T07:48:31.064 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:30.350+0000 7f1a1ce64700 1 -- 172.21.15.133:0/3068475003 learned_addr learned my addr 172.21.15.133:0/3068475003 (peer_addr_for_me v2:172.21.15.133:0/0) 2024-02-20T07:48:31.064 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:30.351+0000 7f1a1ce64700 1 -- 172.21.15.133:0/3068475003 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a0c0094f0 con 0x7f1a18078d20 2024-02-20T07:48:31.064 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:30.351+0000 7f1a1ce64700 1 --2- 172.21.15.133:0/3068475003 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f1a18078d20 0x7f1a18113790 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f1a0c013030 tx=0x7f1a0c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:48:31.064 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:30.351+0000 7f1a157fa700 1 -- 172.21.15.133:0/3068475003 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1a0c010760 con 0x7f1a18078d20 2024-02-20T07:48:31.064 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:30.351+0000 7f1a1f0c8700 1 -- 172.21.15.133:0/3068475003 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1a18113ed0 con 0x7f1a18078d20 2024-02-20T07:48:31.064 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:30.351+0000 7f1a1f0c8700 1 -- 172.21.15.133:0/3068475003 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1a18114b70 con 0x7f1a18078d20 2024-02-20T07:48:31.064 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:30.351+0000 7f1a157fa700 1 -- 172.21.15.133:0/3068475003 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f1a0c010d50 con 0x7f1a18078d20 2024-02-20T07:48:31.064 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:30.351+0000 7f1a157fa700 1 -- 172.21.15.133:0/3068475003 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1a0c01ae20 con 0x7f1a18078d20 2024-02-20T07:48:31.064 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:30.351+0000 7f1a157fa700 1 -- 172.21.15.133:0/3068475003 <== mon.0 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7f1a0c019850 con 0x7f1a18078d20 2024-02-20T07:48:31.064 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:30.351+0000 7f1a157fa700 1 -- 172.21.15.133:0/3068475003 <== mon.0 v2:172.21.15.133:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7f1a0c024b00 con 0x7f1a18078d20 2024-02-20T07:48:31.064 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:30.352+0000 7f1a1f0c8700 1 -- 172.21.15.133:0/3068475003 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f19fc0052e0 con 0x7f1a18078d20 2024-02-20T07:48:31.064 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:30.354+0000 7f1a157fa700 1 -- 172.21.15.133:0/3068475003 <== mon.0 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71349 (secure 0 0 0) 0x7f1a0c029060 con 0x7f1a18078d20 2024-02-20T07:48:31.064 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:30.426+0000 7f1a1f0c8700 1 -- 172.21.15.133:0/3068475003 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f19fc005590 con 0x7f1a18078d20 2024-02-20T07:48:31.064 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:30.426+0000 7f1a157fa700 1 -- 172.21.15.133:0/3068475003 <== mon.0 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1248 (secure 0 0 0) 0x7f1a0c032090 con 0x7f1a18078d20 2024-02-20T07:48:31.064 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:30.427+0000 7f1a1f0c8700 1 -- 172.21.15.133:0/3068475003 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f1a18078d20 msgr2=0x7f1a18113790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:48:31.065 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:30.427+0000 7f1a1f0c8700 1 --2- 172.21.15.133:0/3068475003 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f1a18078d20 0x7f1a18113790 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f1a0c013030 tx=0x7f1a0c009250 comp rx=0 tx=0).stop 2024-02-20T07:48:31.065 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:30.427+0000 7f1a1f0c8700 1 -- 172.21.15.133:0/3068475003 shutdown_connections 2024-02-20T07:48:31.065 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:30.427+0000 7f1a1f0c8700 1 --2- 172.21.15.133:0/3068475003 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f1a18078d20 0x7f1a18113790 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:48:31.065 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:30.427+0000 7f1a1f0c8700 1 -- 172.21.15.133:0/3068475003 >> 172.21.15.133:0/3068475003 conn(0x7f1a180fded0 msgr2=0x7f1a180ff570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:48:31.065 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:30.427+0000 7f1a1f0c8700 1 -- 172.21.15.133:0/3068475003 shutdown_connections 2024-02-20T07:48:31.065 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:30.427+0000 7f1a1f0c8700 1 -- 172.21.15.133:0/3068475003 wait complete. 2024-02-20T07:48:31.065 INFO:teuthology.orchestra.run.smithi133.stdout:mgr not available, waiting (1/15)... 2024-02-20T07:48:33.702 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:48:33 smithi133 ceph-mon[29117]: from='client.? 172.21.15.133:0/1973869881' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-02-20T07:48:34.107 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stdout 2024-02-20T07:48:34.107 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stdout { 2024-02-20T07:48:34.107 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stdout "fsid": "4537327c-cfc4-11ee-95bb-87774f69a715", 2024-02-20T07:48:34.107 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stdout "health": { 2024-02-20T07:48:34.107 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-02-20T07:48:34.108 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-02-20T07:48:34.108 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-02-20T07:48:34.108 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stdout }, 2024-02-20T07:48:34.108 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-02-20T07:48:34.108 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-02-20T07:48:34.109 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stdout 0 2024-02-20T07:48:34.109 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stdout ], 2024-02-20T07:48:34.109 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-02-20T07:48:34.109 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stdout "smithi133" 2024-02-20T07:48:34.109 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stdout ], 2024-02-20T07:48:34.109 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stdout "quorum_age": 6, 2024-02-20T07:48:34.110 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stdout "monmap": { 2024-02-20T07:48:34.110 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-02-20T07:48:34.110 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "quincy", 2024-02-20T07:48:34.110 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-02-20T07:48:34.110 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stdout }, 2024-02-20T07:48:34.111 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-02-20T07:48:34.111 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-02-20T07:48:34.111 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-02-20T07:48:34.111 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-02-20T07:48:34.111 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-02-20T07:48:34.111 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-02-20T07:48:34.111 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-02-20T07:48:34.112 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-02-20T07:48:34.112 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stdout }, 2024-02-20T07:48:34.112 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-02-20T07:48:34.112 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-02-20T07:48:34.112 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-02-20T07:48:34.112 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-02-20T07:48:34.112 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-02-20T07:48:34.113 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-02-20T07:48:34.113 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-02-20T07:48:34.113 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-02-20T07:48:34.113 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-02-20T07:48:34.113 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stdout }, 2024-02-20T07:48:34.113 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-02-20T07:48:34.113 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-02-20T07:48:34.113 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-02-20T07:48:34.114 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-02-20T07:48:34.114 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stdout }, 2024-02-20T07:48:34.114 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-02-20T07:48:34.114 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stdout "available": false, 2024-02-20T07:48:34.114 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-02-20T07:48:34.114 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stdout "modules": [ 2024-02-20T07:48:34.115 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stdout "iostat", 2024-02-20T07:48:34.115 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stdout "nfs", 2024-02-20T07:48:34.115 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stdout "restful" 2024-02-20T07:48:34.115 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stdout ], 2024-02-20T07:48:34.115 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stdout "services": {} 2024-02-20T07:48:34.115 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stdout }, 2024-02-20T07:48:34.115 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-02-20T07:48:34.116 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-02-20T07:48:34.116 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stdout "modified": "2024-02-20T07:48:13.550997+0000", 2024-02-20T07:48:34.116 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stdout "services": {} 2024-02-20T07:48:34.116 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stdout }, 2024-02-20T07:48:34.116 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-02-20T07:48:34.116 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stdout } 2024-02-20T07:48:34.116 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:33.457+0000 7fa0772ea700 1 Processor -- start 2024-02-20T07:48:34.117 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:33.457+0000 7fa0772ea700 1 -- start start 2024-02-20T07:48:34.117 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:33.457+0000 7fa0772ea700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa070076b80 0x7fa070105420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:48:34.117 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:33.457+0000 7fa0772ea700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa070105960 con 0x7fa070076b80 2024-02-20T07:48:34.117 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:33.457+0000 7fa075086700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa070076b80 0x7fa070105420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:48:34.117 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:33.457+0000 7fa075086700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa070076b80 0x7fa070105420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.133:48192/0 (socket says 172.21.15.133:48192) 2024-02-20T07:48:34.117 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:33.457+0000 7fa075086700 1 -- 172.21.15.133:0/3100718990 learned_addr learned my addr 172.21.15.133:0/3100718990 (peer_addr_for_me v2:172.21.15.133:0/0) 2024-02-20T07:48:34.117 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:33.457+0000 7fa075086700 1 -- 172.21.15.133:0/3100718990 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa070105aa0 con 0x7fa070076b80 2024-02-20T07:48:34.118 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:33.458+0000 7fa075086700 1 --2- 172.21.15.133:0/3100718990 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa070076b80 0x7fa070105420 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fa06c0098e0 tx=0x7fa06c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=209496eef4305128 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:48:34.118 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:33.458+0000 7fa067fff700 1 -- 172.21.15.133:0/3100718990 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa06c00b860 con 0x7fa070076b80 2024-02-20T07:48:34.118 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:33.458+0000 7fa067fff700 1 -- 172.21.15.133:0/3100718990 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7fa06c00be50 con 0x7fa070076b80 2024-02-20T07:48:34.118 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:33.458+0000 7fa067fff700 1 -- 172.21.15.133:0/3100718990 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa06c010380 con 0x7fa070076b80 2024-02-20T07:48:34.118 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:33.458+0000 7fa0772ea700 1 -- 172.21.15.133:0/3100718990 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa070076b80 msgr2=0x7fa070105420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:48:34.118 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:33.458+0000 7fa0772ea700 1 --2- 172.21.15.133:0/3100718990 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa070076b80 0x7fa070105420 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fa06c0098e0 tx=0x7fa06c009250 comp rx=0 tx=0).stop 2024-02-20T07:48:34.118 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:33.458+0000 7fa0772ea700 1 -- 172.21.15.133:0/3100718990 shutdown_connections 2024-02-20T07:48:34.118 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:33.458+0000 7fa0772ea700 1 --2- 172.21.15.133:0/3100718990 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa070076b80 0x7fa070105420 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:48:34.119 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:33.458+0000 7fa0772ea700 1 -- 172.21.15.133:0/3100718990 >> 172.21.15.133:0/3100718990 conn(0x7fa0700fcdd0 msgr2=0x7fa0700ff200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:48:34.119 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:33.459+0000 7fa0772ea700 1 -- 172.21.15.133:0/3100718990 shutdown_connections 2024-02-20T07:48:34.119 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:33.459+0000 7fa0772ea700 1 -- 172.21.15.133:0/3100718990 wait complete. 2024-02-20T07:48:34.119 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:33.459+0000 7fa0772ea700 1 Processor -- start 2024-02-20T07:48:34.119 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:33.459+0000 7fa0772ea700 1 -- start start 2024-02-20T07:48:34.119 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:33.459+0000 7fa0772ea700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa070076b80 0x7fa070111540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:48:34.120 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:33.459+0000 7fa0772ea700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa070111a80 con 0x7fa070076b80 2024-02-20T07:48:34.120 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:33.459+0000 7fa075086700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa070076b80 0x7fa070111540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:48:34.120 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:33.459+0000 7fa075086700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa070076b80 0x7fa070111540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.133:48200/0 (socket says 172.21.15.133:48200) 2024-02-20T07:48:34.120 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:33.459+0000 7fa075086700 1 -- 172.21.15.133:0/1973869881 learned_addr learned my addr 172.21.15.133:0/1973869881 (peer_addr_for_me v2:172.21.15.133:0/0) 2024-02-20T07:48:34.120 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:33.460+0000 7fa075086700 1 -- 172.21.15.133:0/1973869881 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa06c0094f0 con 0x7fa070076b80 2024-02-20T07:48:34.120 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:33.460+0000 7fa075086700 1 --2- 172.21.15.133:0/1973869881 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa070076b80 0x7fa070111540 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fa06c006d40 tx=0x7fa06c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:48:34.120 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:33.460+0000 7fa065ffb700 1 -- 172.21.15.133:0/1973869881 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa06c010760 con 0x7fa070076b80 2024-02-20T07:48:34.120 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:33.460+0000 7fa0772ea700 1 -- 172.21.15.133:0/1973869881 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa070111c80 con 0x7fa070076b80 2024-02-20T07:48:34.121 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:33.460+0000 7fa065ffb700 1 -- 172.21.15.133:0/1973869881 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7fa06c010d50 con 0x7fa070076b80 2024-02-20T07:48:34.121 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:33.460+0000 7fa065ffb700 1 -- 172.21.15.133:0/1973869881 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa06c024d20 con 0x7fa070076b80 2024-02-20T07:48:34.121 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:33.460+0000 7fa0772ea700 1 -- 172.21.15.133:0/1973869881 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa070112920 con 0x7fa070076b80 2024-02-20T07:48:34.121 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:33.460+0000 7fa065ffb700 1 -- 172.21.15.133:0/1973869881 <== mon.0 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7fa06c02e6c0 con 0x7fa070076b80 2024-02-20T07:48:34.121 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:33.460+0000 7fa065ffb700 1 -- 172.21.15.133:0/1973869881 <== mon.0 v2:172.21.15.133:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7fa06c01a940 con 0x7fa070076b80 2024-02-20T07:48:34.121 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:33.461+0000 7fa0772ea700 1 -- 172.21.15.133:0/1973869881 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa07005f5c0 con 0x7fa070076b80 2024-02-20T07:48:34.122 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:33.465+0000 7fa065ffb700 1 -- 172.21.15.133:0/1973869881 <== mon.0 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71349 (secure 0 0 0) 0x7fa06c029070 con 0x7fa070076b80 2024-02-20T07:48:34.122 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:33.534+0000 7fa0772ea700 1 -- 172.21.15.133:0/1973869881 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7fa070106820 con 0x7fa070076b80 2024-02-20T07:48:34.122 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:33.535+0000 7fa065ffb700 1 -- 172.21.15.133:0/1973869881 <== mon.0 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1248 (secure 0 0 0) 0x7fa06c0108c0 con 0x7fa070076b80 2024-02-20T07:48:34.122 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:33.536+0000 7fa0772ea700 1 -- 172.21.15.133:0/1973869881 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa070076b80 msgr2=0x7fa070111540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:48:34.122 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:33.536+0000 7fa0772ea700 1 --2- 172.21.15.133:0/1973869881 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa070076b80 0x7fa070111540 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fa06c006d40 tx=0x7fa06c009250 comp rx=0 tx=0).stop 2024-02-20T07:48:34.122 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:33.536+0000 7fa0772ea700 1 -- 172.21.15.133:0/1973869881 shutdown_connections 2024-02-20T07:48:34.122 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:33.536+0000 7fa0772ea700 1 --2- 172.21.15.133:0/1973869881 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa070076b80 0x7fa070111540 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:48:34.123 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:33.536+0000 7fa0772ea700 1 -- 172.21.15.133:0/1973869881 >> 172.21.15.133:0/1973869881 conn(0x7fa0700fcdd0 msgr2=0x7fa0700fe470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:48:34.123 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:33.536+0000 7fa0772ea700 1 -- 172.21.15.133:0/1973869881 shutdown_connections 2024-02-20T07:48:34.123 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:33.536+0000 7fa0772ea700 1 -- 172.21.15.133:0/1973869881 wait complete. 2024-02-20T07:48:34.123 INFO:teuthology.orchestra.run.smithi133.stdout:mgr not available, waiting (2/15)... 2024-02-20T07:48:36.743 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:48:36 smithi133 ceph-mon[29117]: Activating manager daemon smithi133.mrvcbp 2024-02-20T07:48:36.743 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:48:36 smithi133 ceph-mon[29117]: mgrmap e2: smithi133.mrvcbp(active, starting, since 0.00374763s) 2024-02-20T07:48:36.743 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:48:36 smithi133 ceph-mon[29117]: from='mgr.14100 172.21.15.133:0/1017691764' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-02-20T07:48:36.743 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:48:36 smithi133 ceph-mon[29117]: from='mgr.14100 172.21.15.133:0/1017691764' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-02-20T07:48:36.743 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:48:36 smithi133 ceph-mon[29117]: from='mgr.14100 172.21.15.133:0/1017691764' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-02-20T07:48:36.743 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:48:36 smithi133 ceph-mon[29117]: from='mgr.14100 172.21.15.133:0/1017691764' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "mon metadata", "id": "smithi133"}]: dispatch 2024-02-20T07:48:36.743 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:48:36 smithi133 ceph-mon[29117]: from='mgr.14100 172.21.15.133:0/1017691764' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "mgr metadata", "who": "smithi133.mrvcbp", "id": "smithi133.mrvcbp"}]: dispatch 2024-02-20T07:48:36.743 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:48:36 smithi133 ceph-mon[29117]: Manager daemon smithi133.mrvcbp is now available 2024-02-20T07:48:36.743 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:48:36 smithi133 ceph-mon[29117]: from='mgr.14100 172.21.15.133:0/1017691764' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi133.mrvcbp/mirror_snapshot_schedule"}]: dispatch 2024-02-20T07:48:36.743 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:48:36 smithi133 ceph-mon[29117]: from='mgr.14100 172.21.15.133:0/1017691764' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:48:36.744 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:48:36 smithi133 ceph-mon[29117]: from='mgr.14100 172.21.15.133:0/1017691764' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi133.mrvcbp/trash_purge_schedule"}]: dispatch 2024-02-20T07:48:36.744 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:48:36 smithi133 ceph-mon[29117]: from='mgr.14100 172.21.15.133:0/1017691764' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:48:36.744 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:48:36 smithi133 ceph-mon[29117]: from='mgr.14100 172.21.15.133:0/1017691764' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:48:37.191 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stdout 2024-02-20T07:48:37.191 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stdout { 2024-02-20T07:48:37.191 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stdout "fsid": "4537327c-cfc4-11ee-95bb-87774f69a715", 2024-02-20T07:48:37.191 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stdout "health": { 2024-02-20T07:48:37.191 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-02-20T07:48:37.191 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-02-20T07:48:37.191 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-02-20T07:48:37.191 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stdout }, 2024-02-20T07:48:37.191 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-02-20T07:48:37.192 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-02-20T07:48:37.192 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stdout 0 2024-02-20T07:48:37.192 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stdout ], 2024-02-20T07:48:37.192 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-02-20T07:48:37.192 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stdout "smithi133" 2024-02-20T07:48:37.192 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stdout ], 2024-02-20T07:48:37.192 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stdout "quorum_age": 10, 2024-02-20T07:48:37.192 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stdout "monmap": { 2024-02-20T07:48:37.192 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-02-20T07:48:37.192 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "quincy", 2024-02-20T07:48:37.192 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-02-20T07:48:37.192 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stdout }, 2024-02-20T07:48:37.192 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-02-20T07:48:37.192 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-02-20T07:48:37.192 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-02-20T07:48:37.193 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-02-20T07:48:37.193 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-02-20T07:48:37.193 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-02-20T07:48:37.193 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-02-20T07:48:37.193 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-02-20T07:48:37.193 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stdout }, 2024-02-20T07:48:37.193 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-02-20T07:48:37.193 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-02-20T07:48:37.194 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-02-20T07:48:37.194 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-02-20T07:48:37.194 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-02-20T07:48:37.194 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-02-20T07:48:37.194 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-02-20T07:48:37.194 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-02-20T07:48:37.194 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-02-20T07:48:37.194 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stdout }, 2024-02-20T07:48:37.194 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-02-20T07:48:37.194 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-02-20T07:48:37.201 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-02-20T07:48:37.201 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-02-20T07:48:37.201 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stdout }, 2024-02-20T07:48:37.201 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-02-20T07:48:37.201 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stdout "available": false, 2024-02-20T07:48:37.201 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-02-20T07:48:37.201 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stdout "modules": [ 2024-02-20T07:48:37.202 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stdout "iostat", 2024-02-20T07:48:37.202 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stdout "nfs", 2024-02-20T07:48:37.202 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stdout "restful" 2024-02-20T07:48:37.202 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stdout ], 2024-02-20T07:48:37.202 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stdout "services": {} 2024-02-20T07:48:37.202 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stdout }, 2024-02-20T07:48:37.202 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-02-20T07:48:37.202 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-02-20T07:48:37.202 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stdout "modified": "2024-02-20T07:48:13.550997+0000", 2024-02-20T07:48:37.202 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stdout "services": {} 2024-02-20T07:48:37.202 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stdout }, 2024-02-20T07:48:37.202 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-02-20T07:48:37.202 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stdout } 2024-02-20T07:48:37.202 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:36.548+0000 7fe847f86700 1 Processor -- start 2024-02-20T07:48:37.202 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:36.548+0000 7fe847f86700 1 -- start start 2024-02-20T07:48:37.203 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:36.548+0000 7fe847f86700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fe840102770 0x7fe840102b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:48:37.203 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:36.548+0000 7fe847f86700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8401030c0 con 0x7fe840102770 2024-02-20T07:48:37.203 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:36.548+0000 7fe845d22700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fe840102770 0x7fe840102b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:48:37.203 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:36.548+0000 7fe845d22700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fe840102770 0x7fe840102b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.133:53352/0 (socket says 172.21.15.133:53352) 2024-02-20T07:48:37.203 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:36.548+0000 7fe845d22700 1 -- 172.21.15.133:0/2885082004 learned_addr learned my addr 172.21.15.133:0/2885082004 (peer_addr_for_me v2:172.21.15.133:0/0) 2024-02-20T07:48:37.203 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:36.549+0000 7fe845d22700 1 -- 172.21.15.133:0/2885082004 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe840103200 con 0x7fe840102770 2024-02-20T07:48:37.203 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:36.549+0000 7fe845d22700 1 --2- 172.21.15.133:0/2885082004 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fe840102770 0x7fe840102b80 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fe83c0098e0 tx=0x7fe83c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7dda061b72279fc9 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:48:37.203 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:36.549+0000 7fe844d20700 1 -- 172.21.15.133:0/2885082004 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe83c00b860 con 0x7fe840102770 2024-02-20T07:48:37.203 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:36.549+0000 7fe844d20700 1 -- 172.21.15.133:0/2885082004 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7fe83c00be50 con 0x7fe840102770 2024-02-20T07:48:37.203 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:36.549+0000 7fe844d20700 1 -- 172.21.15.133:0/2885082004 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe83c010380 con 0x7fe840102770 2024-02-20T07:48:37.203 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:36.550+0000 7fe847f86700 1 -- 172.21.15.133:0/2885082004 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fe840102770 msgr2=0x7fe840102b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:48:37.203 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:36.550+0000 7fe847f86700 1 --2- 172.21.15.133:0/2885082004 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fe840102770 0x7fe840102b80 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fe83c0098e0 tx=0x7fe83c009250 comp rx=0 tx=0).stop 2024-02-20T07:48:37.203 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:36.550+0000 7fe847f86700 1 -- 172.21.15.133:0/2885082004 shutdown_connections 2024-02-20T07:48:37.203 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:36.550+0000 7fe847f86700 1 --2- 172.21.15.133:0/2885082004 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fe840102770 0x7fe840102b80 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:48:37.203 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:36.550+0000 7fe847f86700 1 -- 172.21.15.133:0/2885082004 >> 172.21.15.133:0/2885082004 conn(0x7fe8400fded0 msgr2=0x7fe840100300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:48:37.204 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:36.550+0000 7fe847f86700 1 -- 172.21.15.133:0/2885082004 shutdown_connections 2024-02-20T07:48:37.204 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:36.550+0000 7fe847f86700 1 -- 172.21.15.133:0/2885082004 wait complete. 2024-02-20T07:48:37.204 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:36.550+0000 7fe847f86700 1 Processor -- start 2024-02-20T07:48:37.204 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:36.550+0000 7fe847f86700 1 -- start start 2024-02-20T07:48:37.204 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:36.550+0000 7fe847f86700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fe840102770 0x7fe840111500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:48:37.204 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:36.550+0000 7fe847f86700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe840111a40 con 0x7fe840102770 2024-02-20T07:48:37.204 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:36.551+0000 7fe845d22700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fe840102770 0x7fe840111500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:48:37.204 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:36.551+0000 7fe845d22700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fe840102770 0x7fe840111500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.133:53360/0 (socket says 172.21.15.133:53360) 2024-02-20T07:48:37.204 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:36.551+0000 7fe845d22700 1 -- 172.21.15.133:0/3988805513 learned_addr learned my addr 172.21.15.133:0/3988805513 (peer_addr_for_me v2:172.21.15.133:0/0) 2024-02-20T07:48:37.204 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:36.551+0000 7fe845d22700 1 -- 172.21.15.133:0/3988805513 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe83c0094f0 con 0x7fe840102770 2024-02-20T07:48:37.204 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:36.551+0000 7fe845d22700 1 --2- 172.21.15.133:0/3988805513 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fe840102770 0x7fe840111500 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fe83c00b440 tx=0x7fe83c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:48:37.204 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:36.551+0000 7fe8367fc700 1 -- 172.21.15.133:0/3988805513 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe83c010760 con 0x7fe840102770 2024-02-20T07:48:37.204 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:36.551+0000 7fe8367fc700 1 -- 172.21.15.133:0/3988805513 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7fe83c010d50 con 0x7fe840102770 2024-02-20T07:48:37.204 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:36.551+0000 7fe8367fc700 1 -- 172.21.15.133:0/3988805513 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe83c024d80 con 0x7fe840102770 2024-02-20T07:48:37.204 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:36.551+0000 7fe847f86700 1 -- 172.21.15.133:0/3988805513 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe840111c40 con 0x7fe840102770 2024-02-20T07:48:37.205 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:36.551+0000 7fe847f86700 1 -- 172.21.15.133:0/3988805513 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe8401128e0 con 0x7fe840102770 2024-02-20T07:48:37.205 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:36.552+0000 7fe8367fc700 1 -- 172.21.15.133:0/3988805513 <== mon.0 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 2) v1 ==== 43124+0+0 (secure 0 0 0) 0x7fe83c0108c0 con 0x7fe840102770 2024-02-20T07:48:37.205 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:36.552+0000 7fe8367fc700 1 -- 172.21.15.133:0/3988805513 <== mon.0 v2:172.21.15.133:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7fe83c04a1c0 con 0x7fe840102770 2024-02-20T07:48:37.205 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:36.552+0000 7fe847f86700 1 -- 172.21.15.133:0/3988805513 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe84005f5c0 con 0x7fe840102770 2024-02-20T07:48:37.205 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:36.554+0000 7fe8367fc700 1 -- 172.21.15.133:0/3988805513 <== mon.0 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71349 (secure 0 0 0) 0x7fe83c010b70 con 0x7fe840102770 2024-02-20T07:48:37.205 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:36.624+0000 7fe847f86700 1 -- 172.21.15.133:0/3988805513 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7fe840112b50 con 0x7fe840102770 2024-02-20T07:48:37.205 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:36.624+0000 7fe8367fc700 1 -- 172.21.15.133:0/3988805513 <== mon.0 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1249 (secure 0 0 0) 0x7fe83c022340 con 0x7fe840102770 2024-02-20T07:48:37.205 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:36.625+0000 7fe847f86700 1 -- 172.21.15.133:0/3988805513 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fe840102770 msgr2=0x7fe840111500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:48:37.205 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:36.625+0000 7fe847f86700 1 --2- 172.21.15.133:0/3988805513 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fe840102770 0x7fe840111500 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fe83c00b440 tx=0x7fe83c009250 comp rx=0 tx=0).stop 2024-02-20T07:48:37.205 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:36.625+0000 7fe847f86700 1 -- 172.21.15.133:0/3988805513 shutdown_connections 2024-02-20T07:48:37.205 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:36.625+0000 7fe847f86700 1 --2- 172.21.15.133:0/3988805513 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fe840102770 0x7fe840111500 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:48:37.205 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:36.625+0000 7fe847f86700 1 -- 172.21.15.133:0/3988805513 >> 172.21.15.133:0/3988805513 conn(0x7fe8400fded0 msgr2=0x7fe8400fec70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:48:37.205 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:36.625+0000 7fe847f86700 1 -- 172.21.15.133:0/3988805513 shutdown_connections 2024-02-20T07:48:37.205 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:36.625+0000 7fe847f86700 1 -- 172.21.15.133:0/3988805513 wait complete. 2024-02-20T07:48:37.205 INFO:teuthology.orchestra.run.smithi133.stdout:mgr not available, waiting (3/15)... 2024-02-20T07:48:37.992 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:48:37 smithi133 ceph-mon[29117]: from='client.? 172.21.15.133:0/3988805513' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-02-20T07:48:37.992 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:48:37 smithi133 ceph-mon[29117]: mgrmap e3: smithi133.mrvcbp(active, since 1.00631s) 2024-02-20T07:48:39.875 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:48:39 smithi133 ceph-mon[29117]: mgrmap e4: smithi133.mrvcbp(active, since 2s) 2024-02-20T07:48:40.411 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stdout 2024-02-20T07:48:40.411 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stdout { 2024-02-20T07:48:40.412 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stdout "fsid": "4537327c-cfc4-11ee-95bb-87774f69a715", 2024-02-20T07:48:40.412 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stdout "health": { 2024-02-20T07:48:40.412 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-02-20T07:48:40.412 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-02-20T07:48:40.412 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-02-20T07:48:40.412 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stdout }, 2024-02-20T07:48:40.412 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-02-20T07:48:40.412 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-02-20T07:48:40.412 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stdout 0 2024-02-20T07:48:40.412 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stdout ], 2024-02-20T07:48:40.412 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-02-20T07:48:40.412 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stdout "smithi133" 2024-02-20T07:48:40.412 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stdout ], 2024-02-20T07:48:40.413 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stdout "quorum_age": 13, 2024-02-20T07:48:40.413 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stdout "monmap": { 2024-02-20T07:48:40.413 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-02-20T07:48:40.413 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "quincy", 2024-02-20T07:48:40.413 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-02-20T07:48:40.413 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stdout }, 2024-02-20T07:48:40.413 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-02-20T07:48:40.413 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-02-20T07:48:40.413 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-02-20T07:48:40.413 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-02-20T07:48:40.413 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-02-20T07:48:40.424 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-02-20T07:48:40.424 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-02-20T07:48:40.424 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-02-20T07:48:40.424 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stdout }, 2024-02-20T07:48:40.424 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-02-20T07:48:40.424 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-02-20T07:48:40.424 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-02-20T07:48:40.424 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-02-20T07:48:40.424 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-02-20T07:48:40.425 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-02-20T07:48:40.425 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-02-20T07:48:40.425 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-02-20T07:48:40.425 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-02-20T07:48:40.425 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stdout }, 2024-02-20T07:48:40.425 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-02-20T07:48:40.425 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-02-20T07:48:40.425 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-02-20T07:48:40.425 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-02-20T07:48:40.425 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stdout }, 2024-02-20T07:48:40.425 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-02-20T07:48:40.425 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stdout "available": true, 2024-02-20T07:48:40.425 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-02-20T07:48:40.425 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stdout "modules": [ 2024-02-20T07:48:40.425 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stdout "iostat", 2024-02-20T07:48:40.425 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stdout "nfs", 2024-02-20T07:48:40.426 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stdout "restful" 2024-02-20T07:48:40.426 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stdout ], 2024-02-20T07:48:40.426 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stdout "services": {} 2024-02-20T07:48:40.426 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stdout }, 2024-02-20T07:48:40.426 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-02-20T07:48:40.426 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-02-20T07:48:40.426 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stdout "modified": "2024-02-20T07:48:13.550997+0000", 2024-02-20T07:48:40.426 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stdout "services": {} 2024-02-20T07:48:40.426 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stdout }, 2024-02-20T07:48:40.426 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-02-20T07:48:40.426 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stdout } 2024-02-20T07:48:40.426 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:39.609+0000 7f5bbbb83700 1 Processor -- start 2024-02-20T07:48:40.426 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:39.609+0000 7f5bbbb83700 1 -- start start 2024-02-20T07:48:40.426 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:39.609+0000 7f5bbbb83700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f5bb4100e80 0x7f5bb4101290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:48:40.426 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:39.609+0000 7f5bbbb83700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5bb41017d0 con 0x7f5bb4100e80 2024-02-20T07:48:40.427 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:39.609+0000 7f5bb991f700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f5bb4100e80 0x7f5bb4101290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:48:40.427 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:39.609+0000 7f5bb991f700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f5bb4100e80 0x7f5bb4101290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.133:53366/0 (socket says 172.21.15.133:53366) 2024-02-20T07:48:40.427 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:39.609+0000 7f5bb991f700 1 -- 172.21.15.133:0/3065561278 learned_addr learned my addr 172.21.15.133:0/3065561278 (peer_addr_for_me v2:172.21.15.133:0/0) 2024-02-20T07:48:40.427 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:39.610+0000 7f5bb991f700 1 -- 172.21.15.133:0/3065561278 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5bb4101910 con 0x7f5bb4100e80 2024-02-20T07:48:40.427 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:39.610+0000 7f5bb991f700 1 --2- 172.21.15.133:0/3065561278 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f5bb4100e80 0x7f5bb4101290 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f5bb00098e0 tx=0x7f5bb0009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=310cf0afcce8587d server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:48:40.427 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:39.610+0000 7f5bb891d700 1 -- 172.21.15.133:0/3065561278 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5bb000b860 con 0x7f5bb4100e80 2024-02-20T07:48:40.427 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:39.610+0000 7f5bb891d700 1 -- 172.21.15.133:0/3065561278 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f5bb000be50 con 0x7f5bb4100e80 2024-02-20T07:48:40.427 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:39.610+0000 7f5bb891d700 1 -- 172.21.15.133:0/3065561278 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5bb0010380 con 0x7f5bb4100e80 2024-02-20T07:48:40.427 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:39.610+0000 7f5bbbb83700 1 -- 172.21.15.133:0/3065561278 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f5bb4100e80 msgr2=0x7f5bb4101290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:48:40.428 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:39.610+0000 7f5bbbb83700 1 --2- 172.21.15.133:0/3065561278 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f5bb4100e80 0x7f5bb4101290 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f5bb00098e0 tx=0x7f5bb0009250 comp rx=0 tx=0).stop 2024-02-20T07:48:40.428 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:39.610+0000 7f5bbbb83700 1 -- 172.21.15.133:0/3065561278 shutdown_connections 2024-02-20T07:48:40.428 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:39.611+0000 7f5bbbb83700 1 --2- 172.21.15.133:0/3065561278 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f5bb4100e80 0x7f5bb4101290 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:48:40.428 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:39.611+0000 7f5bbbb83700 1 -- 172.21.15.133:0/3065561278 >> 172.21.15.133:0/3065561278 conn(0x7f5bb40fc580 msgr2=0x7f5bb40fe990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:48:40.428 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:39.611+0000 7f5bbbb83700 1 -- 172.21.15.133:0/3065561278 shutdown_connections 2024-02-20T07:48:40.428 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:39.611+0000 7f5bbbb83700 1 -- 172.21.15.133:0/3065561278 wait complete. 2024-02-20T07:48:40.428 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:39.611+0000 7f5bbbb83700 1 Processor -- start 2024-02-20T07:48:40.428 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:39.611+0000 7f5bbbb83700 1 -- start start 2024-02-20T07:48:40.428 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:39.611+0000 7f5bbbb83700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f5bb4100e80 0x7f5bb41117e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:48:40.429 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:39.611+0000 7f5bbbb83700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5bb4111d20 con 0x7f5bb4100e80 2024-02-20T07:48:40.429 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:39.611+0000 7f5bb991f700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f5bb4100e80 0x7f5bb41117e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:48:40.429 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:39.612+0000 7f5bb991f700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f5bb4100e80 0x7f5bb41117e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.133:53372/0 (socket says 172.21.15.133:53372) 2024-02-20T07:48:40.429 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:39.612+0000 7f5bb991f700 1 -- 172.21.15.133:0/4090938878 learned_addr learned my addr 172.21.15.133:0/4090938878 (peer_addr_for_me v2:172.21.15.133:0/0) 2024-02-20T07:48:40.429 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:39.612+0000 7f5bb991f700 1 -- 172.21.15.133:0/4090938878 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5bb00094f0 con 0x7f5bb4100e80 2024-02-20T07:48:40.429 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:39.612+0000 7f5bb991f700 1 --2- 172.21.15.133:0/4090938878 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f5bb4100e80 0x7f5bb41117e0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f5bb000b440 tx=0x7f5bb0009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:48:40.429 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:39.612+0000 7f5baa7fc700 1 -- 172.21.15.133:0/4090938878 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5bb0010760 con 0x7f5bb4100e80 2024-02-20T07:48:40.429 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:39.612+0000 7f5bbbb83700 1 -- 172.21.15.133:0/4090938878 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5bb4111f20 con 0x7f5bb4100e80 2024-02-20T07:48:40.429 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:39.612+0000 7f5baa7fc700 1 -- 172.21.15.133:0/4090938878 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f5bb0010d50 con 0x7f5bb4100e80 2024-02-20T07:48:40.429 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:39.612+0000 7f5baa7fc700 1 -- 172.21.15.133:0/4090938878 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5bb0024d80 con 0x7f5bb4100e80 2024-02-20T07:48:40.429 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:39.612+0000 7f5bbbb83700 1 -- 172.21.15.133:0/4090938878 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5bb4112bc0 con 0x7f5bb4100e80 2024-02-20T07:48:40.429 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:39.613+0000 7f5baa7fc700 1 -- 172.21.15.133:0/4090938878 <== mon.0 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 4) v1 ==== 43357+0+0 (secure 0 0 0) 0x7f5bb001a470 con 0x7f5bb4100e80 2024-02-20T07:48:40.429 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:39.613+0000 7f5bbbb83700 1 -- 172.21.15.133:0/4090938878 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5bb405f5c0 con 0x7f5bb4100e80 2024-02-20T07:48:40.429 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:39.613+0000 7f5baa7fc700 1 --2- 172.21.15.133:0/4090938878 >> [v2:172.21.15.133:6800/3480413474,v1:172.21.15.133:6801/3480413474] conn(0x7f5ba00366e0 0x7f5ba0038b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:48:40.430 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:39.613+0000 7f5baa7fc700 1 -- 172.21.15.133:0/4090938878 <== mon.0 v2:172.21.15.133:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7f5bb004a5c0 con 0x7f5bb4100e80 2024-02-20T07:48:40.430 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:39.613+0000 7f5bb911e700 1 --2- 172.21.15.133:0/4090938878 >> [v2:172.21.15.133:6800/3480413474,v1:172.21.15.133:6801/3480413474] conn(0x7f5ba00366e0 0x7f5ba0038b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:48:40.430 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:39.614+0000 7f5bb911e700 1 --2- 172.21.15.133:0/4090938878 >> [v2:172.21.15.133:6800/3480413474,v1:172.21.15.133:6801/3480413474] conn(0x7f5ba00366e0 0x7f5ba0038b90 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f5ba4009990 tx=0x7f5ba4006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:48:40.430 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:39.618+0000 7f5baa7fc700 1 -- 172.21.15.133:0/4090938878 <== mon.0 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f5bb001a720 con 0x7f5bb4100e80 2024-02-20T07:48:40.430 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:39.872+0000 7f5bbbb83700 1 -- 172.21.15.133:0/4090938878 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f5bb404bed0 con 0x7f5bb4100e80 2024-02-20T07:48:40.430 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:39.872+0000 7f5baa7fc700 1 -- 172.21.15.133:0/4090938878 <== mon.0 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1248 (secure 0 0 0) 0x7f5bb0018350 con 0x7f5bb4100e80 2024-02-20T07:48:40.430 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:39.875+0000 7f5bbbb83700 1 -- 172.21.15.133:0/4090938878 >> [v2:172.21.15.133:6800/3480413474,v1:172.21.15.133:6801/3480413474] conn(0x7f5ba00366e0 msgr2=0x7f5ba0038b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:48:40.430 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:39.875+0000 7f5bbbb83700 1 --2- 172.21.15.133:0/4090938878 >> [v2:172.21.15.133:6800/3480413474,v1:172.21.15.133:6801/3480413474] conn(0x7f5ba00366e0 0x7f5ba0038b90 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f5ba4009990 tx=0x7f5ba4006e40 comp rx=0 tx=0).stop 2024-02-20T07:48:40.430 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:39.875+0000 7f5bbbb83700 1 -- 172.21.15.133:0/4090938878 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f5bb4100e80 msgr2=0x7f5bb41117e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:48:40.430 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:39.875+0000 7f5bbbb83700 1 --2- 172.21.15.133:0/4090938878 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f5bb4100e80 0x7f5bb41117e0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f5bb000b440 tx=0x7f5bb0009250 comp rx=0 tx=0).stop 2024-02-20T07:48:40.430 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:39.875+0000 7f5bbbb83700 1 -- 172.21.15.133:0/4090938878 shutdown_connections 2024-02-20T07:48:40.430 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:39.875+0000 7f5bbbb83700 1 --2- 172.21.15.133:0/4090938878 >> [v2:172.21.15.133:6800/3480413474,v1:172.21.15.133:6801/3480413474] conn(0x7f5ba00366e0 0x7f5ba0038b90 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:48:40.430 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:39.875+0000 7f5bbbb83700 1 --2- 172.21.15.133:0/4090938878 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f5bb4100e80 0x7f5bb41117e0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:48:40.430 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:39.875+0000 7f5bbbb83700 1 -- 172.21.15.133:0/4090938878 >> 172.21.15.133:0/4090938878 conn(0x7f5bb40fc580 msgr2=0x7f5bb41063b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:48:40.430 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:39.875+0000 7f5bbbb83700 1 -- 172.21.15.133:0/4090938878 shutdown_connections 2024-02-20T07:48:40.431 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:39.875+0000 7f5bbbb83700 1 -- 172.21.15.133:0/4090938878 wait complete. 2024-02-20T07:48:40.431 INFO:teuthology.orchestra.run.smithi133.stdout:mgr is available 2024-02-20T07:48:40.976 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:48:40 smithi133 ceph-mon[29117]: from='client.? 172.21.15.133:0/4090938878' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-02-20T07:48:41.559 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stdout 2024-02-20T07:48:41.560 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stdout [global] 2024-02-20T07:48:41.560 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stdout fsid = 4537327c-cfc4-11ee-95bb-87774f69a715 2024-02-20T07:48:41.560 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-02-20T07:48:41.560 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-02-20T07:48:41.560 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-02-20T07:48:41.560 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-02-20T07:48:41.560 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stdout 2024-02-20T07:48:41.560 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stdout [mgr] 2024-02-20T07:48:41.560 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-02-20T07:48:41.560 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stdout 2024-02-20T07:48:41.560 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stdout [osd] 2024-02-20T07:48:41.560 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-02-20T07:48:41.560 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-02-20T07:48:41.560 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-02-20T07:48:41.561 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-02-20T07:48:41.561 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-02-20T07:48:41.561 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:40.783+0000 7fb344e37700 1 Processor -- start 2024-02-20T07:48:41.561 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:40.784+0000 7fb344e37700 1 -- start start 2024-02-20T07:48:41.561 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:40.784+0000 7fb344e37700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb340102770 0x7fb340102b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:48:41.561 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:40.784+0000 7fb344e37700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3401030c0 con 0x7fb340102770 2024-02-20T07:48:41.561 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:40.784+0000 7fb33e59c700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb340102770 0x7fb340102b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:48:41.561 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:40.784+0000 7fb33e59c700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb340102770 0x7fb340102b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.133:53378/0 (socket says 172.21.15.133:53378) 2024-02-20T07:48:41.561 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:40.784+0000 7fb33e59c700 1 -- 172.21.15.133:0/414774304 learned_addr learned my addr 172.21.15.133:0/414774304 (peer_addr_for_me v2:172.21.15.133:0/0) 2024-02-20T07:48:41.561 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:40.784+0000 7fb33e59c700 1 -- 172.21.15.133:0/414774304 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb340103200 con 0x7fb340102770 2024-02-20T07:48:41.561 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:40.784+0000 7fb33e59c700 1 --2- 172.21.15.133:0/414774304 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb340102770 0x7fb340102b80 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fb3340098e0 tx=0x7fb334009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ee32fdb0f292aea2 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:48:41.561 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:40.785+0000 7fb33d59a700 1 -- 172.21.15.133:0/414774304 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb33400b860 con 0x7fb340102770 2024-02-20T07:48:41.561 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:40.785+0000 7fb33d59a700 1 -- 172.21.15.133:0/414774304 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7fb33400be50 con 0x7fb340102770 2024-02-20T07:48:41.561 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:40.785+0000 7fb33d59a700 1 -- 172.21.15.133:0/414774304 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb334010380 con 0x7fb340102770 2024-02-20T07:48:41.561 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:40.785+0000 7fb344e37700 1 -- 172.21.15.133:0/414774304 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb340102770 msgr2=0x7fb340102b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:48:41.562 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:40.785+0000 7fb344e37700 1 --2- 172.21.15.133:0/414774304 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb340102770 0x7fb340102b80 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fb3340098e0 tx=0x7fb334009250 comp rx=0 tx=0).stop 2024-02-20T07:48:41.562 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:40.785+0000 7fb344e37700 1 -- 172.21.15.133:0/414774304 shutdown_connections 2024-02-20T07:48:41.562 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:40.785+0000 7fb344e37700 1 --2- 172.21.15.133:0/414774304 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb340102770 0x7fb340102b80 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:48:41.562 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:40.785+0000 7fb344e37700 1 -- 172.21.15.133:0/414774304 >> 172.21.15.133:0/414774304 conn(0x7fb3400fded0 msgr2=0x7fb340100300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:48:41.562 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:40.785+0000 7fb344e37700 1 -- 172.21.15.133:0/414774304 shutdown_connections 2024-02-20T07:48:41.562 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:40.785+0000 7fb344e37700 1 -- 172.21.15.133:0/414774304 wait complete. 2024-02-20T07:48:41.562 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:40.786+0000 7fb344e37700 1 Processor -- start 2024-02-20T07:48:41.562 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:40.786+0000 7fb344e37700 1 -- start start 2024-02-20T07:48:41.562 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:40.786+0000 7fb344e37700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb340102770 0x7fb340111560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:48:41.562 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:40.786+0000 7fb344e37700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb340111aa0 con 0x7fb340102770 2024-02-20T07:48:41.562 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:40.786+0000 7fb33e59c700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb340102770 0x7fb340111560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:48:41.563 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:40.786+0000 7fb33e59c700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb340102770 0x7fb340111560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.133:53382/0 (socket says 172.21.15.133:53382) 2024-02-20T07:48:41.563 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:40.786+0000 7fb33e59c700 1 -- 172.21.15.133:0/4069549869 learned_addr learned my addr 172.21.15.133:0/4069549869 (peer_addr_for_me v2:172.21.15.133:0/0) 2024-02-20T07:48:41.563 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:40.786+0000 7fb33e59c700 1 -- 172.21.15.133:0/4069549869 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3340094f0 con 0x7fb340102770 2024-02-20T07:48:41.563 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:40.786+0000 7fb33e59c700 1 --2- 172.21.15.133:0/4069549869 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb340102770 0x7fb340111560 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fb334000c00 tx=0x7fb334009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:48:41.563 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:40.787+0000 7fb32effd700 1 -- 172.21.15.133:0/4069549869 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb3340107c0 con 0x7fb340102770 2024-02-20T07:48:41.563 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:40.787+0000 7fb32effd700 1 -- 172.21.15.133:0/4069549869 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7fb334010db0 con 0x7fb340102770 2024-02-20T07:48:41.563 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:40.787+0000 7fb344e37700 1 -- 172.21.15.133:0/4069549869 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb340111ca0 con 0x7fb340102770 2024-02-20T07:48:41.563 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:40.787+0000 7fb32effd700 1 -- 172.21.15.133:0/4069549869 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb33401ad00 con 0x7fb340102770 2024-02-20T07:48:41.563 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:40.787+0000 7fb344e37700 1 -- 172.21.15.133:0/4069549869 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb340112940 con 0x7fb340102770 2024-02-20T07:48:41.563 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:40.787+0000 7fb32effd700 1 -- 172.21.15.133:0/4069549869 <== mon.0 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 4) v1 ==== 43357+0+0 (secure 0 0 0) 0x7fb334010920 con 0x7fb340102770 2024-02-20T07:48:41.563 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:40.788+0000 7fb32effd700 1 --2- 172.21.15.133:0/4069549869 >> [v2:172.21.15.133:6800/3480413474,v1:172.21.15.133:6801/3480413474] conn(0x7fb328036740 0x7fb328038bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:48:41.564 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:40.788+0000 7fb32effd700 1 -- 172.21.15.133:0/4069549869 <== mon.0 v2:172.21.15.133:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7fb33404a630 con 0x7fb340102770 2024-02-20T07:48:41.564 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:40.788+0000 7fb344e37700 1 -- 172.21.15.133:0/4069549869 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb34005f5c0 con 0x7fb340102770 2024-02-20T07:48:41.564 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:40.788+0000 7fb33dd9b700 1 --2- 172.21.15.133:0/4069549869 >> [v2:172.21.15.133:6800/3480413474,v1:172.21.15.133:6801/3480413474] conn(0x7fb328036740 0x7fb328038bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:48:41.564 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:40.788+0000 7fb33dd9b700 1 --2- 172.21.15.133:0/4069549869 >> [v2:172.21.15.133:6800/3480413474,v1:172.21.15.133:6801/3480413474] conn(0x7fb328036740 0x7fb328038bf0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fb330009990 tx=0x7fb330006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:48:41.564 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:40.792+0000 7fb32effd700 1 -- 172.21.15.133:0/4069549869 <== mon.0 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7fb334010bd0 con 0x7fb340102770 2024-02-20T07:48:41.564 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:40.973+0000 7fb344e37700 1 -- 172.21.15.133:0/4069549869 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) v1 -- 0x7fb34004bed0 con 0x7fb340102770 2024-02-20T07:48:41.565 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:40.974+0000 7fb32effd700 1 -- 172.21.15.133:0/4069549869 <== mon.0 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v3) v1 ==== 70+0+373 (secure 0 0 0) 0x7fb334023810 con 0x7fb340102770 2024-02-20T07:48:41.565 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:40.976+0000 7fb344e37700 1 -- 172.21.15.133:0/4069549869 >> [v2:172.21.15.133:6800/3480413474,v1:172.21.15.133:6801/3480413474] conn(0x7fb328036740 msgr2=0x7fb328038bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:48:41.565 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:40.976+0000 7fb344e37700 1 --2- 172.21.15.133:0/4069549869 >> [v2:172.21.15.133:6800/3480413474,v1:172.21.15.133:6801/3480413474] conn(0x7fb328036740 0x7fb328038bf0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fb330009990 tx=0x7fb330006e40 comp rx=0 tx=0).stop 2024-02-20T07:48:41.565 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:40.976+0000 7fb344e37700 1 -- 172.21.15.133:0/4069549869 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb340102770 msgr2=0x7fb340111560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:48:41.565 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:40.976+0000 7fb344e37700 1 --2- 172.21.15.133:0/4069549869 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb340102770 0x7fb340111560 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fb334000c00 tx=0x7fb334009250 comp rx=0 tx=0).stop 2024-02-20T07:48:41.565 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:40.976+0000 7fb344e37700 1 -- 172.21.15.133:0/4069549869 shutdown_connections 2024-02-20T07:48:41.565 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:40.976+0000 7fb344e37700 1 --2- 172.21.15.133:0/4069549869 >> [v2:172.21.15.133:6800/3480413474,v1:172.21.15.133:6801/3480413474] conn(0x7fb328036740 0x7fb328038bf0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:48:41.566 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:40.976+0000 7fb344e37700 1 --2- 172.21.15.133:0/4069549869 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb340102770 0x7fb340111560 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:48:41.566 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:40.976+0000 7fb344e37700 1 -- 172.21.15.133:0/4069549869 >> 172.21.15.133:0/4069549869 conn(0x7fb3400fded0 msgr2=0x7fb3400ff570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:48:41.566 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:40.977+0000 7fb344e37700 1 -- 172.21.15.133:0/4069549869 shutdown_connections 2024-02-20T07:48:41.566 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:40.977+0000 7fb344e37700 1 -- 172.21.15.133:0/4069549869 wait complete. 2024-02-20T07:48:41.566 INFO:teuthology.orchestra.run.smithi133.stdout:Enabling cephadm module... 2024-02-20T07:48:41.865 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:48:41 smithi133 ceph-mon[29117]: from='client.? 172.21.15.133:0/4069549869' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-02-20T07:48:42.851 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:48:42 smithi133 ceph-mon[29117]: from='client.? 172.21.15.133:0/3656436405' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-02-20T07:48:43.161 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:41.961+0000 7f750b95d700 1 Processor -- start 2024-02-20T07:48:43.161 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:41.961+0000 7f750b95d700 1 -- start start 2024-02-20T07:48:43.161 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:41.961+0000 7f750b95d700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f7504103e40 0x7f7504104250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:48:43.161 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:41.961+0000 7f750b95d700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7504104790 con 0x7f7504103e40 2024-02-20T07:48:43.161 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:41.961+0000 7f75096f9700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f7504103e40 0x7f7504104250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:48:43.161 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:41.961+0000 7f75096f9700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f7504103e40 0x7f7504104250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.133:53398/0 (socket says 172.21.15.133:53398) 2024-02-20T07:48:43.161 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:41.961+0000 7f75096f9700 1 -- 172.21.15.133:0/1942553826 learned_addr learned my addr 172.21.15.133:0/1942553826 (peer_addr_for_me v2:172.21.15.133:0/0) 2024-02-20T07:48:43.162 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:41.961+0000 7f75096f9700 1 -- 172.21.15.133:0/1942553826 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75041048d0 con 0x7f7504103e40 2024-02-20T07:48:43.162 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:41.962+0000 7f75096f9700 1 --2- 172.21.15.133:0/1942553826 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f7504103e40 0x7f7504104250 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f75000098e0 tx=0x7f7500009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=64008f3b317646e0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:48:43.162 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:41.962+0000 7f74fbfff700 1 -- 172.21.15.133:0/1942553826 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f750000b860 con 0x7f7504103e40 2024-02-20T07:48:43.162 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:41.962+0000 7f74fbfff700 1 -- 172.21.15.133:0/1942553826 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f750000be50 con 0x7f7504103e40 2024-02-20T07:48:43.162 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:41.962+0000 7f74fbfff700 1 -- 172.21.15.133:0/1942553826 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7500010380 con 0x7f7504103e40 2024-02-20T07:48:43.162 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:41.962+0000 7f750b95d700 1 -- 172.21.15.133:0/1942553826 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f7504103e40 msgr2=0x7f7504104250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:48:43.162 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:41.962+0000 7f750b95d700 1 --2- 172.21.15.133:0/1942553826 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f7504103e40 0x7f7504104250 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f75000098e0 tx=0x7f7500009250 comp rx=0 tx=0).stop 2024-02-20T07:48:43.162 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:41.962+0000 7f750b95d700 1 -- 172.21.15.133:0/1942553826 shutdown_connections 2024-02-20T07:48:43.163 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:41.962+0000 7f750b95d700 1 --2- 172.21.15.133:0/1942553826 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f7504103e40 0x7f7504104250 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:48:43.163 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:41.962+0000 7f750b95d700 1 -- 172.21.15.133:0/1942553826 >> 172.21.15.133:0/1942553826 conn(0x7f75040fc580 msgr2=0x7f75040fe990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:48:43.163 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:41.962+0000 7f750b95d700 1 -- 172.21.15.133:0/1942553826 shutdown_connections 2024-02-20T07:48:43.163 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:41.962+0000 7f750b95d700 1 -- 172.21.15.133:0/1942553826 wait complete. 2024-02-20T07:48:43.163 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:41.963+0000 7f750b95d700 1 Processor -- start 2024-02-20T07:48:43.163 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:41.963+0000 7f750b95d700 1 -- start start 2024-02-20T07:48:43.163 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:41.963+0000 7f750b95d700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f7504103e40 0x7f7504103680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:48:43.163 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:41.963+0000 7f750b95d700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7504101b00 con 0x7f7504103e40 2024-02-20T07:48:43.163 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:41.963+0000 7f75096f9700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f7504103e40 0x7f7504103680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:48:43.163 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:41.963+0000 7f75096f9700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f7504103e40 0x7f7504103680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.133:53400/0 (socket says 172.21.15.133:53400) 2024-02-20T07:48:43.163 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:41.963+0000 7f75096f9700 1 -- 172.21.15.133:0/3656436405 learned_addr learned my addr 172.21.15.133:0/3656436405 (peer_addr_for_me v2:172.21.15.133:0/0) 2024-02-20T07:48:43.164 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:41.963+0000 7f75096f9700 1 -- 172.21.15.133:0/3656436405 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75000094f0 con 0x7f7504103e40 2024-02-20T07:48:43.164 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:41.964+0000 7f75096f9700 1 --2- 172.21.15.133:0/3656436405 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f7504103e40 0x7f7504103680 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f750000b440 tx=0x7f7500009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:48:43.164 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:41.964+0000 7f74f9ffb700 1 -- 172.21.15.133:0/3656436405 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f75000107c0 con 0x7f7504103e40 2024-02-20T07:48:43.164 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:41.964+0000 7f74f9ffb700 1 -- 172.21.15.133:0/3656436405 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f7500010db0 con 0x7f7504103e40 2024-02-20T07:48:43.164 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:41.964+0000 7f750b95d700 1 -- 172.21.15.133:0/3656436405 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7504101d00 con 0x7f7504103e40 2024-02-20T07:48:43.164 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:41.964+0000 7f74f9ffb700 1 -- 172.21.15.133:0/3656436405 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f750001ada0 con 0x7f7504103e40 2024-02-20T07:48:43.164 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:41.964+0000 7f750b95d700 1 -- 172.21.15.133:0/3656436405 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7504102190 con 0x7f7504103e40 2024-02-20T07:48:43.164 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:41.965+0000 7f74f9ffb700 1 -- 172.21.15.133:0/3656436405 <== mon.0 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 4) v1 ==== 43357+0+0 (secure 0 0 0) 0x7f7500024480 con 0x7f7504103e40 2024-02-20T07:48:43.164 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:41.965+0000 7f74f9ffb700 1 --2- 172.21.15.133:0/3656436405 >> [v2:172.21.15.133:6800/3480413474,v1:172.21.15.133:6801/3480413474] conn(0x7f74f0036740 0x7f74f0038bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:48:43.164 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:41.965+0000 7f750b95d700 1 -- 172.21.15.133:0/3656436405 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f750410b6b0 con 0x7f7504103e40 2024-02-20T07:48:43.164 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:41.965+0000 7f74f9ffb700 1 -- 172.21.15.133:0/3656436405 <== mon.0 v2:172.21.15.133:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7f750004af00 con 0x7f7504103e40 2024-02-20T07:48:43.165 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:41.965+0000 7f7508ef8700 1 --2- 172.21.15.133:0/3656436405 >> [v2:172.21.15.133:6800/3480413474,v1:172.21.15.133:6801/3480413474] conn(0x7f74f0036740 0x7f74f0038bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:48:43.165 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:41.966+0000 7f7508ef8700 1 --2- 172.21.15.133:0/3656436405 >> [v2:172.21.15.133:6800/3480413474,v1:172.21.15.133:6801/3480413474] conn(0x7f74f0036740 0x7f74f0038bf0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f74f4009990 tx=0x7f74f4006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:48:43.165 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:41.970+0000 7f74f9ffb700 1 -- 172.21.15.133:0/3656436405 <== mon.0 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f750002e440 con 0x7f7504103e40 2024-02-20T07:48:43.165 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:42.200+0000 7f750b95d700 1 -- 172.21.15.133:0/3656436405 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "cephadm"} v 0) v1 -- 0x7f7504064ad0 con 0x7f7504103e40 2024-02-20T07:48:43.165 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:42.588+0000 7f74f9ffb700 1 -- 172.21.15.133:0/3656436405 <== mon.0 v2:172.21.15.133:3300/0 7 ==== mgrmap(e 5) v1 ==== 43368+0+0 (secure 0 0 0) 0x7f750002e7c0 con 0x7f7504103e40 2024-02-20T07:48:43.165 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:42.589+0000 7f74f9ffb700 1 -- 172.21.15.133:0/3656436405 <== mon.0 v2:172.21.15.133:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "cephadm"}]=0 v5) v1 ==== 86+0+0 (secure 0 0 0) 0x7f7500019c50 con 0x7f7504103e40 2024-02-20T07:48:43.165 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:42.591+0000 7f750b95d700 1 -- 172.21.15.133:0/3656436405 >> [v2:172.21.15.133:6800/3480413474,v1:172.21.15.133:6801/3480413474] conn(0x7f74f0036740 msgr2=0x7f74f0038bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:48:43.165 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:42.591+0000 7f750b95d700 1 --2- 172.21.15.133:0/3656436405 >> [v2:172.21.15.133:6800/3480413474,v1:172.21.15.133:6801/3480413474] conn(0x7f74f0036740 0x7f74f0038bf0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f74f4009990 tx=0x7f74f4006e40 comp rx=0 tx=0).stop 2024-02-20T07:48:43.165 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:42.591+0000 7f750b95d700 1 -- 172.21.15.133:0/3656436405 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f7504103e40 msgr2=0x7f7504103680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:48:43.165 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:42.591+0000 7f750b95d700 1 --2- 172.21.15.133:0/3656436405 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f7504103e40 0x7f7504103680 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f750000b440 tx=0x7f7500009250 comp rx=0 tx=0).stop 2024-02-20T07:48:43.165 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:42.591+0000 7f750b95d700 1 -- 172.21.15.133:0/3656436405 shutdown_connections 2024-02-20T07:48:43.165 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:42.591+0000 7f750b95d700 1 --2- 172.21.15.133:0/3656436405 >> [v2:172.21.15.133:6800/3480413474,v1:172.21.15.133:6801/3480413474] conn(0x7f74f0036740 0x7f74f0038bf0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:48:43.166 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:42.591+0000 7f750b95d700 1 --2- 172.21.15.133:0/3656436405 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f7504103e40 0x7f7504103680 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:48:43.166 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:42.591+0000 7f750b95d700 1 -- 172.21.15.133:0/3656436405 >> 172.21.15.133:0/3656436405 conn(0x7f75040fc580 msgr2=0x7f75040fd3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:48:43.166 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:42.591+0000 7f750b95d700 1 -- 172.21.15.133:0/3656436405 shutdown_connections 2024-02-20T07:48:43.166 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:42.591+0000 7f750b95d700 1 -- 172.21.15.133:0/3656436405 wait complete. 2024-02-20T07:48:43.742 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:48:43 smithi133 ceph-mon[29117]: from='client.? 172.21.15.133:0/3656436405' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-02-20T07:48:43.743 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:48:43 smithi133 ceph-mon[29117]: mgrmap e5: smithi133.mrvcbp(active, since 6s) 2024-02-20T07:48:44.417 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stdout { 2024-02-20T07:48:44.417 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stdout "epoch": 5, 2024-02-20T07:48:44.417 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stdout "available": true, 2024-02-20T07:48:44.418 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stdout "active_name": "smithi133.mrvcbp", 2024-02-20T07:48:44.418 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-02-20T07:48:44.418 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stdout } 2024-02-20T07:48:44.418 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:43.548+0000 7f3e9e7e7700 1 Processor -- start 2024-02-20T07:48:44.418 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:43.548+0000 7f3e9e7e7700 1 -- start start 2024-02-20T07:48:44.418 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:43.548+0000 7f3e9e7e7700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f3e98078d20 0x7f3e98077230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:48:44.418 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:43.548+0000 7f3e9e7e7700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e98079130 con 0x7f3e98078d20 2024-02-20T07:48:44.418 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:43.549+0000 7f3e97fff700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f3e98078d20 0x7f3e98077230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:48:44.418 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:43.549+0000 7f3e97fff700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f3e98078d20 0x7f3e98077230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.133:53424/0 (socket says 172.21.15.133:53424) 2024-02-20T07:48:44.418 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:43.549+0000 7f3e97fff700 1 -- 172.21.15.133:0/132592866 learned_addr learned my addr 172.21.15.133:0/132592866 (peer_addr_for_me v2:172.21.15.133:0/0) 2024-02-20T07:48:44.419 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:43.549+0000 7f3e97fff700 1 -- 172.21.15.133:0/132592866 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e98077770 con 0x7f3e98078d20 2024-02-20T07:48:44.419 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:43.549+0000 7f3e97fff700 1 --2- 172.21.15.133:0/132592866 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f3e98078d20 0x7f3e98077230 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f3e880098e0 tx=0x7f3e88009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2ceef6274509cda9 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:48:44.419 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:43.549+0000 7f3e96ffd700 1 -- 172.21.15.133:0/132592866 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3e8800b860 con 0x7f3e98078d20 2024-02-20T07:48:44.419 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:43.549+0000 7f3e96ffd700 1 -- 172.21.15.133:0/132592866 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f3e8800be50 con 0x7f3e98078d20 2024-02-20T07:48:44.419 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:43.549+0000 7f3e96ffd700 1 -- 172.21.15.133:0/132592866 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3e88010380 con 0x7f3e98078d20 2024-02-20T07:48:44.419 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:43.550+0000 7f3e9e7e7700 1 -- 172.21.15.133:0/132592866 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f3e98078d20 msgr2=0x7f3e98077230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:48:44.419 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:43.550+0000 7f3e9e7e7700 1 --2- 172.21.15.133:0/132592866 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f3e98078d20 0x7f3e98077230 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f3e880098e0 tx=0x7f3e88009250 comp rx=0 tx=0).stop 2024-02-20T07:48:44.419 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:43.550+0000 7f3e9e7e7700 1 -- 172.21.15.133:0/132592866 shutdown_connections 2024-02-20T07:48:44.420 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:43.550+0000 7f3e9e7e7700 1 --2- 172.21.15.133:0/132592866 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f3e98078d20 0x7f3e98077230 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:48:44.420 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:43.550+0000 7f3e9e7e7700 1 -- 172.21.15.133:0/132592866 >> 172.21.15.133:0/132592866 conn(0x7f3e980fded0 msgr2=0x7f3e98100300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:48:44.420 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:43.550+0000 7f3e9e7e7700 1 -- 172.21.15.133:0/132592866 shutdown_connections 2024-02-20T07:48:44.420 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:43.550+0000 7f3e9e7e7700 1 -- 172.21.15.133:0/132592866 wait complete. 2024-02-20T07:48:44.420 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:43.550+0000 7f3e9e7e7700 1 Processor -- start 2024-02-20T07:48:44.420 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:43.551+0000 7f3e9e7e7700 1 -- start start 2024-02-20T07:48:44.420 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:43.551+0000 7f3e9e7e7700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f3e98078d20 0x7f3e98113790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:48:44.420 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:43.551+0000 7f3e9e7e7700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e98113cd0 con 0x7f3e98078d20 2024-02-20T07:48:44.420 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:43.551+0000 7f3e97fff700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f3e98078d20 0x7f3e98113790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:48:44.420 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:43.551+0000 7f3e97fff700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f3e98078d20 0x7f3e98113790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.133:53440/0 (socket says 172.21.15.133:53440) 2024-02-20T07:48:44.420 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:43.551+0000 7f3e97fff700 1 -- 172.21.15.133:0/1409218203 learned_addr learned my addr 172.21.15.133:0/1409218203 (peer_addr_for_me v2:172.21.15.133:0/0) 2024-02-20T07:48:44.420 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:43.551+0000 7f3e97fff700 1 -- 172.21.15.133:0/1409218203 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e880094f0 con 0x7f3e98078d20 2024-02-20T07:48:44.420 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:43.551+0000 7f3e97fff700 1 --2- 172.21.15.133:0/1409218203 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f3e98078d20 0x7f3e98113790 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f3e88006fd0 tx=0x7f3e88009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:48:44.420 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:43.551+0000 7f3e94ff9700 1 -- 172.21.15.133:0/1409218203 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3e880107c0 con 0x7f3e98078d20 2024-02-20T07:48:44.421 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:43.551+0000 7f3e94ff9700 1 -- 172.21.15.133:0/1409218203 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f3e88010db0 con 0x7f3e98078d20 2024-02-20T07:48:44.421 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:43.551+0000 7f3e94ff9700 1 -- 172.21.15.133:0/1409218203 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3e8801add0 con 0x7f3e98078d20 2024-02-20T07:48:44.421 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:43.552+0000 7f3e9e7e7700 1 -- 172.21.15.133:0/1409218203 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3e98113ed0 con 0x7f3e98078d20 2024-02-20T07:48:44.421 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:43.552+0000 7f3e9e7e7700 1 -- 172.21.15.133:0/1409218203 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e98114b70 con 0x7f3e98078d20 2024-02-20T07:48:44.421 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:43.552+0000 7f3e94ff9700 1 -- 172.21.15.133:0/1409218203 <== mon.0 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 5) v1 ==== 43368+0+0 (secure 0 0 0) 0x7f3e88010920 con 0x7f3e98078d20 2024-02-20T07:48:44.421 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:43.553+0000 7f3e94ff9700 1 --2- 172.21.15.133:0/1409218203 >> [v2:172.21.15.133:6800/3480413474,v1:172.21.15.133:6801/3480413474] conn(0x7f3e80036790 0x7f3e80038c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:48:44.421 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:43.553+0000 7f3e94ff9700 1 -- 172.21.15.133:0/1409218203 <== mon.0 v2:172.21.15.133:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7f3e8801e070 con 0x7f3e98078d20 2024-02-20T07:48:44.422 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:43.553+0000 7f3e977fe700 1 -- 172.21.15.133:0/1409218203 >> [v2:172.21.15.133:6800/3480413474,v1:172.21.15.133:6801/3480413474] conn(0x7f3e80036790 msgr2=0x7f3e80038c40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.133:6800/3480413474 2024-02-20T07:48:44.422 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:43.553+0000 7f3e977fe700 1 --2- 172.21.15.133:0/1409218203 >> [v2:172.21.15.133:6800/3480413474,v1:172.21.15.133:6801/3480413474] conn(0x7f3e80036790 0x7f3e80038c40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-02-20T07:48:44.422 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:43.553+0000 7f3e9e7e7700 1 -- 172.21.15.133:0/1409218203 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3e98114360 con 0x7f3e98078d20 2024-02-20T07:48:44.422 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:43.558+0000 7f3e94ff9700 1 -- 172.21.15.133:0/1409218203 <== mon.0 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f3e8801a5c0 con 0x7f3e98078d20 2024-02-20T07:48:44.422 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:43.754+0000 7f3e977fe700 1 -- 172.21.15.133:0/1409218203 >> [v2:172.21.15.133:6800/3480413474,v1:172.21.15.133:6801/3480413474] conn(0x7f3e80036790 msgr2=0x7f3e80038c40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.133:6800/3480413474 2024-02-20T07:48:44.422 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:43.754+0000 7f3e977fe700 1 --2- 172.21.15.133:0/1409218203 >> [v2:172.21.15.133:6800/3480413474,v1:172.21.15.133:6801/3480413474] conn(0x7f3e80036790 0x7f3e80038c40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-02-20T07:48:44.422 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:43.815+0000 7f3e9e7e7700 1 -- 172.21.15.133:0/1409218203 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7f3e98114db0 con 0x7f3e98078d20 2024-02-20T07:48:44.422 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:43.815+0000 7f3e94ff9700 1 -- 172.21.15.133:0/1409218203 <== mon.0 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v5) v1 ==== 56+0+103 (secure 0 0 0) 0x7f3e88029340 con 0x7f3e98078d20 2024-02-20T07:48:44.423 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:43.818+0000 7f3e9e7e7700 1 -- 172.21.15.133:0/1409218203 >> [v2:172.21.15.133:6800/3480413474,v1:172.21.15.133:6801/3480413474] conn(0x7f3e80036790 msgr2=0x7f3e80038c40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-02-20T07:48:44.423 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:43.818+0000 7f3e9e7e7700 1 --2- 172.21.15.133:0/1409218203 >> [v2:172.21.15.133:6800/3480413474,v1:172.21.15.133:6801/3480413474] conn(0x7f3e80036790 0x7f3e80038c40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:48:44.423 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:43.818+0000 7f3e9e7e7700 1 -- 172.21.15.133:0/1409218203 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f3e98078d20 msgr2=0x7f3e98113790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:48:44.423 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:43.818+0000 7f3e9e7e7700 1 --2- 172.21.15.133:0/1409218203 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f3e98078d20 0x7f3e98113790 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f3e88006fd0 tx=0x7f3e88009250 comp rx=0 tx=0).stop 2024-02-20T07:48:44.423 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:43.818+0000 7f3e9e7e7700 1 -- 172.21.15.133:0/1409218203 shutdown_connections 2024-02-20T07:48:44.423 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:43.818+0000 7f3e9e7e7700 1 --2- 172.21.15.133:0/1409218203 >> [v2:172.21.15.133:6800/3480413474,v1:172.21.15.133:6801/3480413474] conn(0x7f3e80036790 0x7f3e80038c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:48:44.423 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:43.818+0000 7f3e9e7e7700 1 --2- 172.21.15.133:0/1409218203 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f3e98078d20 0x7f3e98113790 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:48:44.424 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:43.818+0000 7f3e9e7e7700 1 -- 172.21.15.133:0/1409218203 >> 172.21.15.133:0/1409218203 conn(0x7f3e980fded0 msgr2=0x7f3e980ff570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:48:44.424 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:43.818+0000 7f3e9e7e7700 1 -- 172.21.15.133:0/1409218203 shutdown_connections 2024-02-20T07:48:44.424 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:43.818+0000 7f3e9e7e7700 1 -- 172.21.15.133:0/1409218203 wait complete. 2024-02-20T07:48:44.424 INFO:teuthology.orchestra.run.smithi133.stdout:Waiting for the mgr to restart... 2024-02-20T07:48:44.424 INFO:teuthology.orchestra.run.smithi133.stdout:Waiting for mgr epoch 5... 2024-02-20T07:48:44.659 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:48:44 smithi133 ceph-mon[29117]: from='client.? 172.21.15.133:0/1409218203' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-02-20T07:48:49.492 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:48:49 smithi133 ceph-mon[29117]: Active manager daemon smithi133.mrvcbp restarted 2024-02-20T07:48:49.493 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:48:49 smithi133 ceph-mon[29117]: Activating manager daemon smithi133.mrvcbp 2024-02-20T07:48:49.493 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:48:49 smithi133 ceph-mon[29117]: osdmap e2: 0 total, 0 up, 0 in 2024-02-20T07:48:49.493 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:48:49 smithi133 ceph-mon[29117]: mgrmap e6: smithi133.mrvcbp(active, starting, since 0.00384245s) 2024-02-20T07:48:49.493 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:48:49 smithi133 ceph-mon[29117]: from='mgr.14120 172.21.15.133:0/2965705206' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "mon metadata", "id": "smithi133"}]: dispatch 2024-02-20T07:48:49.493 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:48:49 smithi133 ceph-mon[29117]: from='mgr.14120 172.21.15.133:0/2965705206' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "mgr metadata", "who": "smithi133.mrvcbp", "id": "smithi133.mrvcbp"}]: dispatch 2024-02-20T07:48:49.493 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:48:49 smithi133 ceph-mon[29117]: from='mgr.14120 172.21.15.133:0/2965705206' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-02-20T07:48:49.493 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:48:49 smithi133 ceph-mon[29117]: from='mgr.14120 172.21.15.133:0/2965705206' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-02-20T07:48:49.493 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:48:49 smithi133 ceph-mon[29117]: from='mgr.14120 172.21.15.133:0/2965705206' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-02-20T07:48:49.493 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:48:49 smithi133 ceph-mon[29117]: Manager daemon smithi133.mrvcbp is now available 2024-02-20T07:48:49.493 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:48:49 smithi133 ceph-mon[29117]: from='mgr.14120 172.21.15.133:0/2965705206' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:48:49.493 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:48:49 smithi133 ceph-mon[29117]: from='mgr.14120 172.21.15.133:0/2965705206' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:48:49.493 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:48:49 smithi133 ceph-mon[29117]: from='mgr.14120 172.21.15.133:0/2965705206' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-20T07:48:49.493 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:48:49 smithi133 ceph-mon[29117]: from='mgr.14120 172.21.15.133:0/2965705206' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-20T07:48:49.493 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:48:49 smithi133 ceph-mon[29117]: from='mgr.14120 172.21.15.133:0/2965705206' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-20T07:48:50.439 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:48:50 smithi133 ceph-mon[29117]: Found migration_current of "None". Setting to last migration. 2024-02-20T07:48:50.440 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:48:50 smithi133 ceph-mon[29117]: from='mgr.14120 172.21.15.133:0/2965705206' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi133.mrvcbp/mirror_snapshot_schedule"}]: dispatch 2024-02-20T07:48:50.440 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:48:50 smithi133 ceph-mon[29117]: from='mgr.14120 172.21.15.133:0/2965705206' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi133.mrvcbp/trash_purge_schedule"}]: dispatch 2024-02-20T07:48:50.440 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:48:50 smithi133 ceph-mon[29117]: mgrmap e7: smithi133.mrvcbp(active, since 1.00535s) 2024-02-20T07:48:50.752 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stdout { 2024-02-20T07:48:50.752 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 7, 2024-02-20T07:48:50.752 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stdout "initialized": true 2024-02-20T07:48:50.752 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stdout } 2024-02-20T07:48:50.752 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:44.857+0000 7f9c79916700 1 Processor -- start 2024-02-20T07:48:50.752 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:44.857+0000 7f9c79916700 1 -- start start 2024-02-20T07:48:50.752 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:44.857+0000 7f9c79916700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f9c74076be0 0x7f9c741077c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:48:50.752 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:44.857+0000 7f9c79916700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c741031b0 con 0x7f9c74076be0 2024-02-20T07:48:50.752 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:44.857+0000 7f9c72ffd700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f9c74076be0 0x7f9c741077c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:48:50.752 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:44.857+0000 7f9c72ffd700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f9c74076be0 0x7f9c741077c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.133:53450/0 (socket says 172.21.15.133:53450) 2024-02-20T07:48:50.753 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:44.857+0000 7f9c72ffd700 1 -- 172.21.15.133:0/860357052 learned_addr learned my addr 172.21.15.133:0/860357052 (peer_addr_for_me v2:172.21.15.133:0/0) 2024-02-20T07:48:50.753 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:44.858+0000 7f9c72ffd700 1 -- 172.21.15.133:0/860357052 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c74107d00 con 0x7f9c74076be0 2024-02-20T07:48:50.753 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:44.858+0000 7f9c72ffd700 1 --2- 172.21.15.133:0/860357052 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f9c74076be0 0x7f9c741077c0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f9c680098e0 tx=0x7f9c68009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d0651464f6aafa72 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:48:50.753 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:44.858+0000 7f9c71ffb700 1 -- 172.21.15.133:0/860357052 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9c6800b860 con 0x7f9c74076be0 2024-02-20T07:48:50.753 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:44.858+0000 7f9c71ffb700 1 -- 172.21.15.133:0/860357052 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f9c6800be50 con 0x7f9c74076be0 2024-02-20T07:48:50.753 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:44.858+0000 7f9c71ffb700 1 -- 172.21.15.133:0/860357052 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9c68010380 con 0x7f9c74076be0 2024-02-20T07:48:50.753 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:44.859+0000 7f9c79916700 1 -- 172.21.15.133:0/860357052 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f9c74076be0 msgr2=0x7f9c741077c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:48:50.753 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:44.859+0000 7f9c79916700 1 --2- 172.21.15.133:0/860357052 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f9c74076be0 0x7f9c741077c0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f9c680098e0 tx=0x7f9c68009250 comp rx=0 tx=0).stop 2024-02-20T07:48:50.753 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:44.859+0000 7f9c79916700 1 -- 172.21.15.133:0/860357052 shutdown_connections 2024-02-20T07:48:50.753 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:44.859+0000 7f9c79916700 1 --2- 172.21.15.133:0/860357052 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f9c74076be0 0x7f9c741077c0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:48:50.753 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:44.859+0000 7f9c79916700 1 -- 172.21.15.133:0/860357052 >> 172.21.15.133:0/860357052 conn(0x7f9c740fcdd0 msgr2=0x7f9c740ff200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:48:50.753 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:44.859+0000 7f9c79916700 1 -- 172.21.15.133:0/860357052 shutdown_connections 2024-02-20T07:48:50.753 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:44.859+0000 7f9c79916700 1 -- 172.21.15.133:0/860357052 wait complete. 2024-02-20T07:48:50.753 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:44.859+0000 7f9c79916700 1 Processor -- start 2024-02-20T07:48:50.753 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:44.859+0000 7f9c79916700 1 -- start start 2024-02-20T07:48:50.754 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:44.860+0000 7f9c79916700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f9c74076be0 0x7f9c74113780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:48:50.754 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:44.860+0000 7f9c79916700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c74113cc0 con 0x7f9c74076be0 2024-02-20T07:48:50.754 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:44.860+0000 7f9c72ffd700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f9c74076be0 0x7f9c74113780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:48:50.754 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:44.860+0000 7f9c72ffd700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f9c74076be0 0x7f9c74113780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.133:53462/0 (socket says 172.21.15.133:53462) 2024-02-20T07:48:50.754 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:44.860+0000 7f9c72ffd700 1 -- 172.21.15.133:0/1934355445 learned_addr learned my addr 172.21.15.133:0/1934355445 (peer_addr_for_me v2:172.21.15.133:0/0) 2024-02-20T07:48:50.754 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:44.860+0000 7f9c72ffd700 1 -- 172.21.15.133:0/1934355445 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c680094f0 con 0x7f9c74076be0 2024-02-20T07:48:50.754 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:44.860+0000 7f9c72ffd700 1 --2- 172.21.15.133:0/1934355445 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f9c74076be0 0x7f9c74113780 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f9c68003050 tx=0x7f9c68009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:48:50.754 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:44.860+0000 7f9c5b7fe700 1 -- 172.21.15.133:0/1934355445 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9c68010760 con 0x7f9c74076be0 2024-02-20T07:48:50.754 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:44.860+0000 7f9c5b7fe700 1 -- 172.21.15.133:0/1934355445 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f9c68010d50 con 0x7f9c74076be0 2024-02-20T07:48:50.754 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:44.860+0000 7f9c79916700 1 -- 172.21.15.133:0/1934355445 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9c74113ec0 con 0x7f9c74076be0 2024-02-20T07:48:50.754 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:44.860+0000 7f9c5b7fe700 1 -- 172.21.15.133:0/1934355445 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9c6801ae20 con 0x7f9c74076be0 2024-02-20T07:48:50.754 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:44.860+0000 7f9c79916700 1 -- 172.21.15.133:0/1934355445 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9c74114b60 con 0x7f9c74076be0 2024-02-20T07:48:50.754 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:44.861+0000 7f9c5b7fe700 1 -- 172.21.15.133:0/1934355445 <== mon.0 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 5) v1 ==== 43368+0+0 (secure 0 0 0) 0x7f9c6802e440 con 0x7f9c74076be0 2024-02-20T07:48:50.754 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:44.861+0000 7f9c5b7fe700 1 --2- 172.21.15.133:0/1934355445 >> [v2:172.21.15.133:6800/3480413474,v1:172.21.15.133:6801/3480413474] conn(0x7f9c5c0367e0 0x7f9c5c038c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:48:50.754 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:44.861+0000 7f9c5b7fe700 1 -- 172.21.15.133:0/1934355445 --> [v2:172.21.15.133:6800/3480413474,v1:172.21.15.133:6801/3480413474] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f9c5c0393a0 con 0x7f9c5c0367e0 2024-02-20T07:48:50.755 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:44.861+0000 7f9c5b7fe700 1 -- 172.21.15.133:0/1934355445 <== mon.0 v2:172.21.15.133:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7f9c6804a4c0 con 0x7f9c74076be0 2024-02-20T07:48:50.755 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:44.861+0000 7f9c727fc700 1 -- 172.21.15.133:0/1934355445 >> [v2:172.21.15.133:6800/3480413474,v1:172.21.15.133:6801/3480413474] conn(0x7f9c5c0367e0 msgr2=0x7f9c5c038c90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.133:6800/3480413474 2024-02-20T07:48:50.755 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:44.861+0000 7f9c727fc700 1 --2- 172.21.15.133:0/1934355445 >> [v2:172.21.15.133:6800/3480413474,v1:172.21.15.133:6801/3480413474] conn(0x7f9c5c0367e0 0x7f9c5c038c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-02-20T07:48:50.755 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:45.062+0000 7f9c727fc700 1 -- 172.21.15.133:0/1934355445 >> [v2:172.21.15.133:6800/3480413474,v1:172.21.15.133:6801/3480413474] conn(0x7f9c5c0367e0 msgr2=0x7f9c5c038c90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.133:6800/3480413474 2024-02-20T07:48:50.755 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:45.062+0000 7f9c727fc700 1 --2- 172.21.15.133:0/1934355445 >> [v2:172.21.15.133:6800/3480413474,v1:172.21.15.133:6801/3480413474] conn(0x7f9c5c0367e0 0x7f9c5c038c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-02-20T07:48:50.755 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:45.463+0000 7f9c727fc700 1 -- 172.21.15.133:0/1934355445 >> [v2:172.21.15.133:6800/3480413474,v1:172.21.15.133:6801/3480413474] conn(0x7f9c5c0367e0 msgr2=0x7f9c5c038c90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.133:6800/3480413474 2024-02-20T07:48:50.755 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:45.463+0000 7f9c727fc700 1 --2- 172.21.15.133:0/1934355445 >> [v2:172.21.15.133:6800/3480413474,v1:172.21.15.133:6801/3480413474] conn(0x7f9c5c0367e0 0x7f9c5c038c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-02-20T07:48:50.755 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:46.264+0000 7f9c727fc700 1 -- 172.21.15.133:0/1934355445 >> [v2:172.21.15.133:6800/3480413474,v1:172.21.15.133:6801/3480413474] conn(0x7f9c5c0367e0 msgr2=0x7f9c5c038c90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.133:6800/3480413474 2024-02-20T07:48:50.755 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:46.264+0000 7f9c727fc700 1 --2- 172.21.15.133:0/1934355445 >> [v2:172.21.15.133:6800/3480413474,v1:172.21.15.133:6801/3480413474] conn(0x7f9c5c0367e0 0x7f9c5c038c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-02-20T07:48:50.755 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:47.865+0000 7f9c727fc700 1 -- 172.21.15.133:0/1934355445 >> [v2:172.21.15.133:6800/3480413474,v1:172.21.15.133:6801/3480413474] conn(0x7f9c5c0367e0 msgr2=0x7f9c5c038c90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.133:6800/3480413474 2024-02-20T07:48:50.755 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:47.865+0000 7f9c727fc700 1 --2- 172.21.15.133:0/1934355445 >> [v2:172.21.15.133:6800/3480413474,v1:172.21.15.133:6801/3480413474] conn(0x7f9c5c0367e0 0x7f9c5c038c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-02-20T07:48:50.755 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:49.131+0000 7f9c5b7fe700 1 -- 172.21.15.133:0/1934355445 <== mon.0 v2:172.21.15.133:3300/0 6 ==== mgrmap(e 6) v1 ==== 43135+0+0 (secure 0 0 0) 0x7f9c6802e740 con 0x7f9c74076be0 2024-02-20T07:48:50.755 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:49.131+0000 7f9c5b7fe700 1 -- 172.21.15.133:0/1934355445 >> [v2:172.21.15.133:6800/3480413474,v1:172.21.15.133:6801/3480413474] conn(0x7f9c5c0367e0 msgr2=0x7f9c5c038c90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-02-20T07:48:50.756 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:49.131+0000 7f9c5b7fe700 1 --2- 172.21.15.133:0/1934355445 >> [v2:172.21.15.133:6800/3480413474,v1:172.21.15.133:6801/3480413474] conn(0x7f9c5c0367e0 0x7f9c5c038c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:48:50.756 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:50.133+0000 7f9c5b7fe700 1 -- 172.21.15.133:0/1934355445 <== mon.0 v2:172.21.15.133:3300/0 7 ==== mgrmap(e 7) v1 ==== 43262+0+0 (secure 0 0 0) 0x7f9c6802eac0 con 0x7f9c74076be0 2024-02-20T07:48:50.756 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:50.133+0000 7f9c5b7fe700 1 --2- 172.21.15.133:0/1934355445 >> [v2:172.21.15.133:6800/1670376163,v1:172.21.15.133:6801/1670376163] conn(0x7f9c5c039fc0 0x7f9c5c03c3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:48:50.756 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:50.133+0000 7f9c5b7fe700 1 -- 172.21.15.133:0/1934355445 --> [v2:172.21.15.133:6800/1670376163,v1:172.21.15.133:6801/1670376163] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f9c5c0393a0 con 0x7f9c5c039fc0 2024-02-20T07:48:50.756 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:50.134+0000 7f9c727fc700 1 --2- 172.21.15.133:0/1934355445 >> [v2:172.21.15.133:6800/1670376163,v1:172.21.15.133:6801/1670376163] conn(0x7f9c5c039fc0 0x7f9c5c03c3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:48:50.756 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:50.134+0000 7f9c727fc700 1 --2- 172.21.15.133:0/1934355445 >> [v2:172.21.15.133:6800/1670376163,v1:172.21.15.133:6801/1670376163] conn(0x7f9c5c039fc0 0x7f9c5c03c3a0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f9c640039d0 tx=0x7f9c64009350 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:48:50.756 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:50.135+0000 7f9c5b7fe700 1 -- 172.21.15.133:0/1934355445 <== mgr.14120 v2:172.21.15.133:6800/1670376163 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+6549 (secure 0 0 0) 0x7f9c5c0393a0 con 0x7f9c5c039fc0 2024-02-20T07:48:50.756 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:50.141+0000 7f9c79916700 1 -- 172.21.15.133:0/1934355445 --> [v2:172.21.15.133:6800/1670376163,v1:172.21.15.133:6801/1670376163] -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7f9c74114350 con 0x7f9c5c039fc0 2024-02-20T07:48:50.756 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:50.141+0000 7f9c5b7fe700 1 -- 172.21.15.133:0/1934355445 <== mgr.14120 v2:172.21.15.133:6800/1670376163 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7f9c74114350 con 0x7f9c5c039fc0 2024-02-20T07:48:50.756 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:50.141+0000 7f9c79916700 1 -- 172.21.15.133:0/1934355445 >> [v2:172.21.15.133:6800/1670376163,v1:172.21.15.133:6801/1670376163] conn(0x7f9c5c039fc0 msgr2=0x7f9c5c03c3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:48:50.756 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:50.141+0000 7f9c79916700 1 --2- 172.21.15.133:0/1934355445 >> [v2:172.21.15.133:6800/1670376163,v1:172.21.15.133:6801/1670376163] conn(0x7f9c5c039fc0 0x7f9c5c03c3a0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f9c640039d0 tx=0x7f9c64009350 comp rx=0 tx=0).stop 2024-02-20T07:48:50.756 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:50.141+0000 7f9c79916700 1 -- 172.21.15.133:0/1934355445 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f9c74076be0 msgr2=0x7f9c74113780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:48:50.756 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:50.141+0000 7f9c79916700 1 --2- 172.21.15.133:0/1934355445 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f9c74076be0 0x7f9c74113780 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f9c68003050 tx=0x7f9c68009250 comp rx=0 tx=0).stop 2024-02-20T07:48:50.757 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:50.141+0000 7f9c79916700 1 -- 172.21.15.133:0/1934355445 shutdown_connections 2024-02-20T07:48:50.757 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:50.141+0000 7f9c79916700 1 --2- 172.21.15.133:0/1934355445 >> [v2:172.21.15.133:6800/1670376163,v1:172.21.15.133:6801/1670376163] conn(0x7f9c5c039fc0 0x7f9c5c03c3a0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:48:50.757 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:50.141+0000 7f9c79916700 1 --2- 172.21.15.133:0/1934355445 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f9c74076be0 0x7f9c74113780 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:48:50.757 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:50.141+0000 7f9c79916700 1 --2- 172.21.15.133:0/1934355445 >> [v2:172.21.15.133:6800/3480413474,v1:172.21.15.133:6801/3480413474] conn(0x7f9c5c0367e0 0x7f9c5c038c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:48:50.757 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:50.141+0000 7f9c79916700 1 -- 172.21.15.133:0/1934355445 >> 172.21.15.133:0/1934355445 conn(0x7f9c740fcdd0 msgr2=0x7f9c740fe4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:48:50.757 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:50.141+0000 7f9c79916700 1 -- 172.21.15.133:0/1934355445 shutdown_connections 2024-02-20T07:48:50.757 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:50.141+0000 7f9c79916700 1 -- 172.21.15.133:0/1934355445 wait complete. 2024-02-20T07:48:50.757 INFO:teuthology.orchestra.run.smithi133.stdout:mgr epoch 5 is available 2024-02-20T07:48:50.757 INFO:teuthology.orchestra.run.smithi133.stdout:Setting orchestrator backend to cephadm... 2024-02-20T07:48:51.721 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:48:51 smithi133 ceph-mon[29117]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-02-20T07:48:51.721 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:48:51 smithi133 ceph-mon[29117]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-02-20T07:48:51.722 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:48:51 smithi133 ceph-mon[29117]: [20/Feb/2024:07:48:50] ENGINE Bus STARTING 2024-02-20T07:48:51.722 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:48:51 smithi133 ceph-mon[29117]: [20/Feb/2024:07:48:50] ENGINE Serving on https://172.21.15.133:7150 2024-02-20T07:48:51.722 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:48:51 smithi133 ceph-mon[29117]: [20/Feb/2024:07:48:50] ENGINE Bus STARTED 2024-02-20T07:48:51.722 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:48:51 smithi133 ceph-mon[29117]: from='mgr.14120 172.21.15.133:0/2965705206' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:48:51.722 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:48:51 smithi133 ceph-mon[29117]: from='mgr.14120 172.21.15.133:0/2965705206' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-20T07:48:51.722 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:48:51 smithi133 ceph-mon[29117]: from='mgr.14120 172.21.15.133:0/2965705206' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:48:51.722 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:48:51 smithi133 ceph-mon[29117]: from='mgr.14120 172.21.15.133:0/2965705206' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-20T07:48:51.995 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:51.178+0000 7fd3ac513700 1 Processor -- start 2024-02-20T07:48:51.996 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:51.178+0000 7fd3ac513700 1 -- start start 2024-02-20T07:48:51.996 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:51.178+0000 7fd3ac513700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fd3a4102770 0x7fd3a4102b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:48:51.996 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:51.178+0000 7fd3ac513700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3a41030c0 con 0x7fd3a4102770 2024-02-20T07:48:51.996 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:51.179+0000 7fd3aa2af700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fd3a4102770 0x7fd3a4102b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:48:51.996 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:51.179+0000 7fd3aa2af700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fd3a4102770 0x7fd3a4102b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.133:55454/0 (socket says 172.21.15.133:55454) 2024-02-20T07:48:51.996 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:51.179+0000 7fd3aa2af700 1 -- 172.21.15.133:0/2656607317 learned_addr learned my addr 172.21.15.133:0/2656607317 (peer_addr_for_me v2:172.21.15.133:0/0) 2024-02-20T07:48:51.996 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:51.179+0000 7fd3aa2af700 1 -- 172.21.15.133:0/2656607317 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3a4103200 con 0x7fd3a4102770 2024-02-20T07:48:51.996 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:51.179+0000 7fd3aa2af700 1 --2- 172.21.15.133:0/2656607317 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fd3a4102770 0x7fd3a4102b80 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fd3a0009e50 tx=0x7fd3a0009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=69e90bd25e6bd73b server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:48:51.996 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:51.179+0000 7fd3a92ad700 1 -- 172.21.15.133:0/2656607317 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd3a00092e0 con 0x7fd3a4102770 2024-02-20T07:48:51.996 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:51.179+0000 7fd3a92ad700 1 -- 172.21.15.133:0/2656607317 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7fd3a0012430 con 0x7fd3a4102770 2024-02-20T07:48:51.996 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:51.179+0000 7fd3a92ad700 1 -- 172.21.15.133:0/2656607317 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd3a00075c0 con 0x7fd3a4102770 2024-02-20T07:48:51.997 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:51.180+0000 7fd3ac513700 1 -- 172.21.15.133:0/2656607317 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fd3a4102770 msgr2=0x7fd3a4102b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:48:51.997 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:51.180+0000 7fd3ac513700 1 --2- 172.21.15.133:0/2656607317 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fd3a4102770 0x7fd3a4102b80 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fd3a0009e50 tx=0x7fd3a0009840 comp rx=0 tx=0).stop 2024-02-20T07:48:51.997 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:51.180+0000 7fd3ac513700 1 -- 172.21.15.133:0/2656607317 shutdown_connections 2024-02-20T07:48:51.997 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:51.180+0000 7fd3ac513700 1 --2- 172.21.15.133:0/2656607317 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fd3a4102770 0x7fd3a4102b80 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:48:51.997 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:51.180+0000 7fd3ac513700 1 -- 172.21.15.133:0/2656607317 >> 172.21.15.133:0/2656607317 conn(0x7fd3a40fded0 msgr2=0x7fd3a4100300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:48:51.997 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:51.180+0000 7fd3ac513700 1 -- 172.21.15.133:0/2656607317 shutdown_connections 2024-02-20T07:48:51.997 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:51.180+0000 7fd3ac513700 1 -- 172.21.15.133:0/2656607317 wait complete. 2024-02-20T07:48:51.997 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:51.180+0000 7fd3ac513700 1 Processor -- start 2024-02-20T07:48:51.997 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:51.180+0000 7fd3ac513700 1 -- start start 2024-02-20T07:48:51.997 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:51.181+0000 7fd3ac513700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fd3a4102770 0x7fd3a4111550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:48:51.997 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:51.181+0000 7fd3ac513700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3a4111a90 con 0x7fd3a4102770 2024-02-20T07:48:51.997 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:51.181+0000 7fd3aa2af700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fd3a4102770 0x7fd3a4111550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:48:51.997 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:51.181+0000 7fd3aa2af700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fd3a4102770 0x7fd3a4111550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.133:55462/0 (socket says 172.21.15.133:55462) 2024-02-20T07:48:51.997 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:51.181+0000 7fd3aa2af700 1 -- 172.21.15.133:0/2997477506 learned_addr learned my addr 172.21.15.133:0/2997477506 (peer_addr_for_me v2:172.21.15.133:0/0) 2024-02-20T07:48:51.997 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:51.181+0000 7fd3aa2af700 1 -- 172.21.15.133:0/2997477506 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3a0009ae0 con 0x7fd3a4102770 2024-02-20T07:48:51.998 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:51.181+0000 7fd3aa2af700 1 --2- 172.21.15.133:0/2997477506 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fd3a4102770 0x7fd3a4111550 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fd3a00127d0 tx=0x7fd3a0009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:48:51.998 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:51.181+0000 7fd39affd700 1 -- 172.21.15.133:0/2997477506 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd3a000bc30 con 0x7fd3a4102770 2024-02-20T07:48:51.998 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:51.181+0000 7fd39affd700 1 -- 172.21.15.133:0/2997477506 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7fd3a000bd90 con 0x7fd3a4102770 2024-02-20T07:48:51.998 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:51.181+0000 7fd3ac513700 1 -- 172.21.15.133:0/2997477506 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd3a4111c90 con 0x7fd3a4102770 2024-02-20T07:48:51.998 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:51.181+0000 7fd39affd700 1 -- 172.21.15.133:0/2997477506 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd3a001c9d0 con 0x7fd3a4102770 2024-02-20T07:48:51.998 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:51.181+0000 7fd3ac513700 1 -- 172.21.15.133:0/2997477506 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd3a4112930 con 0x7fd3a4102770 2024-02-20T07:48:51.998 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:51.182+0000 7fd39affd700 1 -- 172.21.15.133:0/2997477506 <== mon.0 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 7) v1 ==== 43262+0+0 (secure 0 0 0) 0x7fd3a0019070 con 0x7fd3a4102770 2024-02-20T07:48:51.998 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:51.182+0000 7fd39affd700 1 --2- 172.21.15.133:0/2997477506 >> [v2:172.21.15.133:6800/1670376163,v1:172.21.15.133:6801/1670376163] conn(0x7fd390036660 0x7fd390038b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:48:51.998 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:51.182+0000 7fd3ac513700 1 -- 172.21.15.133:0/2997477506 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd3a405f5c0 con 0x7fd3a4102770 2024-02-20T07:48:51.998 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:51.182+0000 7fd39affd700 1 -- 172.21.15.133:0/2997477506 <== mon.0 v2:172.21.15.133:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7fd3a0049cb0 con 0x7fd3a4102770 2024-02-20T07:48:51.998 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:51.182+0000 7fd3a9aae700 1 --2- 172.21.15.133:0/2997477506 >> [v2:172.21.15.133:6800/1670376163,v1:172.21.15.133:6801/1670376163] conn(0x7fd390036660 0x7fd390038b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:48:51.998 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:51.183+0000 7fd3a9aae700 1 --2- 172.21.15.133:0/2997477506 >> [v2:172.21.15.133:6800/1670376163,v1:172.21.15.133:6801/1670376163] conn(0x7fd390036660 0x7fd390038b10 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fd394009990 tx=0x7fd394006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:48:51.998 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:51.187+0000 7fd39affd700 1 -- 172.21.15.133:0/2997477506 <== mon.0 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7fd3a0020080 con 0x7fd3a4102770 2024-02-20T07:48:51.998 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:51.383+0000 7fd3ac513700 1 -- 172.21.15.133:0/2997477506 --> [v2:172.21.15.133:6800/1670376163,v1:172.21.15.133:6801/1670376163] -- mgr_command(tid 0: {"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}) v1 -- 0x7fd3a41001c0 con 0x7fd390036660 2024-02-20T07:48:51.999 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:51.390+0000 7fd39affd700 1 -- 172.21.15.133:0/2997477506 <== mgr.14120 v2:172.21.15.133:6800/1670376163 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fd3a41001c0 con 0x7fd390036660 2024-02-20T07:48:51.999 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:51.392+0000 7fd3ac513700 1 -- 172.21.15.133:0/2997477506 >> [v2:172.21.15.133:6800/1670376163,v1:172.21.15.133:6801/1670376163] conn(0x7fd390036660 msgr2=0x7fd390038b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:48:51.999 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:51.392+0000 7fd3ac513700 1 --2- 172.21.15.133:0/2997477506 >> [v2:172.21.15.133:6800/1670376163,v1:172.21.15.133:6801/1670376163] conn(0x7fd390036660 0x7fd390038b10 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fd394009990 tx=0x7fd394006e40 comp rx=0 tx=0).stop 2024-02-20T07:48:51.999 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:51.392+0000 7fd3ac513700 1 -- 172.21.15.133:0/2997477506 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fd3a4102770 msgr2=0x7fd3a4111550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:48:51.999 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:51.392+0000 7fd3ac513700 1 --2- 172.21.15.133:0/2997477506 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fd3a4102770 0x7fd3a4111550 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fd3a00127d0 tx=0x7fd3a0009840 comp rx=0 tx=0).stop 2024-02-20T07:48:51.999 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:51.392+0000 7fd3ac513700 1 -- 172.21.15.133:0/2997477506 shutdown_connections 2024-02-20T07:48:51.999 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:51.392+0000 7fd3ac513700 1 --2- 172.21.15.133:0/2997477506 >> [v2:172.21.15.133:6800/1670376163,v1:172.21.15.133:6801/1670376163] conn(0x7fd390036660 0x7fd390038b10 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:48:51.999 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:51.392+0000 7fd3ac513700 1 --2- 172.21.15.133:0/2997477506 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fd3a4102770 0x7fd3a4111550 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:48:51.999 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:51.392+0000 7fd3ac513700 1 -- 172.21.15.133:0/2997477506 >> 172.21.15.133:0/2997477506 conn(0x7fd3a40fded0 msgr2=0x7fd3a40ff5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:48:51.999 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:51.392+0000 7fd3ac513700 1 -- 172.21.15.133:0/2997477506 shutdown_connections 2024-02-20T07:48:51.999 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:51.392+0000 7fd3ac513700 1 -- 172.21.15.133:0/2997477506 wait complete. 2024-02-20T07:48:52.902 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:48:52 smithi133 ceph-mon[29117]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-02-20T07:48:52.902 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:48:52 smithi133 ceph-mon[29117]: mgrmap e8: smithi133.mrvcbp(active, since 2s) 2024-02-20T07:48:53.169 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stdout value unchanged 2024-02-20T07:48:53.169 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:52.412+0000 7fb005899700 1 Processor -- start 2024-02-20T07:48:53.169 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:52.412+0000 7fb005899700 1 -- start start 2024-02-20T07:48:53.169 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:52.412+0000 7fb005899700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb000101670 0x7fb000101a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:48:53.169 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:52.412+0000 7fb005899700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb000101fc0 con 0x7fb000101670 2024-02-20T07:48:53.169 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:52.412+0000 7faffeffd700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb000101670 0x7fb000101a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:48:53.169 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:52.412+0000 7faffeffd700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb000101670 0x7fb000101a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.133:55466/0 (socket says 172.21.15.133:55466) 2024-02-20T07:48:53.169 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:52.412+0000 7faffeffd700 1 -- 172.21.15.133:0/3578573063 learned_addr learned my addr 172.21.15.133:0/3578573063 (peer_addr_for_me v2:172.21.15.133:0/0) 2024-02-20T07:48:53.169 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:52.412+0000 7faffeffd700 1 -- 172.21.15.133:0/3578573063 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb000102100 con 0x7fb000101670 2024-02-20T07:48:53.169 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:52.412+0000 7faffeffd700 1 --2- 172.21.15.133:0/3578573063 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb000101670 0x7fb000101a80 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7faff000bb80 tx=0x7faff000b4f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4c7d5ba44222be76 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:48:53.169 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:52.413+0000 7faffdffb700 1 -- 172.21.15.133:0/3578573063 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7faff000dd00 con 0x7fb000101670 2024-02-20T07:48:53.169 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:52.413+0000 7faffdffb700 1 -- 172.21.15.133:0/3578573063 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7faff000de60 con 0x7fb000101670 2024-02-20T07:48:53.170 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:52.413+0000 7faffdffb700 1 -- 172.21.15.133:0/3578573063 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7faff00188a0 con 0x7fb000101670 2024-02-20T07:48:53.170 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:52.413+0000 7fb005899700 1 -- 172.21.15.133:0/3578573063 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb000101670 msgr2=0x7fb000101a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:48:53.170 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:52.413+0000 7fb005899700 1 --2- 172.21.15.133:0/3578573063 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb000101670 0x7fb000101a80 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7faff000bb80 tx=0x7faff000b4f0 comp rx=0 tx=0).stop 2024-02-20T07:48:53.170 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:52.413+0000 7fb005899700 1 -- 172.21.15.133:0/3578573063 shutdown_connections 2024-02-20T07:48:53.170 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:52.413+0000 7fb005899700 1 --2- 172.21.15.133:0/3578573063 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb000101670 0x7fb000101a80 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:48:53.170 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:52.413+0000 7fb005899700 1 -- 172.21.15.133:0/3578573063 >> 172.21.15.133:0/3578573063 conn(0x7fb0000fcdd0 msgr2=0x7fb0000ff200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:48:53.170 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:52.413+0000 7fb005899700 1 -- 172.21.15.133:0/3578573063 shutdown_connections 2024-02-20T07:48:53.170 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:52.413+0000 7fb005899700 1 -- 172.21.15.133:0/3578573063 wait complete. 2024-02-20T07:48:53.170 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:52.414+0000 7fb005899700 1 Processor -- start 2024-02-20T07:48:53.170 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:52.414+0000 7fb005899700 1 -- start start 2024-02-20T07:48:53.170 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:52.414+0000 7fb005899700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb000101670 0x7fb00010f280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:48:53.170 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:52.414+0000 7fb005899700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb00010f7c0 con 0x7fb000101670 2024-02-20T07:48:53.170 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:52.414+0000 7faffeffd700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb000101670 0x7fb00010f280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:48:53.170 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:52.414+0000 7faffeffd700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb000101670 0x7fb00010f280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.133:55476/0 (socket says 172.21.15.133:55476) 2024-02-20T07:48:53.170 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:52.414+0000 7faffeffd700 1 -- 172.21.15.133:0/2098831741 learned_addr learned my addr 172.21.15.133:0/2098831741 (peer_addr_for_me v2:172.21.15.133:0/0) 2024-02-20T07:48:53.171 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:52.415+0000 7faffeffd700 1 -- 172.21.15.133:0/2098831741 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faff000b790 con 0x7fb000101670 2024-02-20T07:48:53.171 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:52.415+0000 7faffeffd700 1 --2- 172.21.15.133:0/2098831741 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb000101670 0x7fb00010f280 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7faff0010fa0 tx=0x7faff000b4f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:48:53.171 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:52.415+0000 7fafe77fe700 1 -- 172.21.15.133:0/2098831741 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7faff0010b90 con 0x7fb000101670 2024-02-20T07:48:53.171 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:52.415+0000 7fb005899700 1 -- 172.21.15.133:0/2098831741 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb00010f9c0 con 0x7fb000101670 2024-02-20T07:48:53.171 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:52.415+0000 7fafe77fe700 1 -- 172.21.15.133:0/2098831741 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7faff001d050 con 0x7fb000101670 2024-02-20T07:48:53.171 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:52.415+0000 7fafe77fe700 1 -- 172.21.15.133:0/2098831741 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7faff0022590 con 0x7fb000101670 2024-02-20T07:48:53.171 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:52.415+0000 7fb005899700 1 -- 172.21.15.133:0/2098831741 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb000110660 con 0x7fb000101670 2024-02-20T07:48:53.171 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:52.416+0000 7fafe77fe700 1 -- 172.21.15.133:0/2098831741 <== mon.0 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 8) v1 ==== 43368+0+0 (secure 0 0 0) 0x7faff0022790 con 0x7fb000101670 2024-02-20T07:48:53.171 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:52.416+0000 7fb005899700 1 -- 172.21.15.133:0/2098831741 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb00005f5c0 con 0x7fb000101670 2024-02-20T07:48:53.171 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:52.416+0000 7fafe77fe700 1 --2- 172.21.15.133:0/2098831741 >> [v2:172.21.15.133:6800/1670376163,v1:172.21.15.133:6801/1670376163] conn(0x7fafe8036770 0x7fafe8038c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:48:53.171 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:52.416+0000 7fafe77fe700 1 -- 172.21.15.133:0/2098831741 <== mon.0 v2:172.21.15.133:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7faff0014070 con 0x7fb000101670 2024-02-20T07:48:53.171 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:52.416+0000 7faffe7fc700 1 --2- 172.21.15.133:0/2098831741 >> [v2:172.21.15.133:6800/1670376163,v1:172.21.15.133:6801/1670376163] conn(0x7fafe8036770 0x7fafe8038c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:48:53.171 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:52.416+0000 7faffe7fc700 1 --2- 172.21.15.133:0/2098831741 >> [v2:172.21.15.133:6800/1670376163,v1:172.21.15.133:6801/1670376163] conn(0x7fafe8036770 0x7fafe8038c20 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7faff4009990 tx=0x7faff4006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:48:53.171 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:52.421+0000 7fafe77fe700 1 -- 172.21.15.133:0/2098831741 <== mon.0 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7faff002d460 con 0x7fb000101670 2024-02-20T07:48:53.171 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:52.604+0000 7fb005899700 1 -- 172.21.15.133:0/2098831741 --> [v2:172.21.15.133:6800/1670376163,v1:172.21.15.133:6801/1670376163] -- mgr_command(tid 0: {"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}) v1 -- 0x7fb000111240 con 0x7fafe8036770 2024-02-20T07:48:53.172 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:52.605+0000 7fafe77fe700 1 -- 172.21.15.133:0/2098831741 <== mgr.14120 v2:172.21.15.133:6800/1670376163 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+16 (secure 0 0 0) 0x7fb000111240 con 0x7fafe8036770 2024-02-20T07:48:53.172 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:52.607+0000 7fb005899700 1 -- 172.21.15.133:0/2098831741 >> [v2:172.21.15.133:6800/1670376163,v1:172.21.15.133:6801/1670376163] conn(0x7fafe8036770 msgr2=0x7fafe8038c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:48:53.172 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:52.607+0000 7fb005899700 1 --2- 172.21.15.133:0/2098831741 >> [v2:172.21.15.133:6800/1670376163,v1:172.21.15.133:6801/1670376163] conn(0x7fafe8036770 0x7fafe8038c20 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7faff4009990 tx=0x7faff4006e40 comp rx=0 tx=0).stop 2024-02-20T07:48:53.172 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:52.607+0000 7fb005899700 1 -- 172.21.15.133:0/2098831741 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb000101670 msgr2=0x7fb00010f280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:48:53.172 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:52.607+0000 7fb005899700 1 --2- 172.21.15.133:0/2098831741 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb000101670 0x7fb00010f280 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7faff0010fa0 tx=0x7faff000b4f0 comp rx=0 tx=0).stop 2024-02-20T07:48:53.172 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:52.607+0000 7fb005899700 1 -- 172.21.15.133:0/2098831741 shutdown_connections 2024-02-20T07:48:53.172 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:52.607+0000 7fb005899700 1 --2- 172.21.15.133:0/2098831741 >> [v2:172.21.15.133:6800/1670376163,v1:172.21.15.133:6801/1670376163] conn(0x7fafe8036770 0x7fafe8038c20 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:48:53.172 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:52.607+0000 7fb005899700 1 --2- 172.21.15.133:0/2098831741 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb000101670 0x7fb00010f280 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:48:53.172 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:52.607+0000 7fb005899700 1 -- 172.21.15.133:0/2098831741 >> 172.21.15.133:0/2098831741 conn(0x7fb0000fcdd0 msgr2=0x7fb0000ff110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:48:53.172 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:52.607+0000 7fb005899700 1 -- 172.21.15.133:0/2098831741 shutdown_connections 2024-02-20T07:48:53.172 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:52.607+0000 7fb005899700 1 -- 172.21.15.133:0/2098831741 wait complete. 2024-02-20T07:48:53.172 INFO:teuthology.orchestra.run.smithi133.stdout:Generating ssh key... 2024-02-20T07:48:53.730 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:48:53 smithi133 ceph-mon[29117]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-02-20T07:48:54.320 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:53.535+0000 7fe9cc8b0700 1 Processor -- start 2024-02-20T07:48:54.321 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:53.535+0000 7fe9cc8b0700 1 -- start start 2024-02-20T07:48:54.321 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:53.536+0000 7fe9cc8b0700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fe9c8102770 0x7fe9c8102b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:48:54.321 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:53.536+0000 7fe9cc8b0700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9c81030c0 con 0x7fe9c8102770 2024-02-20T07:48:54.322 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:53.536+0000 7fe9c659c700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fe9c8102770 0x7fe9c8102b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:48:54.323 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:53.536+0000 7fe9c659c700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fe9c8102770 0x7fe9c8102b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.133:55488/0 (socket says 172.21.15.133:55488) 2024-02-20T07:48:54.323 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:53.536+0000 7fe9c659c700 1 -- 172.21.15.133:0/2509541889 learned_addr learned my addr 172.21.15.133:0/2509541889 (peer_addr_for_me v2:172.21.15.133:0/0) 2024-02-20T07:48:54.323 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:53.536+0000 7fe9c659c700 1 -- 172.21.15.133:0/2509541889 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9c8103200 con 0x7fe9c8102770 2024-02-20T07:48:54.323 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:53.536+0000 7fe9c659c700 1 --2- 172.21.15.133:0/2509541889 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fe9c8102770 0x7fe9c8102b80 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fe9bc0098e0 tx=0x7fe9bc009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1ca374b96bfb0926 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:48:54.324 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:53.536+0000 7fe9c559a700 1 -- 172.21.15.133:0/2509541889 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe9bc00b860 con 0x7fe9c8102770 2024-02-20T07:48:54.324 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:53.536+0000 7fe9c559a700 1 -- 172.21.15.133:0/2509541889 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7fe9bc00be50 con 0x7fe9c8102770 2024-02-20T07:48:54.324 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:53.536+0000 7fe9c559a700 1 -- 172.21.15.133:0/2509541889 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe9bc010380 con 0x7fe9c8102770 2024-02-20T07:48:54.324 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:53.537+0000 7fe9cc8b0700 1 -- 172.21.15.133:0/2509541889 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fe9c8102770 msgr2=0x7fe9c8102b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:48:54.324 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:53.537+0000 7fe9cc8b0700 1 --2- 172.21.15.133:0/2509541889 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fe9c8102770 0x7fe9c8102b80 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fe9bc0098e0 tx=0x7fe9bc009250 comp rx=0 tx=0).stop 2024-02-20T07:48:54.324 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:53.537+0000 7fe9cc8b0700 1 -- 172.21.15.133:0/2509541889 shutdown_connections 2024-02-20T07:48:54.325 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:53.537+0000 7fe9cc8b0700 1 --2- 172.21.15.133:0/2509541889 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fe9c8102770 0x7fe9c8102b80 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:48:54.325 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:53.537+0000 7fe9cc8b0700 1 -- 172.21.15.133:0/2509541889 >> 172.21.15.133:0/2509541889 conn(0x7fe9c80fded0 msgr2=0x7fe9c8100300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:48:54.325 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:53.537+0000 7fe9cc8b0700 1 -- 172.21.15.133:0/2509541889 shutdown_connections 2024-02-20T07:48:54.325 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:53.537+0000 7fe9cc8b0700 1 -- 172.21.15.133:0/2509541889 wait complete. 2024-02-20T07:48:54.325 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:53.538+0000 7fe9cc8b0700 1 Processor -- start 2024-02-20T07:48:54.326 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:53.538+0000 7fe9cc8b0700 1 -- start start 2024-02-20T07:48:54.327 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:53.538+0000 7fe9cc8b0700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fe9c8102770 0x7fe9c8111560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:48:54.327 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:53.538+0000 7fe9cc8b0700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9c8111aa0 con 0x7fe9c8102770 2024-02-20T07:48:54.327 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:53.538+0000 7fe9c659c700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fe9c8102770 0x7fe9c8111560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:48:54.327 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:53.538+0000 7fe9c659c700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fe9c8102770 0x7fe9c8111560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.133:55490/0 (socket says 172.21.15.133:55490) 2024-02-20T07:48:54.327 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:53.538+0000 7fe9c659c700 1 -- 172.21.15.133:0/190195475 learned_addr learned my addr 172.21.15.133:0/190195475 (peer_addr_for_me v2:172.21.15.133:0/0) 2024-02-20T07:48:54.327 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:53.538+0000 7fe9c659c700 1 -- 172.21.15.133:0/190195475 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9bc0094f0 con 0x7fe9c8102770 2024-02-20T07:48:54.328 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:53.538+0000 7fe9c659c700 1 --2- 172.21.15.133:0/190195475 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fe9c8102770 0x7fe9c8111560 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fe9bc000c00 tx=0x7fe9bc009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:48:54.328 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:53.539+0000 7fe9b6ffd700 1 -- 172.21.15.133:0/190195475 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe9bc0107c0 con 0x7fe9c8102770 2024-02-20T07:48:54.328 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:53.539+0000 7fe9b6ffd700 1 -- 172.21.15.133:0/190195475 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7fe9bc010db0 con 0x7fe9c8102770 2024-02-20T07:48:54.328 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:53.539+0000 7fe9cc8b0700 1 -- 172.21.15.133:0/190195475 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe9c8111ca0 con 0x7fe9c8102770 2024-02-20T07:48:54.328 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:53.539+0000 7fe9b6ffd700 1 -- 172.21.15.133:0/190195475 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe9bc01ad00 con 0x7fe9c8102770 2024-02-20T07:48:54.328 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:53.539+0000 7fe9cc8b0700 1 -- 172.21.15.133:0/190195475 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe9c8112940 con 0x7fe9c8102770 2024-02-20T07:48:54.328 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:53.539+0000 7fe9b6ffd700 1 -- 172.21.15.133:0/190195475 <== mon.0 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 8) v1 ==== 43368+0+0 (secure 0 0 0) 0x7fe9bc0193c0 con 0x7fe9c8102770 2024-02-20T07:48:54.329 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:53.540+0000 7fe9b6ffd700 1 --2- 172.21.15.133:0/190195475 >> [v2:172.21.15.133:6800/1670376163,v1:172.21.15.133:6801/1670376163] conn(0x7fe9b0036790 0x7fe9b0038c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:48:54.329 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:53.540+0000 7fe9b6ffd700 1 -- 172.21.15.133:0/190195475 <== mon.0 v2:172.21.15.133:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7fe9bc04a650 con 0x7fe9c8102770 2024-02-20T07:48:54.329 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:53.540+0000 7fe9cc8b0700 1 -- 172.21.15.133:0/190195475 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe9c805f5c0 con 0x7fe9c8102770 2024-02-20T07:48:54.329 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:53.540+0000 7fe9c5d9b700 1 --2- 172.21.15.133:0/190195475 >> [v2:172.21.15.133:6800/1670376163,v1:172.21.15.133:6801/1670376163] conn(0x7fe9b0036790 0x7fe9b0038c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:48:54.329 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:53.540+0000 7fe9c5d9b700 1 --2- 172.21.15.133:0/190195475 >> [v2:172.21.15.133:6800/1670376163,v1:172.21.15.133:6801/1670376163] conn(0x7fe9b0036790 0x7fe9b0038c40 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fe9b8009990 tx=0x7fe9b8006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:48:54.330 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:53.544+0000 7fe9b6ffd700 1 -- 172.21.15.133:0/190195475 <== mon.0 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7fe9bc0196c0 con 0x7fe9c8102770 2024-02-20T07:48:54.330 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:53.728+0000 7fe9cc8b0700 1 -- 172.21.15.133:0/190195475 --> [v2:172.21.15.133:6800/1670376163,v1:172.21.15.133:6801/1670376163] -- mgr_command(tid 0: {"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}) v1 -- 0x7fe9c8100190 con 0x7fe9b0036790 2024-02-20T07:48:54.330 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:53.788+0000 7fe9b6ffd700 1 -- 172.21.15.133:0/190195475 <== mgr.14120 v2:172.21.15.133:6800/1670376163 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fe9c8100190 con 0x7fe9b0036790 2024-02-20T07:48:54.330 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:53.790+0000 7fe9cc8b0700 1 -- 172.21.15.133:0/190195475 >> [v2:172.21.15.133:6800/1670376163,v1:172.21.15.133:6801/1670376163] conn(0x7fe9b0036790 msgr2=0x7fe9b0038c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:48:54.330 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:53.790+0000 7fe9cc8b0700 1 --2- 172.21.15.133:0/190195475 >> [v2:172.21.15.133:6800/1670376163,v1:172.21.15.133:6801/1670376163] conn(0x7fe9b0036790 0x7fe9b0038c40 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fe9b8009990 tx=0x7fe9b8006e40 comp rx=0 tx=0).stop 2024-02-20T07:48:54.331 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:53.790+0000 7fe9cc8b0700 1 -- 172.21.15.133:0/190195475 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fe9c8102770 msgr2=0x7fe9c8111560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:48:54.331 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:53.790+0000 7fe9cc8b0700 1 --2- 172.21.15.133:0/190195475 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fe9c8102770 0x7fe9c8111560 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fe9bc000c00 tx=0x7fe9bc009250 comp rx=0 tx=0).stop 2024-02-20T07:48:54.331 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:53.790+0000 7fe9cc8b0700 1 -- 172.21.15.133:0/190195475 shutdown_connections 2024-02-20T07:48:54.331 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:53.790+0000 7fe9cc8b0700 1 --2- 172.21.15.133:0/190195475 >> [v2:172.21.15.133:6800/1670376163,v1:172.21.15.133:6801/1670376163] conn(0x7fe9b0036790 0x7fe9b0038c40 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:48:54.332 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:53.790+0000 7fe9cc8b0700 1 --2- 172.21.15.133:0/190195475 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fe9c8102770 0x7fe9c8111560 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:48:54.332 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:53.790+0000 7fe9cc8b0700 1 -- 172.21.15.133:0/190195475 >> 172.21.15.133:0/190195475 conn(0x7fe9c80fded0 msgr2=0x7fe9c80ff570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:48:54.332 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:53.790+0000 7fe9cc8b0700 1 -- 172.21.15.133:0/190195475 shutdown_connections 2024-02-20T07:48:54.332 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:53.790+0000 7fe9cc8b0700 1 -- 172.21.15.133:0/190195475 wait complete. 2024-02-20T07:48:54.898 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:48:54 smithi133 ceph-mon[29117]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-02-20T07:48:54.899 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:48:54 smithi133 ceph-mon[29117]: Generating ssh key... 2024-02-20T07:48:54.899 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:48:54 smithi133 ceph-mon[29117]: from='mgr.14120 172.21.15.133:0/2965705206' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:48:54.899 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:48:54 smithi133 ceph-mon[29117]: from='mgr.14120 172.21.15.133:0/2965705206' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:48:55.489 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stdout ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDGmLCWNF8ryxVrFc1CLvy0ci3t3+CkHbdSl8viWPRc0u2fVKDkEI+fNpGs/CWbzWvods5a6sfpm+MXyeD8CkoBM0fuTuHTNQzWkGwy5ecGhSh9i30t5SRD3j/eumX1NQnr8SjLzoMx+exUJHnSB+q5ySlqJ4C+b+yBKYxGez/Mjc8R9Ep9C/RBVclIH7Mt2DBoprCFzz0hA2WdGN0oBRpad2bHHLB0yvFmUfP5h/ese1ftQ7QIjQ6T6dd61TJuBZUAQlwKMQNI3zejD/ibG/mybeh9QOt1Z554nyjKKhLHsKh/eyTEJjhehXDu3oiAl1vwI1b9s95+56/FDre98WQDcQjfTd23eLO5QcaSk1hfJjdyzRUHt1PlitxnzXDzbuptyFabAv2DN4CQoI/j4qUFRvkuDm7JX/p7933w5y5lhw/Y6fhVlcvSYMRZbNjC7Y96Zt7hLHQRzHGdyIZNHpKzock607TCJwMWBDE5xky6b9MrgwZsaSaZnmgA7CJctwk= ceph-4537327c-cfc4-11ee-95bb-87774f69a715 2024-02-20T07:48:55.489 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:54.705+0000 7f3d92f86700 1 Processor -- start 2024-02-20T07:48:55.490 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:54.705+0000 7f3d92f86700 1 -- start start 2024-02-20T07:48:55.490 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:54.705+0000 7f3d92f86700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f3d8c078cc0 0x7f3d8c0771d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:48:55.490 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:54.705+0000 7f3d92f86700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d8c0790d0 con 0x7f3d8c078cc0 2024-02-20T07:48:55.490 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:54.705+0000 7f3d90d22700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f3d8c078cc0 0x7f3d8c0771d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:48:55.490 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:54.705+0000 7f3d90d22700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f3d8c078cc0 0x7f3d8c0771d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.133:55500/0 (socket says 172.21.15.133:55500) 2024-02-20T07:48:55.490 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:54.705+0000 7f3d90d22700 1 -- 172.21.15.133:0/1060065218 learned_addr learned my addr 172.21.15.133:0/1060065218 (peer_addr_for_me v2:172.21.15.133:0/0) 2024-02-20T07:48:55.490 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:54.706+0000 7f3d90d22700 1 -- 172.21.15.133:0/1060065218 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d8c077710 con 0x7f3d8c078cc0 2024-02-20T07:48:55.490 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:54.706+0000 7f3d90d22700 1 --2- 172.21.15.133:0/1060065218 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f3d8c078cc0 0x7f3d8c0771d0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f3d800098e0 tx=0x7f3d80009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e5eb6c49ee7b2b1 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:48:55.490 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:54.706+0000 7f3d8b7fe700 1 -- 172.21.15.133:0/1060065218 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3d8000b860 con 0x7f3d8c078cc0 2024-02-20T07:48:55.490 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:54.706+0000 7f3d8b7fe700 1 -- 172.21.15.133:0/1060065218 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f3d8000be50 con 0x7f3d8c078cc0 2024-02-20T07:48:55.490 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:54.706+0000 7f3d8b7fe700 1 -- 172.21.15.133:0/1060065218 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3d80010380 con 0x7f3d8c078cc0 2024-02-20T07:48:55.490 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:54.706+0000 7f3d92f86700 1 -- 172.21.15.133:0/1060065218 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f3d8c078cc0 msgr2=0x7f3d8c0771d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:48:55.490 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:54.706+0000 7f3d92f86700 1 --2- 172.21.15.133:0/1060065218 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f3d8c078cc0 0x7f3d8c0771d0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f3d800098e0 tx=0x7f3d80009250 comp rx=0 tx=0).stop 2024-02-20T07:48:55.491 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:54.707+0000 7f3d92f86700 1 -- 172.21.15.133:0/1060065218 shutdown_connections 2024-02-20T07:48:55.491 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:54.707+0000 7f3d92f86700 1 --2- 172.21.15.133:0/1060065218 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f3d8c078cc0 0x7f3d8c0771d0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:48:55.491 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:54.707+0000 7f3d92f86700 1 -- 172.21.15.133:0/1060065218 >> 172.21.15.133:0/1060065218 conn(0x7f3d8c0fded0 msgr2=0x7f3d8c100300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:48:55.491 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:54.707+0000 7f3d92f86700 1 -- 172.21.15.133:0/1060065218 shutdown_connections 2024-02-20T07:48:55.491 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:54.707+0000 7f3d92f86700 1 -- 172.21.15.133:0/1060065218 wait complete. 2024-02-20T07:48:55.491 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:54.707+0000 7f3d92f86700 1 Processor -- start 2024-02-20T07:48:55.491 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:54.707+0000 7f3d92f86700 1 -- start start 2024-02-20T07:48:55.491 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:54.707+0000 7f3d92f86700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f3d8c078cc0 0x7f3d8c113810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:48:55.491 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:54.707+0000 7f3d92f86700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d8c113d50 con 0x7f3d8c078cc0 2024-02-20T07:48:55.491 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:54.707+0000 7f3d90d22700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f3d8c078cc0 0x7f3d8c113810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:48:55.491 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:54.707+0000 7f3d90d22700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f3d8c078cc0 0x7f3d8c113810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.133:55506/0 (socket says 172.21.15.133:55506) 2024-02-20T07:48:55.491 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:54.707+0000 7f3d90d22700 1 -- 172.21.15.133:0/295969821 learned_addr learned my addr 172.21.15.133:0/295969821 (peer_addr_for_me v2:172.21.15.133:0/0) 2024-02-20T07:48:55.492 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:54.708+0000 7f3d90d22700 1 -- 172.21.15.133:0/295969821 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d800094f0 con 0x7f3d8c078cc0 2024-02-20T07:48:55.492 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:54.708+0000 7f3d90d22700 1 --2- 172.21.15.133:0/295969821 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f3d8c078cc0 0x7f3d8c113810 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f3d80006fd0 tx=0x7f3d80009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:48:55.492 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:54.708+0000 7f3d897fa700 1 -- 172.21.15.133:0/295969821 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3d800107c0 con 0x7f3d8c078cc0 2024-02-20T07:48:55.492 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:54.708+0000 7f3d897fa700 1 -- 172.21.15.133:0/295969821 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f3d80010db0 con 0x7f3d8c078cc0 2024-02-20T07:48:55.492 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:54.708+0000 7f3d92f86700 1 -- 172.21.15.133:0/295969821 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d8c113f50 con 0x7f3d8c078cc0 2024-02-20T07:48:55.492 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:54.708+0000 7f3d897fa700 1 -- 172.21.15.133:0/295969821 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3d8001ada0 con 0x7f3d8c078cc0 2024-02-20T07:48:55.492 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:54.708+0000 7f3d92f86700 1 -- 172.21.15.133:0/295969821 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d8c114bf0 con 0x7f3d8c078cc0 2024-02-20T07:48:55.492 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:54.709+0000 7f3d897fa700 1 -- 172.21.15.133:0/295969821 <== mon.0 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 8) v1 ==== 43368+0+0 (secure 0 0 0) 0x7f3d80019410 con 0x7f3d8c078cc0 2024-02-20T07:48:55.492 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:54.709+0000 7f3d92f86700 1 -- 172.21.15.133:0/295969821 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3d8c05f5c0 con 0x7f3d8c078cc0 2024-02-20T07:48:55.492 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:54.709+0000 7f3d897fa700 1 --2- 172.21.15.133:0/295969821 >> [v2:172.21.15.133:6800/1670376163,v1:172.21.15.133:6801/1670376163] conn(0x7f3d7c0367e0 0x7f3d7c038c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:48:55.492 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:54.709+0000 7f3d897fa700 1 -- 172.21.15.133:0/295969821 <== mon.0 v2:172.21.15.133:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f3d80049e30 con 0x7f3d8c078cc0 2024-02-20T07:48:55.492 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:54.709+0000 7f3d8bfff700 1 --2- 172.21.15.133:0/295969821 >> [v2:172.21.15.133:6800/1670376163,v1:172.21.15.133:6801/1670376163] conn(0x7f3d7c0367e0 0x7f3d7c038c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:48:55.492 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:54.710+0000 7f3d8bfff700 1 --2- 172.21.15.133:0/295969821 >> [v2:172.21.15.133:6800/1670376163,v1:172.21.15.133:6801/1670376163] conn(0x7f3d7c0367e0 0x7f3d7c038c90 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f3d78009990 tx=0x7f3d78006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:48:55.493 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:54.714+0000 7f3d897fa700 1 -- 172.21.15.133:0/295969821 <== mon.0 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f3d80023cf0 con 0x7f3d8c078cc0 2024-02-20T07:48:55.493 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:54.897+0000 7f3d92f86700 1 -- 172.21.15.133:0/295969821 --> [v2:172.21.15.133:6800/1670376163,v1:172.21.15.133:6801/1670376163] -- mgr_command(tid 0: {"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}) v1 -- 0x7f3d8c100210 con 0x7f3d7c0367e0 2024-02-20T07:48:55.493 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:54.898+0000 7f3d897fa700 1 -- 172.21.15.133:0/295969821 <== mgr.14120 v2:172.21.15.133:6800/1670376163 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+595 (secure 0 0 0) 0x7f3d8c100210 con 0x7f3d7c0367e0 2024-02-20T07:48:55.493 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:54.900+0000 7f3d92f86700 1 -- 172.21.15.133:0/295969821 >> [v2:172.21.15.133:6800/1670376163,v1:172.21.15.133:6801/1670376163] conn(0x7f3d7c0367e0 msgr2=0x7f3d7c038c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:48:55.493 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:54.900+0000 7f3d92f86700 1 --2- 172.21.15.133:0/295969821 >> [v2:172.21.15.133:6800/1670376163,v1:172.21.15.133:6801/1670376163] conn(0x7f3d7c0367e0 0x7f3d7c038c90 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f3d78009990 tx=0x7f3d78006e40 comp rx=0 tx=0).stop 2024-02-20T07:48:55.493 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:54.900+0000 7f3d92f86700 1 -- 172.21.15.133:0/295969821 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f3d8c078cc0 msgr2=0x7f3d8c113810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:48:55.493 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:54.900+0000 7f3d92f86700 1 --2- 172.21.15.133:0/295969821 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f3d8c078cc0 0x7f3d8c113810 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f3d80006fd0 tx=0x7f3d80009250 comp rx=0 tx=0).stop 2024-02-20T07:48:55.493 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:54.900+0000 7f3d92f86700 1 -- 172.21.15.133:0/295969821 shutdown_connections 2024-02-20T07:48:55.493 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:54.900+0000 7f3d92f86700 1 --2- 172.21.15.133:0/295969821 >> [v2:172.21.15.133:6800/1670376163,v1:172.21.15.133:6801/1670376163] conn(0x7f3d7c0367e0 0x7f3d7c038c90 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:48:55.493 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:54.900+0000 7f3d92f86700 1 --2- 172.21.15.133:0/295969821 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f3d8c078cc0 0x7f3d8c113810 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:48:55.493 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:54.900+0000 7f3d92f86700 1 -- 172.21.15.133:0/295969821 >> 172.21.15.133:0/295969821 conn(0x7f3d8c0fded0 msgr2=0x7f3d8c0ff5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:48:55.493 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:54.900+0000 7f3d92f86700 1 -- 172.21.15.133:0/295969821 shutdown_connections 2024-02-20T07:48:55.494 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:54.900+0000 7f3d92f86700 1 -- 172.21.15.133:0/295969821 wait complete. 2024-02-20T07:48:55.494 INFO:teuthology.orchestra.run.smithi133.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-02-20T07:48:55.494 INFO:teuthology.orchestra.run.smithi133.stdout:Adding key to root@localhost authorized_keys... 2024-02-20T07:48:55.494 INFO:teuthology.orchestra.run.smithi133.stdout:Adding host smithi133... 2024-02-20T07:48:56.086 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:48:55 smithi133 ceph-mon[29117]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-02-20T07:48:57.992 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:48:57 smithi133 ceph-mon[29117]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi133", "addr": "172.21.15.133", "target": ["mon-mgr", ""]}]: dispatch 2024-02-20T07:48:58.690 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:48:58 smithi133 ceph-mon[29117]: Deploying cephadm binary to smithi133 2024-02-20T07:48:58.690 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:48:58 smithi133 ceph-mon[29117]: from='mgr.14120 172.21.15.133:0/2965705206' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:48:58.690 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:48:58 smithi133 ceph-mon[29117]: from='mgr.14120 172.21.15.133:0/2965705206' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-20T07:48:58.698 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stdout Added host 'smithi133' with addr '172.21.15.133' 2024-02-20T07:48:58.698 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:55.888+0000 7f2a8c785700 1 Processor -- start 2024-02-20T07:48:58.698 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:55.888+0000 7f2a8c785700 1 -- start start 2024-02-20T07:48:58.698 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:55.888+0000 7f2a8c785700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f2a84103e00 0x7f2a84104210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:48:58.698 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:55.888+0000 7f2a8c785700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a84104750 con 0x7f2a84103e00 2024-02-20T07:48:58.698 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:55.888+0000 7f2a8a521700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f2a84103e00 0x7f2a84104210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:48:58.698 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:55.888+0000 7f2a8a521700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f2a84103e00 0x7f2a84104210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.133:55510/0 (socket says 172.21.15.133:55510) 2024-02-20T07:48:58.698 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:55.888+0000 7f2a8a521700 1 -- 172.21.15.133:0/3603370380 learned_addr learned my addr 172.21.15.133:0/3603370380 (peer_addr_for_me v2:172.21.15.133:0/0) 2024-02-20T07:48:58.699 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:55.888+0000 7f2a8a521700 1 -- 172.21.15.133:0/3603370380 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a84104890 con 0x7f2a84103e00 2024-02-20T07:48:58.699 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:55.888+0000 7f2a8a521700 1 --2- 172.21.15.133:0/3603370380 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f2a84103e00 0x7f2a84104210 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f2a800098e0 tx=0x7f2a80009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8e0e2eb35201b7a5 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:48:58.699 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:55.889+0000 7f2a8951f700 1 -- 172.21.15.133:0/3603370380 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2a8000b860 con 0x7f2a84103e00 2024-02-20T07:48:58.699 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:55.889+0000 7f2a8951f700 1 -- 172.21.15.133:0/3603370380 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f2a8000be50 con 0x7f2a84103e00 2024-02-20T07:48:58.699 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:55.889+0000 7f2a8951f700 1 -- 172.21.15.133:0/3603370380 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2a80010380 con 0x7f2a84103e00 2024-02-20T07:48:58.699 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:55.889+0000 7f2a8c785700 1 -- 172.21.15.133:0/3603370380 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f2a84103e00 msgr2=0x7f2a84104210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:48:58.699 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:55.889+0000 7f2a8c785700 1 --2- 172.21.15.133:0/3603370380 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f2a84103e00 0x7f2a84104210 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f2a800098e0 tx=0x7f2a80009250 comp rx=0 tx=0).stop 2024-02-20T07:48:58.699 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:55.889+0000 7f2a8c785700 1 -- 172.21.15.133:0/3603370380 shutdown_connections 2024-02-20T07:48:58.699 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:55.889+0000 7f2a8c785700 1 --2- 172.21.15.133:0/3603370380 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f2a84103e00 0x7f2a84104210 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:48:58.699 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:55.889+0000 7f2a8c785700 1 -- 172.21.15.133:0/3603370380 >> 172.21.15.133:0/3603370380 conn(0x7f2a840fc580 msgr2=0x7f2a840fe990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:48:58.699 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:55.889+0000 7f2a8c785700 1 -- 172.21.15.133:0/3603370380 shutdown_connections 2024-02-20T07:48:58.699 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:55.889+0000 7f2a8c785700 1 -- 172.21.15.133:0/3603370380 wait complete. 2024-02-20T07:48:58.699 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:55.890+0000 7f2a8c785700 1 Processor -- start 2024-02-20T07:48:58.699 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:55.890+0000 7f2a8c785700 1 -- start start 2024-02-20T07:48:58.699 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:55.890+0000 7f2a8c785700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f2a84103e00 0x7f2a84111820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:48:58.699 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:55.890+0000 7f2a8c785700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a84111d60 con 0x7f2a84103e00 2024-02-20T07:48:58.700 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:55.890+0000 7f2a8a521700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f2a84103e00 0x7f2a84111820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:48:58.700 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:55.890+0000 7f2a8a521700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f2a84103e00 0x7f2a84111820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.133:55512/0 (socket says 172.21.15.133:55512) 2024-02-20T07:48:58.700 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:55.890+0000 7f2a8a521700 1 -- 172.21.15.133:0/3604985221 learned_addr learned my addr 172.21.15.133:0/3604985221 (peer_addr_for_me v2:172.21.15.133:0/0) 2024-02-20T07:48:58.700 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:55.890+0000 7f2a8a521700 1 -- 172.21.15.133:0/3604985221 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a800094f0 con 0x7f2a84103e00 2024-02-20T07:48:58.700 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:55.891+0000 7f2a8a521700 1 --2- 172.21.15.133:0/3604985221 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f2a84103e00 0x7f2a84111820 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f2a8000b440 tx=0x7f2a80009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:48:58.700 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:55.891+0000 7f2a7affd700 1 -- 172.21.15.133:0/3604985221 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2a80010760 con 0x7f2a84103e00 2024-02-20T07:48:58.700 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:55.891+0000 7f2a7affd700 1 -- 172.21.15.133:0/3604985221 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f2a80010d50 con 0x7f2a84103e00 2024-02-20T07:48:58.700 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:55.891+0000 7f2a8c785700 1 -- 172.21.15.133:0/3604985221 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2a84111f60 con 0x7f2a84103e00 2024-02-20T07:48:58.700 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:55.891+0000 7f2a7affd700 1 -- 172.21.15.133:0/3604985221 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2a80024d80 con 0x7f2a84103e00 2024-02-20T07:48:58.700 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:55.891+0000 7f2a8c785700 1 -- 172.21.15.133:0/3604985221 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2a84112c00 con 0x7f2a84103e00 2024-02-20T07:48:58.700 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:55.891+0000 7f2a7affd700 1 -- 172.21.15.133:0/3604985221 <== mon.0 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 8) v1 ==== 43368+0+0 (secure 0 0 0) 0x7f2a8001a430 con 0x7f2a84103e00 2024-02-20T07:48:58.700 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:55.892+0000 7f2a7affd700 1 --2- 172.21.15.133:0/3604985221 >> [v2:172.21.15.133:6800/1670376163,v1:172.21.15.133:6801/1670376163] conn(0x7f2a700367e0 0x7f2a70038c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:48:58.700 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:55.892+0000 7f2a8c785700 1 -- 172.21.15.133:0/3604985221 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2a8405f5c0 con 0x7f2a84103e00 2024-02-20T07:48:58.700 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:55.892+0000 7f2a7affd700 1 -- 172.21.15.133:0/3604985221 <== mon.0 v2:172.21.15.133:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f2a8004a5c0 con 0x7f2a84103e00 2024-02-20T07:48:58.701 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:55.892+0000 7f2a89d20700 1 --2- 172.21.15.133:0/3604985221 >> [v2:172.21.15.133:6800/1670376163,v1:172.21.15.133:6801/1670376163] conn(0x7f2a700367e0 0x7f2a70038c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:48:58.701 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:55.892+0000 7f2a89d20700 1 --2- 172.21.15.133:0/3604985221 >> [v2:172.21.15.133:6800/1670376163,v1:172.21.15.133:6801/1670376163] conn(0x7f2a700367e0 0x7f2a70038c90 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f2a74009990 tx=0x7f2a74006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:48:58.701 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:55.896+0000 7f2a7affd700 1 -- 172.21.15.133:0/3604985221 <== mon.0 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f2a80018940 con 0x7f2a84103e00 2024-02-20T07:48:58.701 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:56.084+0000 7f2a8c785700 1 -- 172.21.15.133:0/3604985221 --> [v2:172.21.15.133:6800/1670376163,v1:172.21.15.133:6801/1670376163] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi133", "addr": "172.21.15.133", "target": ["mon-mgr", ""]}) v1 -- 0x7f2a84101470 con 0x7f2a700367e0 2024-02-20T07:48:58.701 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:58.069+0000 7f2a7affd700 1 -- 172.21.15.133:0/3604985221 <== mgr.14120 v2:172.21.15.133:6800/1670376163 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (secure 0 0 0) 0x7f2a84101470 con 0x7f2a700367e0 2024-02-20T07:48:58.701 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:58.071+0000 7f2a8c785700 1 -- 172.21.15.133:0/3604985221 >> [v2:172.21.15.133:6800/1670376163,v1:172.21.15.133:6801/1670376163] conn(0x7f2a700367e0 msgr2=0x7f2a70038c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:48:58.701 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:58.071+0000 7f2a8c785700 1 --2- 172.21.15.133:0/3604985221 >> [v2:172.21.15.133:6800/1670376163,v1:172.21.15.133:6801/1670376163] conn(0x7f2a700367e0 0x7f2a70038c90 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f2a74009990 tx=0x7f2a74006e40 comp rx=0 tx=0).stop 2024-02-20T07:48:58.701 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:58.071+0000 7f2a8c785700 1 -- 172.21.15.133:0/3604985221 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f2a84103e00 msgr2=0x7f2a84111820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:48:58.701 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:58.071+0000 7f2a8c785700 1 --2- 172.21.15.133:0/3604985221 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f2a84103e00 0x7f2a84111820 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f2a8000b440 tx=0x7f2a80009250 comp rx=0 tx=0).stop 2024-02-20T07:48:58.701 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:58.071+0000 7f2a8c785700 1 -- 172.21.15.133:0/3604985221 shutdown_connections 2024-02-20T07:48:58.701 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:58.071+0000 7f2a8c785700 1 --2- 172.21.15.133:0/3604985221 >> [v2:172.21.15.133:6800/1670376163,v1:172.21.15.133:6801/1670376163] conn(0x7f2a700367e0 0x7f2a70038c90 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:48:58.702 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:58.071+0000 7f2a8c785700 1 --2- 172.21.15.133:0/3604985221 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f2a84103e00 0x7f2a84111820 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:48:58.702 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:58.071+0000 7f2a8c785700 1 -- 172.21.15.133:0/3604985221 >> 172.21.15.133:0/3604985221 conn(0x7f2a840fc580 msgr2=0x7f2a840fd400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:48:58.702 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:58.072+0000 7f2a8c785700 1 -- 172.21.15.133:0/3604985221 shutdown_connections 2024-02-20T07:48:58.702 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:58.072+0000 7f2a8c785700 1 -- 172.21.15.133:0/3604985221 wait complete. 2024-02-20T07:48:58.702 INFO:teuthology.orchestra.run.smithi133.stdout:Deploying mon service with default placement... 2024-02-20T07:48:59.860 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:48:59 smithi133 ceph-mon[29117]: Added host smithi133 2024-02-20T07:48:59.861 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:48:59 smithi133 ceph-mon[29117]: from='mgr.14120 172.21.15.133:0/2965705206' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:49:00.040 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2024-02-20T07:49:00.041 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:59.089+0000 7f707649d700 1 Processor -- start 2024-02-20T07:49:00.041 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:59.089+0000 7f707649d700 1 -- start start 2024-02-20T07:49:00.041 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:59.089+0000 7f707649d700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f7070102c60 0x7f7070103070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:49:00.041 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:59.089+0000 7f707649d700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70701035b0 con 0x7f7070102c60 2024-02-20T07:49:00.041 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:59.090+0000 7f706ffff700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f7070102c60 0x7f7070103070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:49:00.041 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:59.090+0000 7f706ffff700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f7070102c60 0x7f7070103070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.133:49924/0 (socket says 172.21.15.133:49924) 2024-02-20T07:49:00.041 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:59.090+0000 7f706ffff700 1 -- 172.21.15.133:0/1160124616 learned_addr learned my addr 172.21.15.133:0/1160124616 (peer_addr_for_me v2:172.21.15.133:0/0) 2024-02-20T07:49:00.042 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:59.090+0000 7f706ffff700 1 -- 172.21.15.133:0/1160124616 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70701036f0 con 0x7f7070102c60 2024-02-20T07:49:00.042 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:59.090+0000 7f706ffff700 1 --2- 172.21.15.133:0/1160124616 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f7070102c60 0x7f7070103070 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f706400bb80 tx=0x7f706400b4f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9fa612e521af5625 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:49:00.042 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:59.090+0000 7f706effd700 1 -- 172.21.15.133:0/1160124616 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f706400dd00 con 0x7f7070102c60 2024-02-20T07:49:00.042 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:59.090+0000 7f706effd700 1 -- 172.21.15.133:0/1160124616 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f706400de60 con 0x7f7070102c60 2024-02-20T07:49:00.042 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:59.090+0000 7f706effd700 1 -- 172.21.15.133:0/1160124616 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f70640188a0 con 0x7f7070102c60 2024-02-20T07:49:00.042 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:59.091+0000 7f707649d700 1 -- 172.21.15.133:0/1160124616 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f7070102c60 msgr2=0x7f7070103070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:49:00.042 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:59.091+0000 7f707649d700 1 --2- 172.21.15.133:0/1160124616 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f7070102c60 0x7f7070103070 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f706400bb80 tx=0x7f706400b4f0 comp rx=0 tx=0).stop 2024-02-20T07:49:00.042 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:59.091+0000 7f707649d700 1 -- 172.21.15.133:0/1160124616 shutdown_connections 2024-02-20T07:49:00.042 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:59.091+0000 7f707649d700 1 --2- 172.21.15.133:0/1160124616 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f7070102c60 0x7f7070103070 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:49:00.042 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:59.091+0000 7f707649d700 1 -- 172.21.15.133:0/1160124616 >> 172.21.15.133:0/1160124616 conn(0x7f70700fc580 msgr2=0x7f70700fe990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:49:00.042 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:59.091+0000 7f707649d700 1 -- 172.21.15.133:0/1160124616 shutdown_connections 2024-02-20T07:49:00.043 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:59.091+0000 7f707649d700 1 -- 172.21.15.133:0/1160124616 wait complete. 2024-02-20T07:49:00.043 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:59.091+0000 7f707649d700 1 Processor -- start 2024-02-20T07:49:00.043 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:59.092+0000 7f707649d700 1 -- start start 2024-02-20T07:49:00.043 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:59.092+0000 7f707649d700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f7070102c60 0x7f707010f570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:49:00.043 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:59.092+0000 7f707649d700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f707010fab0 con 0x7f7070102c60 2024-02-20T07:49:00.043 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:59.092+0000 7f706ffff700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f7070102c60 0x7f707010f570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:49:00.043 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:59.092+0000 7f706ffff700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f7070102c60 0x7f707010f570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.133:49938/0 (socket says 172.21.15.133:49938) 2024-02-20T07:49:00.043 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:59.092+0000 7f706ffff700 1 -- 172.21.15.133:0/2082698256 learned_addr learned my addr 172.21.15.133:0/2082698256 (peer_addr_for_me v2:172.21.15.133:0/0) 2024-02-20T07:49:00.043 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:59.092+0000 7f706ffff700 1 -- 172.21.15.133:0/2082698256 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f706400b790 con 0x7f7070102c60 2024-02-20T07:49:00.044 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:59.092+0000 7f706ffff700 1 --2- 172.21.15.133:0/2082698256 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f7070102c60 0x7f707010f570 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f7064010fa0 tx=0x7f706400b4f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:49:00.044 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:59.092+0000 7f706cff9700 1 -- 172.21.15.133:0/2082698256 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7064010b90 con 0x7f7070102c60 2024-02-20T07:49:00.044 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:59.092+0000 7f706cff9700 1 -- 172.21.15.133:0/2082698256 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f706401d050 con 0x7f7070102c60 2024-02-20T07:49:00.044 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:59.092+0000 7f707649d700 1 -- 172.21.15.133:0/2082698256 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f707010fcb0 con 0x7f7070102c60 2024-02-20T07:49:00.044 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:59.092+0000 7f706cff9700 1 -- 172.21.15.133:0/2082698256 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7064022590 con 0x7f7070102c60 2024-02-20T07:49:00.044 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:59.092+0000 7f707649d700 1 -- 172.21.15.133:0/2082698256 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7070110950 con 0x7f7070102c60 2024-02-20T07:49:00.044 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:59.093+0000 7f706cff9700 1 -- 172.21.15.133:0/2082698256 <== mon.0 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 8) v1 ==== 43368+0+0 (secure 0 0 0) 0x7f7064022740 con 0x7f7070102c60 2024-02-20T07:49:00.044 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:59.093+0000 7f706cff9700 1 --2- 172.21.15.133:0/2082698256 >> [v2:172.21.15.133:6800/1670376163,v1:172.21.15.133:6801/1670376163] conn(0x7f70580367e0 0x7f7058038c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:49:00.044 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:59.093+0000 7f707649d700 1 -- 172.21.15.133:0/2082698256 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f707005f5c0 con 0x7f7070102c60 2024-02-20T07:49:00.044 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:59.093+0000 7f706cff9700 1 -- 172.21.15.133:0/2082698256 <== mon.0 v2:172.21.15.133:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f7064014070 con 0x7f7070102c60 2024-02-20T07:49:00.044 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:59.093+0000 7f706f7fe700 1 --2- 172.21.15.133:0/2082698256 >> [v2:172.21.15.133:6800/1670376163,v1:172.21.15.133:6801/1670376163] conn(0x7f70580367e0 0x7f7058038c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:49:00.045 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:59.094+0000 7f706f7fe700 1 --2- 172.21.15.133:0/2082698256 >> [v2:172.21.15.133:6800/1670376163,v1:172.21.15.133:6801/1670376163] conn(0x7f70580367e0 0x7f7058038c90 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f7060009990 tx=0x7f7060006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:49:00.045 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:59.098+0000 7f706cff9700 1 -- 172.21.15.133:0/2082698256 <== mon.0 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f7064021d70 con 0x7f7070102c60 2024-02-20T07:49:00.045 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:59.322+0000 7f707649d700 1 -- 172.21.15.133:0/2082698256 --> [v2:172.21.15.133:6800/1670376163,v1:172.21.15.133:6801/1670376163] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "target": ["mon-mgr", ""]}) v1 -- 0x7f7070101550 con 0x7f70580367e0 2024-02-20T07:49:00.045 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:59.327+0000 7f706cff9700 1 -- 172.21.15.133:0/2082698256 <== mgr.14120 v2:172.21.15.133:6800/1670376163 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f7070101550 con 0x7f70580367e0 2024-02-20T07:49:00.045 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:59.329+0000 7f707649d700 1 -- 172.21.15.133:0/2082698256 >> [v2:172.21.15.133:6800/1670376163,v1:172.21.15.133:6801/1670376163] conn(0x7f70580367e0 msgr2=0x7f7058038c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:49:00.045 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:59.329+0000 7f707649d700 1 --2- 172.21.15.133:0/2082698256 >> [v2:172.21.15.133:6800/1670376163,v1:172.21.15.133:6801/1670376163] conn(0x7f70580367e0 0x7f7058038c90 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f7060009990 tx=0x7f7060006e40 comp rx=0 tx=0).stop 2024-02-20T07:49:00.045 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:59.329+0000 7f707649d700 1 -- 172.21.15.133:0/2082698256 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f7070102c60 msgr2=0x7f707010f570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:49:00.045 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:59.329+0000 7f707649d700 1 --2- 172.21.15.133:0/2082698256 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f7070102c60 0x7f707010f570 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f7064010fa0 tx=0x7f706400b4f0 comp rx=0 tx=0).stop 2024-02-20T07:49:00.045 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:59.329+0000 7f707649d700 1 -- 172.21.15.133:0/2082698256 shutdown_connections 2024-02-20T07:49:00.045 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:59.329+0000 7f707649d700 1 --2- 172.21.15.133:0/2082698256 >> [v2:172.21.15.133:6800/1670376163,v1:172.21.15.133:6801/1670376163] conn(0x7f70580367e0 0x7f7058038c90 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:49:00.045 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:59.329+0000 7f707649d700 1 --2- 172.21.15.133:0/2082698256 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f7070102c60 0x7f707010f570 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:49:00.045 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:59.329+0000 7f707649d700 1 -- 172.21.15.133:0/2082698256 >> 172.21.15.133:0/2082698256 conn(0x7f70700fc580 msgr2=0x7f7070105da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:49:00.045 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:59.329+0000 7f707649d700 1 -- 172.21.15.133:0/2082698256 shutdown_connections 2024-02-20T07:49:00.045 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:48:59.329+0000 7f707649d700 1 -- 172.21.15.133:0/2082698256 wait complete. 2024-02-20T07:49:00.045 INFO:teuthology.orchestra.run.smithi133.stdout:Deploying mgr service with default placement... 2024-02-20T07:49:00.722 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:49:00 smithi133 ceph-mon[29117]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "target": ["mon-mgr", ""]}]: dispatch 2024-02-20T07:49:00.722 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:49:00 smithi133 ceph-mon[29117]: Saving service mon spec with placement count:5 2024-02-20T07:49:01.992 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:49:01 smithi133 ceph-mon[29117]: from='client.14144 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "target": ["mon-mgr", ""]}]: dispatch 2024-02-20T07:49:01.992 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:49:01 smithi133 ceph-mon[29117]: Saving service mgr spec with placement count:2 2024-02-20T07:49:01.992 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:49:01 smithi133 ceph-mon[29117]: from='mgr.14120 172.21.15.133:0/2965705206' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:49:02.003 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2024-02-20T07:49:02.003 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:00.446+0000 7f74e0c3a700 1 Processor -- start 2024-02-20T07:49:02.003 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:00.446+0000 7f74e0c3a700 1 -- start start 2024-02-20T07:49:02.003 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:00.446+0000 7f74e0c3a700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f74dc103e20 0x7f74dc104230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:49:02.003 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:00.446+0000 7f74e0c3a700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74dc104770 con 0x7f74dc103e20 2024-02-20T07:49:02.003 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:00.446+0000 7f74da59c700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f74dc103e20 0x7f74dc104230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:49:02.003 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:00.446+0000 7f74da59c700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f74dc103e20 0x7f74dc104230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.133:49946/0 (socket says 172.21.15.133:49946) 2024-02-20T07:49:02.003 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:00.446+0000 7f74da59c700 1 -- 172.21.15.133:0/1134341166 learned_addr learned my addr 172.21.15.133:0/1134341166 (peer_addr_for_me v2:172.21.15.133:0/0) 2024-02-20T07:49:02.003 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:00.447+0000 7f74da59c700 1 -- 172.21.15.133:0/1134341166 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74dc1048b0 con 0x7f74dc103e20 2024-02-20T07:49:02.003 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:00.447+0000 7f74da59c700 1 --2- 172.21.15.133:0/1134341166 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f74dc103e20 0x7f74dc104230 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f74cc0098e0 tx=0x7f74cc009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=62cebd29da909817 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:49:02.003 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:00.447+0000 7f74d959a700 1 -- 172.21.15.133:0/1134341166 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f74cc00b860 con 0x7f74dc103e20 2024-02-20T07:49:02.004 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:00.447+0000 7f74d959a700 1 -- 172.21.15.133:0/1134341166 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f74cc00be50 con 0x7f74dc103e20 2024-02-20T07:49:02.004 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:00.447+0000 7f74d959a700 1 -- 172.21.15.133:0/1134341166 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f74cc010380 con 0x7f74dc103e20 2024-02-20T07:49:02.004 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:00.447+0000 7f74e0c3a700 1 -- 172.21.15.133:0/1134341166 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f74dc103e20 msgr2=0x7f74dc104230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:49:02.004 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:00.447+0000 7f74e0c3a700 1 --2- 172.21.15.133:0/1134341166 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f74dc103e20 0x7f74dc104230 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f74cc0098e0 tx=0x7f74cc009250 comp rx=0 tx=0).stop 2024-02-20T07:49:02.004 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:00.448+0000 7f74e0c3a700 1 -- 172.21.15.133:0/1134341166 shutdown_connections 2024-02-20T07:49:02.004 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:00.448+0000 7f74e0c3a700 1 --2- 172.21.15.133:0/1134341166 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f74dc103e20 0x7f74dc104230 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:49:02.004 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:00.448+0000 7f74e0c3a700 1 -- 172.21.15.133:0/1134341166 >> 172.21.15.133:0/1134341166 conn(0x7f74dc0fc580 msgr2=0x7f74dc0fe990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:49:02.004 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:00.448+0000 7f74e0c3a700 1 -- 172.21.15.133:0/1134341166 shutdown_connections 2024-02-20T07:49:02.004 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:00.448+0000 7f74e0c3a700 1 -- 172.21.15.133:0/1134341166 wait complete. 2024-02-20T07:49:02.004 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:00.448+0000 7f74e0c3a700 1 Processor -- start 2024-02-20T07:49:02.004 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:00.448+0000 7f74e0c3a700 1 -- start start 2024-02-20T07:49:02.004 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:00.448+0000 7f74e0c3a700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f74dc103e20 0x7f74dc111800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:49:02.004 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:00.448+0000 7f74e0c3a700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74dc111d40 con 0x7f74dc103e20 2024-02-20T07:49:02.004 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:00.448+0000 7f74da59c700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f74dc103e20 0x7f74dc111800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:49:02.004 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:00.448+0000 7f74da59c700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f74dc103e20 0x7f74dc111800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.133:49950/0 (socket says 172.21.15.133:49950) 2024-02-20T07:49:02.005 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:00.448+0000 7f74da59c700 1 -- 172.21.15.133:0/841063019 learned_addr learned my addr 172.21.15.133:0/841063019 (peer_addr_for_me v2:172.21.15.133:0/0) 2024-02-20T07:49:02.005 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:00.449+0000 7f74da59c700 1 -- 172.21.15.133:0/841063019 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74cc0094f0 con 0x7f74dc103e20 2024-02-20T07:49:02.005 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:00.449+0000 7f74da59c700 1 --2- 172.21.15.133:0/841063019 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f74dc103e20 0x7f74dc111800 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f74cc00b440 tx=0x7f74cc009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:49:02.005 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:00.449+0000 7f74caffd700 1 -- 172.21.15.133:0/841063019 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f74cc010760 con 0x7f74dc103e20 2024-02-20T07:49:02.005 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:00.449+0000 7f74caffd700 1 -- 172.21.15.133:0/841063019 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f74cc010d50 con 0x7f74dc103e20 2024-02-20T07:49:02.005 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:00.449+0000 7f74caffd700 1 -- 172.21.15.133:0/841063019 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f74cc024d80 con 0x7f74dc103e20 2024-02-20T07:49:02.005 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:00.449+0000 7f74e0c3a700 1 -- 172.21.15.133:0/841063019 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f74dc111f40 con 0x7f74dc103e20 2024-02-20T07:49:02.005 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:00.449+0000 7f74e0c3a700 1 -- 172.21.15.133:0/841063019 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f74dc112be0 con 0x7f74dc103e20 2024-02-20T07:49:02.005 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:00.450+0000 7f74caffd700 1 -- 172.21.15.133:0/841063019 <== mon.0 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 8) v1 ==== 43368+0+0 (secure 0 0 0) 0x7f74cc01a430 con 0x7f74dc103e20 2024-02-20T07:49:02.005 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:00.450+0000 7f74e0c3a700 1 -- 172.21.15.133:0/841063019 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f74dc05f5c0 con 0x7f74dc103e20 2024-02-20T07:49:02.005 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:00.450+0000 7f74caffd700 1 --2- 172.21.15.133:0/841063019 >> [v2:172.21.15.133:6800/1670376163,v1:172.21.15.133:6801/1670376163] conn(0x7f74c40367e0 0x7f74c4038c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:49:02.005 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:00.450+0000 7f74caffd700 1 -- 172.21.15.133:0/841063019 <== mon.0 v2:172.21.15.133:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f74cc04a800 con 0x7f74dc103e20 2024-02-20T07:49:02.005 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:00.450+0000 7f74d9d9b700 1 --2- 172.21.15.133:0/841063019 >> [v2:172.21.15.133:6800/1670376163,v1:172.21.15.133:6801/1670376163] conn(0x7f74c40367e0 0x7f74c4038c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:49:02.005 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:00.450+0000 7f74d9d9b700 1 --2- 172.21.15.133:0/841063019 >> [v2:172.21.15.133:6800/1670376163,v1:172.21.15.133:6801/1670376163] conn(0x7f74c40367e0 0x7f74c4038c90 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f74d0009990 tx=0x7f74d0006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:49:02.005 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:00.455+0000 7f74caffd700 1 -- 172.21.15.133:0/841063019 <== mon.0 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f74cc02d770 con 0x7f74dc103e20 2024-02-20T07:49:02.006 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:00.649+0000 7f74e0c3a700 1 -- 172.21.15.133:0/841063019 --> [v2:172.21.15.133:6800/1670376163,v1:172.21.15.133:6801/1670376163] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "target": ["mon-mgr", ""]}) v1 -- 0x7f74dc101440 con 0x7f74c40367e0 2024-02-20T07:49:02.006 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:00.654+0000 7f74caffd700 1 -- 172.21.15.133:0/841063019 <== mgr.14120 v2:172.21.15.133:6800/1670376163 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f74dc101440 con 0x7f74c40367e0 2024-02-20T07:49:02.006 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:00.656+0000 7f74e0c3a700 1 -- 172.21.15.133:0/841063019 >> [v2:172.21.15.133:6800/1670376163,v1:172.21.15.133:6801/1670376163] conn(0x7f74c40367e0 msgr2=0x7f74c4038c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:49:02.006 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:00.656+0000 7f74e0c3a700 1 --2- 172.21.15.133:0/841063019 >> [v2:172.21.15.133:6800/1670376163,v1:172.21.15.133:6801/1670376163] conn(0x7f74c40367e0 0x7f74c4038c90 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f74d0009990 tx=0x7f74d0006e40 comp rx=0 tx=0).stop 2024-02-20T07:49:02.006 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:00.656+0000 7f74e0c3a700 1 -- 172.21.15.133:0/841063019 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f74dc103e20 msgr2=0x7f74dc111800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:49:02.006 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:00.656+0000 7f74e0c3a700 1 --2- 172.21.15.133:0/841063019 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f74dc103e20 0x7f74dc111800 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f74cc00b440 tx=0x7f74cc009250 comp rx=0 tx=0).stop 2024-02-20T07:49:02.006 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:00.656+0000 7f74e0c3a700 1 -- 172.21.15.133:0/841063019 shutdown_connections 2024-02-20T07:49:02.006 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:00.656+0000 7f74e0c3a700 1 --2- 172.21.15.133:0/841063019 >> [v2:172.21.15.133:6800/1670376163,v1:172.21.15.133:6801/1670376163] conn(0x7f74c40367e0 0x7f74c4038c90 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:49:02.006 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:00.656+0000 7f74e0c3a700 1 --2- 172.21.15.133:0/841063019 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f74dc103e20 0x7f74dc111800 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:49:02.006 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:00.656+0000 7f74e0c3a700 1 -- 172.21.15.133:0/841063019 >> 172.21.15.133:0/841063019 conn(0x7f74dc0fc580 msgr2=0x7f74dc0fd440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:49:02.006 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:00.656+0000 7f74e0c3a700 1 -- 172.21.15.133:0/841063019 shutdown_connections 2024-02-20T07:49:02.006 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:00.656+0000 7f74e0c3a700 1 -- 172.21.15.133:0/841063019 wait complete. 2024-02-20T07:49:02.006 INFO:teuthology.orchestra.run.smithi133.stdout:Deploying crash service with default placement... 2024-02-20T07:49:02.962 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:49:02 smithi133 ceph-mon[29117]: from='mgr.14120 172.21.15.133:0/2965705206' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:49:02.962 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:49:02 smithi133 ceph-mon[29117]: from='mgr.14120 172.21.15.133:0/2965705206' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:49:03.825 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stdout Scheduled crash update... 2024-02-20T07:49:03.825 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:02.491+0000 7f803f7a9700 1 Processor -- start 2024-02-20T07:49:03.825 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:02.492+0000 7f803f7a9700 1 -- start start 2024-02-20T07:49:03.825 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:02.492+0000 7f803f7a9700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f8038102770 0x7f8038102b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:49:03.825 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:02.492+0000 7f803f7a9700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80381030c0 con 0x7f8038102770 2024-02-20T07:49:03.826 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:02.492+0000 7f803d545700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f8038102770 0x7f8038102b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:49:03.826 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:02.492+0000 7f803d545700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f8038102770 0x7f8038102b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.133:49952/0 (socket says 172.21.15.133:49952) 2024-02-20T07:49:03.826 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:02.492+0000 7f803d545700 1 -- 172.21.15.133:0/391641635 learned_addr learned my addr 172.21.15.133:0/391641635 (peer_addr_for_me v2:172.21.15.133:0/0) 2024-02-20T07:49:03.826 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:02.492+0000 7f803d545700 1 -- 172.21.15.133:0/391641635 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8038103200 con 0x7f8038102770 2024-02-20T07:49:03.827 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:02.492+0000 7f803d545700 1 --2- 172.21.15.133:0/391641635 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f8038102770 0x7f8038102b80 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f80340098e0 tx=0x7f8034009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=73b9a9c3f58c1d9b server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:49:03.827 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:02.493+0000 7f802ffff700 1 -- 172.21.15.133:0/391641635 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f803400b860 con 0x7f8038102770 2024-02-20T07:49:03.827 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:02.493+0000 7f802ffff700 1 -- 172.21.15.133:0/391641635 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f803400be90 con 0x7f8038102770 2024-02-20T07:49:03.827 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:02.493+0000 7f802ffff700 1 -- 172.21.15.133:0/391641635 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f80340103c0 con 0x7f8038102770 2024-02-20T07:49:03.827 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:02.493+0000 7f803f7a9700 1 -- 172.21.15.133:0/391641635 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f8038102770 msgr2=0x7f8038102b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:49:03.827 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:02.493+0000 7f803f7a9700 1 --2- 172.21.15.133:0/391641635 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f8038102770 0x7f8038102b80 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f80340098e0 tx=0x7f8034009250 comp rx=0 tx=0).stop 2024-02-20T07:49:03.827 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:02.493+0000 7f803f7a9700 1 -- 172.21.15.133:0/391641635 shutdown_connections 2024-02-20T07:49:03.827 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:02.493+0000 7f803f7a9700 1 --2- 172.21.15.133:0/391641635 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f8038102770 0x7f8038102b80 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:49:03.827 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:02.493+0000 7f803f7a9700 1 -- 172.21.15.133:0/391641635 >> 172.21.15.133:0/391641635 conn(0x7f80380fded0 msgr2=0x7f8038100300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:49:03.827 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:02.493+0000 7f803f7a9700 1 -- 172.21.15.133:0/391641635 shutdown_connections 2024-02-20T07:49:03.827 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:02.493+0000 7f803f7a9700 1 -- 172.21.15.133:0/391641635 wait complete. 2024-02-20T07:49:03.827 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:02.494+0000 7f803f7a9700 1 Processor -- start 2024-02-20T07:49:03.827 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:02.494+0000 7f803f7a9700 1 -- start start 2024-02-20T07:49:03.828 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:02.494+0000 7f803f7a9700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f8038102770 0x7f80381115b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:49:03.828 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:02.494+0000 7f803f7a9700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8038111af0 con 0x7f8038102770 2024-02-20T07:49:03.828 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:02.494+0000 7f803d545700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f8038102770 0x7f80381115b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:49:03.828 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:02.494+0000 7f803d545700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f8038102770 0x7f80381115b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.133:49954/0 (socket says 172.21.15.133:49954) 2024-02-20T07:49:03.828 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:02.494+0000 7f803d545700 1 -- 172.21.15.133:0/3290693648 learned_addr learned my addr 172.21.15.133:0/3290693648 (peer_addr_for_me v2:172.21.15.133:0/0) 2024-02-20T07:49:03.828 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:02.494+0000 7f803d545700 1 -- 172.21.15.133:0/3290693648 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80340094f0 con 0x7f8038102770 2024-02-20T07:49:03.828 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:02.494+0000 7f803d545700 1 --2- 172.21.15.133:0/3290693648 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f8038102770 0x7f80381115b0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f80340090b0 tx=0x7f8034009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:49:03.828 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:02.495+0000 7f802dffb700 1 -- 172.21.15.133:0/3290693648 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8034010840 con 0x7f8038102770 2024-02-20T07:49:03.828 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:02.495+0000 7f802dffb700 1 -- 172.21.15.133:0/3290693648 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8034010e70 con 0x7f8038102770 2024-02-20T07:49:03.828 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:02.495+0000 7f802dffb700 1 -- 172.21.15.133:0/3290693648 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8034024e40 con 0x7f8038102770 2024-02-20T07:49:03.828 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:02.495+0000 7f803f7a9700 1 -- 172.21.15.133:0/3290693648 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8038111cf0 con 0x7f8038102770 2024-02-20T07:49:03.828 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:02.495+0000 7f803f7a9700 1 -- 172.21.15.133:0/3290693648 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8038112990 con 0x7f8038102770 2024-02-20T07:49:03.828 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:02.496+0000 7f802dffb700 1 -- 172.21.15.133:0/3290693648 <== mon.0 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 8) v1 ==== 43368+0+0 (secure 0 0 0) 0x7f8034024430 con 0x7f8038102770 2024-02-20T07:49:03.828 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:02.496+0000 7f803f7a9700 1 -- 172.21.15.133:0/3290693648 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f803805f5c0 con 0x7f8038102770 2024-02-20T07:49:03.829 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:02.496+0000 7f802dffb700 1 --2- 172.21.15.133:0/3290693648 >> [v2:172.21.15.133:6800/1670376163,v1:172.21.15.133:6801/1670376163] conn(0x7f8024036790 0x7f8024038c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:49:03.829 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:02.496+0000 7f802dffb700 1 -- 172.21.15.133:0/3290693648 <== mon.0 v2:172.21.15.133:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f803404a810 con 0x7f8038102770 2024-02-20T07:49:03.829 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:02.497+0000 7f803cd44700 1 --2- 172.21.15.133:0/3290693648 >> [v2:172.21.15.133:6800/1670376163,v1:172.21.15.133:6801/1670376163] conn(0x7f8024036790 0x7f8024038c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:49:03.829 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:02.497+0000 7f803cd44700 1 --2- 172.21.15.133:0/3290693648 >> [v2:172.21.15.133:6800/1670376163,v1:172.21.15.133:6801/1670376163] conn(0x7f8024036790 0x7f8024038c40 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f8028009990 tx=0x7f8028006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:49:03.829 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:02.501+0000 7f802dffb700 1 -- 172.21.15.133:0/3290693648 <== mon.0 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f803402c930 con 0x7f8038102770 2024-02-20T07:49:03.829 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:02.714+0000 7f803f7a9700 1 -- 172.21.15.133:0/3290693648 --> [v2:172.21.15.133:6800/1670376163,v1:172.21.15.133:6801/1670376163] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "crash", "target": ["mon-mgr", ""]}) v1 -- 0x7f80381001b0 con 0x7f8024036790 2024-02-20T07:49:03.829 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:02.719+0000 7f802dffb700 1 -- 172.21.15.133:0/3290693648 <== mgr.14120 v2:172.21.15.133:6800/1670376163 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+26 (secure 0 0 0) 0x7f80381001b0 con 0x7f8024036790 2024-02-20T07:49:03.829 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:02.721+0000 7f803f7a9700 1 -- 172.21.15.133:0/3290693648 >> [v2:172.21.15.133:6800/1670376163,v1:172.21.15.133:6801/1670376163] conn(0x7f8024036790 msgr2=0x7f8024038c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:49:03.829 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:02.721+0000 7f803f7a9700 1 --2- 172.21.15.133:0/3290693648 >> [v2:172.21.15.133:6800/1670376163,v1:172.21.15.133:6801/1670376163] conn(0x7f8024036790 0x7f8024038c40 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f8028009990 tx=0x7f8028006e40 comp rx=0 tx=0).stop 2024-02-20T07:49:03.829 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:02.721+0000 7f803f7a9700 1 -- 172.21.15.133:0/3290693648 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f8038102770 msgr2=0x7f80381115b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:49:03.829 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:02.721+0000 7f803f7a9700 1 --2- 172.21.15.133:0/3290693648 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f8038102770 0x7f80381115b0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f80340090b0 tx=0x7f8034009250 comp rx=0 tx=0).stop 2024-02-20T07:49:03.829 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:02.721+0000 7f803f7a9700 1 -- 172.21.15.133:0/3290693648 shutdown_connections 2024-02-20T07:49:03.829 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:02.721+0000 7f803f7a9700 1 --2- 172.21.15.133:0/3290693648 >> [v2:172.21.15.133:6800/1670376163,v1:172.21.15.133:6801/1670376163] conn(0x7f8024036790 0x7f8024038c40 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:49:03.829 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:02.721+0000 7f803f7a9700 1 --2- 172.21.15.133:0/3290693648 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f8038102770 0x7f80381115b0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:49:03.830 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:02.721+0000 7f803f7a9700 1 -- 172.21.15.133:0/3290693648 >> 172.21.15.133:0/3290693648 conn(0x7f80380fded0 msgr2=0x7f80380ff590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:49:03.830 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:02.721+0000 7f803f7a9700 1 -- 172.21.15.133:0/3290693648 shutdown_connections 2024-02-20T07:49:03.830 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:02.721+0000 7f803f7a9700 1 -- 172.21.15.133:0/3290693648 wait complete. 2024-02-20T07:49:03.830 INFO:teuthology.orchestra.run.smithi133.stdout:Deploying prometheus service with default placement... 2024-02-20T07:49:03.844 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:49:03 smithi133 ceph-mon[29117]: from='client.14146 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "crash", "target": ["mon-mgr", ""]}]: dispatch 2024-02-20T07:49:03.844 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:49:03 smithi133 ceph-mon[29117]: Saving service crash spec with placement * 2024-02-20T07:49:03.844 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:49:03 smithi133 ceph-mon[29117]: from='mgr.14120 172.21.15.133:0/2965705206' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:49:03.844 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:49:03 smithi133 ceph-mon[29117]: from='mgr.14120 172.21.15.133:0/2965705206' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:49:05.098 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stdout Scheduled prometheus update... 2024-02-20T07:49:05.098 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:04.275+0000 7f48a1aca700 1 Processor -- start 2024-02-20T07:49:05.099 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:04.275+0000 7f48a1aca700 1 -- start start 2024-02-20T07:49:05.099 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:04.275+0000 7f48a1aca700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f489c076b80 0x7f489c105420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:49:05.099 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:04.275+0000 7f48a1aca700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f489c105960 con 0x7f489c076b80 2024-02-20T07:49:05.099 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:04.275+0000 7f489b7fe700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f489c076b80 0x7f489c105420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:49:05.099 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:04.275+0000 7f489b7fe700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f489c076b80 0x7f489c105420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.133:49960/0 (socket says 172.21.15.133:49960) 2024-02-20T07:49:05.099 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:04.275+0000 7f489b7fe700 1 -- 172.21.15.133:0/1584895887 learned_addr learned my addr 172.21.15.133:0/1584895887 (peer_addr_for_me v2:172.21.15.133:0/0) 2024-02-20T07:49:05.100 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:04.276+0000 7f489b7fe700 1 -- 172.21.15.133:0/1584895887 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f489c105aa0 con 0x7f489c076b80 2024-02-20T07:49:05.100 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:04.276+0000 7f489b7fe700 1 --2- 172.21.15.133:0/1584895887 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f489c076b80 0x7f489c105420 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f488c0098e0 tx=0x7f488c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=af8e809e2b04c8f0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:49:05.100 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:04.276+0000 7f489a7fc700 1 -- 172.21.15.133:0/1584895887 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f488c00b860 con 0x7f489c076b80 2024-02-20T07:49:05.100 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:04.276+0000 7f489a7fc700 1 -- 172.21.15.133:0/1584895887 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f488c00be90 con 0x7f489c076b80 2024-02-20T07:49:05.100 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:04.276+0000 7f489a7fc700 1 -- 172.21.15.133:0/1584895887 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f488c0103c0 con 0x7f489c076b80 2024-02-20T07:49:05.101 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:04.277+0000 7f48a1aca700 1 -- 172.21.15.133:0/1584895887 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f489c076b80 msgr2=0x7f489c105420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:49:05.101 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:04.277+0000 7f48a1aca700 1 --2- 172.21.15.133:0/1584895887 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f489c076b80 0x7f489c105420 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f488c0098e0 tx=0x7f488c009250 comp rx=0 tx=0).stop 2024-02-20T07:49:05.101 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:04.277+0000 7f48a1aca700 1 -- 172.21.15.133:0/1584895887 shutdown_connections 2024-02-20T07:49:05.101 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:04.277+0000 7f48a1aca700 1 --2- 172.21.15.133:0/1584895887 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f489c076b80 0x7f489c105420 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:49:05.101 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:04.277+0000 7f48a1aca700 1 -- 172.21.15.133:0/1584895887 >> 172.21.15.133:0/1584895887 conn(0x7f489c0fcdd0 msgr2=0x7f489c0ff200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:49:05.101 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:04.277+0000 7f48a1aca700 1 -- 172.21.15.133:0/1584895887 shutdown_connections 2024-02-20T07:49:05.102 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:04.277+0000 7f48a1aca700 1 -- 172.21.15.133:0/1584895887 wait complete. 2024-02-20T07:49:05.102 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:04.277+0000 7f48a1aca700 1 Processor -- start 2024-02-20T07:49:05.102 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:04.278+0000 7f48a1aca700 1 -- start start 2024-02-20T07:49:05.102 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:04.278+0000 7f48a1aca700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f489c076b80 0x7f489c102ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:49:05.102 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:04.278+0000 7f48a1aca700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f489c101320 con 0x7f489c076b80 2024-02-20T07:49:05.102 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:04.278+0000 7f489b7fe700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f489c076b80 0x7f489c102ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:49:05.102 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:04.278+0000 7f489b7fe700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f489c076b80 0x7f489c102ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.133:49972/0 (socket says 172.21.15.133:49972) 2024-02-20T07:49:05.103 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:04.278+0000 7f489b7fe700 1 -- 172.21.15.133:0/1812815935 learned_addr learned my addr 172.21.15.133:0/1812815935 (peer_addr_for_me v2:172.21.15.133:0/0) 2024-02-20T07:49:05.103 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:04.278+0000 7f489b7fe700 1 -- 172.21.15.133:0/1812815935 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f488c0094f0 con 0x7f489c076b80 2024-02-20T07:49:05.103 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:04.278+0000 7f489b7fe700 1 --2- 172.21.15.133:0/1812815935 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f489c076b80 0x7f489c102ea0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f488c0090b0 tx=0x7f488c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:49:05.104 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:04.278+0000 7f4883fff700 1 -- 172.21.15.133:0/1812815935 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f488c0118f0 con 0x7f489c076b80 2024-02-20T07:49:05.104 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:04.278+0000 7f4883fff700 1 -- 172.21.15.133:0/1812815935 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f488c010d10 con 0x7f489c076b80 2024-02-20T07:49:05.104 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:04.279+0000 7f48a1aca700 1 -- 172.21.15.133:0/1812815935 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f489c101520 con 0x7f489c076b80 2024-02-20T07:49:05.104 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:04.279+0000 7f4883fff700 1 -- 172.21.15.133:0/1812815935 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f488c01aea0 con 0x7f489c076b80 2024-02-20T07:49:05.104 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:04.279+0000 7f48a1aca700 1 -- 172.21.15.133:0/1812815935 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f489c1019b0 con 0x7f489c076b80 2024-02-20T07:49:05.104 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:04.279+0000 7f4883fff700 1 -- 172.21.15.133:0/1812815935 <== mon.0 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 8) v1 ==== 43368+0+0 (secure 0 0 0) 0x7f488c0194e0 con 0x7f489c076b80 2024-02-20T07:49:05.105 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:04.280+0000 7f48a1aca700 1 -- 172.21.15.133:0/1812815935 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f489c04bed0 con 0x7f489c076b80 2024-02-20T07:49:05.105 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:04.280+0000 7f4883fff700 1 --2- 172.21.15.133:0/1812815935 >> [v2:172.21.15.133:6800/1670376163,v1:172.21.15.133:6801/1670376163] conn(0x7f4884036740 0x7f4884038bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:49:05.105 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:04.280+0000 7f4883fff700 1 -- 172.21.15.133:0/1812815935 <== mon.0 v2:172.21.15.133:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f488c010e80 con 0x7f489c076b80 2024-02-20T07:49:05.105 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:04.280+0000 7f489affd700 1 --2- 172.21.15.133:0/1812815935 >> [v2:172.21.15.133:6800/1670376163,v1:172.21.15.133:6801/1670376163] conn(0x7f4884036740 0x7f4884038bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:49:05.105 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:04.280+0000 7f489affd700 1 --2- 172.21.15.133:0/1812815935 >> [v2:172.21.15.133:6800/1670376163,v1:172.21.15.133:6801/1670376163] conn(0x7f4884036740 0x7f4884038bf0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f4890009990 tx=0x7f4890006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:49:05.105 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:04.284+0000 7f4883fff700 1 -- 172.21.15.133:0/1812815935 <== mon.0 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f488c022d90 con 0x7f489c076b80 2024-02-20T07:49:05.106 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:04.485+0000 7f48a1aca700 1 -- 172.21.15.133:0/1812815935 --> [v2:172.21.15.133:6800/1670376163,v1:172.21.15.133:6801/1670376163] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "prometheus", "target": ["mon-mgr", ""]}) v1 -- 0x7f489c0ff550 con 0x7f4884036740 2024-02-20T07:49:05.106 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:04.492+0000 7f4883fff700 1 -- 172.21.15.133:0/1812815935 <== mgr.14120 v2:172.21.15.133:6800/1670376163 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+31 (secure 0 0 0) 0x7f489c0ff550 con 0x7f4884036740 2024-02-20T07:49:05.106 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:04.494+0000 7f48a1aca700 1 -- 172.21.15.133:0/1812815935 >> [v2:172.21.15.133:6800/1670376163,v1:172.21.15.133:6801/1670376163] conn(0x7f4884036740 msgr2=0x7f4884038bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:49:05.106 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:04.494+0000 7f48a1aca700 1 --2- 172.21.15.133:0/1812815935 >> [v2:172.21.15.133:6800/1670376163,v1:172.21.15.133:6801/1670376163] conn(0x7f4884036740 0x7f4884038bf0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f4890009990 tx=0x7f4890006e40 comp rx=0 tx=0).stop 2024-02-20T07:49:05.106 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:04.494+0000 7f48a1aca700 1 -- 172.21.15.133:0/1812815935 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f489c076b80 msgr2=0x7f489c102ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:49:05.107 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:04.494+0000 7f48a1aca700 1 --2- 172.21.15.133:0/1812815935 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f489c076b80 0x7f489c102ea0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f488c0090b0 tx=0x7f488c009250 comp rx=0 tx=0).stop 2024-02-20T07:49:05.107 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:04.494+0000 7f48a1aca700 1 -- 172.21.15.133:0/1812815935 shutdown_connections 2024-02-20T07:49:05.107 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:04.494+0000 7f48a1aca700 1 --2- 172.21.15.133:0/1812815935 >> [v2:172.21.15.133:6800/1670376163,v1:172.21.15.133:6801/1670376163] conn(0x7f4884036740 0x7f4884038bf0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:49:05.107 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:04.494+0000 7f48a1aca700 1 --2- 172.21.15.133:0/1812815935 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f489c076b80 0x7f489c102ea0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:49:05.108 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:04.494+0000 7f48a1aca700 1 -- 172.21.15.133:0/1812815935 >> 172.21.15.133:0/1812815935 conn(0x7f489c0fcdd0 msgr2=0x7f489c0fe470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:49:05.108 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:04.494+0000 7f48a1aca700 1 -- 172.21.15.133:0/1812815935 shutdown_connections 2024-02-20T07:49:05.108 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:04.494+0000 7f48a1aca700 1 -- 172.21.15.133:0/1812815935 wait complete. 2024-02-20T07:49:05.108 INFO:teuthology.orchestra.run.smithi133.stdout:Deploying grafana service with default placement... 2024-02-20T07:49:05.399 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:49:05 smithi133 ceph-mon[29117]: from='mgr.14120 172.21.15.133:0/2965705206' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:49:05.399 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:49:05 smithi133 ceph-mon[29117]: from='client.14148 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "target": ["mon-mgr", ""]}]: dispatch 2024-02-20T07:49:05.399 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:49:05 smithi133 ceph-mon[29117]: Saving service prometheus spec with placement count:1 2024-02-20T07:49:05.399 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:49:05 smithi133 ceph-mon[29117]: from='mgr.14120 172.21.15.133:0/2965705206' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:49:06.766 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:49:06 smithi133 ceph-mon[29117]: from='client.14150 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "target": ["mon-mgr", ""]}]: dispatch 2024-02-20T07:49:06.766 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:49:06 smithi133 ceph-mon[29117]: Saving service grafana spec with placement count:1 2024-02-20T07:49:06.766 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:49:06 smithi133 ceph-mon[29117]: from='mgr.14120 172.21.15.133:0/2965705206' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:49:07.021 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stdout Scheduled grafana update... 2024-02-20T07:49:07.021 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:05.496+0000 7ff596cc6700 1 Processor -- start 2024-02-20T07:49:07.021 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:05.496+0000 7ff596cc6700 1 -- start start 2024-02-20T07:49:07.021 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:05.496+0000 7ff596cc6700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ff590102780 0x7ff590102b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:49:07.021 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:05.496+0000 7ff596cc6700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5901030d0 con 0x7ff590102780 2024-02-20T07:49:07.021 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:05.496+0000 7ff594a62700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ff590102780 0x7ff590102b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:49:07.022 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:05.496+0000 7ff594a62700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ff590102780 0x7ff590102b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.133:49980/0 (socket says 172.21.15.133:49980) 2024-02-20T07:49:07.022 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:05.496+0000 7ff594a62700 1 -- 172.21.15.133:0/3771171120 learned_addr learned my addr 172.21.15.133:0/3771171120 (peer_addr_for_me v2:172.21.15.133:0/0) 2024-02-20T07:49:07.022 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:05.496+0000 7ff594a62700 1 -- 172.21.15.133:0/3771171120 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff590103210 con 0x7ff590102780 2024-02-20T07:49:07.022 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:05.497+0000 7ff594a62700 1 --2- 172.21.15.133:0/3771171120 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ff590102780 0x7ff590102b90 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7ff5840098e0 tx=0x7ff584009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b6fe1bea6dbfda2e server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:49:07.022 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:05.497+0000 7ff58f7fe700 1 -- 172.21.15.133:0/3771171120 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff58400b860 con 0x7ff590102780 2024-02-20T07:49:07.022 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:05.497+0000 7ff58f7fe700 1 -- 172.21.15.133:0/3771171120 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff58400be90 con 0x7ff590102780 2024-02-20T07:49:07.022 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:05.497+0000 7ff58f7fe700 1 -- 172.21.15.133:0/3771171120 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff5840103c0 con 0x7ff590102780 2024-02-20T07:49:07.022 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:05.498+0000 7ff596cc6700 1 -- 172.21.15.133:0/3771171120 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ff590102780 msgr2=0x7ff590102b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:49:07.022 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:05.498+0000 7ff596cc6700 1 --2- 172.21.15.133:0/3771171120 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ff590102780 0x7ff590102b90 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7ff5840098e0 tx=0x7ff584009250 comp rx=0 tx=0).stop 2024-02-20T07:49:07.022 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:05.498+0000 7ff596cc6700 1 -- 172.21.15.133:0/3771171120 shutdown_connections 2024-02-20T07:49:07.023 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:05.498+0000 7ff596cc6700 1 --2- 172.21.15.133:0/3771171120 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ff590102780 0x7ff590102b90 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:49:07.023 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:05.498+0000 7ff596cc6700 1 -- 172.21.15.133:0/3771171120 >> 172.21.15.133:0/3771171120 conn(0x7ff5900fdf00 msgr2=0x7ff590100310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:49:07.023 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:05.498+0000 7ff596cc6700 1 -- 172.21.15.133:0/3771171120 shutdown_connections 2024-02-20T07:49:07.023 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:05.498+0000 7ff596cc6700 1 -- 172.21.15.133:0/3771171120 wait complete. 2024-02-20T07:49:07.023 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:05.499+0000 7ff596cc6700 1 Processor -- start 2024-02-20T07:49:07.023 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:05.499+0000 7ff596cc6700 1 -- start start 2024-02-20T07:49:07.024 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:05.499+0000 7ff596cc6700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ff590102780 0x7ff590111590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:49:07.024 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:05.499+0000 7ff596cc6700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff590111ad0 con 0x7ff590102780 2024-02-20T07:49:07.024 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:05.499+0000 7ff594a62700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ff590102780 0x7ff590111590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:49:07.024 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:05.499+0000 7ff594a62700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ff590102780 0x7ff590111590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.133:49992/0 (socket says 172.21.15.133:49992) 2024-02-20T07:49:07.024 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:05.499+0000 7ff594a62700 1 -- 172.21.15.133:0/1865187840 learned_addr learned my addr 172.21.15.133:0/1865187840 (peer_addr_for_me v2:172.21.15.133:0/0) 2024-02-20T07:49:07.024 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:05.499+0000 7ff594a62700 1 -- 172.21.15.133:0/1865187840 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5840094f0 con 0x7ff590102780 2024-02-20T07:49:07.024 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:05.500+0000 7ff594a62700 1 --2- 172.21.15.133:0/1865187840 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ff590102780 0x7ff590111590 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7ff5840090b0 tx=0x7ff584009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:49:07.024 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:05.500+0000 7ff58d7fa700 1 -- 172.21.15.133:0/1865187840 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff584010840 con 0x7ff590102780 2024-02-20T07:49:07.024 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:05.500+0000 7ff58d7fa700 1 -- 172.21.15.133:0/1865187840 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff584010e70 con 0x7ff590102780 2024-02-20T07:49:07.024 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:05.500+0000 7ff596cc6700 1 -- 172.21.15.133:0/1865187840 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff590111cd0 con 0x7ff590102780 2024-02-20T07:49:07.025 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:05.500+0000 7ff596cc6700 1 -- 172.21.15.133:0/1865187840 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff590112970 con 0x7ff590102780 2024-02-20T07:49:07.025 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:05.500+0000 7ff58d7fa700 1 -- 172.21.15.133:0/1865187840 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff584024e40 con 0x7ff590102780 2024-02-20T07:49:07.025 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:05.501+0000 7ff58d7fa700 1 -- 172.21.15.133:0/1865187840 <== mon.0 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 8) v1 ==== 43368+0+0 (secure 0 0 0) 0x7ff58401a570 con 0x7ff590102780 2024-02-20T07:49:07.025 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:05.501+0000 7ff596cc6700 1 -- 172.21.15.133:0/1865187840 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff59005f5c0 con 0x7ff590102780 2024-02-20T07:49:07.025 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:05.501+0000 7ff58d7fa700 1 --2- 172.21.15.133:0/1865187840 >> [v2:172.21.15.133:6800/1670376163,v1:172.21.15.133:6801/1670376163] conn(0x7ff580036740 0x7ff580038bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:49:07.025 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:05.501+0000 7ff58d7fa700 1 -- 172.21.15.133:0/1865187840 <== mon.0 v2:172.21.15.133:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7ff58404a840 con 0x7ff590102780 2024-02-20T07:49:07.025 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:05.501+0000 7ff58ffff700 1 --2- 172.21.15.133:0/1865187840 >> [v2:172.21.15.133:6800/1670376163,v1:172.21.15.133:6801/1670376163] conn(0x7ff580036740 0x7ff580038bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:49:07.025 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:05.502+0000 7ff58ffff700 1 --2- 172.21.15.133:0/1865187840 >> [v2:172.21.15.133:6800/1670376163,v1:172.21.15.133:6801/1670376163] conn(0x7ff580036740 0x7ff580038bf0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7ff57c009990 tx=0x7ff57c006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:49:07.025 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:05.506+0000 7ff58d7fa700 1 -- 172.21.15.133:0/1865187840 <== mon.0 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7ff584018940 con 0x7ff590102780 2024-02-20T07:49:07.025 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:05.720+0000 7ff596cc6700 1 -- 172.21.15.133:0/1865187840 --> [v2:172.21.15.133:6800/1670376163,v1:172.21.15.133:6801/1670376163] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "grafana", "target": ["mon-mgr", ""]}) v1 -- 0x7ff5901001b0 con 0x7ff580036740 2024-02-20T07:49:07.025 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:05.724+0000 7ff58d7fa700 1 -- 172.21.15.133:0/1865187840 <== mgr.14120 v2:172.21.15.133:6800/1670376163 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+28 (secure 0 0 0) 0x7ff5901001b0 con 0x7ff580036740 2024-02-20T07:49:07.025 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:05.726+0000 7ff596cc6700 1 -- 172.21.15.133:0/1865187840 >> [v2:172.21.15.133:6800/1670376163,v1:172.21.15.133:6801/1670376163] conn(0x7ff580036740 msgr2=0x7ff580038bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:49:07.025 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:05.726+0000 7ff596cc6700 1 --2- 172.21.15.133:0/1865187840 >> [v2:172.21.15.133:6800/1670376163,v1:172.21.15.133:6801/1670376163] conn(0x7ff580036740 0x7ff580038bf0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7ff57c009990 tx=0x7ff57c006e40 comp rx=0 tx=0).stop 2024-02-20T07:49:07.026 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:05.726+0000 7ff596cc6700 1 -- 172.21.15.133:0/1865187840 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ff590102780 msgr2=0x7ff590111590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:49:07.026 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:05.726+0000 7ff596cc6700 1 --2- 172.21.15.133:0/1865187840 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ff590102780 0x7ff590111590 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7ff5840090b0 tx=0x7ff584009250 comp rx=0 tx=0).stop 2024-02-20T07:49:07.026 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:05.727+0000 7ff596cc6700 1 -- 172.21.15.133:0/1865187840 shutdown_connections 2024-02-20T07:49:07.026 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:05.727+0000 7ff596cc6700 1 --2- 172.21.15.133:0/1865187840 >> [v2:172.21.15.133:6800/1670376163,v1:172.21.15.133:6801/1670376163] conn(0x7ff580036740 0x7ff580038bf0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:49:07.026 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:05.727+0000 7ff596cc6700 1 --2- 172.21.15.133:0/1865187840 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ff590102780 0x7ff590111590 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:49:07.026 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:05.727+0000 7ff596cc6700 1 -- 172.21.15.133:0/1865187840 >> 172.21.15.133:0/1865187840 conn(0x7ff5900fdf00 msgr2=0x7ff5900ff590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:49:07.026 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:05.727+0000 7ff596cc6700 1 -- 172.21.15.133:0/1865187840 shutdown_connections 2024-02-20T07:49:07.026 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:05.727+0000 7ff596cc6700 1 -- 172.21.15.133:0/1865187840 wait complete. 2024-02-20T07:49:07.026 INFO:teuthology.orchestra.run.smithi133.stdout:Deploying node-exporter service with default placement... 2024-02-20T07:49:08.214 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stdout Scheduled node-exporter update... 2024-02-20T07:49:08.214 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:07.410+0000 7fe9c8281700 1 Processor -- start 2024-02-20T07:49:08.214 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:07.410+0000 7fe9c8281700 1 -- start start 2024-02-20T07:49:08.214 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:07.411+0000 7fe9c8281700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fe9c00714c0 0x7fe9c0105340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:49:08.215 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:07.411+0000 7fe9c8281700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9c0105880 con 0x7fe9c00714c0 2024-02-20T07:49:08.215 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:07.411+0000 7fe9c601d700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fe9c00714c0 0x7fe9c0105340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:49:08.215 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:07.411+0000 7fe9c601d700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fe9c00714c0 0x7fe9c0105340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.133:43416/0 (socket says 172.21.15.133:43416) 2024-02-20T07:49:08.215 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:07.411+0000 7fe9c601d700 1 -- 172.21.15.133:0/2604254581 learned_addr learned my addr 172.21.15.133:0/2604254581 (peer_addr_for_me v2:172.21.15.133:0/0) 2024-02-20T07:49:08.215 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:07.411+0000 7fe9c601d700 1 -- 172.21.15.133:0/2604254581 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9c01059c0 con 0x7fe9c00714c0 2024-02-20T07:49:08.215 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:07.411+0000 7fe9c601d700 1 --2- 172.21.15.133:0/2604254581 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fe9c00714c0 0x7fe9c0105340 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fe9b00098e0 tx=0x7fe9b0009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=75abe4b679bb4133 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:49:08.215 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:07.411+0000 7fe9c501b700 1 -- 172.21.15.133:0/2604254581 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe9b000b860 con 0x7fe9c00714c0 2024-02-20T07:49:08.215 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:07.412+0000 7fe9c501b700 1 -- 172.21.15.133:0/2604254581 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe9b000be90 con 0x7fe9c00714c0 2024-02-20T07:49:08.215 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:07.412+0000 7fe9c501b700 1 -- 172.21.15.133:0/2604254581 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe9b00103c0 con 0x7fe9c00714c0 2024-02-20T07:49:08.215 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:07.412+0000 7fe9c8281700 1 -- 172.21.15.133:0/2604254581 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fe9c00714c0 msgr2=0x7fe9c0105340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:49:08.215 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:07.412+0000 7fe9c8281700 1 --2- 172.21.15.133:0/2604254581 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fe9c00714c0 0x7fe9c0105340 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fe9b00098e0 tx=0x7fe9b0009250 comp rx=0 tx=0).stop 2024-02-20T07:49:08.215 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:07.412+0000 7fe9c8281700 1 -- 172.21.15.133:0/2604254581 shutdown_connections 2024-02-20T07:49:08.215 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:07.412+0000 7fe9c8281700 1 --2- 172.21.15.133:0/2604254581 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fe9c00714c0 0x7fe9c0105340 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:49:08.215 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:07.412+0000 7fe9c8281700 1 -- 172.21.15.133:0/2604254581 >> 172.21.15.133:0/2604254581 conn(0x7fe9c00fcdd0 msgr2=0x7fe9c00ff200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:49:08.216 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:07.412+0000 7fe9c8281700 1 -- 172.21.15.133:0/2604254581 shutdown_connections 2024-02-20T07:49:08.216 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:07.412+0000 7fe9c8281700 1 -- 172.21.15.133:0/2604254581 wait complete. 2024-02-20T07:49:08.216 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:07.413+0000 7fe9c8281700 1 Processor -- start 2024-02-20T07:49:08.216 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:07.413+0000 7fe9c8281700 1 -- start start 2024-02-20T07:49:08.216 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:07.413+0000 7fe9c8281700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fe9c00714c0 0x7fe9c0102dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:49:08.216 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:07.413+0000 7fe9c8281700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9c0101240 con 0x7fe9c00714c0 2024-02-20T07:49:08.216 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:07.413+0000 7fe9c601d700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fe9c00714c0 0x7fe9c0102dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:49:08.216 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:07.413+0000 7fe9c601d700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fe9c00714c0 0x7fe9c0102dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.133:43420/0 (socket says 172.21.15.133:43420) 2024-02-20T07:49:08.216 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:07.413+0000 7fe9c601d700 1 -- 172.21.15.133:0/2573422082 learned_addr learned my addr 172.21.15.133:0/2573422082 (peer_addr_for_me v2:172.21.15.133:0/0) 2024-02-20T07:49:08.216 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:07.413+0000 7fe9c601d700 1 -- 172.21.15.133:0/2573422082 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9b00094f0 con 0x7fe9c00714c0 2024-02-20T07:49:08.216 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:07.413+0000 7fe9c601d700 1 --2- 172.21.15.133:0/2573422082 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fe9c00714c0 0x7fe9c0102dc0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fe9b00094c0 tx=0x7fe9b0009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:49:08.216 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:07.414+0000 7fe9b6ffd700 1 -- 172.21.15.133:0/2573422082 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe9b0010890 con 0x7fe9c00714c0 2024-02-20T07:49:08.216 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:07.414+0000 7fe9c8281700 1 -- 172.21.15.133:0/2573422082 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe9c0101440 con 0x7fe9c00714c0 2024-02-20T07:49:08.216 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:07.414+0000 7fe9b6ffd700 1 -- 172.21.15.133:0/2573422082 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe9b0010ec0 con 0x7fe9c00714c0 2024-02-20T07:49:08.217 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:07.414+0000 7fe9b6ffd700 1 -- 172.21.15.133:0/2573422082 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe9b00243f0 con 0x7fe9c00714c0 2024-02-20T07:49:08.217 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:07.414+0000 7fe9c8281700 1 -- 172.21.15.133:0/2573422082 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe9c01018d0 con 0x7fe9c00714c0 2024-02-20T07:49:08.217 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:07.414+0000 7fe9b6ffd700 1 -- 172.21.15.133:0/2573422082 <== mon.0 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 8) v1 ==== 43368+0+0 (secure 0 0 0) 0x7fe9b002d3f0 con 0x7fe9c00714c0 2024-02-20T07:49:08.217 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:07.414+0000 7fe9c8281700 1 -- 172.21.15.133:0/2573422082 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe9c010b680 con 0x7fe9c00714c0 2024-02-20T07:49:08.217 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:07.415+0000 7fe9b6ffd700 1 --2- 172.21.15.133:0/2573422082 >> [v2:172.21.15.133:6800/1670376163,v1:172.21.15.133:6801/1670376163] conn(0x7fe9ac036790 0x7fe9ac038c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:49:08.217 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:07.415+0000 7fe9b6ffd700 1 -- 172.21.15.133:0/2573422082 <== mon.0 v2:172.21.15.133:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7fe9b004a3f0 con 0x7fe9c00714c0 2024-02-20T07:49:08.217 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:07.415+0000 7fe9c581c700 1 --2- 172.21.15.133:0/2573422082 >> [v2:172.21.15.133:6800/1670376163,v1:172.21.15.133:6801/1670376163] conn(0x7fe9ac036790 0x7fe9ac038c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:49:08.217 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:07.415+0000 7fe9c581c700 1 --2- 172.21.15.133:0/2573422082 >> [v2:172.21.15.133:6800/1670376163,v1:172.21.15.133:6801/1670376163] conn(0x7fe9ac036790 0x7fe9ac038c40 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fe9bc009990 tx=0x7fe9bc006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:49:08.217 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:07.419+0000 7fe9b6ffd700 1 -- 172.21.15.133:0/2573422082 <== mon.0 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7fe9b002b8e0 con 0x7fe9c00714c0 2024-02-20T07:49:08.217 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:07.614+0000 7fe9c8281700 1 -- 172.21.15.133:0/2573422082 --> [v2:172.21.15.133:6800/1670376163,v1:172.21.15.133:6801/1670376163] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "node-exporter", "target": ["mon-mgr", ""]}) v1 -- 0x7fe9c002b410 con 0x7fe9ac036790 2024-02-20T07:49:08.217 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:07.618+0000 7fe9b6ffd700 1 -- 172.21.15.133:0/2573422082 <== mgr.14120 v2:172.21.15.133:6800/1670376163 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+34 (secure 0 0 0) 0x7fe9c002b410 con 0x7fe9ac036790 2024-02-20T07:49:08.217 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:07.620+0000 7fe9c8281700 1 -- 172.21.15.133:0/2573422082 >> [v2:172.21.15.133:6800/1670376163,v1:172.21.15.133:6801/1670376163] conn(0x7fe9ac036790 msgr2=0x7fe9ac038c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:49:08.217 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:07.620+0000 7fe9c8281700 1 --2- 172.21.15.133:0/2573422082 >> [v2:172.21.15.133:6800/1670376163,v1:172.21.15.133:6801/1670376163] conn(0x7fe9ac036790 0x7fe9ac038c40 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fe9bc009990 tx=0x7fe9bc006e40 comp rx=0 tx=0).stop 2024-02-20T07:49:08.218 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:07.620+0000 7fe9c8281700 1 -- 172.21.15.133:0/2573422082 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fe9c00714c0 msgr2=0x7fe9c0102dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:49:08.218 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:07.620+0000 7fe9c8281700 1 --2- 172.21.15.133:0/2573422082 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fe9c00714c0 0x7fe9c0102dc0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fe9b00094c0 tx=0x7fe9b0009250 comp rx=0 tx=0).stop 2024-02-20T07:49:08.218 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:07.620+0000 7fe9c8281700 1 -- 172.21.15.133:0/2573422082 shutdown_connections 2024-02-20T07:49:08.218 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:07.620+0000 7fe9c8281700 1 --2- 172.21.15.133:0/2573422082 >> [v2:172.21.15.133:6800/1670376163,v1:172.21.15.133:6801/1670376163] conn(0x7fe9ac036790 0x7fe9ac038c40 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:49:08.218 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:07.620+0000 7fe9c8281700 1 --2- 172.21.15.133:0/2573422082 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fe9c00714c0 0x7fe9c0102dc0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:49:08.218 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:07.620+0000 7fe9c8281700 1 -- 172.21.15.133:0/2573422082 >> 172.21.15.133:0/2573422082 conn(0x7fe9c00fcdd0 msgr2=0x7fe9c00fe540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:49:08.218 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:07.620+0000 7fe9c8281700 1 -- 172.21.15.133:0/2573422082 shutdown_connections 2024-02-20T07:49:08.218 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:07.620+0000 7fe9c8281700 1 -- 172.21.15.133:0/2573422082 wait complete. 2024-02-20T07:49:08.218 INFO:teuthology.orchestra.run.smithi133.stdout:Deploying alertmanager service with default placement... 2024-02-20T07:49:08.869 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:49:08 smithi133 ceph-mon[29117]: from='client.14152 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "target": ["mon-mgr", ""]}]: dispatch 2024-02-20T07:49:08.870 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:49:08 smithi133 ceph-mon[29117]: Saving service node-exporter spec with placement * 2024-02-20T07:49:08.870 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:49:08 smithi133 ceph-mon[29117]: from='mgr.14120 172.21.15.133:0/2965705206' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:49:09.446 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stdout Scheduled alertmanager update... 2024-02-20T07:49:09.446 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:08.663+0000 7fcd6c4e6700 1 Processor -- start 2024-02-20T07:49:09.446 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:08.663+0000 7fcd6c4e6700 1 -- start start 2024-02-20T07:49:09.446 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:08.663+0000 7fcd6c4e6700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fcd64076c00 0x7fcd641077a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:49:09.446 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:08.663+0000 7fcd6c4e6700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd64103190 con 0x7fcd64076c00 2024-02-20T07:49:09.446 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:08.663+0000 7fcd6a282700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fcd64076c00 0x7fcd641077a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:49:09.446 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:08.663+0000 7fcd6a282700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fcd64076c00 0x7fcd641077a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.133:43430/0 (socket says 172.21.15.133:43430) 2024-02-20T07:49:09.446 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:08.663+0000 7fcd6a282700 1 -- 172.21.15.133:0/1577016957 learned_addr learned my addr 172.21.15.133:0/1577016957 (peer_addr_for_me v2:172.21.15.133:0/0) 2024-02-20T07:49:09.446 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:08.664+0000 7fcd6a282700 1 -- 172.21.15.133:0/1577016957 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd64107ce0 con 0x7fcd64076c00 2024-02-20T07:49:09.446 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:08.664+0000 7fcd6a282700 1 --2- 172.21.15.133:0/1577016957 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fcd64076c00 0x7fcd641077a0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fcd600098e0 tx=0x7fcd60009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4a73daa2ce5fad70 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:49:09.447 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:08.664+0000 7fcd69280700 1 -- 172.21.15.133:0/1577016957 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcd6000b860 con 0x7fcd64076c00 2024-02-20T07:49:09.447 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:08.664+0000 7fcd69280700 1 -- 172.21.15.133:0/1577016957 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcd6000be90 con 0x7fcd64076c00 2024-02-20T07:49:09.447 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:08.664+0000 7fcd69280700 1 -- 172.21.15.133:0/1577016957 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcd600103c0 con 0x7fcd64076c00 2024-02-20T07:49:09.447 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:08.665+0000 7fcd6c4e6700 1 -- 172.21.15.133:0/1577016957 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fcd64076c00 msgr2=0x7fcd641077a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:49:09.447 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:08.665+0000 7fcd6c4e6700 1 --2- 172.21.15.133:0/1577016957 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fcd64076c00 0x7fcd641077a0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fcd600098e0 tx=0x7fcd60009250 comp rx=0 tx=0).stop 2024-02-20T07:49:09.447 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:08.665+0000 7fcd6c4e6700 1 -- 172.21.15.133:0/1577016957 shutdown_connections 2024-02-20T07:49:09.447 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:08.665+0000 7fcd6c4e6700 1 --2- 172.21.15.133:0/1577016957 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fcd64076c00 0x7fcd641077a0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:49:09.447 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:08.665+0000 7fcd6c4e6700 1 -- 172.21.15.133:0/1577016957 >> 172.21.15.133:0/1577016957 conn(0x7fcd640fcdd0 msgr2=0x7fcd640ff200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:49:09.447 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:08.665+0000 7fcd6c4e6700 1 -- 172.21.15.133:0/1577016957 shutdown_connections 2024-02-20T07:49:09.447 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:08.665+0000 7fcd6c4e6700 1 -- 172.21.15.133:0/1577016957 wait complete. 2024-02-20T07:49:09.447 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:08.665+0000 7fcd6c4e6700 1 Processor -- start 2024-02-20T07:49:09.447 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:08.665+0000 7fcd6c4e6700 1 -- start start 2024-02-20T07:49:09.447 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:08.665+0000 7fcd6c4e6700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fcd64076c00 0x7fcd64113a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:49:09.447 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:08.665+0000 7fcd6c4e6700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd64113fb0 con 0x7fcd64076c00 2024-02-20T07:49:09.447 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:08.665+0000 7fcd6a282700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fcd64076c00 0x7fcd64113a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:49:09.448 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:08.665+0000 7fcd6a282700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fcd64076c00 0x7fcd64113a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.133:43434/0 (socket says 172.21.15.133:43434) 2024-02-20T07:49:09.448 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:08.665+0000 7fcd6a282700 1 -- 172.21.15.133:0/1217485863 learned_addr learned my addr 172.21.15.133:0/1217485863 (peer_addr_for_me v2:172.21.15.133:0/0) 2024-02-20T07:49:09.448 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:08.666+0000 7fcd6a282700 1 -- 172.21.15.133:0/1217485863 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd600094f0 con 0x7fcd64076c00 2024-02-20T07:49:09.448 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:08.666+0000 7fcd6a282700 1 --2- 172.21.15.133:0/1217485863 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fcd64076c00 0x7fcd64113a70 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fcd60003080 tx=0x7fcd60009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:49:09.448 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:08.666+0000 7fcd5affd700 1 -- 172.21.15.133:0/1217485863 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcd60011920 con 0x7fcd64076c00 2024-02-20T07:49:09.448 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:08.666+0000 7fcd6c4e6700 1 -- 172.21.15.133:0/1217485863 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcd641141b0 con 0x7fcd64076c00 2024-02-20T07:49:09.448 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:08.666+0000 7fcd5affd700 1 -- 172.21.15.133:0/1217485863 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcd60010d60 con 0x7fcd64076c00 2024-02-20T07:49:09.448 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:08.666+0000 7fcd5affd700 1 -- 172.21.15.133:0/1217485863 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcd600243f0 con 0x7fcd64076c00 2024-02-20T07:49:09.448 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:08.666+0000 7fcd6c4e6700 1 -- 172.21.15.133:0/1217485863 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcd64114e50 con 0x7fcd64076c00 2024-02-20T07:49:09.448 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:08.667+0000 7fcd5affd700 1 -- 172.21.15.133:0/1217485863 <== mon.0 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 8) v1 ==== 43368+0+0 (secure 0 0 0) 0x7fcd6002d3f0 con 0x7fcd64076c00 2024-02-20T07:49:09.448 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:08.667+0000 7fcd6c4e6700 1 -- 172.21.15.133:0/1217485863 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcd6405f5c0 con 0x7fcd64076c00 2024-02-20T07:49:09.448 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:08.667+0000 7fcd5affd700 1 --2- 172.21.15.133:0/1217485863 >> [v2:172.21.15.133:6800/1670376163,v1:172.21.15.133:6801/1670376163] conn(0x7fcd50036790 0x7fcd50038c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:49:09.448 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:08.667+0000 7fcd5affd700 1 -- 172.21.15.133:0/1217485863 <== mon.0 v2:172.21.15.133:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7fcd6004a340 con 0x7fcd64076c00 2024-02-20T07:49:09.448 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:08.667+0000 7fcd69a81700 1 --2- 172.21.15.133:0/1217485863 >> [v2:172.21.15.133:6800/1670376163,v1:172.21.15.133:6801/1670376163] conn(0x7fcd50036790 0x7fcd50038c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:49:09.449 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:08.668+0000 7fcd69a81700 1 --2- 172.21.15.133:0/1217485863 >> [v2:172.21.15.133:6800/1670376163,v1:172.21.15.133:6801/1670376163] conn(0x7fcd50036790 0x7fcd50038c40 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fcd54009990 tx=0x7fcd54006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:49:09.449 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:08.672+0000 7fcd5affd700 1 -- 172.21.15.133:0/1217485863 <== mon.0 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7fcd6002b8e0 con 0x7fcd64076c00 2024-02-20T07:49:09.449 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:08.863+0000 7fcd6c4e6700 1 -- 172.21.15.133:0/1217485863 --> [v2:172.21.15.133:6800/1670376163,v1:172.21.15.133:6801/1670376163] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "alertmanager", "target": ["mon-mgr", ""]}) v1 -- 0x7fcd64108b70 con 0x7fcd50036790 2024-02-20T07:49:09.449 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:08.868+0000 7fcd5affd700 1 -- 172.21.15.133:0/1217485863 <== mgr.14120 v2:172.21.15.133:6800/1670376163 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+33 (secure 0 0 0) 0x7fcd64108b70 con 0x7fcd50036790 2024-02-20T07:49:09.449 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:08.870+0000 7fcd6c4e6700 1 -- 172.21.15.133:0/1217485863 >> [v2:172.21.15.133:6800/1670376163,v1:172.21.15.133:6801/1670376163] conn(0x7fcd50036790 msgr2=0x7fcd50038c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:49:09.449 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:08.870+0000 7fcd6c4e6700 1 --2- 172.21.15.133:0/1217485863 >> [v2:172.21.15.133:6800/1670376163,v1:172.21.15.133:6801/1670376163] conn(0x7fcd50036790 0x7fcd50038c40 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fcd54009990 tx=0x7fcd54006e40 comp rx=0 tx=0).stop 2024-02-20T07:49:09.449 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:08.870+0000 7fcd6c4e6700 1 -- 172.21.15.133:0/1217485863 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fcd64076c00 msgr2=0x7fcd64113a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:49:09.449 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:08.870+0000 7fcd6c4e6700 1 --2- 172.21.15.133:0/1217485863 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fcd64076c00 0x7fcd64113a70 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fcd60003080 tx=0x7fcd60009250 comp rx=0 tx=0).stop 2024-02-20T07:49:09.449 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:08.870+0000 7fcd6c4e6700 1 -- 172.21.15.133:0/1217485863 shutdown_connections 2024-02-20T07:49:09.449 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:08.870+0000 7fcd6c4e6700 1 --2- 172.21.15.133:0/1217485863 >> [v2:172.21.15.133:6800/1670376163,v1:172.21.15.133:6801/1670376163] conn(0x7fcd50036790 0x7fcd50038c40 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:49:09.449 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:08.870+0000 7fcd6c4e6700 1 --2- 172.21.15.133:0/1217485863 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fcd64076c00 0x7fcd64113a70 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:49:09.449 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:08.870+0000 7fcd6c4e6700 1 -- 172.21.15.133:0/1217485863 >> 172.21.15.133:0/1217485863 conn(0x7fcd640fcdd0 msgr2=0x7fcd640fdd80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:49:09.449 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:08.871+0000 7fcd6c4e6700 1 -- 172.21.15.133:0/1217485863 shutdown_connections 2024-02-20T07:49:09.449 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:08.871+0000 7fcd6c4e6700 1 -- 172.21.15.133:0/1217485863 wait complete. 2024-02-20T07:49:09.992 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:49:09 smithi133 ceph-mon[29117]: from='client.14154 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "target": ["mon-mgr", ""]}]: dispatch 2024-02-20T07:49:09.992 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:49:09 smithi133 ceph-mon[29117]: Saving service alertmanager spec with placement count:1 2024-02-20T07:49:09.992 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:49:09 smithi133 ceph-mon[29117]: from='mgr.14120 172.21.15.133:0/2965705206' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:49:09.992 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:49:09 smithi133 ceph-mon[29117]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T07:49:10.551 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:09.813+0000 7f6dc24c2700 1 Processor -- start 2024-02-20T07:49:10.551 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:09.813+0000 7f6dc24c2700 1 -- start start 2024-02-20T07:49:10.551 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:09.813+0000 7f6dc24c2700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f6dbc076c20 0x7f6dbc105380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:49:10.552 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:09.813+0000 7f6dc24c2700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6dbc1058c0 con 0x7f6dbc076c20 2024-02-20T07:49:10.552 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:09.813+0000 7f6dbbfff700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f6dbc076c20 0x7f6dbc105380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:49:10.552 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:09.813+0000 7f6dbbfff700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f6dbc076c20 0x7f6dbc105380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.133:43448/0 (socket says 172.21.15.133:43448) 2024-02-20T07:49:10.552 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:09.813+0000 7f6dbbfff700 1 -- 172.21.15.133:0/813704813 learned_addr learned my addr 172.21.15.133:0/813704813 (peer_addr_for_me v2:172.21.15.133:0/0) 2024-02-20T07:49:10.552 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:09.814+0000 7f6dbbfff700 1 -- 172.21.15.133:0/813704813 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6dbc105a00 con 0x7f6dbc076c20 2024-02-20T07:49:10.552 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:09.814+0000 7f6dbbfff700 1 --2- 172.21.15.133:0/813704813 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f6dbc076c20 0x7f6dbc105380 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f6dac0098e0 tx=0x7f6dac009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9f92b4fb3307726d server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:49:10.552 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:09.814+0000 7f6dbaffd700 1 -- 172.21.15.133:0/813704813 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6dac00b860 con 0x7f6dbc076c20 2024-02-20T07:49:10.552 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:09.814+0000 7f6dbaffd700 1 -- 172.21.15.133:0/813704813 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6dac00be90 con 0x7f6dbc076c20 2024-02-20T07:49:10.552 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:09.814+0000 7f6dbaffd700 1 -- 172.21.15.133:0/813704813 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6dac0103c0 con 0x7f6dbc076c20 2024-02-20T07:49:10.552 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:09.814+0000 7f6dc24c2700 1 -- 172.21.15.133:0/813704813 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f6dbc076c20 msgr2=0x7f6dbc105380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:49:10.552 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:09.814+0000 7f6dc24c2700 1 --2- 172.21.15.133:0/813704813 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f6dbc076c20 0x7f6dbc105380 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f6dac0098e0 tx=0x7f6dac009250 comp rx=0 tx=0).stop 2024-02-20T07:49:10.552 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:09.814+0000 7f6dc24c2700 1 -- 172.21.15.133:0/813704813 shutdown_connections 2024-02-20T07:49:10.552 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:09.814+0000 7f6dc24c2700 1 --2- 172.21.15.133:0/813704813 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f6dbc076c20 0x7f6dbc105380 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:49:10.552 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:09.814+0000 7f6dc24c2700 1 -- 172.21.15.133:0/813704813 >> 172.21.15.133:0/813704813 conn(0x7f6dbc0fcdd0 msgr2=0x7f6dbc0ff200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:49:10.553 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:09.815+0000 7f6dc24c2700 1 -- 172.21.15.133:0/813704813 shutdown_connections 2024-02-20T07:49:10.553 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:09.815+0000 7f6dc24c2700 1 -- 172.21.15.133:0/813704813 wait complete. 2024-02-20T07:49:10.553 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:09.815+0000 7f6dc24c2700 1 Processor -- start 2024-02-20T07:49:10.553 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:09.815+0000 7f6dc24c2700 1 -- start start 2024-02-20T07:49:10.553 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:09.815+0000 7f6dc24c2700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f6dbc076c20 0x7f6dbc111870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:49:10.553 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:09.815+0000 7f6dc24c2700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6dbc111db0 con 0x7f6dbc076c20 2024-02-20T07:49:10.553 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:09.815+0000 7f6dbbfff700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f6dbc076c20 0x7f6dbc111870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:49:10.553 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:09.815+0000 7f6dbbfff700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f6dbc076c20 0x7f6dbc111870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.133:43464/0 (socket says 172.21.15.133:43464) 2024-02-20T07:49:10.553 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:09.815+0000 7f6dbbfff700 1 -- 172.21.15.133:0/4052953670 learned_addr learned my addr 172.21.15.133:0/4052953670 (peer_addr_for_me v2:172.21.15.133:0/0) 2024-02-20T07:49:10.553 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:09.816+0000 7f6dbbfff700 1 -- 172.21.15.133:0/4052953670 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6dac0094f0 con 0x7f6dbc076c20 2024-02-20T07:49:10.553 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:09.816+0000 7f6dbbfff700 1 --2- 172.21.15.133:0/4052953670 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f6dbc076c20 0x7f6dbc111870 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f6dac003080 tx=0x7f6dac009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:49:10.553 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:09.816+0000 7f6db8ff9700 1 -- 172.21.15.133:0/4052953670 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6dac011920 con 0x7f6dbc076c20 2024-02-20T07:49:10.553 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:09.816+0000 7f6db8ff9700 1 -- 172.21.15.133:0/4052953670 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6dac010d60 con 0x7f6dbc076c20 2024-02-20T07:49:10.553 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:09.816+0000 7f6dc24c2700 1 -- 172.21.15.133:0/4052953670 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6dbc111fb0 con 0x7f6dbc076c20 2024-02-20T07:49:10.553 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:09.816+0000 7f6db8ff9700 1 -- 172.21.15.133:0/4052953670 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6dac0243f0 con 0x7f6dbc076c20 2024-02-20T07:49:10.553 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:09.816+0000 7f6dc24c2700 1 -- 172.21.15.133:0/4052953670 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6dbc112c50 con 0x7f6dbc076c20 2024-02-20T07:49:10.554 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:09.817+0000 7f6db8ff9700 1 -- 172.21.15.133:0/4052953670 <== mon.0 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 8) v1 ==== 43368+0+0 (secure 0 0 0) 0x7f6dac02d3f0 con 0x7f6dbc076c20 2024-02-20T07:49:10.554 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:09.817+0000 7f6dc24c2700 1 -- 172.21.15.133:0/4052953670 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6dbc05f5c0 con 0x7f6dbc076c20 2024-02-20T07:49:10.554 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:09.817+0000 7f6db8ff9700 1 --2- 172.21.15.133:0/4052953670 >> [v2:172.21.15.133:6800/1670376163,v1:172.21.15.133:6801/1670376163] conn(0x7f6da4036790 0x7f6da4038c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:49:10.554 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:09.817+0000 7f6db8ff9700 1 -- 172.21.15.133:0/4052953670 <== mon.0 v2:172.21.15.133:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f6dac04a5f0 con 0x7f6dbc076c20 2024-02-20T07:49:10.554 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:09.817+0000 7f6dbb7fe700 1 --2- 172.21.15.133:0/4052953670 >> [v2:172.21.15.133:6800/1670376163,v1:172.21.15.133:6801/1670376163] conn(0x7f6da4036790 0x7f6da4038c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:49:10.554 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:09.818+0000 7f6dbb7fe700 1 --2- 172.21.15.133:0/4052953670 >> [v2:172.21.15.133:6800/1670376163,v1:172.21.15.133:6801/1670376163] conn(0x7f6da4036790 0x7f6da4038c40 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f6db0009990 tx=0x7f6db0006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:49:10.554 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:09.822+0000 7f6db8ff9700 1 -- 172.21.15.133:0/4052953670 <== mon.0 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f6dac02b8e0 con 0x7f6dbc076c20 2024-02-20T07:49:10.554 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:10.009+0000 7f6dc24c2700 1 -- 172.21.15.133:0/4052953670 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/container_init}] v 0) v1 -- 0x7f6dbc1068e0 con 0x7f6dbc076c20 2024-02-20T07:49:10.554 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:10.013+0000 7f6db8ff9700 1 -- 172.21.15.133:0/4052953670 <== mon.0 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/container_init}]=0 v7)=0 v7) v1 ==== 142+0+0 (secure 0 0 0) 0x7f6dac009350 con 0x7f6dbc076c20 2024-02-20T07:49:10.554 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:10.015+0000 7f6dc24c2700 1 -- 172.21.15.133:0/4052953670 >> [v2:172.21.15.133:6800/1670376163,v1:172.21.15.133:6801/1670376163] conn(0x7f6da4036790 msgr2=0x7f6da4038c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:49:10.554 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:10.015+0000 7f6dc24c2700 1 --2- 172.21.15.133:0/4052953670 >> [v2:172.21.15.133:6800/1670376163,v1:172.21.15.133:6801/1670376163] conn(0x7f6da4036790 0x7f6da4038c40 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f6db0009990 tx=0x7f6db0006e40 comp rx=0 tx=0).stop 2024-02-20T07:49:10.554 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:10.015+0000 7f6dc24c2700 1 -- 172.21.15.133:0/4052953670 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f6dbc076c20 msgr2=0x7f6dbc111870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:49:10.554 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:10.015+0000 7f6dc24c2700 1 --2- 172.21.15.133:0/4052953670 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f6dbc076c20 0x7f6dbc111870 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f6dac003080 tx=0x7f6dac009250 comp rx=0 tx=0).stop 2024-02-20T07:49:10.554 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:10.015+0000 7f6dc24c2700 1 -- 172.21.15.133:0/4052953670 shutdown_connections 2024-02-20T07:49:10.555 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:10.015+0000 7f6dc24c2700 1 --2- 172.21.15.133:0/4052953670 >> [v2:172.21.15.133:6800/1670376163,v1:172.21.15.133:6801/1670376163] conn(0x7f6da4036790 0x7f6da4038c40 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:49:10.555 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:10.015+0000 7f6dc24c2700 1 --2- 172.21.15.133:0/4052953670 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f6dbc076c20 0x7f6dbc111870 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:49:10.555 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:10.015+0000 7f6dc24c2700 1 -- 172.21.15.133:0/4052953670 >> 172.21.15.133:0/4052953670 conn(0x7f6dbc0fcdd0 msgr2=0x7f6dbc0fdd90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:49:10.555 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:10.015+0000 7f6dc24c2700 1 -- 172.21.15.133:0/4052953670 shutdown_connections 2024-02-20T07:49:10.555 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:10.015+0000 7f6dc24c2700 1 -- 172.21.15.133:0/4052953670 wait complete. 2024-02-20T07:49:11.135 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:49:11 smithi133 ceph-mon[29117]: from='client.? 172.21.15.133:0/4052953670' entity='client.admin' 2024-02-20T07:49:11.716 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:10.944+0000 7feaa5d91700 1 Processor -- start 2024-02-20T07:49:11.716 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:10.944+0000 7feaa5d91700 1 -- start start 2024-02-20T07:49:11.716 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:10.944+0000 7feaa5d91700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7feaa0102770 0x7feaa0102b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:49:11.716 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:10.944+0000 7feaa5d91700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feaa01030c0 con 0x7feaa0102770 2024-02-20T07:49:11.716 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:10.944+0000 7fea9f7fe700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7feaa0102770 0x7feaa0102b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:49:11.716 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:10.944+0000 7fea9f7fe700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7feaa0102770 0x7feaa0102b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.133:43468/0 (socket says 172.21.15.133:43468) 2024-02-20T07:49:11.716 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:10.944+0000 7fea9f7fe700 1 -- 172.21.15.133:0/173477818 learned_addr learned my addr 172.21.15.133:0/173477818 (peer_addr_for_me v2:172.21.15.133:0/0) 2024-02-20T07:49:11.716 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:10.945+0000 7fea9f7fe700 1 -- 172.21.15.133:0/173477818 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feaa0103200 con 0x7feaa0102770 2024-02-20T07:49:11.716 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:10.945+0000 7fea9f7fe700 1 --2- 172.21.15.133:0/173477818 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7feaa0102770 0x7feaa0102b80 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fea900098e0 tx=0x7fea90009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f478b3006e81d33 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:49:11.717 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:10.945+0000 7fea9effd700 1 -- 172.21.15.133:0/173477818 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fea9000b860 con 0x7feaa0102770 2024-02-20T07:49:11.717 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:10.945+0000 7fea9effd700 1 -- 172.21.15.133:0/173477818 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fea9000be90 con 0x7feaa0102770 2024-02-20T07:49:11.717 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:10.945+0000 7fea9effd700 1 -- 172.21.15.133:0/173477818 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fea900103c0 con 0x7feaa0102770 2024-02-20T07:49:11.717 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:10.945+0000 7feaa5d91700 1 -- 172.21.15.133:0/173477818 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7feaa0102770 msgr2=0x7feaa0102b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:49:11.717 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:10.945+0000 7feaa5d91700 1 --2- 172.21.15.133:0/173477818 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7feaa0102770 0x7feaa0102b80 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fea900098e0 tx=0x7fea90009250 comp rx=0 tx=0).stop 2024-02-20T07:49:11.717 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:10.945+0000 7feaa5d91700 1 -- 172.21.15.133:0/173477818 shutdown_connections 2024-02-20T07:49:11.717 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:10.945+0000 7feaa5d91700 1 --2- 172.21.15.133:0/173477818 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7feaa0102770 0x7feaa0102b80 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:49:11.717 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:10.945+0000 7feaa5d91700 1 -- 172.21.15.133:0/173477818 >> 172.21.15.133:0/173477818 conn(0x7feaa00fded0 msgr2=0x7feaa0100300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:49:11.717 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:10.946+0000 7feaa5d91700 1 -- 172.21.15.133:0/173477818 shutdown_connections 2024-02-20T07:49:11.717 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:10.946+0000 7feaa5d91700 1 -- 172.21.15.133:0/173477818 wait complete. 2024-02-20T07:49:11.717 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:10.946+0000 7feaa5d91700 1 Processor -- start 2024-02-20T07:49:11.717 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:10.946+0000 7feaa5d91700 1 -- start start 2024-02-20T07:49:11.717 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:10.946+0000 7feaa5d91700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7feaa0102770 0x7feaa01115c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:49:11.717 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:10.946+0000 7feaa5d91700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feaa0111b00 con 0x7feaa0102770 2024-02-20T07:49:11.717 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:10.946+0000 7fea9f7fe700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7feaa0102770 0x7feaa01115c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:49:11.718 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:10.946+0000 7fea9f7fe700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7feaa0102770 0x7feaa01115c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.133:43470/0 (socket says 172.21.15.133:43470) 2024-02-20T07:49:11.718 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:10.946+0000 7fea9f7fe700 1 -- 172.21.15.133:0/2163650153 learned_addr learned my addr 172.21.15.133:0/2163650153 (peer_addr_for_me v2:172.21.15.133:0/0) 2024-02-20T07:49:11.718 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:10.946+0000 7fea9f7fe700 1 -- 172.21.15.133:0/2163650153 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea900094f0 con 0x7feaa0102770 2024-02-20T07:49:11.718 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:10.947+0000 7fea9f7fe700 1 --2- 172.21.15.133:0/2163650153 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7feaa0102770 0x7feaa01115c0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fea900090b0 tx=0x7fea90009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:49:11.718 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:10.947+0000 7fea9cff9700 1 -- 172.21.15.133:0/2163650153 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fea90010840 con 0x7feaa0102770 2024-02-20T07:49:11.718 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:10.947+0000 7fea9cff9700 1 -- 172.21.15.133:0/2163650153 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fea90010e70 con 0x7feaa0102770 2024-02-20T07:49:11.718 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:10.947+0000 7feaa5d91700 1 -- 172.21.15.133:0/2163650153 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feaa0111d00 con 0x7feaa0102770 2024-02-20T07:49:11.718 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:10.947+0000 7fea9cff9700 1 -- 172.21.15.133:0/2163650153 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fea90024e90 con 0x7feaa0102770 2024-02-20T07:49:11.718 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:10.947+0000 7feaa5d91700 1 -- 172.21.15.133:0/2163650153 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feaa01129a0 con 0x7feaa0102770 2024-02-20T07:49:11.718 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:10.948+0000 7feaa5d91700 1 -- 172.21.15.133:0/2163650153 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feaa005f5c0 con 0x7feaa0102770 2024-02-20T07:49:11.719 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:10.948+0000 7fea9cff9700 1 -- 172.21.15.133:0/2163650153 <== mon.0 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 8) v1 ==== 43368+0+0 (secure 0 0 0) 0x7fea900109a0 con 0x7feaa0102770 2024-02-20T07:49:11.719 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:10.948+0000 7fea9cff9700 1 --2- 172.21.15.133:0/2163650153 >> [v2:172.21.15.133:6800/1670376163,v1:172.21.15.133:6801/1670376163] conn(0x7fea8c036740 0x7fea8c038bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:49:11.719 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:10.948+0000 7fea9cff9700 1 -- 172.21.15.133:0/2163650153 <== mon.0 v2:172.21.15.133:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7fea9004a7c0 con 0x7feaa0102770 2024-02-20T07:49:11.719 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:10.948+0000 7fea97fff700 1 --2- 172.21.15.133:0/2163650153 >> [v2:172.21.15.133:6800/1670376163,v1:172.21.15.133:6801/1670376163] conn(0x7fea8c036740 0x7fea8c038bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:49:11.719 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:10.948+0000 7fea97fff700 1 --2- 172.21.15.133:0/2163650153 >> [v2:172.21.15.133:6800/1670376163,v1:172.21.15.133:6801/1670376163] conn(0x7fea8c036740 0x7fea8c038bf0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fea88009990 tx=0x7fea88006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:49:11.719 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:10.952+0000 7fea9cff9700 1 -- 172.21.15.133:0/2163650153 <== mon.0 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7fea9002c990 con 0x7feaa0102770 2024-02-20T07:49:11.719 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:11.134+0000 7feaa5d91700 1 -- 172.21.15.133:0/2163650153 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command([{prefix=config set, name=mgr/dashboard/ssl_server_port}] v 0) v1 -- 0x7feaa004bed0 con 0x7feaa0102770 2024-02-20T07:49:11.719 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:11.137+0000 7fea9cff9700 1 -- 172.21.15.133:0/2163650153 <== mon.0 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/dashboard/ssl_server_port}]=0 v8)=0 v8) v1 ==== 130+0+0 (secure 0 0 0) 0x7fea9002c7b0 con 0x7feaa0102770 2024-02-20T07:49:11.719 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:11.139+0000 7feaa5d91700 1 -- 172.21.15.133:0/2163650153 >> [v2:172.21.15.133:6800/1670376163,v1:172.21.15.133:6801/1670376163] conn(0x7fea8c036740 msgr2=0x7fea8c038bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:49:11.719 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:11.139+0000 7feaa5d91700 1 --2- 172.21.15.133:0/2163650153 >> [v2:172.21.15.133:6800/1670376163,v1:172.21.15.133:6801/1670376163] conn(0x7fea8c036740 0x7fea8c038bf0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fea88009990 tx=0x7fea88006e40 comp rx=0 tx=0).stop 2024-02-20T07:49:11.719 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:11.139+0000 7feaa5d91700 1 -- 172.21.15.133:0/2163650153 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7feaa0102770 msgr2=0x7feaa01115c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:49:11.719 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:11.139+0000 7feaa5d91700 1 --2- 172.21.15.133:0/2163650153 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7feaa0102770 0x7feaa01115c0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fea900090b0 tx=0x7fea90009250 comp rx=0 tx=0).stop 2024-02-20T07:49:11.719 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:11.139+0000 7feaa5d91700 1 -- 172.21.15.133:0/2163650153 shutdown_connections 2024-02-20T07:49:11.719 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:11.139+0000 7feaa5d91700 1 --2- 172.21.15.133:0/2163650153 >> [v2:172.21.15.133:6800/1670376163,v1:172.21.15.133:6801/1670376163] conn(0x7fea8c036740 0x7fea8c038bf0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:49:11.719 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:11.139+0000 7feaa5d91700 1 --2- 172.21.15.133:0/2163650153 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7feaa0102770 0x7feaa01115c0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:49:11.720 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:11.139+0000 7feaa5d91700 1 -- 172.21.15.133:0/2163650153 >> 172.21.15.133:0/2163650153 conn(0x7feaa00fded0 msgr2=0x7feaa00ff5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:49:11.720 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:11.139+0000 7feaa5d91700 1 -- 172.21.15.133:0/2163650153 shutdown_connections 2024-02-20T07:49:11.720 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:11.139+0000 7feaa5d91700 1 -- 172.21.15.133:0/2163650153 wait complete. 2024-02-20T07:49:11.720 INFO:teuthology.orchestra.run.smithi133.stdout:Enabling the dashboard module... 2024-02-20T07:49:12.346 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:49:12 smithi133 ceph-mon[29117]: from='client.? 172.21.15.133:0/2163650153' entity='client.admin' 2024-02-20T07:49:12.346 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:49:12 smithi133 ceph-mon[29117]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T07:49:13.456 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:49:13 smithi133 ceph-mon[29117]: from='client.? 172.21.15.133:0/2520995199' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-02-20T07:49:13.707 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:12.099+0000 7fd1b2e4a700 1 Processor -- start 2024-02-20T07:49:13.707 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:12.099+0000 7fd1b2e4a700 1 -- start start 2024-02-20T07:49:13.707 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:12.099+0000 7fd1b2e4a700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fd1ac076be0 0x7fd1ac1053c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:49:13.707 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:12.099+0000 7fd1b2e4a700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1ac105900 con 0x7fd1ac076be0 2024-02-20T07:49:13.707 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:12.100+0000 7fd1b0be6700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fd1ac076be0 0x7fd1ac1053c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:49:13.707 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:12.100+0000 7fd1b0be6700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fd1ac076be0 0x7fd1ac1053c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.133:43472/0 (socket says 172.21.15.133:43472) 2024-02-20T07:49:13.707 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:12.100+0000 7fd1b0be6700 1 -- 172.21.15.133:0/1907357078 learned_addr learned my addr 172.21.15.133:0/1907357078 (peer_addr_for_me v2:172.21.15.133:0/0) 2024-02-20T07:49:13.707 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:12.100+0000 7fd1b0be6700 1 -- 172.21.15.133:0/1907357078 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1ac105a40 con 0x7fd1ac076be0 2024-02-20T07:49:13.707 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:12.100+0000 7fd1b0be6700 1 --2- 172.21.15.133:0/1907357078 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fd1ac076be0 0x7fd1ac1053c0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fd19c009e50 tx=0x7fd19c009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=229a3050e74bcf8f server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:49:13.708 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:12.100+0000 7fd1ab7fe700 1 -- 172.21.15.133:0/1907357078 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd19c0092e0 con 0x7fd1ac076be0 2024-02-20T07:49:13.708 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:12.100+0000 7fd1ab7fe700 1 -- 172.21.15.133:0/1907357078 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd19c012430 con 0x7fd1ac076be0 2024-02-20T07:49:13.708 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:12.100+0000 7fd1ab7fe700 1 -- 172.21.15.133:0/1907357078 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd19c0075a0 con 0x7fd1ac076be0 2024-02-20T07:49:13.708 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:12.101+0000 7fd1b2e4a700 1 -- 172.21.15.133:0/1907357078 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fd1ac076be0 msgr2=0x7fd1ac1053c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:49:13.708 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:12.101+0000 7fd1b2e4a700 1 --2- 172.21.15.133:0/1907357078 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fd1ac076be0 0x7fd1ac1053c0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fd19c009e50 tx=0x7fd19c009840 comp rx=0 tx=0).stop 2024-02-20T07:49:13.708 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:12.101+0000 7fd1b2e4a700 1 -- 172.21.15.133:0/1907357078 shutdown_connections 2024-02-20T07:49:13.708 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:12.101+0000 7fd1b2e4a700 1 --2- 172.21.15.133:0/1907357078 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fd1ac076be0 0x7fd1ac1053c0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:49:13.708 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:12.101+0000 7fd1b2e4a700 1 -- 172.21.15.133:0/1907357078 >> 172.21.15.133:0/1907357078 conn(0x7fd1ac0fcdd0 msgr2=0x7fd1ac0ff200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:49:13.708 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:12.101+0000 7fd1b2e4a700 1 -- 172.21.15.133:0/1907357078 shutdown_connections 2024-02-20T07:49:13.708 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:12.101+0000 7fd1b2e4a700 1 -- 172.21.15.133:0/1907357078 wait complete. 2024-02-20T07:49:13.708 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:12.101+0000 7fd1b2e4a700 1 Processor -- start 2024-02-20T07:49:13.708 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:12.101+0000 7fd1b2e4a700 1 -- start start 2024-02-20T07:49:13.708 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:12.102+0000 7fd1b2e4a700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fd1ac076be0 0x7fd1ac1115f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:49:13.708 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:12.102+0000 7fd1b2e4a700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1ac111b30 con 0x7fd1ac076be0 2024-02-20T07:49:13.709 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:12.102+0000 7fd1b0be6700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fd1ac076be0 0x7fd1ac1115f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:49:13.709 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:12.102+0000 7fd1b0be6700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fd1ac076be0 0x7fd1ac1115f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.133:43474/0 (socket says 172.21.15.133:43474) 2024-02-20T07:49:13.709 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:12.102+0000 7fd1b0be6700 1 -- 172.21.15.133:0/2520995199 learned_addr learned my addr 172.21.15.133:0/2520995199 (peer_addr_for_me v2:172.21.15.133:0/0) 2024-02-20T07:49:13.709 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:12.102+0000 7fd1b0be6700 1 -- 172.21.15.133:0/2520995199 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd19c009ae0 con 0x7fd1ac076be0 2024-02-20T07:49:13.709 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:12.102+0000 7fd1b0be6700 1 --2- 172.21.15.133:0/2520995199 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fd1ac076be0 0x7fd1ac1115f0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fd19c012de0 tx=0x7fd19c009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:49:13.709 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:12.102+0000 7fd1a97fa700 1 -- 172.21.15.133:0/2520995199 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd19c0092e0 con 0x7fd1ac076be0 2024-02-20T07:49:13.709 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:12.102+0000 7fd1a97fa700 1 -- 172.21.15.133:0/2520995199 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd19c007d30 con 0x7fd1ac076be0 2024-02-20T07:49:13.709 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:12.102+0000 7fd1a97fa700 1 -- 172.21.15.133:0/2520995199 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd19c025a00 con 0x7fd1ac076be0 2024-02-20T07:49:13.709 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:12.102+0000 7fd1b2e4a700 1 -- 172.21.15.133:0/2520995199 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd1ac111d30 con 0x7fd1ac076be0 2024-02-20T07:49:13.709 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:12.102+0000 7fd1b2e4a700 1 -- 172.21.15.133:0/2520995199 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd1ac1129d0 con 0x7fd1ac076be0 2024-02-20T07:49:13.709 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:12.103+0000 7fd1a97fa700 1 -- 172.21.15.133:0/2520995199 <== mon.0 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 8) v1 ==== 43368+0+0 (secure 0 0 0) 0x7fd19c019050 con 0x7fd1ac076be0 2024-02-20T07:49:13.709 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:12.103+0000 7fd1b2e4a700 1 -- 172.21.15.133:0/2520995199 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd1ac05f5c0 con 0x7fd1ac076be0 2024-02-20T07:49:13.709 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:12.103+0000 7fd1a97fa700 1 --2- 172.21.15.133:0/2520995199 >> [v2:172.21.15.133:6800/1670376163,v1:172.21.15.133:6801/1670376163] conn(0x7fd1980367e0 0x7fd198038c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:49:13.709 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:12.103+0000 7fd1a97fa700 1 -- 172.21.15.133:0/2520995199 <== mon.0 v2:172.21.15.133:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7fd19c049ff0 con 0x7fd1ac076be0 2024-02-20T07:49:13.709 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:12.104+0000 7fd1abfff700 1 --2- 172.21.15.133:0/2520995199 >> [v2:172.21.15.133:6800/1670376163,v1:172.21.15.133:6801/1670376163] conn(0x7fd1980367e0 0x7fd198038c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:49:13.710 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:12.104+0000 7fd1abfff700 1 --2- 172.21.15.133:0/2520995199 >> [v2:172.21.15.133:6800/1670376163,v1:172.21.15.133:6801/1670376163] conn(0x7fd1980367e0 0x7fd198038c90 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fd194009990 tx=0x7fd194006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:49:13.710 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:12.108+0000 7fd1a97fa700 1 -- 172.21.15.133:0/2520995199 <== mon.0 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7fd19c02d9a0 con 0x7fd1ac076be0 2024-02-20T07:49:13.710 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:12.344+0000 7fd1b2e4a700 1 -- 172.21.15.133:0/2520995199 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "dashboard"} v 0) v1 -- 0x7fd1ac04bed0 con 0x7fd1ac076be0 2024-02-20T07:49:13.710 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:13.142+0000 7fd1a97fa700 1 -- 172.21.15.133:0/2520995199 <== mon.0 v2:172.21.15.133:3300/0 7 ==== mgrmap(e 9) v1 ==== 43381+0+0 (secure 0 0 0) 0x7fd19c01c550 con 0x7fd1ac076be0 2024-02-20T07:49:13.710 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:13.144+0000 7fd1a97fa700 1 -- 172.21.15.133:0/2520995199 <== mon.0 v2:172.21.15.133:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "dashboard"}]=0 v9) v1 ==== 88+0+0 (secure 0 0 0) 0x7fd19c02e960 con 0x7fd1ac076be0 2024-02-20T07:49:13.710 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:13.148+0000 7fd1b2e4a700 1 -- 172.21.15.133:0/2520995199 >> [v2:172.21.15.133:6800/1670376163,v1:172.21.15.133:6801/1670376163] conn(0x7fd1980367e0 msgr2=0x7fd198038c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:49:13.710 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:13.148+0000 7fd1b2e4a700 1 --2- 172.21.15.133:0/2520995199 >> [v2:172.21.15.133:6800/1670376163,v1:172.21.15.133:6801/1670376163] conn(0x7fd1980367e0 0x7fd198038c90 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fd194009990 tx=0x7fd194006e40 comp rx=0 tx=0).stop 2024-02-20T07:49:13.710 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:13.148+0000 7fd1b2e4a700 1 -- 172.21.15.133:0/2520995199 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fd1ac076be0 msgr2=0x7fd1ac1115f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:49:13.710 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:13.148+0000 7fd1b2e4a700 1 --2- 172.21.15.133:0/2520995199 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fd1ac076be0 0x7fd1ac1115f0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fd19c012de0 tx=0x7fd19c009840 comp rx=0 tx=0).stop 2024-02-20T07:49:13.710 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:13.148+0000 7fd1b2e4a700 1 -- 172.21.15.133:0/2520995199 shutdown_connections 2024-02-20T07:49:13.710 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:13.148+0000 7fd1b2e4a700 1 --2- 172.21.15.133:0/2520995199 >> [v2:172.21.15.133:6800/1670376163,v1:172.21.15.133:6801/1670376163] conn(0x7fd1980367e0 0x7fd198038c90 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:49:13.710 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:13.148+0000 7fd1b2e4a700 1 --2- 172.21.15.133:0/2520995199 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fd1ac076be0 0x7fd1ac1115f0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:49:13.710 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:13.148+0000 7fd1b2e4a700 1 -- 172.21.15.133:0/2520995199 >> 172.21.15.133:0/2520995199 conn(0x7fd1ac0fcdd0 msgr2=0x7fd1ac0fe4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:49:13.710 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:13.148+0000 7fd1b2e4a700 1 -- 172.21.15.133:0/2520995199 shutdown_connections 2024-02-20T07:49:13.711 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:13.148+0000 7fd1b2e4a700 1 -- 172.21.15.133:0/2520995199 wait complete. 2024-02-20T07:49:14.386 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:49:14 smithi133 ceph-mon[29117]: from='client.? 172.21.15.133:0/2520995199' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-02-20T07:49:14.386 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:49:14 smithi133 ceph-mon[29117]: mgrmap e9: smithi133.mrvcbp(active, since 24s) 2024-02-20T07:49:14.973 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stdout { 2024-02-20T07:49:14.974 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stdout "epoch": 9, 2024-02-20T07:49:14.974 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stdout "available": true, 2024-02-20T07:49:14.974 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stdout "active_name": "smithi133.mrvcbp", 2024-02-20T07:49:14.974 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-02-20T07:49:14.974 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stdout } 2024-02-20T07:49:14.974 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:14.129+0000 7fd3bcdc6700 1 Processor -- start 2024-02-20T07:49:14.974 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:14.129+0000 7fd3bcdc6700 1 -- start start 2024-02-20T07:49:14.974 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:14.129+0000 7fd3bcdc6700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fd3b8076b80 0x7fd3b81053c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:49:14.975 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:14.129+0000 7fd3bcdc6700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3b8105900 con 0x7fd3b8076b80 2024-02-20T07:49:14.975 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:14.129+0000 7fd3b659c700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fd3b8076b80 0x7fd3b81053c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:49:14.975 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:14.129+0000 7fd3b659c700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fd3b8076b80 0x7fd3b81053c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.133:43498/0 (socket says 172.21.15.133:43498) 2024-02-20T07:49:14.975 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:14.129+0000 7fd3b659c700 1 -- 172.21.15.133:0/3041740152 learned_addr learned my addr 172.21.15.133:0/3041740152 (peer_addr_for_me v2:172.21.15.133:0/0) 2024-02-20T07:49:14.975 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:14.130+0000 7fd3b659c700 1 -- 172.21.15.133:0/3041740152 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3b8105a40 con 0x7fd3b8076b80 2024-02-20T07:49:14.975 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:14.130+0000 7fd3b659c700 1 --2- 172.21.15.133:0/3041740152 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fd3b8076b80 0x7fd3b81053c0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fd3a80098e0 tx=0x7fd3a8009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7dd700d3659440f6 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:49:14.975 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:14.130+0000 7fd3b559a700 1 -- 172.21.15.133:0/3041740152 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd3a800b860 con 0x7fd3b8076b80 2024-02-20T07:49:14.975 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:14.130+0000 7fd3b559a700 1 -- 172.21.15.133:0/3041740152 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd3a800be90 con 0x7fd3b8076b80 2024-02-20T07:49:14.975 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:14.130+0000 7fd3b559a700 1 -- 172.21.15.133:0/3041740152 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd3a80103c0 con 0x7fd3b8076b80 2024-02-20T07:49:14.976 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:14.131+0000 7fd3bcdc6700 1 -- 172.21.15.133:0/3041740152 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fd3b8076b80 msgr2=0x7fd3b81053c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:49:14.976 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:14.131+0000 7fd3bcdc6700 1 --2- 172.21.15.133:0/3041740152 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fd3b8076b80 0x7fd3b81053c0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fd3a80098e0 tx=0x7fd3a8009250 comp rx=0 tx=0).stop 2024-02-20T07:49:14.976 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:14.131+0000 7fd3bcdc6700 1 -- 172.21.15.133:0/3041740152 shutdown_connections 2024-02-20T07:49:14.976 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:14.131+0000 7fd3bcdc6700 1 --2- 172.21.15.133:0/3041740152 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fd3b8076b80 0x7fd3b81053c0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:49:14.976 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:14.131+0000 7fd3bcdc6700 1 -- 172.21.15.133:0/3041740152 >> 172.21.15.133:0/3041740152 conn(0x7fd3b80fcd90 msgr2=0x7fd3b80ff1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:49:14.976 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:14.131+0000 7fd3bcdc6700 1 -- 172.21.15.133:0/3041740152 shutdown_connections 2024-02-20T07:49:14.976 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:14.131+0000 7fd3bcdc6700 1 -- 172.21.15.133:0/3041740152 wait complete. 2024-02-20T07:49:14.976 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:14.131+0000 7fd3bcdc6700 1 Processor -- start 2024-02-20T07:49:14.976 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:14.132+0000 7fd3bcdc6700 1 -- start start 2024-02-20T07:49:14.976 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:14.132+0000 7fd3bcdc6700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fd3b8076b80 0x7fd3b8111570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:49:14.976 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:14.132+0000 7fd3bcdc6700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3b8111ab0 con 0x7fd3b8076b80 2024-02-20T07:49:14.977 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:14.132+0000 7fd3b659c700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fd3b8076b80 0x7fd3b8111570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:49:14.977 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:14.132+0000 7fd3b659c700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fd3b8076b80 0x7fd3b8111570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.133:43500/0 (socket says 172.21.15.133:43500) 2024-02-20T07:49:14.977 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:14.132+0000 7fd3b659c700 1 -- 172.21.15.133:0/2380292316 learned_addr learned my addr 172.21.15.133:0/2380292316 (peer_addr_for_me v2:172.21.15.133:0/0) 2024-02-20T07:49:14.977 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:14.132+0000 7fd3b659c700 1 -- 172.21.15.133:0/2380292316 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3a80094f0 con 0x7fd3b8076b80 2024-02-20T07:49:14.977 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:14.132+0000 7fd3b659c700 1 --2- 172.21.15.133:0/2380292316 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fd3b8076b80 0x7fd3b8111570 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fd3a80090b0 tx=0x7fd3a8009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:49:14.977 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:14.132+0000 7fd3a6ffd700 1 -- 172.21.15.133:0/2380292316 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd3a80118f0 con 0x7fd3b8076b80 2024-02-20T07:49:14.977 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:14.132+0000 7fd3bcdc6700 1 -- 172.21.15.133:0/2380292316 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd3b8111cb0 con 0x7fd3b8076b80 2024-02-20T07:49:14.977 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:14.132+0000 7fd3a6ffd700 1 -- 172.21.15.133:0/2380292316 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd3a8010d10 con 0x7fd3b8076b80 2024-02-20T07:49:14.977 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:14.132+0000 7fd3a6ffd700 1 -- 172.21.15.133:0/2380292316 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd3a801aea0 con 0x7fd3b8076b80 2024-02-20T07:49:14.978 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:14.132+0000 7fd3bcdc6700 1 -- 172.21.15.133:0/2380292316 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd3b8112950 con 0x7fd3b8076b80 2024-02-20T07:49:14.978 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:14.133+0000 7fd3a6ffd700 1 -- 172.21.15.133:0/2380292316 <== mon.0 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 9) v1 ==== 43381+0+0 (secure 0 0 0) 0x7fd3a80194f0 con 0x7fd3b8076b80 2024-02-20T07:49:14.978 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:14.133+0000 7fd3a6ffd700 1 --2- 172.21.15.133:0/2380292316 >> [v2:172.21.15.133:6800/1670376163,v1:172.21.15.133:6801/1670376163] conn(0x7fd3a00367e0 0x7fd3a0038c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:49:14.978 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:14.133+0000 7fd3bcdc6700 1 -- 172.21.15.133:0/2380292316 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd3b805f5c0 con 0x7fd3b8076b80 2024-02-20T07:49:14.978 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:14.134+0000 7fd3b5d9b700 1 -- 172.21.15.133:0/2380292316 >> [v2:172.21.15.133:6800/1670376163,v1:172.21.15.133:6801/1670376163] conn(0x7fd3a00367e0 msgr2=0x7fd3a0038c90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.133:6800/1670376163 2024-02-20T07:49:14.978 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:14.134+0000 7fd3a6ffd700 1 -- 172.21.15.133:0/2380292316 <== mon.0 v2:172.21.15.133:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7fd3a804a620 con 0x7fd3b8076b80 2024-02-20T07:49:14.978 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:14.134+0000 7fd3b5d9b700 1 --2- 172.21.15.133:0/2380292316 >> [v2:172.21.15.133:6800/1670376163,v1:172.21.15.133:6801/1670376163] conn(0x7fd3a00367e0 0x7fd3a0038c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-02-20T07:49:14.978 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:14.138+0000 7fd3a6ffd700 1 -- 172.21.15.133:0/2380292316 <== mon.0 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7fd3a802d3f0 con 0x7fd3b8076b80 2024-02-20T07:49:14.979 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:14.335+0000 7fd3b5d9b700 1 -- 172.21.15.133:0/2380292316 >> [v2:172.21.15.133:6800/1670376163,v1:172.21.15.133:6801/1670376163] conn(0x7fd3a00367e0 msgr2=0x7fd3a0038c90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.133:6800/1670376163 2024-02-20T07:49:14.979 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:14.335+0000 7fd3b5d9b700 1 --2- 172.21.15.133:0/2380292316 >> [v2:172.21.15.133:6800/1670376163,v1:172.21.15.133:6801/1670376163] conn(0x7fd3a00367e0 0x7fd3a0038c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-02-20T07:49:14.979 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:14.383+0000 7fd3bcdc6700 1 -- 172.21.15.133:0/2380292316 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7fd3b8112c90 con 0x7fd3b8076b80 2024-02-20T07:49:14.979 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:14.383+0000 7fd3a6ffd700 1 -- 172.21.15.133:0/2380292316 <== mon.0 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v9) v1 ==== 56+0+103 (secure 0 0 0) 0x7fd3a8022330 con 0x7fd3b8076b80 2024-02-20T07:49:14.979 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:14.386+0000 7fd3bcdc6700 1 -- 172.21.15.133:0/2380292316 >> [v2:172.21.15.133:6800/1670376163,v1:172.21.15.133:6801/1670376163] conn(0x7fd3a00367e0 msgr2=0x7fd3a0038c90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-02-20T07:49:14.980 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:14.386+0000 7fd3bcdc6700 1 --2- 172.21.15.133:0/2380292316 >> [v2:172.21.15.133:6800/1670376163,v1:172.21.15.133:6801/1670376163] conn(0x7fd3a00367e0 0x7fd3a0038c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:49:14.980 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:14.386+0000 7fd3bcdc6700 1 -- 172.21.15.133:0/2380292316 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fd3b8076b80 msgr2=0x7fd3b8111570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:49:14.980 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:14.386+0000 7fd3bcdc6700 1 --2- 172.21.15.133:0/2380292316 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fd3b8076b80 0x7fd3b8111570 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fd3a80090b0 tx=0x7fd3a8009250 comp rx=0 tx=0).stop 2024-02-20T07:49:14.980 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:14.386+0000 7fd3bcdc6700 1 -- 172.21.15.133:0/2380292316 shutdown_connections 2024-02-20T07:49:14.980 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:14.386+0000 7fd3bcdc6700 1 --2- 172.21.15.133:0/2380292316 >> [v2:172.21.15.133:6800/1670376163,v1:172.21.15.133:6801/1670376163] conn(0x7fd3a00367e0 0x7fd3a0038c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:49:14.980 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:14.386+0000 7fd3bcdc6700 1 --2- 172.21.15.133:0/2380292316 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fd3b8076b80 0x7fd3b8111570 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:49:14.980 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:14.386+0000 7fd3bcdc6700 1 -- 172.21.15.133:0/2380292316 >> 172.21.15.133:0/2380292316 conn(0x7fd3b80fcd90 msgr2=0x7fd3b80fe4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:49:14.980 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:14.386+0000 7fd3bcdc6700 1 -- 172.21.15.133:0/2380292316 shutdown_connections 2024-02-20T07:49:14.980 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:14.386+0000 7fd3bcdc6700 1 -- 172.21.15.133:0/2380292316 wait complete. 2024-02-20T07:49:14.980 INFO:teuthology.orchestra.run.smithi133.stdout:Waiting for the mgr to restart... 2024-02-20T07:49:14.980 INFO:teuthology.orchestra.run.smithi133.stdout:Waiting for mgr epoch 9... 2024-02-20T07:49:15.242 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:49:15 smithi133 ceph-mon[29117]: from='client.? 172.21.15.133:0/2380292316' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-02-20T07:49:19.992 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:49:19 smithi133 ceph-mon[29117]: Active manager daemon smithi133.mrvcbp restarted 2024-02-20T07:49:19.993 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:49:19 smithi133 ceph-mon[29117]: Activating manager daemon smithi133.mrvcbp 2024-02-20T07:49:19.993 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:49:19 smithi133 ceph-mon[29117]: osdmap e3: 0 total, 0 up, 0 in 2024-02-20T07:49:19.993 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:49:19 smithi133 ceph-mon[29117]: mgrmap e10: smithi133.mrvcbp(active, starting, since 0.00319511s) 2024-02-20T07:49:19.993 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:49:19 smithi133 ceph-mon[29117]: from='mgr.14162 172.21.15.133:0/3642562278' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "mon metadata", "id": "smithi133"}]: dispatch 2024-02-20T07:49:19.993 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:49:19 smithi133 ceph-mon[29117]: from='mgr.14162 172.21.15.133:0/3642562278' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "mgr metadata", "who": "smithi133.mrvcbp", "id": "smithi133.mrvcbp"}]: dispatch 2024-02-20T07:49:19.993 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:49:19 smithi133 ceph-mon[29117]: from='mgr.14162 172.21.15.133:0/3642562278' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-02-20T07:49:19.993 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:49:19 smithi133 ceph-mon[29117]: from='mgr.14162 172.21.15.133:0/3642562278' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-02-20T07:49:19.993 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:49:19 smithi133 ceph-mon[29117]: from='mgr.14162 172.21.15.133:0/3642562278' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-02-20T07:49:19.993 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:49:19 smithi133 ceph-mon[29117]: Manager daemon smithi133.mrvcbp is now available 2024-02-20T07:49:19.993 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:49:19 smithi133 ceph-mon[29117]: from='mgr.14162 172.21.15.133:0/3642562278' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-20T07:49:19.993 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:49:19 smithi133 ceph-mon[29117]: from='mgr.14162 172.21.15.133:0/3642562278' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-20T07:49:20.992 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:49:20 smithi133 ceph-mon[29117]: from='mgr.14162 172.21.15.133:0/3642562278' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi133.mrvcbp/mirror_snapshot_schedule"}]: dispatch 2024-02-20T07:49:20.992 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:49:20 smithi133 ceph-mon[29117]: from='mgr.14162 172.21.15.133:0/3642562278' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi133.mrvcbp/trash_purge_schedule"}]: dispatch 2024-02-20T07:49:20.992 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:49:20 smithi133 ceph-mon[29117]: from='mgr.14162 172.21.15.133:0/3642562278' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:49:20.992 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:49:20 smithi133 ceph-mon[29117]: mgrmap e11: smithi133.mrvcbp(active, since 1.00622s) 2024-02-20T07:49:21.372 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stdout { 2024-02-20T07:49:21.372 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 11, 2024-02-20T07:49:21.372 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stdout "initialized": true 2024-02-20T07:49:21.372 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stdout } 2024-02-20T07:49:21.372 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:15.403+0000 7fe98189d700 1 Processor -- start 2024-02-20T07:49:21.372 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:15.403+0000 7fe98189d700 1 -- start start 2024-02-20T07:49:21.372 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:15.403+0000 7fe98189d700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fe97c102770 0x7fe97c102b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:49:21.372 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:15.403+0000 7fe98189d700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe97c1030c0 con 0x7fe97c102770 2024-02-20T07:49:21.372 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:15.403+0000 7fe97affd700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fe97c102770 0x7fe97c102b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:49:21.372 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:15.403+0000 7fe97affd700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fe97c102770 0x7fe97c102b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.133:43504/0 (socket says 172.21.15.133:43504) 2024-02-20T07:49:21.372 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:15.403+0000 7fe97affd700 1 -- 172.21.15.133:0/874980454 learned_addr learned my addr 172.21.15.133:0/874980454 (peer_addr_for_me v2:172.21.15.133:0/0) 2024-02-20T07:49:21.373 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:15.404+0000 7fe97affd700 1 -- 172.21.15.133:0/874980454 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe97c103200 con 0x7fe97c102770 2024-02-20T07:49:21.373 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:15.404+0000 7fe97affd700 1 --2- 172.21.15.133:0/874980454 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fe97c102770 0x7fe97c102b80 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fe9700098e0 tx=0x7fe970009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d933408b02db3876 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:49:21.373 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:15.404+0000 7fe979ffb700 1 -- 172.21.15.133:0/874980454 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe97000b860 con 0x7fe97c102770 2024-02-20T07:49:21.373 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:15.404+0000 7fe979ffb700 1 -- 172.21.15.133:0/874980454 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe97000be90 con 0x7fe97c102770 2024-02-20T07:49:21.373 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:15.404+0000 7fe979ffb700 1 -- 172.21.15.133:0/874980454 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe9700103c0 con 0x7fe97c102770 2024-02-20T07:49:21.373 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:15.405+0000 7fe98189d700 1 -- 172.21.15.133:0/874980454 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fe97c102770 msgr2=0x7fe97c102b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:49:21.373 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:15.405+0000 7fe98189d700 1 --2- 172.21.15.133:0/874980454 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fe97c102770 0x7fe97c102b80 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fe9700098e0 tx=0x7fe970009250 comp rx=0 tx=0).stop 2024-02-20T07:49:21.373 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:15.405+0000 7fe98189d700 1 -- 172.21.15.133:0/874980454 shutdown_connections 2024-02-20T07:49:21.373 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:15.405+0000 7fe98189d700 1 --2- 172.21.15.133:0/874980454 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fe97c102770 0x7fe97c102b80 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:49:21.373 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:15.405+0000 7fe98189d700 1 -- 172.21.15.133:0/874980454 >> 172.21.15.133:0/874980454 conn(0x7fe97c0fded0 msgr2=0x7fe97c100300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:49:21.373 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:15.405+0000 7fe98189d700 1 -- 172.21.15.133:0/874980454 shutdown_connections 2024-02-20T07:49:21.373 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:15.405+0000 7fe98189d700 1 -- 172.21.15.133:0/874980454 wait complete. 2024-02-20T07:49:21.373 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:15.405+0000 7fe98189d700 1 Processor -- start 2024-02-20T07:49:21.374 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:15.405+0000 7fe98189d700 1 -- start start 2024-02-20T07:49:21.374 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:15.406+0000 7fe98189d700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fe97c102770 0x7fe97c111540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:49:21.374 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:15.406+0000 7fe98189d700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe97c111a80 con 0x7fe97c102770 2024-02-20T07:49:21.374 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:15.406+0000 7fe97affd700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fe97c102770 0x7fe97c111540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:49:21.374 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:15.406+0000 7fe97affd700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fe97c102770 0x7fe97c111540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.133:43514/0 (socket says 172.21.15.133:43514) 2024-02-20T07:49:21.374 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:15.406+0000 7fe97affd700 1 -- 172.21.15.133:0/3274797933 learned_addr learned my addr 172.21.15.133:0/3274797933 (peer_addr_for_me v2:172.21.15.133:0/0) 2024-02-20T07:49:21.374 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:15.406+0000 7fe97affd700 1 -- 172.21.15.133:0/3274797933 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9700094f0 con 0x7fe97c102770 2024-02-20T07:49:21.374 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:15.406+0000 7fe97affd700 1 --2- 172.21.15.133:0/3274797933 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fe97c102770 0x7fe97c111540 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fe970011650 tx=0x7fe970009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:49:21.374 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:15.406+0000 7fe9637fe700 1 -- 172.21.15.133:0/3274797933 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe970010890 con 0x7fe97c102770 2024-02-20T07:49:21.374 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:15.406+0000 7fe98189d700 1 -- 172.21.15.133:0/3274797933 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe97c111c80 con 0x7fe97c102770 2024-02-20T07:49:21.375 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:15.406+0000 7fe9637fe700 1 -- 172.21.15.133:0/3274797933 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe970010ec0 con 0x7fe97c102770 2024-02-20T07:49:21.375 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:15.406+0000 7fe9637fe700 1 -- 172.21.15.133:0/3274797933 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe9700243f0 con 0x7fe97c102770 2024-02-20T07:49:21.375 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:15.406+0000 7fe98189d700 1 -- 172.21.15.133:0/3274797933 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe97c112920 con 0x7fe97c102770 2024-02-20T07:49:21.375 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:15.407+0000 7fe9637fe700 1 -- 172.21.15.133:0/3274797933 <== mon.0 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 9) v1 ==== 43381+0+0 (secure 0 0 0) 0x7fe970024620 con 0x7fe97c102770 2024-02-20T07:49:21.375 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:15.407+0000 7fe9637fe700 1 --2- 172.21.15.133:0/3274797933 >> [v2:172.21.15.133:6800/1670376163,v1:172.21.15.133:6801/1670376163] conn(0x7fe964036830 0x7fe964038ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:49:21.375 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:15.407+0000 7fe98189d700 1 -- 172.21.15.133:0/3274797933 --> [v2:172.21.15.133:6800/1670376163,v1:172.21.15.133:6801/1670376163] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe97c05f5c0 con 0x7fe964036830 2024-02-20T07:49:21.375 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:15.407+0000 7fe9637fe700 1 -- 172.21.15.133:0/3274797933 <== mon.0 v2:172.21.15.133:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7fe97004b7c0 con 0x7fe97c102770 2024-02-20T07:49:21.375 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:15.407+0000 7fe97a7fc700 1 -- 172.21.15.133:0/3274797933 >> [v2:172.21.15.133:6800/1670376163,v1:172.21.15.133:6801/1670376163] conn(0x7fe964036830 msgr2=0x7fe964038ce0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.133:6800/1670376163 2024-02-20T07:49:21.375 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:15.407+0000 7fe97a7fc700 1 --2- 172.21.15.133:0/3274797933 >> [v2:172.21.15.133:6800/1670376163,v1:172.21.15.133:6801/1670376163] conn(0x7fe964036830 0x7fe964038ce0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-02-20T07:49:21.375 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:15.608+0000 7fe97a7fc700 1 -- 172.21.15.133:0/3274797933 >> [v2:172.21.15.133:6800/1670376163,v1:172.21.15.133:6801/1670376163] conn(0x7fe964036830 msgr2=0x7fe964038ce0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.133:6800/1670376163 2024-02-20T07:49:21.375 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:15.608+0000 7fe97a7fc700 1 --2- 172.21.15.133:0/3274797933 >> [v2:172.21.15.133:6800/1670376163,v1:172.21.15.133:6801/1670376163] conn(0x7fe964036830 0x7fe964038ce0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-02-20T07:49:21.375 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:16.009+0000 7fe97a7fc700 1 -- 172.21.15.133:0/3274797933 >> [v2:172.21.15.133:6800/1670376163,v1:172.21.15.133:6801/1670376163] conn(0x7fe964036830 msgr2=0x7fe964038ce0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.133:6800/1670376163 2024-02-20T07:49:21.375 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:16.009+0000 7fe97a7fc700 1 --2- 172.21.15.133:0/3274797933 >> [v2:172.21.15.133:6800/1670376163,v1:172.21.15.133:6801/1670376163] conn(0x7fe964036830 0x7fe964038ce0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-02-20T07:49:21.376 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:16.810+0000 7fe97a7fc700 1 -- 172.21.15.133:0/3274797933 >> [v2:172.21.15.133:6800/1670376163,v1:172.21.15.133:6801/1670376163] conn(0x7fe964036830 msgr2=0x7fe964038ce0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.133:6800/1670376163 2024-02-20T07:49:21.376 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:16.810+0000 7fe97a7fc700 1 --2- 172.21.15.133:0/3274797933 >> [v2:172.21.15.133:6800/1670376163,v1:172.21.15.133:6801/1670376163] conn(0x7fe964036830 0x7fe964038ce0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-02-20T07:49:21.376 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:18.411+0000 7fe97a7fc700 1 -- 172.21.15.133:0/3274797933 >> [v2:172.21.15.133:6800/1670376163,v1:172.21.15.133:6801/1670376163] conn(0x7fe964036830 msgr2=0x7fe964038ce0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.133:6800/1670376163 2024-02-20T07:49:21.376 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:18.411+0000 7fe97a7fc700 1 --2- 172.21.15.133:0/3274797933 >> [v2:172.21.15.133:6800/1670376163,v1:172.21.15.133:6801/1670376163] conn(0x7fe964036830 0x7fe964038ce0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-02-20T07:49:21.376 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:19.685+0000 7fe9637fe700 1 -- 172.21.15.133:0/3274797933 <== mon.0 v2:172.21.15.133:3300/0 6 ==== mgrmap(e 10) v1 ==== 43148+0+0 (secure 0 0 0) 0x7fe97000fdc0 con 0x7fe97c102770 2024-02-20T07:49:21.376 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:19.685+0000 7fe9637fe700 1 -- 172.21.15.133:0/3274797933 >> [v2:172.21.15.133:6800/1670376163,v1:172.21.15.133:6801/1670376163] conn(0x7fe964036830 msgr2=0x7fe964038ce0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-02-20T07:49:21.376 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:19.685+0000 7fe9637fe700 1 --2- 172.21.15.133:0/3274797933 >> [v2:172.21.15.133:6800/1670376163,v1:172.21.15.133:6801/1670376163] conn(0x7fe964036830 0x7fe964038ce0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:49:21.376 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:20.687+0000 7fe9637fe700 1 -- 172.21.15.133:0/3274797933 <== mon.0 v2:172.21.15.133:3300/0 7 ==== mgrmap(e 11) v1 ==== 43275+0+0 (secure 0 0 0) 0x7fe97001a430 con 0x7fe97c102770 2024-02-20T07:49:21.376 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:20.687+0000 7fe9637fe700 1 --2- 172.21.15.133:0/3274797933 >> [v2:172.21.15.133:6800/3944459195,v1:172.21.15.133:6801/3944459195] conn(0x7fe964039ff0 0x7fe96403c3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:49:21.376 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:20.687+0000 7fe9637fe700 1 -- 172.21.15.133:0/3274797933 --> [v2:172.21.15.133:6800/3944459195,v1:172.21.15.133:6801/3944459195] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe97c05f5c0 con 0x7fe964039ff0 2024-02-20T07:49:21.376 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:20.687+0000 7fe97a7fc700 1 --2- 172.21.15.133:0/3274797933 >> [v2:172.21.15.133:6800/3944459195,v1:172.21.15.133:6801/3944459195] conn(0x7fe964039ff0 0x7fe96403c3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:49:21.376 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:20.688+0000 7fe97a7fc700 1 --2- 172.21.15.133:0/3274797933 >> [v2:172.21.15.133:6800/3944459195,v1:172.21.15.133:6801/3944459195] conn(0x7fe964039ff0 0x7fe96403c3d0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fe96c0039d0 tx=0x7fe96c009350 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:49:21.376 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:20.688+0000 7fe9637fe700 1 -- 172.21.15.133:0/3274797933 <== mgr.14162 v2:172.21.15.133:6800/3944459195 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+6549 (secure 0 0 0) 0x7fe97c05f5c0 con 0x7fe964039ff0 2024-02-20T07:49:21.377 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:20.694+0000 7fe98189d700 1 -- 172.21.15.133:0/3274797933 --> [v2:172.21.15.133:6800/3944459195,v1:172.21.15.133:6801/3944459195] -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7fe97c112110 con 0x7fe964039ff0 2024-02-20T07:49:21.377 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:20.695+0000 7fe9637fe700 1 -- 172.21.15.133:0/3274797933 <== mgr.14162 v2:172.21.15.133:6800/3944459195 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+52 (secure 0 0 0) 0x7fe97c112110 con 0x7fe964039ff0 2024-02-20T07:49:21.377 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:20.695+0000 7fe98189d700 1 -- 172.21.15.133:0/3274797933 >> [v2:172.21.15.133:6800/3944459195,v1:172.21.15.133:6801/3944459195] conn(0x7fe964039ff0 msgr2=0x7fe96403c3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:49:21.377 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:20.695+0000 7fe98189d700 1 --2- 172.21.15.133:0/3274797933 >> [v2:172.21.15.133:6800/3944459195,v1:172.21.15.133:6801/3944459195] conn(0x7fe964039ff0 0x7fe96403c3d0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fe96c0039d0 tx=0x7fe96c009350 comp rx=0 tx=0).stop 2024-02-20T07:49:21.377 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:20.695+0000 7fe98189d700 1 -- 172.21.15.133:0/3274797933 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fe97c102770 msgr2=0x7fe97c111540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:49:21.377 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:20.695+0000 7fe98189d700 1 --2- 172.21.15.133:0/3274797933 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fe97c102770 0x7fe97c111540 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fe970011650 tx=0x7fe970009250 comp rx=0 tx=0).stop 2024-02-20T07:49:21.377 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:20.695+0000 7fe98189d700 1 -- 172.21.15.133:0/3274797933 shutdown_connections 2024-02-20T07:49:21.377 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:20.695+0000 7fe98189d700 1 --2- 172.21.15.133:0/3274797933 >> [v2:172.21.15.133:6800/3944459195,v1:172.21.15.133:6801/3944459195] conn(0x7fe964039ff0 0x7fe96403c3d0 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:49:21.377 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:20.695+0000 7fe98189d700 1 --2- 172.21.15.133:0/3274797933 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fe97c102770 0x7fe97c111540 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:49:21.377 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:20.695+0000 7fe98189d700 1 --2- 172.21.15.133:0/3274797933 >> [v2:172.21.15.133:6800/1670376163,v1:172.21.15.133:6801/1670376163] conn(0x7fe964036830 0x7fe964038ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:49:21.377 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:20.695+0000 7fe98189d700 1 -- 172.21.15.133:0/3274797933 >> 172.21.15.133:0/3274797933 conn(0x7fe97c0fded0 msgr2=0x7fe97c0fec90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:49:21.377 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:20.695+0000 7fe98189d700 1 -- 172.21.15.133:0/3274797933 shutdown_connections 2024-02-20T07:49:21.377 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:20.695+0000 7fe98189d700 1 -- 172.21.15.133:0/3274797933 wait complete. 2024-02-20T07:49:21.377 INFO:teuthology.orchestra.run.smithi133.stdout:mgr epoch 9 is available 2024-02-20T07:49:21.377 INFO:teuthology.orchestra.run.smithi133.stdout:Generating a dashboard self-signed certificate... 2024-02-20T07:49:22.492 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:49:22 smithi133 ceph-mon[29117]: [20/Feb/2024:07:49:20] ENGINE Bus STARTING 2024-02-20T07:49:22.492 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:49:22 smithi133 ceph-mon[29117]: [20/Feb/2024:07:49:20] ENGINE Serving on https://172.21.15.133:7150 2024-02-20T07:49:22.492 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:49:22 smithi133 ceph-mon[29117]: [20/Feb/2024:07:49:20] ENGINE Bus STARTED 2024-02-20T07:49:22.492 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:49:22 smithi133 ceph-mon[29117]: from='client.14166 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-02-20T07:49:22.492 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:49:22 smithi133 ceph-mon[29117]: from='client.14166 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-02-20T07:49:22.492 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:49:22 smithi133 ceph-mon[29117]: from='mgr.14162 172.21.15.133:0/3642562278' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:49:22.779 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stdout Self-signed certificate created 2024-02-20T07:49:22.779 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:21.826+0000 7f60ee03c700 1 Processor -- start 2024-02-20T07:49:22.779 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:21.826+0000 7f60ee03c700 1 -- start start 2024-02-20T07:49:22.779 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:21.826+0000 7f60ee03c700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f60e8103d80 0x7f60e8104190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:49:22.779 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:21.826+0000 7f60ee03c700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60e81046d0 con 0x7f60e8103d80 2024-02-20T07:49:22.779 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:21.826+0000 7f60e77fe700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f60e8103d80 0x7f60e8104190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:49:22.779 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:21.826+0000 7f60e77fe700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f60e8103d80 0x7f60e8104190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.133:49736/0 (socket says 172.21.15.133:49736) 2024-02-20T07:49:22.779 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:21.826+0000 7f60e77fe700 1 -- 172.21.15.133:0/244986692 learned_addr learned my addr 172.21.15.133:0/244986692 (peer_addr_for_me v2:172.21.15.133:0/0) 2024-02-20T07:49:22.780 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:21.827+0000 7f60e77fe700 1 -- 172.21.15.133:0/244986692 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60e8104810 con 0x7f60e8103d80 2024-02-20T07:49:22.780 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:21.827+0000 7f60e77fe700 1 --2- 172.21.15.133:0/244986692 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f60e8103d80 0x7f60e8104190 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f60d80098e0 tx=0x7f60d8009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d95c6feda202943f server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:49:22.780 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:21.827+0000 7f60e6ffd700 1 -- 172.21.15.133:0/244986692 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f60d800b860 con 0x7f60e8103d80 2024-02-20T07:49:22.780 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:21.827+0000 7f60e6ffd700 1 -- 172.21.15.133:0/244986692 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f60d800be90 con 0x7f60e8103d80 2024-02-20T07:49:22.780 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:21.827+0000 7f60e6ffd700 1 -- 172.21.15.133:0/244986692 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f60d80103c0 con 0x7f60e8103d80 2024-02-20T07:49:22.780 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:21.827+0000 7f60ee03c700 1 -- 172.21.15.133:0/244986692 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f60e8103d80 msgr2=0x7f60e8104190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:49:22.780 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:21.827+0000 7f60ee03c700 1 --2- 172.21.15.133:0/244986692 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f60e8103d80 0x7f60e8104190 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f60d80098e0 tx=0x7f60d8009250 comp rx=0 tx=0).stop 2024-02-20T07:49:22.780 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:21.828+0000 7f60ee03c700 1 -- 172.21.15.133:0/244986692 shutdown_connections 2024-02-20T07:49:22.780 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:21.828+0000 7f60ee03c700 1 --2- 172.21.15.133:0/244986692 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f60e8103d80 0x7f60e8104190 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:49:22.780 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:21.828+0000 7f60ee03c700 1 -- 172.21.15.133:0/244986692 >> 172.21.15.133:0/244986692 conn(0x7f60e80fc580 msgr2=0x7f60e80fe990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:49:22.780 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:21.828+0000 7f60ee03c700 1 -- 172.21.15.133:0/244986692 shutdown_connections 2024-02-20T07:49:22.780 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:21.828+0000 7f60ee03c700 1 -- 172.21.15.133:0/244986692 wait complete. 2024-02-20T07:49:22.780 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:21.828+0000 7f60ee03c700 1 Processor -- start 2024-02-20T07:49:22.780 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:21.828+0000 7f60ee03c700 1 -- start start 2024-02-20T07:49:22.781 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:21.829+0000 7f60ee03c700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f60e8103d80 0x7f60e8111870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:49:22.781 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:21.829+0000 7f60ee03c700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60e8111db0 con 0x7f60e8103d80 2024-02-20T07:49:22.781 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:21.829+0000 7f60e77fe700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f60e8103d80 0x7f60e8111870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:49:22.781 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:21.829+0000 7f60e77fe700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f60e8103d80 0x7f60e8111870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.133:49744/0 (socket says 172.21.15.133:49744) 2024-02-20T07:49:22.781 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:21.829+0000 7f60e77fe700 1 -- 172.21.15.133:0/654297129 learned_addr learned my addr 172.21.15.133:0/654297129 (peer_addr_for_me v2:172.21.15.133:0/0) 2024-02-20T07:49:22.781 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:21.829+0000 7f60e77fe700 1 -- 172.21.15.133:0/654297129 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60d80094f0 con 0x7f60e8103d80 2024-02-20T07:49:22.781 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:21.829+0000 7f60e77fe700 1 --2- 172.21.15.133:0/654297129 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f60e8103d80 0x7f60e8111870 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f60d8006b00 tx=0x7f60d8009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:49:22.781 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:21.829+0000 7f60e4ff9700 1 -- 172.21.15.133:0/654297129 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f60d80109a0 con 0x7f60e8103d80 2024-02-20T07:49:22.782 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:21.829+0000 7f60ee03c700 1 -- 172.21.15.133:0/654297129 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f60e8111fb0 con 0x7f60e8103d80 2024-02-20T07:49:22.782 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:21.829+0000 7f60e4ff9700 1 -- 172.21.15.133:0/654297129 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f60d801a430 con 0x7f60e8103d80 2024-02-20T07:49:22.782 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:21.829+0000 7f60e4ff9700 1 -- 172.21.15.133:0/654297129 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f60d80245b0 con 0x7f60e8103d80 2024-02-20T07:49:22.782 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:21.829+0000 7f60ee03c700 1 -- 172.21.15.133:0/654297129 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f60e8112c50 con 0x7f60e8103d80 2024-02-20T07:49:22.782 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:21.830+0000 7f60e4ff9700 1 -- 172.21.15.133:0/654297129 <== mon.0 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 11) v1 ==== 43275+0+0 (secure 0 0 0) 0x7f60d8024830 con 0x7f60e8103d80 2024-02-20T07:49:22.782 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:21.830+0000 7f60e4ff9700 1 --2- 172.21.15.133:0/654297129 >> [v2:172.21.15.133:6800/3944459195,v1:172.21.15.133:6801/3944459195] conn(0x7f60d40366b0 0x7f60d4038b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:49:22.782 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:21.830+0000 7f60ee03c700 1 -- 172.21.15.133:0/654297129 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f60e805f5c0 con 0x7f60e8103d80 2024-02-20T07:49:22.782 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:21.830+0000 7f60e4ff9700 1 -- 172.21.15.133:0/654297129 <== mon.0 v2:172.21.15.133:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f60d804a650 con 0x7f60e8103d80 2024-02-20T07:49:22.782 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:21.830+0000 7f60deffd700 1 --2- 172.21.15.133:0/654297129 >> [v2:172.21.15.133:6800/3944459195,v1:172.21.15.133:6801/3944459195] conn(0x7f60d40366b0 0x7f60d4038b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:49:22.782 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:21.831+0000 7f60deffd700 1 --2- 172.21.15.133:0/654297129 >> [v2:172.21.15.133:6800/3944459195,v1:172.21.15.133:6801/3944459195] conn(0x7f60d40366b0 0x7f60d4038b60 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f60d0009990 tx=0x7f60d0006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:49:22.782 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:21.835+0000 7f60e4ff9700 1 -- 172.21.15.133:0/654297129 <== mon.0 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f60d80189a0 con 0x7f60e8103d80 2024-02-20T07:49:22.782 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:22.081+0000 7f60ee03c700 1 -- 172.21.15.133:0/654297129 --> [v2:172.21.15.133:6800/3944459195,v1:172.21.15.133:6801/3944459195] -- mgr_command(tid 0: {"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}) v1 -- 0x7f60e8101400 con 0x7f60d40366b0 2024-02-20T07:49:22.782 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:22.150+0000 7f60e4ff9700 1 -- 172.21.15.133:0/654297129 <== mgr.14162 v2:172.21.15.133:6800/3944459195 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+32 (secure 0 0 0) 0x7f60e8101400 con 0x7f60d40366b0 2024-02-20T07:49:22.782 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:22.153+0000 7f60ee03c700 1 -- 172.21.15.133:0/654297129 >> [v2:172.21.15.133:6800/3944459195,v1:172.21.15.133:6801/3944459195] conn(0x7f60d40366b0 msgr2=0x7f60d4038b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:49:22.782 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:22.153+0000 7f60ee03c700 1 --2- 172.21.15.133:0/654297129 >> [v2:172.21.15.133:6800/3944459195,v1:172.21.15.133:6801/3944459195] conn(0x7f60d40366b0 0x7f60d4038b60 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f60d0009990 tx=0x7f60d0006e40 comp rx=0 tx=0).stop 2024-02-20T07:49:22.783 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:22.153+0000 7f60ee03c700 1 -- 172.21.15.133:0/654297129 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f60e8103d80 msgr2=0x7f60e8111870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:49:22.783 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:22.153+0000 7f60ee03c700 1 --2- 172.21.15.133:0/654297129 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f60e8103d80 0x7f60e8111870 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f60d8006b00 tx=0x7f60d8009250 comp rx=0 tx=0).stop 2024-02-20T07:49:22.783 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:22.153+0000 7f60ee03c700 1 -- 172.21.15.133:0/654297129 shutdown_connections 2024-02-20T07:49:22.783 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:22.153+0000 7f60ee03c700 1 --2- 172.21.15.133:0/654297129 >> [v2:172.21.15.133:6800/3944459195,v1:172.21.15.133:6801/3944459195] conn(0x7f60d40366b0 0x7f60d4038b60 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:49:22.783 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:22.153+0000 7f60ee03c700 1 --2- 172.21.15.133:0/654297129 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f60e8103d80 0x7f60e8111870 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:49:22.783 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:22.153+0000 7f60ee03c700 1 -- 172.21.15.133:0/654297129 >> 172.21.15.133:0/654297129 conn(0x7f60e80fc580 msgr2=0x7f60e80fd4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:49:22.783 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:22.153+0000 7f60ee03c700 1 -- 172.21.15.133:0/654297129 shutdown_connections 2024-02-20T07:49:22.783 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:22.153+0000 7f60ee03c700 1 -- 172.21.15.133:0/654297129 wait complete. 2024-02-20T07:49:22.783 INFO:teuthology.orchestra.run.smithi133.stdout:Creating initial admin user... 2024-02-20T07:49:23.413 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:49:23 smithi133 ceph-mon[29117]: from='client.14174 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-02-20T07:49:23.413 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:49:23 smithi133 ceph-mon[29117]: from='mgr.14162 172.21.15.133:0/3642562278' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:49:23.413 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:49:23 smithi133 ceph-mon[29117]: from='mgr.14162 172.21.15.133:0/3642562278' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:49:23.414 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:49:23 smithi133 ceph-mon[29117]: from='mgr.14162 172.21.15.133:0/3642562278' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:49:23.414 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:49:23 smithi133 ceph-mon[29117]: mgrmap e12: smithi133.mrvcbp(active, since 2s) 2024-02-20T07:49:24.667 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:49:24 smithi133 ceph-mon[29117]: from='client.14176 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-02-20T07:49:24.667 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:49:24 smithi133 ceph-mon[29117]: from='mgr.14162 172.21.15.133:0/3642562278' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:49:24.675 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$2gLIxuROJpnWw36DnMCYO.BAcPVtgeAl3kLKN.8duBWiaOKgD.F7m", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1708415363, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-02-20T07:49:24.675 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:23.201+0000 7f90da0bc700 1 Processor -- start 2024-02-20T07:49:24.675 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:23.201+0000 7f90da0bc700 1 -- start start 2024-02-20T07:49:24.675 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:23.201+0000 7f90da0bc700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f90d4101670 0x7f90d4101a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:49:24.675 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:23.201+0000 7f90da0bc700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90d4101fc0 con 0x7f90d4101670 2024-02-20T07:49:24.675 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:23.202+0000 7f90d37fe700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f90d4101670 0x7f90d4101a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:49:24.675 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:23.202+0000 7f90d37fe700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f90d4101670 0x7f90d4101a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.133:49758/0 (socket says 172.21.15.133:49758) 2024-02-20T07:49:24.675 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:23.202+0000 7f90d37fe700 1 -- 172.21.15.133:0/730994846 learned_addr learned my addr 172.21.15.133:0/730994846 (peer_addr_for_me v2:172.21.15.133:0/0) 2024-02-20T07:49:24.676 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:23.202+0000 7f90d37fe700 1 -- 172.21.15.133:0/730994846 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90d4102100 con 0x7f90d4101670 2024-02-20T07:49:24.676 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:23.202+0000 7f90d37fe700 1 --2- 172.21.15.133:0/730994846 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f90d4101670 0x7f90d4101a80 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f90cc009e50 tx=0x7f90cc009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b3266c5d7291f07b server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:49:24.676 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:23.202+0000 7f90d27fc700 1 -- 172.21.15.133:0/730994846 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f90cc0092e0 con 0x7f90d4101670 2024-02-20T07:49:24.676 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:23.202+0000 7f90d27fc700 1 -- 172.21.15.133:0/730994846 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f90cc012430 con 0x7f90d4101670 2024-02-20T07:49:24.676 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:23.202+0000 7f90d27fc700 1 -- 172.21.15.133:0/730994846 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f90cc0075a0 con 0x7f90d4101670 2024-02-20T07:49:24.676 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:23.203+0000 7f90da0bc700 1 -- 172.21.15.133:0/730994846 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f90d4101670 msgr2=0x7f90d4101a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:49:24.676 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:23.203+0000 7f90da0bc700 1 --2- 172.21.15.133:0/730994846 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f90d4101670 0x7f90d4101a80 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f90cc009e50 tx=0x7f90cc009840 comp rx=0 tx=0).stop 2024-02-20T07:49:24.676 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:23.203+0000 7f90da0bc700 1 -- 172.21.15.133:0/730994846 shutdown_connections 2024-02-20T07:49:24.676 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:23.203+0000 7f90da0bc700 1 --2- 172.21.15.133:0/730994846 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f90d4101670 0x7f90d4101a80 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:49:24.676 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:23.203+0000 7f90da0bc700 1 -- 172.21.15.133:0/730994846 >> 172.21.15.133:0/730994846 conn(0x7f90d40fcdd0 msgr2=0x7f90d40ff200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:49:24.676 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:23.203+0000 7f90da0bc700 1 -- 172.21.15.133:0/730994846 shutdown_connections 2024-02-20T07:49:24.676 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:23.203+0000 7f90da0bc700 1 -- 172.21.15.133:0/730994846 wait complete. 2024-02-20T07:49:24.676 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:23.204+0000 7f90da0bc700 1 Processor -- start 2024-02-20T07:49:24.676 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:23.204+0000 7f90da0bc700 1 -- start start 2024-02-20T07:49:24.677 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:23.204+0000 7f90da0bc700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f90d4101670 0x7f90d41104f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:49:24.677 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:23.204+0000 7f90da0bc700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90d4110a30 con 0x7f90d4101670 2024-02-20T07:49:24.677 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:23.205+0000 7f90d37fe700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f90d4101670 0x7f90d41104f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:49:24.677 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:23.205+0000 7f90d37fe700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f90d4101670 0x7f90d41104f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.133:49770/0 (socket says 172.21.15.133:49770) 2024-02-20T07:49:24.677 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:23.205+0000 7f90d37fe700 1 -- 172.21.15.133:0/3155406502 learned_addr learned my addr 172.21.15.133:0/3155406502 (peer_addr_for_me v2:172.21.15.133:0/0) 2024-02-20T07:49:24.677 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:23.205+0000 7f90d37fe700 1 -- 172.21.15.133:0/3155406502 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90cc009ae0 con 0x7f90d4101670 2024-02-20T07:49:24.677 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:23.205+0000 7f90d37fe700 1 --2- 172.21.15.133:0/3155406502 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f90d4101670 0x7f90d41104f0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f90cc0127d0 tx=0x7f90cc009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:49:24.677 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:23.205+0000 7f90bbfff700 1 -- 172.21.15.133:0/3155406502 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f90cc0092e0 con 0x7f90d4101670 2024-02-20T07:49:24.677 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:23.205+0000 7f90da0bc700 1 -- 172.21.15.133:0/3155406502 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f90d4110c30 con 0x7f90d4101670 2024-02-20T07:49:24.677 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:23.205+0000 7f90bbfff700 1 -- 172.21.15.133:0/3155406502 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f90cc007d30 con 0x7f90d4101670 2024-02-20T07:49:24.677 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:23.205+0000 7f90bbfff700 1 -- 172.21.15.133:0/3155406502 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f90cc025960 con 0x7f90d4101670 2024-02-20T07:49:24.677 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:23.205+0000 7f90da0bc700 1 -- 172.21.15.133:0/3155406502 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f90d41118d0 con 0x7f90d4101670 2024-02-20T07:49:24.677 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:23.206+0000 7f90bbfff700 1 -- 172.21.15.133:0/3155406502 <== mon.0 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 12) v1 ==== 43381+0+0 (secure 0 0 0) 0x7f90cc019050 con 0x7f90d4101670 2024-02-20T07:49:24.677 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:23.206+0000 7f90da0bc700 1 -- 172.21.15.133:0/3155406502 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f90d405f5c0 con 0x7f90d4101670 2024-02-20T07:49:24.677 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:23.206+0000 7f90bbfff700 1 --2- 172.21.15.133:0/3155406502 >> [v2:172.21.15.133:6800/3944459195,v1:172.21.15.133:6801/3944459195] conn(0x7f90bc0367e0 0x7f90bc038c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:49:24.677 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:23.207+0000 7f90bbfff700 1 -- 172.21.15.133:0/3155406502 <== mon.0 v2:172.21.15.133:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f90cc04a220 con 0x7f90d4101670 2024-02-20T07:49:24.678 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:23.207+0000 7f90d2ffd700 1 --2- 172.21.15.133:0/3155406502 >> [v2:172.21.15.133:6800/3944459195,v1:172.21.15.133:6801/3944459195] conn(0x7f90bc0367e0 0x7f90bc038c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:49:24.678 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:23.207+0000 7f90d2ffd700 1 --2- 172.21.15.133:0/3155406502 >> [v2:172.21.15.133:6800/3944459195,v1:172.21.15.133:6801/3944459195] conn(0x7f90bc0367e0 0x7f90bc038c90 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f90c8009990 tx=0x7f90c8006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:49:24.678 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:23.211+0000 7f90bbfff700 1 -- 172.21.15.133:0/3155406502 <== mon.0 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f90cc02d9a0 con 0x7f90d4101670 2024-02-20T07:49:24.678 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:23.430+0000 7f90da0bc700 1 -- 172.21.15.133:0/3155406502 --> [v2:172.21.15.133:6800/3944459195,v1:172.21.15.133:6801/3944459195] -- mgr_command(tid 0: {"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}) v1 -- 0x7f90d405f3e0 con 0x7f90bc0367e0 2024-02-20T07:49:24.678 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:23.652+0000 7f90bbfff700 1 -- 172.21.15.133:0/3155406502 <== mgr.14162 v2:172.21.15.133:6800/3944459195 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+252 (secure 0 0 0) 0x7f90d405f3e0 con 0x7f90bc0367e0 2024-02-20T07:49:24.678 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:23.654+0000 7f90da0bc700 1 -- 172.21.15.133:0/3155406502 >> [v2:172.21.15.133:6800/3944459195,v1:172.21.15.133:6801/3944459195] conn(0x7f90bc0367e0 msgr2=0x7f90bc038c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:49:24.678 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:23.654+0000 7f90da0bc700 1 --2- 172.21.15.133:0/3155406502 >> [v2:172.21.15.133:6800/3944459195,v1:172.21.15.133:6801/3944459195] conn(0x7f90bc0367e0 0x7f90bc038c90 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f90c8009990 tx=0x7f90c8006e40 comp rx=0 tx=0).stop 2024-02-20T07:49:24.678 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:23.654+0000 7f90da0bc700 1 -- 172.21.15.133:0/3155406502 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f90d4101670 msgr2=0x7f90d41104f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:49:24.678 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:23.654+0000 7f90da0bc700 1 --2- 172.21.15.133:0/3155406502 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f90d4101670 0x7f90d41104f0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f90cc0127d0 tx=0x7f90cc009840 comp rx=0 tx=0).stop 2024-02-20T07:49:24.678 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:23.655+0000 7f90da0bc700 1 -- 172.21.15.133:0/3155406502 shutdown_connections 2024-02-20T07:49:24.678 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:23.655+0000 7f90da0bc700 1 --2- 172.21.15.133:0/3155406502 >> [v2:172.21.15.133:6800/3944459195,v1:172.21.15.133:6801/3944459195] conn(0x7f90bc0367e0 0x7f90bc038c90 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:49:24.678 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:23.655+0000 7f90da0bc700 1 --2- 172.21.15.133:0/3155406502 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f90d4101670 0x7f90d41104f0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:49:24.678 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:23.655+0000 7f90da0bc700 1 -- 172.21.15.133:0/3155406502 >> 172.21.15.133:0/3155406502 conn(0x7f90d40fcdd0 msgr2=0x7f90d40fe4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:49:24.678 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:23.655+0000 7f90da0bc700 1 -- 172.21.15.133:0/3155406502 shutdown_connections 2024-02-20T07:49:24.678 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:23.655+0000 7f90da0bc700 1 -- 172.21.15.133:0/3155406502 wait complete. 2024-02-20T07:49:24.679 INFO:teuthology.orchestra.run.smithi133.stdout:Fetching dashboard port number... 2024-02-20T07:49:25.883 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stdout 8443 2024-02-20T07:49:25.883 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:25.115+0000 7f4d3fdc0700 1 Processor -- start 2024-02-20T07:49:25.883 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:25.115+0000 7f4d3fdc0700 1 -- start start 2024-02-20T07:49:25.883 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:25.115+0000 7f4d3fdc0700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f4d38102770 0x7f4d38102b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:49:25.883 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:25.115+0000 7f4d3fdc0700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d381030c0 con 0x7f4d38102770 2024-02-20T07:49:25.883 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:25.115+0000 7f4d3db5c700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f4d38102770 0x7f4d38102b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:49:25.883 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:25.115+0000 7f4d3db5c700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f4d38102770 0x7f4d38102b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.133:49778/0 (socket says 172.21.15.133:49778) 2024-02-20T07:49:25.883 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:25.115+0000 7f4d3db5c700 1 -- 172.21.15.133:0/2554897741 learned_addr learned my addr 172.21.15.133:0/2554897741 (peer_addr_for_me v2:172.21.15.133:0/0) 2024-02-20T07:49:25.883 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:25.115+0000 7f4d3db5c700 1 -- 172.21.15.133:0/2554897741 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d38103200 con 0x7f4d38102770 2024-02-20T07:49:25.883 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:25.116+0000 7f4d3db5c700 1 --2- 172.21.15.133:0/2554897741 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f4d38102770 0x7f4d38102b80 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f4d280098e0 tx=0x7f4d28009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=362bc1760ca608f9 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:49:25.884 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:25.116+0000 7f4d3cb5a700 1 -- 172.21.15.133:0/2554897741 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4d2800b860 con 0x7f4d38102770 2024-02-20T07:49:25.884 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:25.116+0000 7f4d3cb5a700 1 -- 172.21.15.133:0/2554897741 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4d2800be90 con 0x7f4d38102770 2024-02-20T07:49:25.884 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:25.116+0000 7f4d3cb5a700 1 -- 172.21.15.133:0/2554897741 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4d280103c0 con 0x7f4d38102770 2024-02-20T07:49:25.884 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:25.116+0000 7f4d3fdc0700 1 -- 172.21.15.133:0/2554897741 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f4d38102770 msgr2=0x7f4d38102b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:49:25.884 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:25.116+0000 7f4d3fdc0700 1 --2- 172.21.15.133:0/2554897741 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f4d38102770 0x7f4d38102b80 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f4d280098e0 tx=0x7f4d28009250 comp rx=0 tx=0).stop 2024-02-20T07:49:25.884 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:25.116+0000 7f4d3fdc0700 1 -- 172.21.15.133:0/2554897741 shutdown_connections 2024-02-20T07:49:25.884 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:25.116+0000 7f4d3fdc0700 1 --2- 172.21.15.133:0/2554897741 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f4d38102770 0x7f4d38102b80 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:49:25.884 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:25.116+0000 7f4d3fdc0700 1 -- 172.21.15.133:0/2554897741 >> 172.21.15.133:0/2554897741 conn(0x7f4d380fded0 msgr2=0x7f4d38100300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:49:25.884 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:25.116+0000 7f4d3fdc0700 1 -- 172.21.15.133:0/2554897741 shutdown_connections 2024-02-20T07:49:25.884 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:25.117+0000 7f4d3fdc0700 1 -- 172.21.15.133:0/2554897741 wait complete. 2024-02-20T07:49:25.884 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:25.117+0000 7f4d3fdc0700 1 Processor -- start 2024-02-20T07:49:25.884 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:25.117+0000 7f4d3fdc0700 1 -- start start 2024-02-20T07:49:25.884 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:25.117+0000 7f4d3fdc0700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f4d38102770 0x7f4d381115c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:49:25.885 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:25.117+0000 7f4d3fdc0700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d38111b00 con 0x7f4d38102770 2024-02-20T07:49:25.885 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:25.117+0000 7f4d3db5c700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f4d38102770 0x7f4d381115c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:49:25.885 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:25.117+0000 7f4d3db5c700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f4d38102770 0x7f4d381115c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.133:49780/0 (socket says 172.21.15.133:49780) 2024-02-20T07:49:25.885 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:25.117+0000 7f4d3db5c700 1 -- 172.21.15.133:0/1066980668 learned_addr learned my addr 172.21.15.133:0/1066980668 (peer_addr_for_me v2:172.21.15.133:0/0) 2024-02-20T07:49:25.885 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:25.117+0000 7f4d3db5c700 1 -- 172.21.15.133:0/1066980668 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d280094f0 con 0x7f4d38102770 2024-02-20T07:49:25.885 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:25.118+0000 7f4d3db5c700 1 --2- 172.21.15.133:0/1066980668 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f4d38102770 0x7f4d381115c0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f4d28011680 tx=0x7f4d28009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:49:25.885 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:25.118+0000 7f4d2e7fc700 1 -- 172.21.15.133:0/1066980668 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4d28010890 con 0x7f4d38102770 2024-02-20T07:49:25.885 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:25.118+0000 7f4d2e7fc700 1 -- 172.21.15.133:0/1066980668 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4d28010ec0 con 0x7f4d38102770 2024-02-20T07:49:25.885 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:25.118+0000 7f4d3fdc0700 1 -- 172.21.15.133:0/1066980668 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4d38111d00 con 0x7f4d38102770 2024-02-20T07:49:25.886 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:25.118+0000 7f4d2e7fc700 1 -- 172.21.15.133:0/1066980668 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4d280243f0 con 0x7f4d38102770 2024-02-20T07:49:25.886 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:25.118+0000 7f4d3fdc0700 1 -- 172.21.15.133:0/1066980668 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4d381129a0 con 0x7f4d38102770 2024-02-20T07:49:25.886 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:25.118+0000 7f4d2e7fc700 1 -- 172.21.15.133:0/1066980668 <== mon.0 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 12) v1 ==== 43381+0+0 (secure 0 0 0) 0x7f4d2802d3f0 con 0x7f4d38102770 2024-02-20T07:49:25.886 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:25.119+0000 7f4d2e7fc700 1 --2- 172.21.15.133:0/1066980668 >> [v2:172.21.15.133:6800/3944459195,v1:172.21.15.133:6801/3944459195] conn(0x7f4d240367e0 0x7f4d24038c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:49:25.886 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:25.119+0000 7f4d2e7fc700 1 -- 172.21.15.133:0/1066980668 <== mon.0 v2:172.21.15.133:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f4d2804b5a0 con 0x7f4d38102770 2024-02-20T07:49:25.886 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:25.119+0000 7f4d3d35b700 1 --2- 172.21.15.133:0/1066980668 >> [v2:172.21.15.133:6800/3944459195,v1:172.21.15.133:6801/3944459195] conn(0x7f4d240367e0 0x7f4d24038c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:49:25.886 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:25.119+0000 7f4d3fdc0700 1 -- 172.21.15.133:0/1066980668 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4d38112190 con 0x7f4d38102770 2024-02-20T07:49:25.886 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:25.119+0000 7f4d3d35b700 1 --2- 172.21.15.133:0/1066980668 >> [v2:172.21.15.133:6800/3944459195,v1:172.21.15.133:6801/3944459195] conn(0x7f4d240367e0 0x7f4d24038c90 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f4d34009990 tx=0x7f4d34006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:49:25.886 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:25.123+0000 7f4d2e7fc700 1 -- 172.21.15.133:0/1066980668 <== mon.0 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f4d2802b8e0 con 0x7f4d38102770 2024-02-20T07:49:25.887 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:25.305+0000 7f4d3fdc0700 1 -- 172.21.15.133:0/1066980668 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"} v 0) v1 -- 0x7f4d381045e0 con 0x7f4d38102770 2024-02-20T07:49:25.887 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:25.306+0000 7f4d2e7fc700 1 -- 172.21.15.133:0/1066980668 <== mon.0 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]=0 v8) v1 ==== 112+0+5 (secure 0 0 0) 0x7f4d2802b2f0 con 0x7f4d38102770 2024-02-20T07:49:25.887 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:25.308+0000 7f4d3fdc0700 1 -- 172.21.15.133:0/1066980668 >> [v2:172.21.15.133:6800/3944459195,v1:172.21.15.133:6801/3944459195] conn(0x7f4d240367e0 msgr2=0x7f4d24038c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:49:25.887 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:25.308+0000 7f4d3fdc0700 1 --2- 172.21.15.133:0/1066980668 >> [v2:172.21.15.133:6800/3944459195,v1:172.21.15.133:6801/3944459195] conn(0x7f4d240367e0 0x7f4d24038c90 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f4d34009990 tx=0x7f4d34006e40 comp rx=0 tx=0).stop 2024-02-20T07:49:25.887 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:25.308+0000 7f4d3fdc0700 1 -- 172.21.15.133:0/1066980668 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f4d38102770 msgr2=0x7f4d381115c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:49:25.887 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:25.308+0000 7f4d3fdc0700 1 --2- 172.21.15.133:0/1066980668 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f4d38102770 0x7f4d381115c0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f4d28011680 tx=0x7f4d28009250 comp rx=0 tx=0).stop 2024-02-20T07:49:25.887 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:25.308+0000 7f4d3fdc0700 1 -- 172.21.15.133:0/1066980668 shutdown_connections 2024-02-20T07:49:25.887 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:25.308+0000 7f4d3fdc0700 1 --2- 172.21.15.133:0/1066980668 >> [v2:172.21.15.133:6800/3944459195,v1:172.21.15.133:6801/3944459195] conn(0x7f4d240367e0 0x7f4d24038c90 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:49:25.887 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:25.308+0000 7f4d3fdc0700 1 --2- 172.21.15.133:0/1066980668 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f4d38102770 0x7f4d381115c0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:49:25.887 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:25.308+0000 7f4d3fdc0700 1 -- 172.21.15.133:0/1066980668 >> 172.21.15.133:0/1066980668 conn(0x7f4d380fded0 msgr2=0x7f4d380ff5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:49:25.888 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:25.308+0000 7f4d3fdc0700 1 -- 172.21.15.133:0/1066980668 shutdown_connections 2024-02-20T07:49:25.888 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:25.308+0000 7f4d3fdc0700 1 -- 172.21.15.133:0/1066980668 wait complete. 2024-02-20T07:49:25.888 INFO:teuthology.orchestra.run.smithi133.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-02-20T07:49:25.888 INFO:teuthology.orchestra.run.smithi133.stdout:systemctl: stdout disabled 2024-02-20T07:49:25.895 INFO:teuthology.orchestra.run.smithi133.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-02-20T07:49:25.895 INFO:teuthology.orchestra.run.smithi133.stdout:systemctl: stdout inactive 2024-02-20T07:49:25.895 INFO:teuthology.orchestra.run.smithi133.stdout:firewalld.service is not enabled 2024-02-20T07:49:25.895 INFO:teuthology.orchestra.run.smithi133.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2024-02-20T07:49:25.897 INFO:teuthology.orchestra.run.smithi133.stdout:Ceph Dashboard is now available at: 2024-02-20T07:49:25.897 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-02-20T07:49:25.897 INFO:teuthology.orchestra.run.smithi133.stdout: URL: https://smithi133.front.sepia.ceph.com:8443/ 2024-02-20T07:49:25.897 INFO:teuthology.orchestra.run.smithi133.stdout: User: admin 2024-02-20T07:49:25.897 INFO:teuthology.orchestra.run.smithi133.stdout: Password: b5pa2skgz1 2024-02-20T07:49:25.897 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-02-20T07:49:25.897 INFO:teuthology.orchestra.run.smithi133.stdout:Saving cluster configuration to /var/lib/ceph/4537327c-cfc4-11ee-95bb-87774f69a715/config directory 2024-02-20T07:49:25.898 INFO:teuthology.orchestra.run.smithi133.stdout:Enabling autotune for osd_memory_target 2024-02-20T07:49:25.992 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:49:25 smithi133 ceph-mon[29117]: from='client.? 172.21.15.133:0/1066980668' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-02-20T07:49:27.121 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:26.305+0000 7f4845b8b700 1 Processor -- start 2024-02-20T07:49:27.122 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:26.305+0000 7f4845b8b700 1 -- start start 2024-02-20T07:49:27.122 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:26.306+0000 7f4845b8b700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f4840102c60 0x7f4840103070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:49:27.122 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:26.306+0000 7f4845b8b700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48401035b0 con 0x7f4840102c60 2024-02-20T07:49:27.122 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:26.306+0000 7f483f7fe700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f4840102c60 0x7f4840103070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:49:27.122 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:26.306+0000 7f483f7fe700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f4840102c60 0x7f4840103070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.133:49782/0 (socket says 172.21.15.133:49782) 2024-02-20T07:49:27.122 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:26.306+0000 7f483f7fe700 1 -- 172.21.15.133:0/2288369568 learned_addr learned my addr 172.21.15.133:0/2288369568 (peer_addr_for_me v2:172.21.15.133:0/0) 2024-02-20T07:49:27.122 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:26.306+0000 7f483f7fe700 1 -- 172.21.15.133:0/2288369568 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48401036f0 con 0x7f4840102c60 2024-02-20T07:49:27.122 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:26.306+0000 7f483f7fe700 1 --2- 172.21.15.133:0/2288369568 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f4840102c60 0x7f4840103070 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f48300098e0 tx=0x7f4830009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bd5c91af1b51baa9 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:49:27.122 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:26.306+0000 7f483e7fc700 1 -- 172.21.15.133:0/2288369568 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f483000b860 con 0x7f4840102c60 2024-02-20T07:49:27.122 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:26.306+0000 7f483e7fc700 1 -- 172.21.15.133:0/2288369568 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f483000be90 con 0x7f4840102c60 2024-02-20T07:49:27.122 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:26.306+0000 7f483e7fc700 1 -- 172.21.15.133:0/2288369568 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f48300103c0 con 0x7f4840102c60 2024-02-20T07:49:27.122 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:26.307+0000 7f4845b8b700 1 -- 172.21.15.133:0/2288369568 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f4840102c60 msgr2=0x7f4840103070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:49:27.122 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:26.307+0000 7f4845b8b700 1 --2- 172.21.15.133:0/2288369568 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f4840102c60 0x7f4840103070 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f48300098e0 tx=0x7f4830009250 comp rx=0 tx=0).stop 2024-02-20T07:49:27.122 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:26.307+0000 7f4845b8b700 1 -- 172.21.15.133:0/2288369568 shutdown_connections 2024-02-20T07:49:27.123 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:26.307+0000 7f4845b8b700 1 --2- 172.21.15.133:0/2288369568 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f4840102c60 0x7f4840103070 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:49:27.123 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:26.307+0000 7f4845b8b700 1 -- 172.21.15.133:0/2288369568 >> 172.21.15.133:0/2288369568 conn(0x7f48400fc580 msgr2=0x7f48400fe990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:49:27.123 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:26.307+0000 7f4845b8b700 1 -- 172.21.15.133:0/2288369568 shutdown_connections 2024-02-20T07:49:27.123 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:26.307+0000 7f4845b8b700 1 -- 172.21.15.133:0/2288369568 wait complete. 2024-02-20T07:49:27.123 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:26.308+0000 7f4845b8b700 1 Processor -- start 2024-02-20T07:49:27.123 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:26.308+0000 7f4845b8b700 1 -- start start 2024-02-20T07:49:27.123 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:26.308+0000 7f4845b8b700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f4840102c60 0x7f4840116980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:49:27.123 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:26.308+0000 7f4845b8b700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4840112370 con 0x7f4840102c60 2024-02-20T07:49:27.123 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:26.308+0000 7f483f7fe700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f4840102c60 0x7f4840116980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:49:27.123 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:26.308+0000 7f483f7fe700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f4840102c60 0x7f4840116980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.133:49784/0 (socket says 172.21.15.133:49784) 2024-02-20T07:49:27.123 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:26.308+0000 7f483f7fe700 1 -- 172.21.15.133:0/946421270 learned_addr learned my addr 172.21.15.133:0/946421270 (peer_addr_for_me v2:172.21.15.133:0/0) 2024-02-20T07:49:27.124 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:26.308+0000 7f483f7fe700 1 -- 172.21.15.133:0/946421270 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48300094f0 con 0x7f4840102c60 2024-02-20T07:49:27.124 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:26.308+0000 7f483f7fe700 1 --2- 172.21.15.133:0/946421270 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f4840102c60 0x7f4840116980 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f4830006b00 tx=0x7f4830009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:49:27.124 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:26.308+0000 7f4827fff700 1 -- 172.21.15.133:0/946421270 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f48300109a0 con 0x7f4840102c60 2024-02-20T07:49:27.124 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:26.309+0000 7f4845b8b700 1 -- 172.21.15.133:0/946421270 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4840101540 con 0x7f4840102c60 2024-02-20T07:49:27.124 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:26.309+0000 7f4827fff700 1 -- 172.21.15.133:0/946421270 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f483001a430 con 0x7f4840102c60 2024-02-20T07:49:27.124 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:26.309+0000 7f4827fff700 1 -- 172.21.15.133:0/946421270 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f48300245b0 con 0x7f4840102c60 2024-02-20T07:49:27.124 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:26.309+0000 7f4845b8b700 1 -- 172.21.15.133:0/946421270 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4840116ec0 con 0x7f4840102c60 2024-02-20T07:49:27.124 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:26.309+0000 7f4827fff700 1 -- 172.21.15.133:0/946421270 <== mon.0 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 12) v1 ==== 43381+0+0 (secure 0 0 0) 0x7f48300247e0 con 0x7f4840102c60 2024-02-20T07:49:27.124 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:26.309+0000 7f4845b8b700 1 -- 172.21.15.133:0/946421270 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f484010a570 con 0x7f4840102c60 2024-02-20T07:49:27.125 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:26.310+0000 7f4827fff700 1 --2- 172.21.15.133:0/946421270 >> [v2:172.21.15.133:6800/3944459195,v1:172.21.15.133:6801/3944459195] conn(0x7f4828036830 0x7f4828038ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:49:27.125 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:26.310+0000 7f4827fff700 1 -- 172.21.15.133:0/946421270 <== mon.0 v2:172.21.15.133:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f483004a7d0 con 0x7f4840102c60 2024-02-20T07:49:27.125 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:26.310+0000 7f483effd700 1 --2- 172.21.15.133:0/946421270 >> [v2:172.21.15.133:6800/3944459195,v1:172.21.15.133:6801/3944459195] conn(0x7f4828036830 0x7f4828038ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:49:27.125 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:26.310+0000 7f483effd700 1 --2- 172.21.15.133:0/946421270 >> [v2:172.21.15.133:6800/3944459195,v1:172.21.15.133:6801/3944459195] conn(0x7f4828036830 0x7f4828038ce0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f4834009990 tx=0x7f4834006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:49:27.125 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:26.314+0000 7f4827fff700 1 -- 172.21.15.133:0/946421270 <== mon.0 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f48300189a0 con 0x7f4840102c60 2024-02-20T07:49:27.125 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:26.495+0000 7f4845b8b700 1 -- 172.21.15.133:0/946421270 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command([{prefix=config set, name=osd_memory_target_autotune}] v 0) v1 -- 0x7f484004bed0 con 0x7f4840102c60 2024-02-20T07:49:27.125 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:26.496+0000 7f4827fff700 1 -- 172.21.15.133:0/946421270 <== mon.0 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{prefix=config set, name=osd_memory_target_autotune}]=0 v8)=0 v8) v1 ==== 127+0+0 (secure 0 0 0) 0x7f4830018350 con 0x7f4840102c60 2024-02-20T07:49:27.125 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:26.498+0000 7f4845b8b700 1 -- 172.21.15.133:0/946421270 >> [v2:172.21.15.133:6800/3944459195,v1:172.21.15.133:6801/3944459195] conn(0x7f4828036830 msgr2=0x7f4828038ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:49:27.125 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:26.498+0000 7f4845b8b700 1 --2- 172.21.15.133:0/946421270 >> [v2:172.21.15.133:6800/3944459195,v1:172.21.15.133:6801/3944459195] conn(0x7f4828036830 0x7f4828038ce0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f4834009990 tx=0x7f4834006e40 comp rx=0 tx=0).stop 2024-02-20T07:49:27.125 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:26.498+0000 7f4845b8b700 1 -- 172.21.15.133:0/946421270 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f4840102c60 msgr2=0x7f4840116980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:49:27.125 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:26.498+0000 7f4845b8b700 1 --2- 172.21.15.133:0/946421270 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f4840102c60 0x7f4840116980 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f4830006b00 tx=0x7f4830009250 comp rx=0 tx=0).stop 2024-02-20T07:49:27.125 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:26.498+0000 7f4845b8b700 1 -- 172.21.15.133:0/946421270 shutdown_connections 2024-02-20T07:49:27.125 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:26.498+0000 7f4845b8b700 1 --2- 172.21.15.133:0/946421270 >> [v2:172.21.15.133:6800/3944459195,v1:172.21.15.133:6801/3944459195] conn(0x7f4828036830 0x7f4828038ce0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:49:27.125 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:26.498+0000 7f4845b8b700 1 --2- 172.21.15.133:0/946421270 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f4840102c60 0x7f4840116980 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:49:27.126 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:26.498+0000 7f4845b8b700 1 -- 172.21.15.133:0/946421270 >> 172.21.15.133:0/946421270 conn(0x7f48400fc580 msgr2=0x7f48400fd4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:49:27.126 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:26.498+0000 7f4845b8b700 1 -- 172.21.15.133:0/946421270 shutdown_connections 2024-02-20T07:49:27.126 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:26.498+0000 7f4845b8b700 1 -- 172.21.15.133:0/946421270 wait complete. 2024-02-20T07:49:27.742 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:49:27 smithi133 ceph-mon[29117]: mgrmap e13: smithi133.mrvcbp(active, since 6s) 2024-02-20T07:49:28.377 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:27.502+0000 7f89bc7bb700 1 Processor -- start 2024-02-20T07:49:28.377 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:27.502+0000 7f89bc7bb700 1 -- start start 2024-02-20T07:49:28.378 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:27.502+0000 7f89bc7bb700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f89b4103e60 0x7f89b4104270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:49:28.378 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:27.502+0000 7f89bc7bb700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89b41047b0 con 0x7f89b4103e60 2024-02-20T07:49:28.378 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:27.503+0000 7f89ba557700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f89b4103e60 0x7f89b4104270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:49:28.378 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:27.503+0000 7f89ba557700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f89b4103e60 0x7f89b4104270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.133:33694/0 (socket says 172.21.15.133:33694) 2024-02-20T07:49:28.378 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:27.503+0000 7f89ba557700 1 -- 172.21.15.133:0/2217321213 learned_addr learned my addr 172.21.15.133:0/2217321213 (peer_addr_for_me v2:172.21.15.133:0/0) 2024-02-20T07:49:28.378 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:27.503+0000 7f89ba557700 1 -- 172.21.15.133:0/2217321213 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89b41048f0 con 0x7f89b4103e60 2024-02-20T07:49:28.378 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:27.503+0000 7f89ba557700 1 --2- 172.21.15.133:0/2217321213 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f89b4103e60 0x7f89b4104270 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f89b0009e50 tx=0x7f89b0009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=85103912c3e74ae1 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:49:28.378 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:27.503+0000 7f89b9555700 1 -- 172.21.15.133:0/2217321213 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f89b00092e0 con 0x7f89b4103e60 2024-02-20T07:49:28.378 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:27.503+0000 7f89b9555700 1 -- 172.21.15.133:0/2217321213 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f89b0012430 con 0x7f89b4103e60 2024-02-20T07:49:28.378 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:27.503+0000 7f89b9555700 1 -- 172.21.15.133:0/2217321213 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f89b00075a0 con 0x7f89b4103e60 2024-02-20T07:49:28.378 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:27.504+0000 7f89bc7bb700 1 -- 172.21.15.133:0/2217321213 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f89b4103e60 msgr2=0x7f89b4104270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:49:28.378 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:27.504+0000 7f89bc7bb700 1 --2- 172.21.15.133:0/2217321213 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f89b4103e60 0x7f89b4104270 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f89b0009e50 tx=0x7f89b0009840 comp rx=0 tx=0).stop 2024-02-20T07:49:28.378 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:27.504+0000 7f89bc7bb700 1 -- 172.21.15.133:0/2217321213 shutdown_connections 2024-02-20T07:49:28.378 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:27.504+0000 7f89bc7bb700 1 --2- 172.21.15.133:0/2217321213 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f89b4103e60 0x7f89b4104270 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:49:28.379 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:27.504+0000 7f89bc7bb700 1 -- 172.21.15.133:0/2217321213 >> 172.21.15.133:0/2217321213 conn(0x7f89b40fc580 msgr2=0x7f89b40fe990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:49:28.379 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:27.504+0000 7f89bc7bb700 1 -- 172.21.15.133:0/2217321213 shutdown_connections 2024-02-20T07:49:28.379 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:27.504+0000 7f89bc7bb700 1 -- 172.21.15.133:0/2217321213 wait complete. 2024-02-20T07:49:28.379 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:27.504+0000 7f89bc7bb700 1 Processor -- start 2024-02-20T07:49:28.379 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:27.504+0000 7f89bc7bb700 1 -- start start 2024-02-20T07:49:28.379 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:27.504+0000 7f89bc7bb700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f89b4103e60 0x7f89b41118b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:49:28.379 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:27.504+0000 7f89bc7bb700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89b4111df0 con 0x7f89b4103e60 2024-02-20T07:49:28.379 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:27.505+0000 7f89ba557700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f89b4103e60 0x7f89b41118b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:49:28.379 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:27.505+0000 7f89ba557700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f89b4103e60 0x7f89b41118b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.133:33702/0 (socket says 172.21.15.133:33702) 2024-02-20T07:49:28.379 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:27.505+0000 7f89ba557700 1 -- 172.21.15.133:0/2629952497 learned_addr learned my addr 172.21.15.133:0/2629952497 (peer_addr_for_me v2:172.21.15.133:0/0) 2024-02-20T07:49:28.379 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:27.505+0000 7f89ba557700 1 -- 172.21.15.133:0/2629952497 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89b0009ae0 con 0x7f89b4103e60 2024-02-20T07:49:28.379 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:27.505+0000 7f89ba557700 1 --2- 172.21.15.133:0/2629952497 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f89b4103e60 0x7f89b41118b0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f89b0012800 tx=0x7f89b0009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:49:28.379 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:27.505+0000 7f89aaffd700 1 -- 172.21.15.133:0/2629952497 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f89b00092e0 con 0x7f89b4103e60 2024-02-20T07:49:28.379 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:27.505+0000 7f89aaffd700 1 -- 172.21.15.133:0/2629952497 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f89b0007d30 con 0x7f89b4103e60 2024-02-20T07:49:28.379 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:27.505+0000 7f89aaffd700 1 -- 172.21.15.133:0/2629952497 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f89b00259a0 con 0x7f89b4103e60 2024-02-20T07:49:28.380 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:27.505+0000 7f89bc7bb700 1 -- 172.21.15.133:0/2629952497 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f89b4111ff0 con 0x7f89b4103e60 2024-02-20T07:49:28.380 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:27.505+0000 7f89bc7bb700 1 -- 172.21.15.133:0/2629952497 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f89b4112c90 con 0x7f89b4103e60 2024-02-20T07:49:28.380 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:27.506+0000 7f89aaffd700 1 -- 172.21.15.133:0/2629952497 <== mon.0 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 13) v1 ==== 43425+0+0 (secure 0 0 0) 0x7f89b0019050 con 0x7f89b4103e60 2024-02-20T07:49:28.380 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:27.506+0000 7f89bc7bb700 1 -- 172.21.15.133:0/2629952497 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f89b405f5c0 con 0x7f89b4103e60 2024-02-20T07:49:28.380 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:27.506+0000 7f89aaffd700 1 --2- 172.21.15.133:0/2629952497 >> [v2:172.21.15.133:6800/3944459195,v1:172.21.15.133:6801/3944459195] conn(0x7f89a00368d0 0x7f89a0038d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:49:28.380 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:27.506+0000 7f89aaffd700 1 -- 172.21.15.133:0/2629952497 <== mon.0 v2:172.21.15.133:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f89b004a3e0 con 0x7f89b4103e60 2024-02-20T07:49:28.380 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:27.506+0000 7f89b9d56700 1 --2- 172.21.15.133:0/2629952497 >> [v2:172.21.15.133:6800/3944459195,v1:172.21.15.133:6801/3944459195] conn(0x7f89a00368d0 0x7f89a0038d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:49:28.380 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:27.507+0000 7f89b9d56700 1 --2- 172.21.15.133:0/2629952497 >> [v2:172.21.15.133:6800/3944459195,v1:172.21.15.133:6801/3944459195] conn(0x7f89a00368d0 0x7f89a0038d80 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f89a4009990 tx=0x7f89a4006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:49:28.380 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:27.511+0000 7f89aaffd700 1 -- 172.21.15.133:0/2629952497 <== mon.0 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f89b0011570 con 0x7f89b4103e60 2024-02-20T07:49:28.380 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:27.794+0000 7f89bc7bb700 1 -- 172.21.15.133:0/2629952497 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command([{prefix=config-key set, key=mgr/dashboard/cluster/status}] v 0) v1 -- 0x7f89b404bed0 con 0x7f89b4103e60 2024-02-20T07:49:28.380 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:27.799+0000 7f89aaffd700 1 -- 172.21.15.133:0/2629952497 <== mon.0 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{prefix=config-key set, key=mgr/dashboard/cluster/status}]=0 set mgr/dashboard/cluster/status v33)=0 set mgr/dashboard/cluster/status v33) v1 ==== 153+0+0 (secure 0 0 0) 0x7f89b0020020 con 0x7f89b4103e60 2024-02-20T07:49:28.380 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:27.801+0000 7f89bc7bb700 1 -- 172.21.15.133:0/2629952497 >> [v2:172.21.15.133:6800/3944459195,v1:172.21.15.133:6801/3944459195] conn(0x7f89a00368d0 msgr2=0x7f89a0038d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:49:28.380 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:27.801+0000 7f89bc7bb700 1 --2- 172.21.15.133:0/2629952497 >> [v2:172.21.15.133:6800/3944459195,v1:172.21.15.133:6801/3944459195] conn(0x7f89a00368d0 0x7f89a0038d80 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f89a4009990 tx=0x7f89a4006e40 comp rx=0 tx=0).stop 2024-02-20T07:49:28.380 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:27.801+0000 7f89bc7bb700 1 -- 172.21.15.133:0/2629952497 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f89b4103e60 msgr2=0x7f89b41118b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:49:28.380 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:27.801+0000 7f89bc7bb700 1 --2- 172.21.15.133:0/2629952497 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f89b4103e60 0x7f89b41118b0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f89b0012800 tx=0x7f89b0009840 comp rx=0 tx=0).stop 2024-02-20T07:49:28.381 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:27.801+0000 7f89bc7bb700 1 -- 172.21.15.133:0/2629952497 shutdown_connections 2024-02-20T07:49:28.381 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:27.801+0000 7f89bc7bb700 1 --2- 172.21.15.133:0/2629952497 >> [v2:172.21.15.133:6800/3944459195,v1:172.21.15.133:6801/3944459195] conn(0x7f89a00368d0 0x7f89a0038d80 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:49:28.381 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:27.801+0000 7f89bc7bb700 1 --2- 172.21.15.133:0/2629952497 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f89b4103e60 0x7f89b41118b0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:49:28.381 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:27.801+0000 7f89bc7bb700 1 -- 172.21.15.133:0/2629952497 >> 172.21.15.133:0/2629952497 conn(0x7f89b40fc580 msgr2=0x7f89b40fd530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:49:28.381 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:27.801+0000 7f89bc7bb700 1 -- 172.21.15.133:0/2629952497 shutdown_connections 2024-02-20T07:49:28.381 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr 2024-02-20T07:49:27.802+0000 7f89bc7bb700 1 -- 172.21.15.133:0/2629952497 wait complete. 2024-02-20T07:49:28.381 INFO:teuthology.orchestra.run.smithi133.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2024-02-20T07:49:28.381 INFO:teuthology.orchestra.run.smithi133.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2024-02-20T07:49:28.381 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-02-20T07:49:28.381 INFO:teuthology.orchestra.run.smithi133.stdout: sudo /home/ubuntu/cephtest/cephadm shell --fsid 4537327c-cfc4-11ee-95bb-87774f69a715 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-02-20T07:49:28.381 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-02-20T07:49:28.381 INFO:teuthology.orchestra.run.smithi133.stdout:Or, if you are only running a single cluster on this host: 2024-02-20T07:49:28.381 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-02-20T07:49:28.381 INFO:teuthology.orchestra.run.smithi133.stdout: sudo /home/ubuntu/cephtest/cephadm shell 2024-02-20T07:49:28.381 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-02-20T07:49:28.381 INFO:teuthology.orchestra.run.smithi133.stdout:Please consider enabling telemetry to help improve Ceph: 2024-02-20T07:49:28.382 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-02-20T07:49:28.382 INFO:teuthology.orchestra.run.smithi133.stdout: ceph telemetry on 2024-02-20T07:49:28.382 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-02-20T07:49:28.382 INFO:teuthology.orchestra.run.smithi133.stdout:For more information see: 2024-02-20T07:49:28.382 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-02-20T07:49:28.382 INFO:teuthology.orchestra.run.smithi133.stdout: https://docs.ceph.com/docs/master/mgr/telemetry/ 2024-02-20T07:49:28.382 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-02-20T07:49:28.382 INFO:teuthology.orchestra.run.smithi133.stdout:Bootstrap complete. 2024-02-20T07:49:28.413 INFO:tasks.cephadm:Fetching config... 2024-02-20T07:49:28.414 DEBUG:teuthology.orchestra.run.smithi133:> set -ex 2024-02-20T07:49:28.414 DEBUG:teuthology.orchestra.run.smithi133:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-02-20T07:49:28.468 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-02-20T07:49:28.469 DEBUG:teuthology.orchestra.run.smithi133:> set -ex 2024-02-20T07:49:28.469 DEBUG:teuthology.orchestra.run.smithi133:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-02-20T07:49:28.524 INFO:tasks.cephadm:Fetching mon keyring... 2024-02-20T07:49:28.524 DEBUG:teuthology.orchestra.run.smithi133:> set -ex 2024-02-20T07:49:28.524 DEBUG:teuthology.orchestra.run.smithi133:> sudo dd if=/var/lib/ceph/4537327c-cfc4-11ee-95bb-87774f69a715/mon.smithi133/keyring of=/dev/stdout 2024-02-20T07:49:28.592 INFO:tasks.cephadm:Fetching pub ssh key... 2024-02-20T07:49:28.592 DEBUG:teuthology.orchestra.run.smithi133:> set -ex 2024-02-20T07:49:28.593 DEBUG:teuthology.orchestra.run.smithi133:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-02-20T07:49:28.648 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-02-20T07:49:28.648 DEBUG:teuthology.orchestra.run.smithi133:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDGmLCWNF8ryxVrFc1CLvy0ci3t3+CkHbdSl8viWPRc0u2fVKDkEI+fNpGs/CWbzWvods5a6sfpm+MXyeD8CkoBM0fuTuHTNQzWkGwy5ecGhSh9i30t5SRD3j/eumX1NQnr8SjLzoMx+exUJHnSB+q5ySlqJ4C+b+yBKYxGez/Mjc8R9Ep9C/RBVclIH7Mt2DBoprCFzz0hA2WdGN0oBRpad2bHHLB0yvFmUfP5h/ese1ftQ7QIjQ6T6dd61TJuBZUAQlwKMQNI3zejD/ibG/mybeh9QOt1Z554nyjKKhLHsKh/eyTEJjhehXDu3oiAl1vwI1b9s95+56/FDre98WQDcQjfTd23eLO5QcaSk1hfJjdyzRUHt1PlitxnzXDzbuptyFabAv2DN4CQoI/j4qUFRvkuDm7JX/p7933w5y5lhw/Y6fhVlcvSYMRZbNjC7Y96Zt7hLHQRzHGdyIZNHpKzock607TCJwMWBDE5xky6b9MrgwZsaSaZnmgA7CJctwk= ceph-4537327c-cfc4-11ee-95bb-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-02-20T07:49:28.748 INFO:teuthology.orchestra.run.smithi133.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDGmLCWNF8ryxVrFc1CLvy0ci3t3+CkHbdSl8viWPRc0u2fVKDkEI+fNpGs/CWbzWvods5a6sfpm+MXyeD8CkoBM0fuTuHTNQzWkGwy5ecGhSh9i30t5SRD3j/eumX1NQnr8SjLzoMx+exUJHnSB+q5ySlqJ4C+b+yBKYxGez/Mjc8R9Ep9C/RBVclIH7Mt2DBoprCFzz0hA2WdGN0oBRpad2bHHLB0yvFmUfP5h/ese1ftQ7QIjQ6T6dd61TJuBZUAQlwKMQNI3zejD/ibG/mybeh9QOt1Z554nyjKKhLHsKh/eyTEJjhehXDu3oiAl1vwI1b9s95+56/FDre98WQDcQjfTd23eLO5QcaSk1hfJjdyzRUHt1PlitxnzXDzbuptyFabAv2DN4CQoI/j4qUFRvkuDm7JX/p7933w5y5lhw/Y6fhVlcvSYMRZbNjC7Y96Zt7hLHQRzHGdyIZNHpKzock607TCJwMWBDE5xky6b9MrgwZsaSaZnmgA7CJctwk= ceph-4537327c-cfc4-11ee-95bb-87774f69a715 2024-02-20T07:49:28.761 DEBUG:teuthology.orchestra.run.smithi183:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDGmLCWNF8ryxVrFc1CLvy0ci3t3+CkHbdSl8viWPRc0u2fVKDkEI+fNpGs/CWbzWvods5a6sfpm+MXyeD8CkoBM0fuTuHTNQzWkGwy5ecGhSh9i30t5SRD3j/eumX1NQnr8SjLzoMx+exUJHnSB+q5ySlqJ4C+b+yBKYxGez/Mjc8R9Ep9C/RBVclIH7Mt2DBoprCFzz0hA2WdGN0oBRpad2bHHLB0yvFmUfP5h/ese1ftQ7QIjQ6T6dd61TJuBZUAQlwKMQNI3zejD/ibG/mybeh9QOt1Z554nyjKKhLHsKh/eyTEJjhehXDu3oiAl1vwI1b9s95+56/FDre98WQDcQjfTd23eLO5QcaSk1hfJjdyzRUHt1PlitxnzXDzbuptyFabAv2DN4CQoI/j4qUFRvkuDm7JX/p7933w5y5lhw/Y6fhVlcvSYMRZbNjC7Y96Zt7hLHQRzHGdyIZNHpKzock607TCJwMWBDE5xky6b9MrgwZsaSaZnmgA7CJctwk= ceph-4537327c-cfc4-11ee-95bb-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-02-20T07:49:28.807 INFO:teuthology.orchestra.run.smithi183.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDGmLCWNF8ryxVrFc1CLvy0ci3t3+CkHbdSl8viWPRc0u2fVKDkEI+fNpGs/CWbzWvods5a6sfpm+MXyeD8CkoBM0fuTuHTNQzWkGwy5ecGhSh9i30t5SRD3j/eumX1NQnr8SjLzoMx+exUJHnSB+q5ySlqJ4C+b+yBKYxGez/Mjc8R9Ep9C/RBVclIH7Mt2DBoprCFzz0hA2WdGN0oBRpad2bHHLB0yvFmUfP5h/ese1ftQ7QIjQ6T6dd61TJuBZUAQlwKMQNI3zejD/ibG/mybeh9QOt1Z554nyjKKhLHsKh/eyTEJjhehXDu3oiAl1vwI1b9s95+56/FDre98WQDcQjfTd23eLO5QcaSk1hfJjdyzRUHt1PlitxnzXDzbuptyFabAv2DN4CQoI/j4qUFRvkuDm7JX/p7933w5y5lhw/Y6fhVlcvSYMRZbNjC7Y96Zt7hLHQRzHGdyIZNHpKzock607TCJwMWBDE5xky6b9MrgwZsaSaZnmgA7CJctwk= ceph-4537327c-cfc4-11ee-95bb-87774f69a715 2024-02-20T07:49:28.822 DEBUG:teuthology.orchestra.run.smithi133:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4537327c-cfc4-11ee-95bb-87774f69a715 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-02-20T07:49:28.992 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:49:28 smithi133 ceph-mon[29117]: from='client.? 172.21.15.133:0/2629952497' entity='client.admin' 2024-02-20T07:49:29.075 INFO:teuthology.orchestra.run.smithi133.stderr:Inferring config /var/lib/ceph/4537327c-cfc4-11ee-95bb-87774f69a715/mon.smithi133/config 2024-02-20T07:49:31.335 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:49:31.331+0000 7fa22c3b2700 1 -- 172.21.15.133:0/1949357135 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa2240fe240 msgr2=0x7fa2240fe650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:49:31.336 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:49:31.331+0000 7fa22c3b2700 1 --2- 172.21.15.133:0/1949357135 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa2240fe240 0x7fa2240fe650 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fa2200048c0 tx=0x7fa220009e30 comp rx=0 tx=0).stop 2024-02-20T07:49:31.336 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:49:31.332+0000 7fa22c3b2700 1 -- 172.21.15.133:0/1949357135 shutdown_connections 2024-02-20T07:49:31.336 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:49:31.332+0000 7fa22c3b2700 1 --2- 172.21.15.133:0/1949357135 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa2240fe240 0x7fa2240fe650 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:49:31.336 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:49:31.332+0000 7fa22c3b2700 1 -- 172.21.15.133:0/1949357135 >> 172.21.15.133:0/1949357135 conn(0x7fa2240f6920 msgr2=0x7fa2240f8d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:49:31.336 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:49:31.332+0000 7fa22c3b2700 1 -- 172.21.15.133:0/1949357135 shutdown_connections 2024-02-20T07:49:31.336 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:49:31.332+0000 7fa22c3b2700 1 -- 172.21.15.133:0/1949357135 wait complete. 2024-02-20T07:49:31.336 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:49:31.332+0000 7fa22c3b2700 1 Processor -- start 2024-02-20T07:49:31.336 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:49:31.332+0000 7fa22c3b2700 1 -- start start 2024-02-20T07:49:31.336 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:49:31.332+0000 7fa22c3b2700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa2240fe240 0x7fa22410cf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:49:31.336 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:49:31.332+0000 7fa22c3b2700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa22410d480 con 0x7fa2240fe240 2024-02-20T07:49:31.336 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:49:31.332+0000 7fa22a14e700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa2240fe240 0x7fa22410cf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:49:31.336 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:49:31.332+0000 7fa22a14e700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa2240fe240 0x7fa22410cf40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.133:33724/0 (socket says 172.21.15.133:33724) 2024-02-20T07:49:31.336 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:49:31.332+0000 7fa22a14e700 1 -- 172.21.15.133:0/1665641218 learned_addr learned my addr 172.21.15.133:0/1665641218 (peer_addr_for_me v2:172.21.15.133:0/0) 2024-02-20T07:49:31.336 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:49:31.333+0000 7fa22a14e700 1 -- 172.21.15.133:0/1665641218 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa220005040 con 0x7fa2240fe240 2024-02-20T07:49:31.337 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:49:31.333+0000 7fa22a14e700 1 --2- 172.21.15.133:0/1665641218 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa2240fe240 0x7fa22410cf40 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fa220009300 tx=0x7fa220009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:49:31.337 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:49:31.333+0000 7fa21affd700 1 -- 172.21.15.133:0/1665641218 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa22000be50 con 0x7fa2240fe240 2024-02-20T07:49:31.337 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:49:31.333+0000 7fa21affd700 1 -- 172.21.15.133:0/1665641218 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa220009680 con 0x7fa2240fe240 2024-02-20T07:49:31.337 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:49:31.333+0000 7fa22c3b2700 1 -- 172.21.15.133:0/1665641218 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa22410d680 con 0x7fa2240fe240 2024-02-20T07:49:31.337 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:49:31.333+0000 7fa21affd700 1 -- 172.21.15.133:0/1665641218 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa22001c5b0 con 0x7fa2240fe240 2024-02-20T07:49:31.337 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:49:31.333+0000 7fa22c3b2700 1 -- 172.21.15.133:0/1665641218 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa22410e320 con 0x7fa2240fe240 2024-02-20T07:49:31.338 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:49:31.334+0000 7fa21affd700 1 -- 172.21.15.133:0/1665641218 <== mon.0 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 13) v1 ==== 43425+0+0 (secure 0 0 0) 0x7fa220013420 con 0x7fa2240fe240 2024-02-20T07:49:31.338 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:49:31.334+0000 7fa22c3b2700 1 -- 172.21.15.133:0/1665641218 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa22405f5c0 con 0x7fa2240fe240 2024-02-20T07:49:31.338 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:49:31.334+0000 7fa21affd700 1 --2- 172.21.15.133:0/1665641218 >> [v2:172.21.15.133:6800/3944459195,v1:172.21.15.133:6801/3944459195] conn(0x7fa210036870 0x7fa210038d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:49:31.338 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:49:31.334+0000 7fa21affd700 1 -- 172.21.15.133:0/1665641218 <== mon.0 v2:172.21.15.133:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7fa220049990 con 0x7fa2240fe240 2024-02-20T07:49:31.338 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:49:31.334+0000 7fa22994d700 1 --2- 172.21.15.133:0/1665641218 >> [v2:172.21.15.133:6800/3944459195,v1:172.21.15.133:6801/3944459195] conn(0x7fa210036870 0x7fa210038d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:49:31.338 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:49:31.334+0000 7fa22994d700 1 --2- 172.21.15.133:0/1665641218 >> [v2:172.21.15.133:6800/3944459195,v1:172.21.15.133:6801/3944459195] conn(0x7fa210036870 0x7fa210038d20 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fa214009990 tx=0x7fa214006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:49:31.341 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:49:31.338+0000 7fa21affd700 1 -- 172.21.15.133:0/1665641218 <== mon.0 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7fa220013980 con 0x7fa2240fe240 2024-02-20T07:49:31.528 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:49:31.526+0000 7fa22c3b2700 1 -- 172.21.15.133:0/1665641218 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/allow_ptrace}] v 0) v1 -- 0x7fa22404bed0 con 0x7fa2240fe240 2024-02-20T07:49:31.533 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:49:31.530+0000 7fa21affd700 1 -- 172.21.15.133:0/1665641218 <== mon.0 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/allow_ptrace}]=0 v9)=0 v9) v1 ==== 125+0+0 (secure 0 0 0) 0x7fa220011370 con 0x7fa2240fe240 2024-02-20T07:49:31.535 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:49:31.532+0000 7fa22c3b2700 1 -- 172.21.15.133:0/1665641218 >> [v2:172.21.15.133:6800/3944459195,v1:172.21.15.133:6801/3944459195] conn(0x7fa210036870 msgr2=0x7fa210038d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:49:31.535 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:49:31.533+0000 7fa22c3b2700 1 --2- 172.21.15.133:0/1665641218 >> [v2:172.21.15.133:6800/3944459195,v1:172.21.15.133:6801/3944459195] conn(0x7fa210036870 0x7fa210038d20 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fa214009990 tx=0x7fa214006e40 comp rx=0 tx=0).stop 2024-02-20T07:49:31.535 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:49:31.533+0000 7fa22c3b2700 1 -- 172.21.15.133:0/1665641218 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa2240fe240 msgr2=0x7fa22410cf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:49:31.535 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:49:31.533+0000 7fa22c3b2700 1 --2- 172.21.15.133:0/1665641218 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa2240fe240 0x7fa22410cf40 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fa220009300 tx=0x7fa220009e30 comp rx=0 tx=0).stop 2024-02-20T07:49:31.535 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:49:31.533+0000 7fa22c3b2700 1 -- 172.21.15.133:0/1665641218 shutdown_connections 2024-02-20T07:49:31.535 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:49:31.533+0000 7fa22c3b2700 1 --2- 172.21.15.133:0/1665641218 >> [v2:172.21.15.133:6800/3944459195,v1:172.21.15.133:6801/3944459195] conn(0x7fa210036870 0x7fa210038d20 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:49:31.535 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:49:31.533+0000 7fa22c3b2700 1 --2- 172.21.15.133:0/1665641218 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa2240fe240 0x7fa22410cf40 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:49:31.536 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:49:31.533+0000 7fa22c3b2700 1 -- 172.21.15.133:0/1665641218 >> 172.21.15.133:0/1665641218 conn(0x7fa2240f6920 msgr2=0x7fa2240f76b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:49:31.536 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:49:31.533+0000 7fa22c3b2700 1 -- 172.21.15.133:0/1665641218 shutdown_connections 2024-02-20T07:49:31.536 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:49:31.533+0000 7fa22c3b2700 1 -- 172.21.15.133:0/1665641218 wait complete. 2024-02-20T07:49:31.992 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:49:31 smithi133 ceph-mon[29117]: from='mgr.14162 172.21.15.133:0/3642562278' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:49:31.992 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:49:31 smithi133 ceph-mon[29117]: from='mgr.14162 172.21.15.133:0/3642562278' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:49:31.992 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:49:31 smithi133 ceph-mon[29117]: from='mgr.14162 172.21.15.133:0/3642562278' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "config rm", "who": "osd/host:smithi133", "name": "osd_memory_target"}]: dispatch 2024-02-20T07:49:31.993 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:49:31 smithi133 ceph-mon[29117]: from='mgr.14162 172.21.15.133:0/3642562278' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:49:31.993 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:49:31 smithi133 ceph-mon[29117]: from='mgr.14162 172.21.15.133:0/3642562278' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi133", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-02-20T07:49:31.993 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:49:31 smithi133 ceph-mon[29117]: from='mgr.14162 172.21.15.133:0/3642562278' entity='mgr.smithi133.mrvcbp' cmd='[{"prefix": "auth get-or-create", "entity": "client.crash.smithi133", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]': finished 2024-02-20T07:49:31.993 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:49:31 smithi133 ceph-mon[29117]: from='mgr.14162 172.21.15.133:0/3642562278' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-20T07:49:31.993 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:49:31 smithi133 ceph-mon[29117]: Deploying daemon crash.smithi133 on smithi133 2024-02-20T07:49:31.993 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:49:31 smithi133 ceph-mon[29117]: from='client.? 172.21.15.133:0/1665641218' entity='client.admin' 2024-02-20T07:49:32.221 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-02-20T07:49:32.222 DEBUG:teuthology.orchestra.run.smithi133:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4537327c-cfc4-11ee-95bb-87774f69a715 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-02-20T07:49:32.888 INFO:teuthology.orchestra.run.smithi133.stderr:Inferring config /var/lib/ceph/4537327c-cfc4-11ee-95bb-87774f69a715/mon.smithi133/config 2024-02-20T07:49:35.436 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:49:35.432+0000 7fef07fb9700 1 -- 172.21.15.133:0/391209740 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fef000fcbd0 msgr2=0x7fef000fcfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:49:35.437 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:49:35.432+0000 7fef07fb9700 1 --2- 172.21.15.133:0/391209740 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fef000fcbd0 0x7fef000fcfe0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7feef00048c0 tx=0x7feef0009e30 comp rx=0 tx=0).stop 2024-02-20T07:49:35.437 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:49:35.432+0000 7fef07fb9700 1 -- 172.21.15.133:0/391209740 shutdown_connections 2024-02-20T07:49:35.437 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:49:35.432+0000 7fef07fb9700 1 --2- 172.21.15.133:0/391209740 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fef000fcbd0 0x7fef000fcfe0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:49:35.437 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:49:35.432+0000 7fef07fb9700 1 -- 172.21.15.133:0/391209740 >> 172.21.15.133:0/391209740 conn(0x7fef000f8270 msgr2=0x7fef000fa6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:49:35.437 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:49:35.432+0000 7fef07fb9700 1 -- 172.21.15.133:0/391209740 shutdown_connections 2024-02-20T07:49:35.437 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:49:35.432+0000 7fef07fb9700 1 -- 172.21.15.133:0/391209740 wait complete. 2024-02-20T07:49:35.437 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:49:35.433+0000 7fef07fb9700 1 Processor -- start 2024-02-20T07:49:35.437 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:49:35.433+0000 7fef07fb9700 1 -- start start 2024-02-20T07:49:35.437 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:49:35.433+0000 7fef07fb9700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fef000fcbd0 0x7fef0010cf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:49:35.437 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:49:35.433+0000 7fef07fb9700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef0010d490 con 0x7fef000fcbd0 2024-02-20T07:49:35.437 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:49:35.433+0000 7fef05d55700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fef000fcbd0 0x7fef0010cf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:49:35.437 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:49:35.433+0000 7fef05d55700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fef000fcbd0 0x7fef0010cf50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.133:33744/0 (socket says 172.21.15.133:33744) 2024-02-20T07:49:35.437 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:49:35.433+0000 7fef05d55700 1 -- 172.21.15.133:0/177128654 learned_addr learned my addr 172.21.15.133:0/177128654 (peer_addr_for_me v2:172.21.15.133:0/0) 2024-02-20T07:49:35.437 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:49:35.433+0000 7fef05d55700 1 -- 172.21.15.133:0/177128654 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feef0005040 con 0x7fef000fcbd0 2024-02-20T07:49:35.438 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:49:35.434+0000 7fef05d55700 1 --2- 172.21.15.133:0/177128654 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fef000fcbd0 0x7fef0010cf50 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7feef0009b80 tx=0x7feef0009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:49:35.438 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:49:35.434+0000 7feef67fc700 1 -- 172.21.15.133:0/177128654 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7feef000b5c0 con 0x7fef000fcbd0 2024-02-20T07:49:35.438 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:49:35.434+0000 7fef07fb9700 1 -- 172.21.15.133:0/177128654 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fef0010d690 con 0x7fef000fcbd0 2024-02-20T07:49:35.438 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:49:35.434+0000 7feef67fc700 1 -- 172.21.15.133:0/177128654 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7feef000be70 con 0x7fef000fcbd0 2024-02-20T07:49:35.438 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:49:35.434+0000 7feef67fc700 1 -- 172.21.15.133:0/177128654 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7feef001c660 con 0x7fef000fcbd0 2024-02-20T07:49:35.438 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:49:35.434+0000 7fef07fb9700 1 -- 172.21.15.133:0/177128654 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fef0010e330 con 0x7fef000fcbd0 2024-02-20T07:49:35.439 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:49:35.434+0000 7feef67fc700 1 -- 172.21.15.133:0/177128654 <== mon.0 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 13) v1 ==== 43425+0+0 (secure 0 0 0) 0x7feef001c8d0 con 0x7fef000fcbd0 2024-02-20T07:49:35.439 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:49:35.435+0000 7fef07fb9700 1 -- 172.21.15.133:0/177128654 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fef0005f5c0 con 0x7fef000fcbd0 2024-02-20T07:49:35.439 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:49:35.435+0000 7feef67fc700 1 --2- 172.21.15.133:0/177128654 >> [v2:172.21.15.133:6800/3944459195,v1:172.21.15.133:6801/3944459195] conn(0x7feeec036870 0x7feeec038d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:49:35.439 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:49:35.435+0000 7feef67fc700 1 -- 172.21.15.133:0/177128654 <== mon.0 v2:172.21.15.133:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7feef0049ac0 con 0x7fef000fcbd0 2024-02-20T07:49:35.439 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:49:35.435+0000 7fef05554700 1 --2- 172.21.15.133:0/177128654 >> [v2:172.21.15.133:6800/3944459195,v1:172.21.15.133:6801/3944459195] conn(0x7feeec036870 0x7feeec038d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:49:35.439 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:49:35.435+0000 7fef05554700 1 --2- 172.21.15.133:0/177128654 >> [v2:172.21.15.133:6800/3944459195,v1:172.21.15.133:6801/3944459195] conn(0x7feeec036870 0x7feeec038d20 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7feefc009990 tx=0x7feefc006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:49:35.441 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:49:35.439+0000 7feef67fc700 1 -- 172.21.15.133:0/177128654 <== mon.0 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7feef0011ba0 con 0x7fef000fcbd0 2024-02-20T07:49:35.660 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:49:35.657+0000 7fef07fb9700 1 -- 172.21.15.133:0/177128654 --> [v2:172.21.15.133:6800/3944459195,v1:172.21.15.133:6801/3944459195] -- mgr_command(tid 0: {"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}) v1 -- 0x7fef0010ef10 con 0x7feeec036870 2024-02-20T07:49:35.663 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:49:35.661+0000 7feef67fc700 1 -- 172.21.15.133:0/177128654 <== mgr.14162 v2:172.21.15.133:6800/3944459195 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fef0010ef10 con 0x7feeec036870 2024-02-20T07:49:35.665 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:49:35.663+0000 7fef07fb9700 1 -- 172.21.15.133:0/177128654 >> [v2:172.21.15.133:6800/3944459195,v1:172.21.15.133:6801/3944459195] conn(0x7feeec036870 msgr2=0x7feeec038d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:49:35.665 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:49:35.663+0000 7fef07fb9700 1 --2- 172.21.15.133:0/177128654 >> [v2:172.21.15.133:6800/3944459195,v1:172.21.15.133:6801/3944459195] conn(0x7feeec036870 0x7feeec038d20 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7feefc009990 tx=0x7feefc006e40 comp rx=0 tx=0).stop 2024-02-20T07:49:35.665 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:49:35.663+0000 7fef07fb9700 1 -- 172.21.15.133:0/177128654 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fef000fcbd0 msgr2=0x7fef0010cf50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:49:35.665 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:49:35.663+0000 7fef07fb9700 1 --2- 172.21.15.133:0/177128654 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fef000fcbd0 0x7fef0010cf50 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7feef0009b80 tx=0x7feef0009e30 comp rx=0 tx=0).stop 2024-02-20T07:49:35.665 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:49:35.663+0000 7fef07fb9700 1 -- 172.21.15.133:0/177128654 shutdown_connections 2024-02-20T07:49:35.665 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:49:35.663+0000 7fef07fb9700 1 --2- 172.21.15.133:0/177128654 >> [v2:172.21.15.133:6800/3944459195,v1:172.21.15.133:6801/3944459195] conn(0x7feeec036870 0x7feeec038d20 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:49:35.665 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:49:35.663+0000 7fef07fb9700 1 --2- 172.21.15.133:0/177128654 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fef000fcbd0 0x7fef0010cf50 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:49:35.666 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:49:35.663+0000 7fef07fb9700 1 -- 172.21.15.133:0/177128654 >> 172.21.15.133:0/177128654 conn(0x7fef000f8270 msgr2=0x7fef000f9020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:49:35.666 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:49:35.663+0000 7fef07fb9700 1 -- 172.21.15.133:0/177128654 shutdown_connections 2024-02-20T07:49:35.666 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:49:35.663+0000 7fef07fb9700 1 -- 172.21.15.133:0/177128654 wait complete. 2024-02-20T07:49:36.423 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi183 2024-02-20T07:49:36.423 DEBUG:teuthology.orchestra.run.smithi183:> set -ex 2024-02-20T07:49:36.423 DEBUG:teuthology.orchestra.run.smithi183:> dd of=/etc/ceph/ceph.conf 2024-02-20T07:49:36.442 DEBUG:teuthology.orchestra.run.smithi183:> set -ex 2024-02-20T07:49:36.442 DEBUG:teuthology.orchestra.run.smithi183:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-02-20T07:49:36.500 INFO:tasks.cephadm:Adding host smithi183 to orchestrator... 2024-02-20T07:49:36.500 DEBUG:teuthology.orchestra.run.smithi133:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4537327c-cfc4-11ee-95bb-87774f69a715 -- ceph orch host add smithi183 2024-02-20T07:49:36.774 INFO:teuthology.orchestra.run.smithi133.stderr:Inferring config /var/lib/ceph/4537327c-cfc4-11ee-95bb-87774f69a715/mon.smithi133/config 2024-02-20T07:49:36.898 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:49:36 smithi133 ceph-mon[29117]: from='client.14186 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-02-20T07:49:36.898 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:49:36 smithi133 ceph-mon[29117]: from='mgr.14162 172.21.15.133:0/3642562278' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:49:37.944 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:49:37 smithi133 ceph-mon[29117]: from='mgr.14162 172.21.15.133:0/3642562278' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:49:37.944 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:49:37 smithi133 ceph-mon[29117]: from='mgr.14162 172.21.15.133:0/3642562278' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:49:37.944 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:49:37 smithi133 ceph-mon[29117]: from='mgr.14162 172.21.15.133:0/3642562278' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:49:37.944 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:49:37 smithi133 ceph-mon[29117]: from='mgr.14162 172.21.15.133:0/3642562278' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:49:37.944 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:49:37 smithi133 ceph-mon[29117]: Deploying daemon node-exporter.smithi133 on smithi133 2024-02-20T07:49:38.324 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:49:38.314+0000 7f9f0a8ad700 1 -- 172.21.15.133:0/155655404 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f9f040fbad0 msgr2=0x7f9f040fbee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:49:38.324 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:49:38.314+0000 7f9f0a8ad700 1 --2- 172.21.15.133:0/155655404 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f9f040fbad0 0x7f9f040fbee0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f9ef40048c0 tx=0x7f9ef4009e30 comp rx=0 tx=0).stop 2024-02-20T07:49:38.325 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:49:38.314+0000 7f9f0a8ad700 1 -- 172.21.15.133:0/155655404 shutdown_connections 2024-02-20T07:49:38.325 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:49:38.314+0000 7f9f0a8ad700 1 --2- 172.21.15.133:0/155655404 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f9f040fbad0 0x7f9f040fbee0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:49:38.325 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:49:38.314+0000 7f9f0a8ad700 1 -- 172.21.15.133:0/155655404 >> 172.21.15.133:0/155655404 conn(0x7f9f040f7190 msgr2=0x7f9f040f95c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:49:38.325 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:49:38.314+0000 7f9f0a8ad700 1 -- 172.21.15.133:0/155655404 shutdown_connections 2024-02-20T07:49:38.325 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:49:38.314+0000 7f9f0a8ad700 1 -- 172.21.15.133:0/155655404 wait complete. 2024-02-20T07:49:38.325 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:49:38.315+0000 7f9f0a8ad700 1 Processor -- start 2024-02-20T07:49:38.326 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:49:38.315+0000 7f9f0a8ad700 1 -- start start 2024-02-20T07:49:38.326 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:49:38.315+0000 7f9f0a8ad700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f9f040fbad0 0x7f9f0410fbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:49:38.326 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:49:38.315+0000 7f9f0a8ad700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f04065d60 con 0x7f9f040fbad0 2024-02-20T07:49:38.326 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:49:38.315+0000 7f9f03fff700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f9f040fbad0 0x7f9f0410fbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:49:38.326 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:49:38.315+0000 7f9f03fff700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f9f040fbad0 0x7f9f0410fbd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.133:43616/0 (socket says 172.21.15.133:43616) 2024-02-20T07:49:38.326 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:49:38.315+0000 7f9f03fff700 1 -- 172.21.15.133:0/2485038299 learned_addr learned my addr 172.21.15.133:0/2485038299 (peer_addr_for_me v2:172.21.15.133:0/0) 2024-02-20T07:49:38.326 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:49:38.315+0000 7f9f03fff700 1 -- 172.21.15.133:0/2485038299 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ef4005040 con 0x7f9f040fbad0 2024-02-20T07:49:38.326 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:49:38.316+0000 7f9f03fff700 1 --2- 172.21.15.133:0/2485038299 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f9f040fbad0 0x7f9f0410fbd0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f9ef4009b80 tx=0x7f9ef4009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:49:38.326 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:49:38.316+0000 7f9f00ff9700 1 -- 172.21.15.133:0/2485038299 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9ef400b5c0 con 0x7f9f040fbad0 2024-02-20T07:49:38.326 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:49:38.316+0000 7f9f0a8ad700 1 -- 172.21.15.133:0/2485038299 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9f04065f60 con 0x7f9f040fbad0 2024-02-20T07:49:38.326 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:49:38.316+0000 7f9f00ff9700 1 -- 172.21.15.133:0/2485038299 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9ef400be70 con 0x7f9f040fbad0 2024-02-20T07:49:38.326 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:49:38.316+0000 7f9f00ff9700 1 -- 172.21.15.133:0/2485038299 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9ef401c660 con 0x7f9f040fbad0 2024-02-20T07:49:38.326 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:49:38.316+0000 7f9f0a8ad700 1 -- 172.21.15.133:0/2485038299 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9f04110350 con 0x7f9f040fbad0 2024-02-20T07:49:38.326 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:49:38.317+0000 7f9f00ff9700 1 -- 172.21.15.133:0/2485038299 <== mon.0 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 13) v1 ==== 43425+0+0 (secure 0 0 0) 0x7f9ef401c8d0 con 0x7f9f040fbad0 2024-02-20T07:49:38.327 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:49:38.317+0000 7f9f0a8ad700 1 -- 172.21.15.133:0/2485038299 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9f041058a0 con 0x7f9f040fbad0 2024-02-20T07:49:38.327 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:49:38.317+0000 7f9f00ff9700 1 --2- 172.21.15.133:0/2485038299 >> [v2:172.21.15.133:6800/3944459195,v1:172.21.15.133:6801/3944459195] conn(0x7f9eec036870 0x7f9eec038d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:49:38.327 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:49:38.317+0000 7f9f00ff9700 1 -- 172.21.15.133:0/2485038299 <== mon.0 v2:172.21.15.133:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f9ef40499e0 con 0x7f9f040fbad0 2024-02-20T07:49:38.327 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:49:38.317+0000 7f9f037fe700 1 --2- 172.21.15.133:0/2485038299 >> [v2:172.21.15.133:6800/3944459195,v1:172.21.15.133:6801/3944459195] conn(0x7f9eec036870 0x7f9eec038d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:49:38.327 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:49:38.317+0000 7f9f037fe700 1 --2- 172.21.15.133:0/2485038299 >> [v2:172.21.15.133:6800/3944459195,v1:172.21.15.133:6801/3944459195] conn(0x7f9eec036870 0x7f9eec038d20 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f9ef8009990 tx=0x7f9ef8006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:49:38.328 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:49:38.326+0000 7f9f00ff9700 1 -- 172.21.15.133:0/2485038299 <== mon.0 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f9ef4011dd0 con 0x7f9f040fbad0 2024-02-20T07:49:38.528 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:49:38.523+0000 7f9f0a8ad700 1 -- 172.21.15.133:0/2485038299 --> [v2:172.21.15.133:6800/3944459195,v1:172.21.15.133:6801/3944459195] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi183", "target": ["mon-mgr", ""]}) v1 -- 0x7f9f0402b540 con 0x7f9eec036870 2024-02-20T07:49:39.742 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:49:39 smithi133 ceph-mon[29117]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi183", "target": ["mon-mgr", ""]}]: dispatch 2024-02-20T07:49:40.664 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:49:40.662+0000 7f9f00ff9700 1 -- 172.21.15.133:0/2485038299 <== mgr.14162 v2:172.21.15.133:6800/3944459195 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (secure 0 0 0) 0x7f9f0402b540 con 0x7f9eec036870 2024-02-20T07:49:40.665 INFO:teuthology.orchestra.run.smithi133.stdout:Added host 'smithi183' with addr '172.21.15.183' 2024-02-20T07:49:40.667 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:49:40.664+0000 7f9f0a8ad700 1 -- 172.21.15.133:0/2485038299 >> [v2:172.21.15.133:6800/3944459195,v1:172.21.15.133:6801/3944459195] conn(0x7f9eec036870 msgr2=0x7f9eec038d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:49:40.667 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:49:40.664+0000 7f9f0a8ad700 1 --2- 172.21.15.133:0/2485038299 >> [v2:172.21.15.133:6800/3944459195,v1:172.21.15.133:6801/3944459195] conn(0x7f9eec036870 0x7f9eec038d20 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f9ef8009990 tx=0x7f9ef8006e40 comp rx=0 tx=0).stop 2024-02-20T07:49:40.667 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:49:40.664+0000 7f9f0a8ad700 1 -- 172.21.15.133:0/2485038299 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f9f040fbad0 msgr2=0x7f9f0410fbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:49:40.667 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:49:40.664+0000 7f9f0a8ad700 1 --2- 172.21.15.133:0/2485038299 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f9f040fbad0 0x7f9f0410fbd0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f9ef4009b80 tx=0x7f9ef4009e30 comp rx=0 tx=0).stop 2024-02-20T07:49:40.667 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:49:40.664+0000 7f9f0a8ad700 1 -- 172.21.15.133:0/2485038299 shutdown_connections 2024-02-20T07:49:40.668 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:49:40.664+0000 7f9f0a8ad700 1 --2- 172.21.15.133:0/2485038299 >> [v2:172.21.15.133:6800/3944459195,v1:172.21.15.133:6801/3944459195] conn(0x7f9eec036870 0x7f9eec038d20 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:49:40.668 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:49:40.664+0000 7f9f0a8ad700 1 --2- 172.21.15.133:0/2485038299 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f9f040fbad0 0x7f9f0410fbd0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:49:40.668 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:49:40.664+0000 7f9f0a8ad700 1 -- 172.21.15.133:0/2485038299 >> 172.21.15.133:0/2485038299 conn(0x7f9f040f7190 msgr2=0x7f9f040f7ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:49:40.668 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:49:40.665+0000 7f9f0a8ad700 1 -- 172.21.15.133:0/2485038299 shutdown_connections 2024-02-20T07:49:40.668 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:49:40.665+0000 7f9f0a8ad700 1 -- 172.21.15.133:0/2485038299 wait complete. 2024-02-20T07:49:40.968 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:49:40 smithi133 ceph-mon[29117]: Deploying cephadm binary to smithi183 2024-02-20T07:49:40.968 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:49:40 smithi133 ceph-mon[29117]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T07:49:40.968 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:49:40 smithi133 ceph-mon[29117]: from='mgr.14162 172.21.15.133:0/3642562278' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:49:40.969 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:49:40 smithi133 ceph-mon[29117]: from='mgr.14162 172.21.15.133:0/3642562278' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:49:41.700 DEBUG:teuthology.orchestra.run.smithi133:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4537327c-cfc4-11ee-95bb-87774f69a715 -- ceph orch host ls --format=json 2024-02-20T07:49:41.968 INFO:teuthology.orchestra.run.smithi133.stderr:Inferring config /var/lib/ceph/4537327c-cfc4-11ee-95bb-87774f69a715/mon.smithi133/config 2024-02-20T07:49:42.408 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:49:42 smithi133 ceph-mon[29117]: Added host smithi183 2024-02-20T07:49:42.408 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:49:42 smithi133 ceph-mon[29117]: from='mgr.14162 172.21.15.133:0/3642562278' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:49:42.408 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:49:42 smithi133 ceph-mon[29117]: from='mgr.14162 172.21.15.133:0/3642562278' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:49:42.408 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:49:42 smithi133 ceph-mon[29117]: from='mgr.14162 172.21.15.133:0/3642562278' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:49:42.408 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:49:42 smithi133 ceph-mon[29117]: from='mgr.14162 172.21.15.133:0/3642562278' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:49:42.408 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:49:42 smithi133 ceph-mon[29117]: Deploying daemon alertmanager.smithi133 on smithi133 2024-02-20T07:49:43.242 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:49:43 smithi133 ceph-mon[29117]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T07:49:44.467 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:49:44.463+0000 7fd0f107a700 1 -- 172.21.15.133:0/1409328423 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fd0ec0fcbd0 msgr2=0x7fd0ec0fcfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:49:44.467 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:49:44.463+0000 7fd0f107a700 1 --2- 172.21.15.133:0/1409328423 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fd0ec0fcbd0 0x7fd0ec0fcfe0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fd0dc0048c0 tx=0x7fd0dc009e30 comp rx=0 tx=0).stop 2024-02-20T07:49:44.467 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:49:44.463+0000 7fd0f107a700 1 -- 172.21.15.133:0/1409328423 shutdown_connections 2024-02-20T07:49:44.467 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:49:44.463+0000 7fd0f107a700 1 --2- 172.21.15.133:0/1409328423 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fd0ec0fcbd0 0x7fd0ec0fcfe0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:49:44.467 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:49:44.463+0000 7fd0f107a700 1 -- 172.21.15.133:0/1409328423 >> 172.21.15.133:0/1409328423 conn(0x7fd0ec0f8290 msgr2=0x7fd0ec0fa6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:49:44.467 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:49:44.463+0000 7fd0f107a700 1 -- 172.21.15.133:0/1409328423 shutdown_connections 2024-02-20T07:49:44.468 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:49:44.463+0000 7fd0f107a700 1 -- 172.21.15.133:0/1409328423 wait complete. 2024-02-20T07:49:44.468 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:49:44.464+0000 7fd0f107a700 1 Processor -- start 2024-02-20T07:49:44.468 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:49:44.464+0000 7fd0f107a700 1 -- start start 2024-02-20T07:49:44.468 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:49:44.464+0000 7fd0f107a700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fd0ec0fcbd0 0x7fd0ec071e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:49:44.468 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:49:44.464+0000 7fd0f107a700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0ec0702e0 con 0x7fd0ec0fcbd0 2024-02-20T07:49:44.468 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:49:44.464+0000 7fd0ead9d700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fd0ec0fcbd0 0x7fd0ec071e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:49:44.468 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:49:44.464+0000 7fd0ead9d700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fd0ec0fcbd0 0x7fd0ec071e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.133:43626/0 (socket says 172.21.15.133:43626) 2024-02-20T07:49:44.468 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:49:44.464+0000 7fd0ead9d700 1 -- 172.21.15.133:0/3400497096 learned_addr learned my addr 172.21.15.133:0/3400497096 (peer_addr_for_me v2:172.21.15.133:0/0) 2024-02-20T07:49:44.468 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:49:44.465+0000 7fd0ead9d700 1 -- 172.21.15.133:0/3400497096 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0dc005040 con 0x7fd0ec0fcbd0 2024-02-20T07:49:44.468 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:49:44.465+0000 7fd0ead9d700 1 --2- 172.21.15.133:0/3400497096 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fd0ec0fcbd0 0x7fd0ec071e60 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fd0dc009680 tx=0x7fd0dc009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:49:44.468 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:49:44.465+0000 7fd0d37fe700 1 -- 172.21.15.133:0/3400497096 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd0dc00bed0 con 0x7fd0ec0fcbd0 2024-02-20T07:49:44.468 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:49:44.465+0000 7fd0d37fe700 1 -- 172.21.15.133:0/3400497096 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd0dc00b5c0 con 0x7fd0ec0fcbd0 2024-02-20T07:49:44.468 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:49:44.465+0000 7fd0d37fe700 1 -- 172.21.15.133:0/3400497096 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd0dc01c5b0 con 0x7fd0ec0fcbd0 2024-02-20T07:49:44.468 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:49:44.465+0000 7fd0f107a700 1 -- 172.21.15.133:0/3400497096 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd0ec0704e0 con 0x7fd0ec0fcbd0 2024-02-20T07:49:44.469 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:49:44.465+0000 7fd0f107a700 1 -- 172.21.15.133:0/3400497096 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd0ec0709d0 con 0x7fd0ec0fcbd0 2024-02-20T07:49:44.469 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:49:44.466+0000 7fd0d37fe700 1 -- 172.21.15.133:0/3400497096 <== mon.0 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 13) v1 ==== 43425+0+0 (secure 0 0 0) 0x7fd0dc01c7d0 con 0x7fd0ec0fcbd0 2024-02-20T07:49:44.469 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:49:44.466+0000 7fd0f107a700 1 -- 172.21.15.133:0/3400497096 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd0ec1069a0 con 0x7fd0ec0fcbd0 2024-02-20T07:49:44.469 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:49:44.466+0000 7fd0d37fe700 1 --2- 172.21.15.133:0/3400497096 >> [v2:172.21.15.133:6800/3944459195,v1:172.21.15.133:6801/3944459195] conn(0x7fd0d4036870 0x7fd0d4038d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:49:44.470 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:49:44.466+0000 7fd0d37fe700 1 -- 172.21.15.133:0/3400497096 <== mon.0 v2:172.21.15.133:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7fd0dc049b00 con 0x7fd0ec0fcbd0 2024-02-20T07:49:44.470 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:49:44.467+0000 7fd0ea59c700 1 --2- 172.21.15.133:0/3400497096 >> [v2:172.21.15.133:6800/3944459195,v1:172.21.15.133:6801/3944459195] conn(0x7fd0d4036870 0x7fd0d4038d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:49:44.470 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:49:44.468+0000 7fd0ea59c700 1 --2- 172.21.15.133:0/3400497096 >> [v2:172.21.15.133:6800/3944459195,v1:172.21.15.133:6801/3944459195] conn(0x7fd0d4036870 0x7fd0d4038d20 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fd0e0009990 tx=0x7fd0e0006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:49:44.474 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:49:44.472+0000 7fd0d37fe700 1 -- 172.21.15.133:0/3400497096 <== mon.0 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7fd0dc025400 con 0x7fd0ec0fcbd0 2024-02-20T07:49:44.708 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:49:44.705+0000 7fd0f107a700 1 -- 172.21.15.133:0/3400497096 --> [v2:172.21.15.133:6800/3944459195,v1:172.21.15.133:6801/3944459195] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fd0ec02b310 con 0x7fd0d4036870 2024-02-20T07:49:44.710 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:49:44.707+0000 7fd0d37fe700 1 -- 172.21.15.133:0/3400497096 <== mgr.14162 v2:172.21.15.133:6800/3944459195 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+161 (secure 0 0 0) 0x7fd0ec02b310 con 0x7fd0d4036870 2024-02-20T07:49:44.710 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-02-20T07:49:44.710 INFO:teuthology.orchestra.run.smithi133.stdout:[{"addr": "172.21.15.133", "hostname": "smithi133", "labels": [], "status": ""}, {"addr": "172.21.15.183", "hostname": "smithi183", "labels": [], "status": ""}] 2024-02-20T07:49:44.712 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:49:44.710+0000 7fd0f107a700 1 -- 172.21.15.133:0/3400497096 >> [v2:172.21.15.133:6800/3944459195,v1:172.21.15.133:6801/3944459195] conn(0x7fd0d4036870 msgr2=0x7fd0d4038d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:49:44.713 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:49:44.710+0000 7fd0f107a700 1 --2- 172.21.15.133:0/3400497096 >> [v2:172.21.15.133:6800/3944459195,v1:172.21.15.133:6801/3944459195] conn(0x7fd0d4036870 0x7fd0d4038d20 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fd0e0009990 tx=0x7fd0e0006e40 comp rx=0 tx=0).stop 2024-02-20T07:49:44.713 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:49:44.710+0000 7fd0f107a700 1 -- 172.21.15.133:0/3400497096 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fd0ec0fcbd0 msgr2=0x7fd0ec071e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:49:44.713 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:49:44.710+0000 7fd0f107a700 1 --2- 172.21.15.133:0/3400497096 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fd0ec0fcbd0 0x7fd0ec071e60 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fd0dc009680 tx=0x7fd0dc009e30 comp rx=0 tx=0).stop 2024-02-20T07:49:44.713 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:49:44.710+0000 7fd0f107a700 1 -- 172.21.15.133:0/3400497096 shutdown_connections 2024-02-20T07:49:44.713 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:49:44.710+0000 7fd0f107a700 1 --2- 172.21.15.133:0/3400497096 >> [v2:172.21.15.133:6800/3944459195,v1:172.21.15.133:6801/3944459195] conn(0x7fd0d4036870 0x7fd0d4038d20 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:49:44.713 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:49:44.710+0000 7fd0f107a700 1 --2- 172.21.15.133:0/3400497096 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fd0ec0fcbd0 0x7fd0ec071e60 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:49:44.713 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:49:44.710+0000 7fd0f107a700 1 -- 172.21.15.133:0/3400497096 >> 172.21.15.133:0/3400497096 conn(0x7fd0ec0f8290 msgr2=0x7fd0ec0f8fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:49:44.713 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:49:44.710+0000 7fd0f107a700 1 -- 172.21.15.133:0/3400497096 shutdown_connections 2024-02-20T07:49:44.713 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:49:44.710+0000 7fd0f107a700 1 -- 172.21.15.133:0/3400497096 wait complete. 2024-02-20T07:49:45.492 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:49:45 smithi133 ceph-mon[29117]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T07:49:45.492 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:49:45 smithi133 ceph-mon[29117]: from='mgr.14162 172.21.15.133:0/3642562278' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:49:46.009 INFO:tasks.cephadm:Setting crush tunables to default 2024-02-20T07:49:46.009 DEBUG:teuthology.orchestra.run.smithi133:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4537327c-cfc4-11ee-95bb-87774f69a715 -- ceph osd crush tunables default 2024-02-20T07:49:46.242 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:49:46 smithi133 ceph-mon[29117]: from='client.14190 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-20T07:49:47.145 INFO:teuthology.orchestra.run.smithi133.stderr:Inferring config /var/lib/ceph/4537327c-cfc4-11ee-95bb-87774f69a715/mon.smithi133/config 2024-02-20T07:49:47.433 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:49:47 smithi133 ceph-mon[29117]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T07:49:49.492 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:49:49 smithi133 ceph-mon[29117]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T07:49:50.995 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:49:50.993+0000 7f62aff7d700 1 -- 172.21.15.133:0/570288155 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f62a80fbac0 msgr2=0x7f62a80fbed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:49:50.996 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:49:50.993+0000 7f62aff7d700 1 --2- 172.21.15.133:0/570288155 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f62a80fbac0 0x7f62a80fbed0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f62a40048c0 tx=0x7f62a4009e30 comp rx=0 tx=0).stop 2024-02-20T07:49:50.996 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:49:50.993+0000 7f62aff7d700 1 -- 172.21.15.133:0/570288155 shutdown_connections 2024-02-20T07:49:50.996 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:49:50.993+0000 7f62aff7d700 1 --2- 172.21.15.133:0/570288155 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f62a80fbac0 0x7f62a80fbed0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:49:50.996 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:49:50.993+0000 7f62aff7d700 1 -- 172.21.15.133:0/570288155 >> 172.21.15.133:0/570288155 conn(0x7f62a80f71a0 msgr2=0x7f62a80f95b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:49:50.996 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:49:50.993+0000 7f62aff7d700 1 -- 172.21.15.133:0/570288155 shutdown_connections 2024-02-20T07:49:50.996 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:49:50.993+0000 7f62aff7d700 1 -- 172.21.15.133:0/570288155 wait complete. 2024-02-20T07:49:50.996 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:49:50.994+0000 7f62aff7d700 1 Processor -- start 2024-02-20T07:49:50.996 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:49:50.994+0000 7f62aff7d700 1 -- start start 2024-02-20T07:49:50.996 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:49:50.994+0000 7f62aff7d700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f62a80fbac0 0x7f62a810fc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:49:50.996 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:49:50.994+0000 7f62aff7d700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62a80667f0 con 0x7f62a80fbac0 2024-02-20T07:49:50.996 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:49:50.994+0000 7f62add19700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f62a80fbac0 0x7f62a810fc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:49:50.996 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:49:50.994+0000 7f62add19700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f62a80fbac0 0x7f62a810fc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.133:49292/0 (socket says 172.21.15.133:49292) 2024-02-20T07:49:50.997 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:49:50.994+0000 7f62add19700 1 -- 172.21.15.133:0/745564936 learned_addr learned my addr 172.21.15.133:0/745564936 (peer_addr_for_me v2:172.21.15.133:0/0) 2024-02-20T07:49:50.997 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:49:50.994+0000 7f62add19700 1 -- 172.21.15.133:0/745564936 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62a4005040 con 0x7f62a80fbac0 2024-02-20T07:49:50.997 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:49:50.994+0000 7f62add19700 1 --2- 172.21.15.133:0/745564936 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f62a80fbac0 0x7f62a810fc00 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f62a4009b80 tx=0x7f62a4009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:49:50.997 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:49:50.995+0000 7f629e7fc700 1 -- 172.21.15.133:0/745564936 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f62a400b5c0 con 0x7f62a80fbac0 2024-02-20T07:49:50.997 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:49:50.995+0000 7f629e7fc700 1 -- 172.21.15.133:0/745564936 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f62a400be70 con 0x7f62a80fbac0 2024-02-20T07:49:50.997 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:49:50.995+0000 7f629e7fc700 1 -- 172.21.15.133:0/745564936 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f62a401c660 con 0x7f62a80fbac0 2024-02-20T07:49:50.997 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:49:50.995+0000 7f62aff7d700 1 -- 172.21.15.133:0/745564936 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f62a80669f0 con 0x7f62a80fbac0 2024-02-20T07:49:50.998 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:49:50.995+0000 7f62aff7d700 1 -- 172.21.15.133:0/745564936 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f62a8110380 con 0x7f62a80fbac0 2024-02-20T07:49:50.998 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:49:50.995+0000 7f629e7fc700 1 -- 172.21.15.133:0/745564936 <== mon.0 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 13) v1 ==== 43425+0+0 (secure 0 0 0) 0x7f62a401c7c0 con 0x7f62a80fbac0 2024-02-20T07:49:50.998 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:49:50.996+0000 7f629e7fc700 1 --2- 172.21.15.133:0/745564936 >> [v2:172.21.15.133:6800/3944459195,v1:172.21.15.133:6801/3944459195] conn(0x7f62940368c0 0x7f6294038d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:49:50.998 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:49:50.996+0000 7f629e7fc700 1 -- 172.21.15.133:0/745564936 <== mon.0 v2:172.21.15.133:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f62a40499a0 con 0x7f62a80fbac0 2024-02-20T07:49:50.998 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:49:50.996+0000 7f62aff7d700 1 -- 172.21.15.133:0/745564936 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f62a8105890 con 0x7f62a80fbac0 2024-02-20T07:49:50.998 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:49:50.996+0000 7f62ad518700 1 --2- 172.21.15.133:0/745564936 >> [v2:172.21.15.133:6800/3944459195,v1:172.21.15.133:6801/3944459195] conn(0x7f62940368c0 0x7f6294038d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:49:50.998 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:49:50.996+0000 7f62ad518700 1 --2- 172.21.15.133:0/745564936 >> [v2:172.21.15.133:6800/3944459195,v1:172.21.15.133:6801/3944459195] conn(0x7f62940368c0 0x7f6294038d70 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f6298009990 tx=0x7f6298006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:49:51.002 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:49:51.001+0000 7f629e7fc700 1 -- 172.21.15.133:0/745564936 <== mon.0 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f62a4011dd0 con 0x7f62a80fbac0 2024-02-20T07:49:51.223 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:49:51.221+0000 7f62aff7d700 1 -- 172.21.15.133:0/745564936 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7f62a8101880 con 0x7f62a80fbac0 2024-02-20T07:49:51.223 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:49:51 smithi133 ceph-mon[29117]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T07:49:51.223 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:49:51 smithi133 ceph-mon[29117]: from='mgr.14162 172.21.15.133:0/3642562278' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:49:51.223 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:49:51 smithi133 ceph-mon[29117]: from='mgr.14162 172.21.15.133:0/3642562278' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:49:51.223 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:49:51 smithi133 ceph-mon[29117]: from='mgr.14162 172.21.15.133:0/3642562278' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:49:51.223 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:49:51 smithi133 ceph-mon[29117]: from='mgr.14162 172.21.15.133:0/3642562278' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:49:51.223 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:49:51 smithi133 ceph-mon[29117]: from='mgr.14162 172.21.15.133:0/3642562278' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:49:51.223 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:49:51 smithi133 ceph-mon[29117]: from='mgr.14162 172.21.15.133:0/3642562278' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:49:51.223 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:49:51 smithi133 ceph-mon[29117]: from='mgr.14162 172.21.15.133:0/3642562278' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-02-20T07:49:51.223 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:49:51 smithi133 ceph-mon[29117]: from='mgr.14162 172.21.15.133:0/3642562278' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:49:52.114 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:49:52.113+0000 7f629e7fc700 1 -- 172.21.15.133:0/745564936 <== mon.0 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v4) v1 ==== 124+0+0 (secure 0 0 0) 0x7f62a4011370 con 0x7f62a80fbac0 2024-02-20T07:49:52.116 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:49:52.115+0000 7f62aff7d700 1 -- 172.21.15.133:0/745564936 >> [v2:172.21.15.133:6800/3944459195,v1:172.21.15.133:6801/3944459195] conn(0x7f62940368c0 msgr2=0x7f6294038d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:49:52.117 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:49:52.115+0000 7f62aff7d700 1 --2- 172.21.15.133:0/745564936 >> [v2:172.21.15.133:6800/3944459195,v1:172.21.15.133:6801/3944459195] conn(0x7f62940368c0 0x7f6294038d70 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f6298009990 tx=0x7f6298006e40 comp rx=0 tx=0).stop 2024-02-20T07:49:52.117 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:49:52.115+0000 7f62aff7d700 1 -- 172.21.15.133:0/745564936 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f62a80fbac0 msgr2=0x7f62a810fc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:49:52.117 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:49:52.115+0000 7f62aff7d700 1 --2- 172.21.15.133:0/745564936 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f62a80fbac0 0x7f62a810fc00 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f62a4009b80 tx=0x7f62a4009e30 comp rx=0 tx=0).stop 2024-02-20T07:49:52.117 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:49:52.115+0000 7f62aff7d700 1 -- 172.21.15.133:0/745564936 shutdown_connections 2024-02-20T07:49:52.117 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:49:52.115+0000 7f62aff7d700 1 --2- 172.21.15.133:0/745564936 >> [v2:172.21.15.133:6800/3944459195,v1:172.21.15.133:6801/3944459195] conn(0x7f62940368c0 0x7f6294038d70 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:49:52.117 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:49:52.115+0000 7f62aff7d700 1 --2- 172.21.15.133:0/745564936 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f62a80fbac0 0x7f62a810fc00 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:49:52.117 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:49:52.115+0000 7f62aff7d700 1 -- 172.21.15.133:0/745564936 >> 172.21.15.133:0/745564936 conn(0x7f62a80f71a0 msgr2=0x7f62a80f7f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:49:52.117 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:49:52.115+0000 7f62aff7d700 1 -- 172.21.15.133:0/745564936 shutdown_connections 2024-02-20T07:49:52.117 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:49:52.115+0000 7f62aff7d700 1 -- 172.21.15.133:0/745564936 wait complete. 2024-02-20T07:49:52.117 INFO:teuthology.orchestra.run.smithi133.stderr:adjusted tunables profile to default 2024-02-20T07:49:52.424 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:49:52 smithi133 ceph-mon[29117]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-02-20T07:49:52.424 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:49:52 smithi133 ceph-mon[29117]: Deploying daemon grafana.smithi133 on smithi133 2024-02-20T07:49:52.424 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:49:52 smithi133 ceph-mon[29117]: from='client.? 172.21.15.133:0/745564936' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-02-20T07:49:53.057 INFO:tasks.cephadm:Adding mon.smithi133 on smithi133 2024-02-20T07:49:53.057 INFO:tasks.cephadm:Adding mon.smithi183 on smithi183 2024-02-20T07:49:53.057 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4537327c-cfc4-11ee-95bb-87774f69a715 -- ceph orch apply mon '2;smithi133:172.21.15.133=smithi133;smithi183:172.21.15.183=smithi183' 2024-02-20T07:49:53.305 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-20T07:49:53.351 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-20T07:49:53.492 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:49:53 smithi133 ceph-mon[29117]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T07:49:53.492 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:49:53 smithi133 ceph-mon[29117]: from='client.? 172.21.15.133:0/745564936' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-02-20T07:49:53.492 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:49:53 smithi133 ceph-mon[29117]: osdmap e4: 0 total, 0 up, 0 in 2024-02-20T07:49:54.847 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:49:54.844+0000 7ff2984e0700 1 -- 172.21.15.183:0/3071253266 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ff2900fcbc0 msgr2=0x7ff2900fcfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:49:54.847 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:49:54.844+0000 7ff2984e0700 1 --2- 172.21.15.183:0/3071253266 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ff2900fcbc0 0x7ff2900fcfd0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7ff28c0048c0 tx=0x7ff28c009e30 comp rx=0 tx=0).stop 2024-02-20T07:49:54.847 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:49:54.844+0000 7ff2984e0700 1 -- 172.21.15.183:0/3071253266 shutdown_connections 2024-02-20T07:49:54.847 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:49:54.844+0000 7ff2984e0700 1 --2- 172.21.15.183:0/3071253266 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ff2900fcbc0 0x7ff2900fcfd0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:49:54.848 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:49:54.844+0000 7ff2984e0700 1 -- 172.21.15.183:0/3071253266 >> 172.21.15.183:0/3071253266 conn(0x7ff2900f82a0 msgr2=0x7ff2900fa6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:49:54.848 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:49:54.844+0000 7ff2984e0700 1 -- 172.21.15.183:0/3071253266 shutdown_connections 2024-02-20T07:49:54.848 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:49:54.844+0000 7ff2984e0700 1 -- 172.21.15.183:0/3071253266 wait complete. 2024-02-20T07:49:54.848 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:49:54.845+0000 7ff2984e0700 1 Processor -- start 2024-02-20T07:49:54.848 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:49:54.845+0000 7ff2984e0700 1 -- start start 2024-02-20T07:49:54.848 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:49:54.845+0000 7ff2984e0700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ff2900fcbc0 0x7ff29010cec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:49:54.848 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:49:54.845+0000 7ff2984e0700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff29010d400 con 0x7ff2900fcbc0 2024-02-20T07:49:54.848 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:49:54.845+0000 7ff29627c700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ff2900fcbc0 0x7ff29010cec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:49:54.848 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:49:54.845+0000 7ff29627c700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ff2900fcbc0 0x7ff29010cec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.183:52430/0 (socket says 172.21.15.183:52430) 2024-02-20T07:49:54.848 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:49:54.845+0000 7ff29627c700 1 -- 172.21.15.183:0/3700798696 learned_addr learned my addr 172.21.15.183:0/3700798696 (peer_addr_for_me v2:172.21.15.183:0/0) 2024-02-20T07:49:54.848 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:49:54.845+0000 7ff29627c700 1 -- 172.21.15.183:0/3700798696 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff28c005040 con 0x7ff2900fcbc0 2024-02-20T07:49:54.848 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:49:54.845+0000 7ff29627c700 1 --2- 172.21.15.183:0/3700798696 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ff2900fcbc0 0x7ff29010cec0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7ff28c009b80 tx=0x7ff28c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:49:54.848 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:49:54.846+0000 7ff286ffd700 1 -- 172.21.15.183:0/3700798696 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff28c00b5c0 con 0x7ff2900fcbc0 2024-02-20T07:49:54.848 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:49:54.846+0000 7ff286ffd700 1 -- 172.21.15.183:0/3700798696 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff28c00be70 con 0x7ff2900fcbc0 2024-02-20T07:49:54.848 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:49:54.846+0000 7ff2984e0700 1 -- 172.21.15.183:0/3700798696 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff29010d600 con 0x7ff2900fcbc0 2024-02-20T07:49:54.849 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:49:54.846+0000 7ff286ffd700 1 -- 172.21.15.183:0/3700798696 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff28c01c660 con 0x7ff2900fcbc0 2024-02-20T07:49:54.849 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:49:54.846+0000 7ff2984e0700 1 -- 172.21.15.183:0/3700798696 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff29010e2a0 con 0x7ff2900fcbc0 2024-02-20T07:49:54.849 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:49:54.847+0000 7ff286ffd700 1 -- 172.21.15.183:0/3700798696 <== mon.0 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 13) v1 ==== 43425+0+0 (secure 0 0 0) 0x7ff28c013420 con 0x7ff2900fcbc0 2024-02-20T07:49:54.850 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:49:54.847+0000 7ff2984e0700 1 -- 172.21.15.183:0/3700798696 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff29005f5c0 con 0x7ff2900fcbc0 2024-02-20T07:49:54.850 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:49:54.847+0000 7ff286ffd700 1 --2- 172.21.15.183:0/3700798696 >> [v2:172.21.15.133:6800/3944459195,v1:172.21.15.133:6801/3944459195] conn(0x7ff27c0368c0 0x7ff27c038d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:49:54.850 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:49:54.847+0000 7ff286ffd700 1 -- 172.21.15.183:0/3700798696 <== mon.0 v2:172.21.15.133:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7ff28c0499a0 con 0x7ff2900fcbc0 2024-02-20T07:49:54.850 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:49:54.847+0000 7ff295a7b700 1 --2- 172.21.15.183:0/3700798696 >> [v2:172.21.15.133:6800/3944459195,v1:172.21.15.133:6801/3944459195] conn(0x7ff27c0368c0 0x7ff27c038d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:49:54.850 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:49:54.847+0000 7ff295a7b700 1 --2- 172.21.15.183:0/3700798696 >> [v2:172.21.15.133:6800/3944459195,v1:172.21.15.133:6801/3944459195] conn(0x7ff27c0368c0 0x7ff27c038d70 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7ff280009990 tx=0x7ff280006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:49:54.853 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:49:54.852+0000 7ff286ffd700 1 -- 172.21.15.183:0/3700798696 <== mon.0 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7ff28c013b10 con 0x7ff2900fcbc0 2024-02-20T07:49:55.049 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:49:55.047+0000 7ff2984e0700 1 -- 172.21.15.183:0/3700798696 --> [v2:172.21.15.133:6800/3944459195,v1:172.21.15.133:6801/3944459195] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "placement": "2;smithi133:172.21.15.133=smithi133;smithi183:172.21.15.183=smithi183", "target": ["mon-mgr", ""]}) v1 -- 0x7ff29010ee60 con 0x7ff27c0368c0 2024-02-20T07:49:55.054 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:49:55.052+0000 7ff286ffd700 1 -- 172.21.15.183:0/3700798696 <== mgr.14162 v2:172.21.15.133:6800/3944459195 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7ff29010ee60 con 0x7ff27c0368c0 2024-02-20T07:49:55.054 INFO:teuthology.orchestra.run.smithi183.stdout:Scheduled mon update... 2024-02-20T07:49:55.056 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:49:55.054+0000 7ff2984e0700 1 -- 172.21.15.183:0/3700798696 >> [v2:172.21.15.133:6800/3944459195,v1:172.21.15.133:6801/3944459195] conn(0x7ff27c0368c0 msgr2=0x7ff27c038d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:49:55.056 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:49:55.054+0000 7ff2984e0700 1 --2- 172.21.15.183:0/3700798696 >> [v2:172.21.15.133:6800/3944459195,v1:172.21.15.133:6801/3944459195] conn(0x7ff27c0368c0 0x7ff27c038d70 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7ff280009990 tx=0x7ff280006e40 comp rx=0 tx=0).stop 2024-02-20T07:49:55.056 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:49:55.054+0000 7ff2984e0700 1 -- 172.21.15.183:0/3700798696 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ff2900fcbc0 msgr2=0x7ff29010cec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:49:55.056 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:49:55.054+0000 7ff2984e0700 1 --2- 172.21.15.183:0/3700798696 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ff2900fcbc0 0x7ff29010cec0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7ff28c009b80 tx=0x7ff28c009e30 comp rx=0 tx=0).stop 2024-02-20T07:49:55.056 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:49:55.054+0000 7ff2984e0700 1 -- 172.21.15.183:0/3700798696 shutdown_connections 2024-02-20T07:49:55.056 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:49:55.054+0000 7ff2984e0700 1 --2- 172.21.15.183:0/3700798696 >> [v2:172.21.15.133:6800/3944459195,v1:172.21.15.133:6801/3944459195] conn(0x7ff27c0368c0 0x7ff27c038d70 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:49:55.056 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:49:55.054+0000 7ff2984e0700 1 --2- 172.21.15.183:0/3700798696 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ff2900fcbc0 0x7ff29010cec0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:49:55.056 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:49:55.054+0000 7ff2984e0700 1 -- 172.21.15.183:0/3700798696 >> 172.21.15.183:0/3700798696 conn(0x7ff2900f82a0 msgr2=0x7ff2900f8fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:49:55.056 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:49:55.054+0000 7ff2984e0700 1 -- 172.21.15.183:0/3700798696 shutdown_connections 2024-02-20T07:49:55.056 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:49:55.054+0000 7ff2984e0700 1 -- 172.21.15.183:0/3700798696 wait complete. 2024-02-20T07:49:55.492 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:49:55 smithi133 ceph-mon[29117]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T07:49:55.492 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:49:55 smithi133 ceph-mon[29117]: from='mgr.14162 172.21.15.133:0/3642562278' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:49:55.492 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:49:55 smithi133 ceph-mon[29117]: from='mgr.14162 172.21.15.133:0/3642562278' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:49:55.670 DEBUG:teuthology.orchestra.run.smithi183:mon.smithi183> sudo journalctl -f -n 0 -u ceph-4537327c-cfc4-11ee-95bb-87774f69a715@mon.smithi183.service 2024-02-20T07:49:55.672 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T07:49:55.672 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4537327c-cfc4-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T07:49:55.949 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-20T07:49:55.996 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-20T07:49:56.492 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:49:56 smithi133 ceph-mon[29117]: from='client.14194 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "2;smithi133:172.21.15.133=smithi133;smithi183:172.21.15.183=smithi183", "target": ["mon-mgr", ""]}]: dispatch 2024-02-20T07:49:56.492 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:49:56 smithi133 ceph-mon[29117]: Saving service mon spec with placement smithi133:172.21.15.133=smithi133;smithi183:172.21.15.183=smithi183;count:2 2024-02-20T07:49:57.434 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:49:57.430+0000 7fa682361700 1 -- 172.21.15.183:0/1782206548 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa67c0fcbd0 msgr2=0x7fa67c0fcfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:49:57.434 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:49:57.430+0000 7fa682361700 1 --2- 172.21.15.183:0/1782206548 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa67c0fcbd0 0x7fa67c0fcfe0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fa66c0048c0 tx=0x7fa66c009e30 comp rx=0 tx=0).stop 2024-02-20T07:49:57.434 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:49:57.430+0000 7fa682361700 1 -- 172.21.15.183:0/1782206548 shutdown_connections 2024-02-20T07:49:57.434 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:49:57.430+0000 7fa682361700 1 --2- 172.21.15.183:0/1782206548 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa67c0fcbd0 0x7fa67c0fcfe0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:49:57.434 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:49:57.430+0000 7fa682361700 1 -- 172.21.15.183:0/1782206548 >> 172.21.15.183:0/1782206548 conn(0x7fa67c0f8290 msgr2=0x7fa67c0fa6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:49:57.434 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:49:57.430+0000 7fa682361700 1 -- 172.21.15.183:0/1782206548 shutdown_connections 2024-02-20T07:49:57.434 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:49:57.431+0000 7fa682361700 1 -- 172.21.15.183:0/1782206548 wait complete. 2024-02-20T07:49:57.435 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:49:57.431+0000 7fa682361700 1 Processor -- start 2024-02-20T07:49:57.435 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:49:57.431+0000 7fa682361700 1 -- start start 2024-02-20T07:49:57.435 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:49:57.431+0000 7fa682361700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa67c0fcbd0 0x7fa67c10cec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:49:57.435 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:49:57.431+0000 7fa682361700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa67c10d400 con 0x7fa67c0fcbd0 2024-02-20T07:49:57.435 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:49:57.432+0000 7fa67bfff700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa67c0fcbd0 0x7fa67c10cec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:49:57.435 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:49:57.432+0000 7fa67bfff700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa67c0fcbd0 0x7fa67c10cec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.183:52446/0 (socket says 172.21.15.183:52446) 2024-02-20T07:49:57.435 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:49:57.432+0000 7fa67bfff700 1 -- 172.21.15.183:0/1606674652 learned_addr learned my addr 172.21.15.183:0/1606674652 (peer_addr_for_me v2:172.21.15.183:0/0) 2024-02-20T07:49:57.435 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:49:57.432+0000 7fa67bfff700 1 -- 172.21.15.183:0/1606674652 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa66c005040 con 0x7fa67c0fcbd0 2024-02-20T07:49:57.435 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:49:57.432+0000 7fa67bfff700 1 --2- 172.21.15.183:0/1606674652 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa67c0fcbd0 0x7fa67c10cec0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fa66c009680 tx=0x7fa66c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:49:57.435 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:49:57.432+0000 7fa678ff9700 1 -- 172.21.15.183:0/1606674652 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa66c00bed0 con 0x7fa67c0fcbd0 2024-02-20T07:49:57.435 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:49:57.432+0000 7fa678ff9700 1 -- 172.21.15.183:0/1606674652 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa66c00b5c0 con 0x7fa67c0fcbd0 2024-02-20T07:49:57.435 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:49:57.432+0000 7fa682361700 1 -- 172.21.15.183:0/1606674652 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa67c10d600 con 0x7fa67c0fcbd0 2024-02-20T07:49:57.435 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:49:57.432+0000 7fa678ff9700 1 -- 172.21.15.183:0/1606674652 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa66c01c5b0 con 0x7fa67c0fcbd0 2024-02-20T07:49:57.435 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:49:57.432+0000 7fa682361700 1 -- 172.21.15.183:0/1606674652 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa67c10e2a0 con 0x7fa67c0fcbd0 2024-02-20T07:49:57.436 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:49:57.433+0000 7fa678ff9700 1 -- 172.21.15.183:0/1606674652 <== mon.0 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 13) v1 ==== 43425+0+0 (secure 0 0 0) 0x7fa66c01c820 con 0x7fa67c0fcbd0 2024-02-20T07:49:57.437 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:49:57.433+0000 7fa678ff9700 1 --2- 172.21.15.183:0/1606674652 >> [v2:172.21.15.133:6800/3944459195,v1:172.21.15.133:6801/3944459195] conn(0x7fa664036820 0x7fa664038cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:49:57.437 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:49:57.433+0000 7fa678ff9700 1 -- 172.21.15.183:0/1606674652 <== mon.0 v2:172.21.15.133:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7fa66c049b80 con 0x7fa67c0fcbd0 2024-02-20T07:49:57.437 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:49:57.433+0000 7fa682361700 1 -- 172.21.15.183:0/1606674652 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa67c05f5c0 con 0x7fa67c0fcbd0 2024-02-20T07:49:57.437 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:49:57.433+0000 7fa67b7fe700 1 --2- 172.21.15.183:0/1606674652 >> [v2:172.21.15.133:6800/3944459195,v1:172.21.15.133:6801/3944459195] conn(0x7fa664036820 0x7fa664038cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:49:57.437 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:49:57.434+0000 7fa67b7fe700 1 --2- 172.21.15.183:0/1606674652 >> [v2:172.21.15.133:6800/3944459195,v1:172.21.15.133:6801/3944459195] conn(0x7fa664036820 0x7fa664038cd0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fa670009990 tx=0x7fa670006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:49:57.444 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:49:57.443+0000 7fa678ff9700 1 -- 172.21.15.183:0/1606674652 <== mon.0 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7fa66c023300 con 0x7fa67c0fcbd0 2024-02-20T07:49:57.492 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:49:57 smithi133 ceph-mon[29117]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T07:49:57.702 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:49:57.700+0000 7fa682361700 1 -- 172.21.15.183:0/1606674652 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa67c10ecb0 con 0x7fa67c0fcbd0 2024-02-20T07:49:57.702 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:49:57.701+0000 7fa678ff9700 1 -- 172.21.15.183:0/1606674652 <== mon.0 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+743 (secure 0 0 0) 0x7fa66c011b60 con 0x7fa67c0fcbd0 2024-02-20T07:49:57.702 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-02-20T07:49:57.703 INFO:teuthology.orchestra.run.smithi183.stdout:{"epoch":1,"fsid":"4537327c-cfc4-11ee-95bb-87774f69a715","modified":"2024-02-20T07:48:10.704626Z","created":"2024-02-20T07:48:10.704626Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi133","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:3300","nonce":0},{"type":"v1","addr":"172.21.15.133:6789","nonce":0}]},"addr":"172.21.15.133:6789/0","public_addr":"172.21.15.133:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T07:49:57.704 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:49:57.702+0000 7fa682361700 1 -- 172.21.15.183:0/1606674652 >> [v2:172.21.15.133:6800/3944459195,v1:172.21.15.133:6801/3944459195] conn(0x7fa664036820 msgr2=0x7fa664038cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:49:57.704 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:49:57.702+0000 7fa682361700 1 --2- 172.21.15.183:0/1606674652 >> [v2:172.21.15.133:6800/3944459195,v1:172.21.15.133:6801/3944459195] conn(0x7fa664036820 0x7fa664038cd0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fa670009990 tx=0x7fa670006e40 comp rx=0 tx=0).stop 2024-02-20T07:49:57.704 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:49:57.702+0000 7fa682361700 1 -- 172.21.15.183:0/1606674652 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa67c0fcbd0 msgr2=0x7fa67c10cec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:49:57.704 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:49:57.702+0000 7fa682361700 1 --2- 172.21.15.183:0/1606674652 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa67c0fcbd0 0x7fa67c10cec0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fa66c009680 tx=0x7fa66c009e30 comp rx=0 tx=0).stop 2024-02-20T07:49:57.705 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:49:57.703+0000 7fa682361700 1 -- 172.21.15.183:0/1606674652 shutdown_connections 2024-02-20T07:49:57.705 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:49:57.703+0000 7fa682361700 1 --2- 172.21.15.183:0/1606674652 >> [v2:172.21.15.133:6800/3944459195,v1:172.21.15.133:6801/3944459195] conn(0x7fa664036820 0x7fa664038cd0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:49:57.705 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:49:57.703+0000 7fa682361700 1 --2- 172.21.15.183:0/1606674652 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa67c0fcbd0 0x7fa67c10cec0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:49:57.705 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:49:57.703+0000 7fa682361700 1 -- 172.21.15.183:0/1606674652 >> 172.21.15.183:0/1606674652 conn(0x7fa67c0f8290 msgr2=0x7fa67c0f9000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:49:57.705 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:49:57.703+0000 7fa682361700 1 -- 172.21.15.183:0/1606674652 shutdown_connections 2024-02-20T07:49:57.705 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:49:57.703+0000 7fa682361700 1 -- 172.21.15.183:0/1606674652 wait complete. 2024-02-20T07:49:57.705 INFO:teuthology.orchestra.run.smithi183.stderr:dumped monmap epoch 1 2024-02-20T07:49:58.492 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:49:58 smithi133 ceph-mon[29117]: from='client.? 172.21.15.183:0/1606674652' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T07:49:59.298 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T07:49:59.298 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4537327c-cfc4-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T07:49:59.492 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:49:59 smithi133 ceph-mon[29117]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T07:49:59.562 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-20T07:49:59.611 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-20T07:50:01.004 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:01.000+0000 7fcd03da3700 1 -- 172.21.15.183:0/3408927813 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fccfc0fbae0 msgr2=0x7fccfc0fbef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:50:01.004 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:01.000+0000 7fcd03da3700 1 --2- 172.21.15.183:0/3408927813 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fccfc0fbae0 0x7fccfc0fbef0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fccec0048c0 tx=0x7fccec009e30 comp rx=0 tx=0).stop 2024-02-20T07:50:01.004 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:01.000+0000 7fcd03da3700 1 -- 172.21.15.183:0/3408927813 shutdown_connections 2024-02-20T07:50:01.004 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:01.000+0000 7fcd03da3700 1 --2- 172.21.15.183:0/3408927813 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fccfc0fbae0 0x7fccfc0fbef0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:50:01.004 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:01.000+0000 7fcd03da3700 1 -- 172.21.15.183:0/3408927813 >> 172.21.15.183:0/3408927813 conn(0x7fccfc0f71a0 msgr2=0x7fccfc0f95d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:50:01.004 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:01.000+0000 7fcd03da3700 1 -- 172.21.15.183:0/3408927813 shutdown_connections 2024-02-20T07:50:01.004 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:01.000+0000 7fcd03da3700 1 -- 172.21.15.183:0/3408927813 wait complete. 2024-02-20T07:50:01.004 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:01.001+0000 7fcd03da3700 1 Processor -- start 2024-02-20T07:50:01.004 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:01.001+0000 7fcd03da3700 1 -- start start 2024-02-20T07:50:01.004 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:01.001+0000 7fcd03da3700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fccfc0fbae0 0x7fccfc10fc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:50:01.004 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:01.001+0000 7fcd03da3700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fccfc066870 con 0x7fccfc0fbae0 2024-02-20T07:50:01.005 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:01.001+0000 7fcd01b3f700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fccfc0fbae0 0x7fccfc10fc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:50:01.005 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:01.001+0000 7fcd01b3f700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fccfc0fbae0 0x7fccfc10fc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.183:35464/0 (socket says 172.21.15.183:35464) 2024-02-20T07:50:01.005 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:01.001+0000 7fcd01b3f700 1 -- 172.21.15.183:0/889511596 learned_addr learned my addr 172.21.15.183:0/889511596 (peer_addr_for_me v2:172.21.15.183:0/0) 2024-02-20T07:50:01.005 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:01.001+0000 7fcd01b3f700 1 -- 172.21.15.183:0/889511596 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fccec005040 con 0x7fccfc0fbae0 2024-02-20T07:50:01.005 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:01.002+0000 7fcd01b3f700 1 --2- 172.21.15.183:0/889511596 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fccfc0fbae0 0x7fccfc10fc50 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fccec009680 tx=0x7fccec009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:50:01.005 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:01.002+0000 7fccf27fc700 1 -- 172.21.15.183:0/889511596 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fccec00bed0 con 0x7fccfc0fbae0 2024-02-20T07:50:01.005 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:01.002+0000 7fcd03da3700 1 -- 172.21.15.183:0/889511596 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fccfc066a70 con 0x7fccfc0fbae0 2024-02-20T07:50:01.005 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:01.002+0000 7fccf27fc700 1 -- 172.21.15.183:0/889511596 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fccec00b5c0 con 0x7fccfc0fbae0 2024-02-20T07:50:01.005 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:01.002+0000 7fccf27fc700 1 -- 172.21.15.183:0/889511596 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fccec01c5b0 con 0x7fccfc0fbae0 2024-02-20T07:50:01.005 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:01.002+0000 7fcd03da3700 1 -- 172.21.15.183:0/889511596 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fccfc1103d0 con 0x7fccfc0fbae0 2024-02-20T07:50:01.006 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:01.003+0000 7fccf27fc700 1 -- 172.21.15.183:0/889511596 <== mon.0 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 13) v1 ==== 43425+0+0 (secure 0 0 0) 0x7fccec01c820 con 0x7fccfc0fbae0 2024-02-20T07:50:01.006 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:01.003+0000 7fcd03da3700 1 -- 172.21.15.183:0/889511596 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fccfc1058b0 con 0x7fccfc0fbae0 2024-02-20T07:50:01.006 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:01.003+0000 7fccf27fc700 1 --2- 172.21.15.183:0/889511596 >> [v2:172.21.15.133:6800/3944459195,v1:172.21.15.133:6801/3944459195] conn(0x7fcce8036820 0x7fcce8038cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:50:01.006 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:01.003+0000 7fccf27fc700 1 -- 172.21.15.183:0/889511596 <== mon.0 v2:172.21.15.133:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7fccec049c70 con 0x7fccfc0fbae0 2024-02-20T07:50:01.006 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:01.003+0000 7fcd0133e700 1 --2- 172.21.15.183:0/889511596 >> [v2:172.21.15.133:6800/3944459195,v1:172.21.15.133:6801/3944459195] conn(0x7fcce8036820 0x7fcce8038cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:50:01.006 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:01.003+0000 7fcd0133e700 1 --2- 172.21.15.183:0/889511596 >> [v2:172.21.15.133:6800/3944459195,v1:172.21.15.133:6801/3944459195] conn(0x7fcce8036820 0x7fcce8038cd0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fccf8009990 tx=0x7fccf8006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:50:01.009 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:01.008+0000 7fccf27fc700 1 -- 172.21.15.183:0/889511596 <== mon.0 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7fccec011550 con 0x7fccfc0fbae0 2024-02-20T07:50:01.263 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:01.261+0000 7fcd03da3700 1 -- 172.21.15.183:0/889511596 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fccfc04bed0 con 0x7fccfc0fbae0 2024-02-20T07:50:01.263 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:01.262+0000 7fccf27fc700 1 -- 172.21.15.183:0/889511596 <== mon.0 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+743 (secure 0 0 0) 0x7fccec004d30 con 0x7fccfc0fbae0 2024-02-20T07:50:01.263 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-02-20T07:50:01.264 INFO:teuthology.orchestra.run.smithi183.stdout:{"epoch":1,"fsid":"4537327c-cfc4-11ee-95bb-87774f69a715","modified":"2024-02-20T07:48:10.704626Z","created":"2024-02-20T07:48:10.704626Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi133","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:3300","nonce":0},{"type":"v1","addr":"172.21.15.133:6789","nonce":0}]},"addr":"172.21.15.133:6789/0","public_addr":"172.21.15.133:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T07:50:01.265 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:01.264+0000 7fcd03da3700 1 -- 172.21.15.183:0/889511596 >> [v2:172.21.15.133:6800/3944459195,v1:172.21.15.133:6801/3944459195] conn(0x7fcce8036820 msgr2=0x7fcce8038cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:50:01.266 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:01.264+0000 7fcd03da3700 1 --2- 172.21.15.183:0/889511596 >> [v2:172.21.15.133:6800/3944459195,v1:172.21.15.133:6801/3944459195] conn(0x7fcce8036820 0x7fcce8038cd0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fccf8009990 tx=0x7fccf8006e40 comp rx=0 tx=0).stop 2024-02-20T07:50:01.266 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:01.264+0000 7fcd03da3700 1 -- 172.21.15.183:0/889511596 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fccfc0fbae0 msgr2=0x7fccfc10fc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:50:01.266 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:01.264+0000 7fcd03da3700 1 --2- 172.21.15.183:0/889511596 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fccfc0fbae0 0x7fccfc10fc50 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fccec009680 tx=0x7fccec009e30 comp rx=0 tx=0).stop 2024-02-20T07:50:01.266 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:01.264+0000 7fcd03da3700 1 -- 172.21.15.183:0/889511596 shutdown_connections 2024-02-20T07:50:01.266 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:01.264+0000 7fcd03da3700 1 --2- 172.21.15.183:0/889511596 >> [v2:172.21.15.133:6800/3944459195,v1:172.21.15.133:6801/3944459195] conn(0x7fcce8036820 0x7fcce8038cd0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:50:01.266 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:01.264+0000 7fcd03da3700 1 --2- 172.21.15.183:0/889511596 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fccfc0fbae0 0x7fccfc10fc50 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:50:01.266 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:01.264+0000 7fcd03da3700 1 -- 172.21.15.183:0/889511596 >> 172.21.15.183:0/889511596 conn(0x7fccfc0f71a0 msgr2=0x7fccfc0f7f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:50:01.266 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:01.264+0000 7fcd03da3700 1 -- 172.21.15.183:0/889511596 shutdown_connections 2024-02-20T07:50:01.266 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:01.264+0000 7fcd03da3700 1 -- 172.21.15.183:0/889511596 wait complete. 2024-02-20T07:50:01.266 INFO:teuthology.orchestra.run.smithi183.stderr:dumped monmap epoch 1 2024-02-20T07:50:01.492 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:50:01 smithi133 ceph-mon[29117]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T07:50:02.376 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:50:02 smithi133 ceph-mon[29117]: from='client.? 172.21.15.183:0/889511596' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T07:50:02.843 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T07:50:02.843 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4537327c-cfc4-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T07:50:03.096 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-20T07:50:03.143 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-20T07:50:03.444 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:50:03 smithi133 ceph-mon[29117]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T07:50:04.564 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:04.562+0000 7f9afe4a4700 1 -- 172.21.15.183:0/326266549 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f9af80fcbd0 msgr2=0x7f9af80fcfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:50:04.564 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:04.562+0000 7f9afe4a4700 1 --2- 172.21.15.183:0/326266549 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f9af80fcbd0 0x7f9af80fcfe0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f9ae80048c0 tx=0x7f9ae8009e30 comp rx=0 tx=0).stop 2024-02-20T07:50:04.564 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:04.562+0000 7f9afe4a4700 1 -- 172.21.15.183:0/326266549 shutdown_connections 2024-02-20T07:50:04.564 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:04.562+0000 7f9afe4a4700 1 --2- 172.21.15.183:0/326266549 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f9af80fcbd0 0x7f9af80fcfe0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:50:04.564 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:04.562+0000 7f9afe4a4700 1 -- 172.21.15.183:0/326266549 >> 172.21.15.183:0/326266549 conn(0x7f9af80f8290 msgr2=0x7f9af80fa6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:50:04.565 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:04.562+0000 7f9afe4a4700 1 -- 172.21.15.183:0/326266549 shutdown_connections 2024-02-20T07:50:04.565 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:04.562+0000 7f9afe4a4700 1 -- 172.21.15.183:0/326266549 wait complete. 2024-02-20T07:50:04.565 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:04.562+0000 7f9afe4a4700 1 Processor -- start 2024-02-20T07:50:04.565 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:04.562+0000 7f9afe4a4700 1 -- start start 2024-02-20T07:50:04.565 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:04.562+0000 7f9afe4a4700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f9af80fcbd0 0x7f9af810cec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:50:04.565 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:04.562+0000 7f9afe4a4700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9af810d400 con 0x7f9af80fcbd0 2024-02-20T07:50:04.565 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:04.563+0000 7f9af7fff700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f9af80fcbd0 0x7f9af810cec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:50:04.565 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:04.563+0000 7f9af7fff700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f9af80fcbd0 0x7f9af810cec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.183:35482/0 (socket says 172.21.15.183:35482) 2024-02-20T07:50:04.565 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:04.563+0000 7f9af7fff700 1 -- 172.21.15.183:0/2936460452 learned_addr learned my addr 172.21.15.183:0/2936460452 (peer_addr_for_me v2:172.21.15.183:0/0) 2024-02-20T07:50:04.567 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:04.563+0000 7f9af7fff700 1 -- 172.21.15.183:0/2936460452 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ae8005040 con 0x7f9af80fcbd0 2024-02-20T07:50:04.567 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:04.563+0000 7f9af7fff700 1 --2- 172.21.15.183:0/2936460452 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f9af80fcbd0 0x7f9af810cec0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f9ae8009680 tx=0x7f9ae8009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:50:04.567 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:04.563+0000 7f9af4ff9700 1 -- 172.21.15.183:0/2936460452 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9ae800bed0 con 0x7f9af80fcbd0 2024-02-20T07:50:04.567 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:04.563+0000 7f9af4ff9700 1 -- 172.21.15.183:0/2936460452 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9ae800b5c0 con 0x7f9af80fcbd0 2024-02-20T07:50:04.567 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:04.564+0000 7f9af4ff9700 1 -- 172.21.15.183:0/2936460452 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9ae801c5b0 con 0x7f9af80fcbd0 2024-02-20T07:50:04.567 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:04.564+0000 7f9afe4a4700 1 -- 172.21.15.183:0/2936460452 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9af810d600 con 0x7f9af80fcbd0 2024-02-20T07:50:04.567 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:04.564+0000 7f9afe4a4700 1 -- 172.21.15.183:0/2936460452 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9af810e2a0 con 0x7f9af80fcbd0 2024-02-20T07:50:04.567 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:04.564+0000 7f9af4ff9700 1 -- 172.21.15.183:0/2936460452 <== mon.0 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 13) v1 ==== 43425+0+0 (secure 0 0 0) 0x7f9ae8013420 con 0x7f9af80fcbd0 2024-02-20T07:50:04.567 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:04.564+0000 7f9afe4a4700 1 -- 172.21.15.183:0/2936460452 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9af805f5c0 con 0x7f9af80fcbd0 2024-02-20T07:50:04.567 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:04.564+0000 7f9af4ff9700 1 --2- 172.21.15.183:0/2936460452 >> [v2:172.21.15.133:6800/3944459195,v1:172.21.15.133:6801/3944459195] conn(0x7f9ae0036820 0x7f9ae0038cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:50:04.567 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:04.565+0000 7f9af4ff9700 1 -- 172.21.15.183:0/2936460452 <== mon.0 v2:172.21.15.133:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f9ae8049b60 con 0x7f9af80fcbd0 2024-02-20T07:50:04.567 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:04.565+0000 7f9af77fe700 1 --2- 172.21.15.183:0/2936460452 >> [v2:172.21.15.133:6800/3944459195,v1:172.21.15.133:6801/3944459195] conn(0x7f9ae0036820 0x7f9ae0038cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:50:04.567 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:04.565+0000 7f9af77fe700 1 --2- 172.21.15.183:0/2936460452 >> [v2:172.21.15.133:6800/3944459195,v1:172.21.15.133:6801/3944459195] conn(0x7f9ae0036820 0x7f9ae0038cd0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f9aec009990 tx=0x7f9aec006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:50:04.571 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:04.569+0000 7f9af4ff9700 1 -- 172.21.15.183:0/2936460452 <== mon.0 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f9ae8025400 con 0x7f9af80fcbd0 2024-02-20T07:50:04.825 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:04.823+0000 7f9afe4a4700 1 -- 172.21.15.183:0/2936460452 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f9af810edf0 con 0x7f9af80fcbd0 2024-02-20T07:50:04.825 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:04.824+0000 7f9af4ff9700 1 -- 172.21.15.183:0/2936460452 <== mon.0 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+743 (secure 0 0 0) 0x7f9ae8004d30 con 0x7f9af80fcbd0 2024-02-20T07:50:04.825 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-02-20T07:50:04.825 INFO:teuthology.orchestra.run.smithi183.stdout:{"epoch":1,"fsid":"4537327c-cfc4-11ee-95bb-87774f69a715","modified":"2024-02-20T07:48:10.704626Z","created":"2024-02-20T07:48:10.704626Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi133","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:3300","nonce":0},{"type":"v1","addr":"172.21.15.133:6789","nonce":0}]},"addr":"172.21.15.133:6789/0","public_addr":"172.21.15.133:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T07:50:04.827 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:04.826+0000 7f9afe4a4700 1 -- 172.21.15.183:0/2936460452 >> [v2:172.21.15.133:6800/3944459195,v1:172.21.15.133:6801/3944459195] conn(0x7f9ae0036820 msgr2=0x7f9ae0038cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:50:04.827 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:04.826+0000 7f9afe4a4700 1 --2- 172.21.15.183:0/2936460452 >> [v2:172.21.15.133:6800/3944459195,v1:172.21.15.133:6801/3944459195] conn(0x7f9ae0036820 0x7f9ae0038cd0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f9aec009990 tx=0x7f9aec006e40 comp rx=0 tx=0).stop 2024-02-20T07:50:04.827 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:04.826+0000 7f9afe4a4700 1 -- 172.21.15.183:0/2936460452 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f9af80fcbd0 msgr2=0x7f9af810cec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:50:04.827 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:04.826+0000 7f9afe4a4700 1 --2- 172.21.15.183:0/2936460452 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f9af80fcbd0 0x7f9af810cec0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f9ae8009680 tx=0x7f9ae8009e30 comp rx=0 tx=0).stop 2024-02-20T07:50:04.828 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:04.826+0000 7f9afe4a4700 1 -- 172.21.15.183:0/2936460452 shutdown_connections 2024-02-20T07:50:04.828 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:04.826+0000 7f9afe4a4700 1 --2- 172.21.15.183:0/2936460452 >> [v2:172.21.15.133:6800/3944459195,v1:172.21.15.133:6801/3944459195] conn(0x7f9ae0036820 0x7f9ae0038cd0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:50:04.828 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:04.826+0000 7f9afe4a4700 1 --2- 172.21.15.183:0/2936460452 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f9af80fcbd0 0x7f9af810cec0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:50:04.828 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:04.826+0000 7f9afe4a4700 1 -- 172.21.15.183:0/2936460452 >> 172.21.15.183:0/2936460452 conn(0x7f9af80f8290 msgr2=0x7f9af80f9000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:50:04.828 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:04.826+0000 7f9afe4a4700 1 -- 172.21.15.183:0/2936460452 shutdown_connections 2024-02-20T07:50:04.828 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:04.826+0000 7f9afe4a4700 1 -- 172.21.15.183:0/2936460452 wait complete. 2024-02-20T07:50:04.828 INFO:teuthology.orchestra.run.smithi183.stderr:dumped monmap epoch 1 2024-02-20T07:50:05.195 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:50:05 smithi133 ceph-mon[29117]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T07:50:05.195 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:50:05 smithi133 ceph-mon[29117]: from='client.? 172.21.15.183:0/2936460452' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T07:50:05.195 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:50:05 smithi133 ceph-mon[29117]: from='mgr.14162 172.21.15.133:0/3642562278' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:50:05.195 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:50:05 smithi133 ceph-mon[29117]: from='mgr.14162 172.21.15.133:0/3642562278' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:50:05.196 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:50:05 smithi133 ceph-mon[29117]: from='mgr.14162 172.21.15.133:0/3642562278' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:50:05.196 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:50:05 smithi133 ceph-mon[29117]: from='mgr.14162 172.21.15.133:0/3642562278' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:50:05.196 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:50:05 smithi133 ceph-mon[29117]: from='mgr.14162 172.21.15.133:0/3642562278' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:50:05.196 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:50:05 smithi133 ceph-mon[29117]: from='mgr.14162 172.21.15.133:0/3642562278' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:50:06.270 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T07:50:06.271 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4537327c-cfc4-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T07:50:06.359 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:50:06 smithi133 ceph-mon[29117]: Deploying daemon prometheus.smithi133 on smithi133 2024-02-20T07:50:06.520 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-20T07:50:06.568 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-20T07:50:07.427 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:50:07 smithi133 ceph-mon[29117]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T07:50:07.940 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:07.938+0000 7f511b642700 1 -- 172.21.15.183:0/2196000264 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f51140fcbc0 msgr2=0x7f51140fcfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:50:07.940 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:07.938+0000 7f511b642700 1 --2- 172.21.15.183:0/2196000264 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f51140fcbc0 0x7f51140fcfd0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f51040048c0 tx=0x7f5104009e30 comp rx=0 tx=0).stop 2024-02-20T07:50:07.940 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:07.938+0000 7f511b642700 1 -- 172.21.15.183:0/2196000264 shutdown_connections 2024-02-20T07:50:07.941 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:07.938+0000 7f511b642700 1 --2- 172.21.15.183:0/2196000264 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f51140fcbc0 0x7f51140fcfd0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:50:07.941 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:07.938+0000 7f511b642700 1 -- 172.21.15.183:0/2196000264 >> 172.21.15.183:0/2196000264 conn(0x7f51140f82a0 msgr2=0x7f51140fa6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:50:07.941 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:07.938+0000 7f511b642700 1 -- 172.21.15.183:0/2196000264 shutdown_connections 2024-02-20T07:50:07.941 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:07.938+0000 7f511b642700 1 -- 172.21.15.183:0/2196000264 wait complete. 2024-02-20T07:50:07.941 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:07.939+0000 7f511b642700 1 Processor -- start 2024-02-20T07:50:07.941 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:07.939+0000 7f511b642700 1 -- start start 2024-02-20T07:50:07.941 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:07.939+0000 7f511b642700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f51140fcbc0 0x7f511410ce90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:50:07.941 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:07.939+0000 7f511b642700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f511410d3d0 con 0x7f51140fcbc0 2024-02-20T07:50:07.943 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:07.939+0000 7f51193de700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f51140fcbc0 0x7f511410ce90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:50:07.943 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:07.939+0000 7f51193de700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f51140fcbc0 0x7f511410ce90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.183:35506/0 (socket says 172.21.15.183:35506) 2024-02-20T07:50:07.943 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:07.939+0000 7f51193de700 1 -- 172.21.15.183:0/2094345143 learned_addr learned my addr 172.21.15.183:0/2094345143 (peer_addr_for_me v2:172.21.15.183:0/0) 2024-02-20T07:50:07.943 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:07.939+0000 7f51193de700 1 -- 172.21.15.183:0/2094345143 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5104005040 con 0x7f51140fcbc0 2024-02-20T07:50:07.943 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:07.940+0000 7f51193de700 1 --2- 172.21.15.183:0/2094345143 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f51140fcbc0 0x7f511410ce90 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f5104009b80 tx=0x7f5104009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:50:07.943 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:07.940+0000 7f5109ffb700 1 -- 172.21.15.183:0/2094345143 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f510400b5c0 con 0x7f51140fcbc0 2024-02-20T07:50:07.943 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:07.940+0000 7f5109ffb700 1 -- 172.21.15.183:0/2094345143 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f510400be70 con 0x7f51140fcbc0 2024-02-20T07:50:07.943 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:07.940+0000 7f5109ffb700 1 -- 172.21.15.183:0/2094345143 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f510401c660 con 0x7f51140fcbc0 2024-02-20T07:50:07.943 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:07.940+0000 7f511b642700 1 -- 172.21.15.183:0/2094345143 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f511410d5d0 con 0x7f51140fcbc0 2024-02-20T07:50:07.944 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:07.940+0000 7f511b642700 1 -- 172.21.15.183:0/2094345143 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f511410e270 con 0x7f51140fcbc0 2024-02-20T07:50:07.944 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:07.941+0000 7f5109ffb700 1 -- 172.21.15.183:0/2094345143 <== mon.0 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 13) v1 ==== 43425+0+0 (secure 0 0 0) 0x7f5104004d30 con 0x7f51140fcbc0 2024-02-20T07:50:07.944 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:07.941+0000 7f511b642700 1 -- 172.21.15.183:0/2094345143 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f511405f5c0 con 0x7f51140fcbc0 2024-02-20T07:50:07.944 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:07.941+0000 7f5109ffb700 1 --2- 172.21.15.183:0/2094345143 >> [v2:172.21.15.133:6800/3944459195,v1:172.21.15.133:6801/3944459195] conn(0x7f5100036870 0x7f5100038d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:50:07.944 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:07.941+0000 7f5109ffb700 1 -- 172.21.15.183:0/2094345143 <== mon.0 v2:172.21.15.133:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f5104049ab0 con 0x7f51140fcbc0 2024-02-20T07:50:07.944 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:07.941+0000 7f5118bdd700 1 --2- 172.21.15.183:0/2094345143 >> [v2:172.21.15.133:6800/3944459195,v1:172.21.15.133:6801/3944459195] conn(0x7f5100036870 0x7f5100038d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:50:07.944 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:07.942+0000 7f5118bdd700 1 --2- 172.21.15.183:0/2094345143 >> [v2:172.21.15.133:6800/3944459195,v1:172.21.15.133:6801/3944459195] conn(0x7f5100036870 0x7f5100038d20 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f5110009990 tx=0x7f5110006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:50:07.947 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:07.946+0000 7f5109ffb700 1 -- 172.21.15.183:0/2094345143 <== mon.0 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f5104025400 con 0x7f51140fcbc0 2024-02-20T07:50:08.203 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:08.202+0000 7f511b642700 1 -- 172.21.15.183:0/2094345143 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f511410edc0 con 0x7f51140fcbc0 2024-02-20T07:50:08.204 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:08.203+0000 7f5109ffb700 1 -- 172.21.15.183:0/2094345143 <== mon.0 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+743 (secure 0 0 0) 0x7f5104011370 con 0x7f51140fcbc0 2024-02-20T07:50:08.204 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-02-20T07:50:08.204 INFO:teuthology.orchestra.run.smithi183.stdout:{"epoch":1,"fsid":"4537327c-cfc4-11ee-95bb-87774f69a715","modified":"2024-02-20T07:48:10.704626Z","created":"2024-02-20T07:48:10.704626Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi133","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:3300","nonce":0},{"type":"v1","addr":"172.21.15.133:6789","nonce":0}]},"addr":"172.21.15.133:6789/0","public_addr":"172.21.15.133:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T07:50:08.206 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:08.204+0000 7f511b642700 1 -- 172.21.15.183:0/2094345143 >> [v2:172.21.15.133:6800/3944459195,v1:172.21.15.133:6801/3944459195] conn(0x7f5100036870 msgr2=0x7f5100038d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:50:08.206 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:08.204+0000 7f511b642700 1 --2- 172.21.15.183:0/2094345143 >> [v2:172.21.15.133:6800/3944459195,v1:172.21.15.133:6801/3944459195] conn(0x7f5100036870 0x7f5100038d20 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f5110009990 tx=0x7f5110006e40 comp rx=0 tx=0).stop 2024-02-20T07:50:08.206 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:08.204+0000 7f511b642700 1 -- 172.21.15.183:0/2094345143 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f51140fcbc0 msgr2=0x7f511410ce90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:50:08.206 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:08.204+0000 7f511b642700 1 --2- 172.21.15.183:0/2094345143 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f51140fcbc0 0x7f511410ce90 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f5104009b80 tx=0x7f5104009e30 comp rx=0 tx=0).stop 2024-02-20T07:50:08.207 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:08.205+0000 7f511b642700 1 -- 172.21.15.183:0/2094345143 shutdown_connections 2024-02-20T07:50:08.207 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:08.205+0000 7f511b642700 1 --2- 172.21.15.183:0/2094345143 >> [v2:172.21.15.133:6800/3944459195,v1:172.21.15.133:6801/3944459195] conn(0x7f5100036870 0x7f5100038d20 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:50:08.207 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:08.205+0000 7f511b642700 1 --2- 172.21.15.183:0/2094345143 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f51140fcbc0 0x7f511410ce90 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:50:08.207 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:08.205+0000 7f511b642700 1 -- 172.21.15.183:0/2094345143 >> 172.21.15.183:0/2094345143 conn(0x7f51140f82a0 msgr2=0x7f51140f8fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:50:08.207 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:08.205+0000 7f511b642700 1 -- 172.21.15.183:0/2094345143 shutdown_connections 2024-02-20T07:50:08.207 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:08.205+0000 7f511b642700 1 -- 172.21.15.183:0/2094345143 wait complete. 2024-02-20T07:50:08.207 INFO:teuthology.orchestra.run.smithi183.stderr:dumped monmap epoch 1 2024-02-20T07:50:09.492 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:50:09 smithi133 ceph-mon[29117]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T07:50:09.492 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:50:09 smithi133 ceph-mon[29117]: from='client.? 172.21.15.183:0/2094345143' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T07:50:09.821 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T07:50:09.821 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4537327c-cfc4-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T07:50:10.071 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-20T07:50:10.117 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-20T07:50:11.242 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:50:10 smithi133 ceph-mon[29117]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T07:50:11.242 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:50:10 smithi133 ceph-mon[29117]: from='mgr.14162 172.21.15.133:0/3642562278' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:50:11.486 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:11.484+0000 7fae9d7e8700 1 -- 172.21.15.183:0/2224548967 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fae980fcbb0 msgr2=0x7fae980fcfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:50:11.487 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:11.484+0000 7fae9d7e8700 1 --2- 172.21.15.183:0/2224548967 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fae980fcbb0 0x7fae980fcfc0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fae880048c0 tx=0x7fae88009e30 comp rx=0 tx=0).stop 2024-02-20T07:50:11.487 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:11.484+0000 7fae9d7e8700 1 -- 172.21.15.183:0/2224548967 shutdown_connections 2024-02-20T07:50:11.487 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:11.484+0000 7fae9d7e8700 1 --2- 172.21.15.183:0/2224548967 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fae980fcbb0 0x7fae980fcfc0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:50:11.487 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:11.484+0000 7fae9d7e8700 1 -- 172.21.15.183:0/2224548967 >> 172.21.15.183:0/2224548967 conn(0x7fae980f8330 msgr2=0x7fae980fa740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:50:11.487 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:11.484+0000 7fae9d7e8700 1 -- 172.21.15.183:0/2224548967 shutdown_connections 2024-02-20T07:50:11.487 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:11.484+0000 7fae9d7e8700 1 -- 172.21.15.183:0/2224548967 wait complete. 2024-02-20T07:50:11.487 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:11.484+0000 7fae9d7e8700 1 Processor -- start 2024-02-20T07:50:11.487 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:11.485+0000 7fae9d7e8700 1 -- start start 2024-02-20T07:50:11.487 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:11.485+0000 7fae9d7e8700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fae980fcbb0 0x7fae9810cec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:50:11.487 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:11.485+0000 7fae9d7e8700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae9810d400 con 0x7fae980fcbb0 2024-02-20T07:50:11.487 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:11.485+0000 7fae96ffd700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fae980fcbb0 0x7fae9810cec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:50:11.487 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:11.485+0000 7fae96ffd700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fae980fcbb0 0x7fae9810cec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.183:43982/0 (socket says 172.21.15.183:43982) 2024-02-20T07:50:11.487 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:11.485+0000 7fae96ffd700 1 -- 172.21.15.183:0/2942458102 learned_addr learned my addr 172.21.15.183:0/2942458102 (peer_addr_for_me v2:172.21.15.183:0/0) 2024-02-20T07:50:11.488 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:11.485+0000 7fae96ffd700 1 -- 172.21.15.183:0/2942458102 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae88005040 con 0x7fae980fcbb0 2024-02-20T07:50:11.492 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:11.486+0000 7fae96ffd700 1 --2- 172.21.15.183:0/2942458102 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fae980fcbb0 0x7fae9810cec0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fae88009680 tx=0x7fae88009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:50:11.492 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:11.486+0000 7fae8ffff700 1 -- 172.21.15.183:0/2942458102 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fae8800b5c0 con 0x7fae980fcbb0 2024-02-20T07:50:11.492 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:11.486+0000 7fae9d7e8700 1 -- 172.21.15.183:0/2942458102 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fae9810d600 con 0x7fae980fcbb0 2024-02-20T07:50:11.492 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:11.486+0000 7fae9d7e8700 1 -- 172.21.15.183:0/2942458102 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fae9810e2a0 con 0x7fae980fcbb0 2024-02-20T07:50:11.493 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:11.486+0000 7fae8ffff700 1 -- 172.21.15.183:0/2942458102 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fae88004e60 con 0x7fae980fcbb0 2024-02-20T07:50:11.493 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:11.486+0000 7fae8ffff700 1 -- 172.21.15.183:0/2942458102 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fae880255a0 con 0x7fae980fcbb0 2024-02-20T07:50:11.493 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:11.487+0000 7fae8ffff700 1 -- 172.21.15.183:0/2942458102 <== mon.0 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 13) v1 ==== 43425+0+0 (secure 0 0 0) 0x7fae8801c3f0 con 0x7fae980fcbb0 2024-02-20T07:50:11.493 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:11.487+0000 7fae8ffff700 1 --2- 172.21.15.183:0/2942458102 >> [v2:172.21.15.133:6800/3944459195,v1:172.21.15.133:6801/3944459195] conn(0x7fae84036870 0x7fae84038d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:50:11.493 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:11.487+0000 7fae8ffff700 1 -- 172.21.15.183:0/2942458102 <== mon.0 v2:172.21.15.133:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7fae88049ee0 con 0x7fae980fcbb0 2024-02-20T07:50:11.493 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:11.487+0000 7fae9d7e8700 1 -- 172.21.15.183:0/2942458102 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fae780052e0 con 0x7fae980fcbb0 2024-02-20T07:50:11.493 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:11.487+0000 7fae8e7fc700 1 --2- 172.21.15.183:0/2942458102 >> [v2:172.21.15.133:6800/3944459195,v1:172.21.15.133:6801/3944459195] conn(0x7fae84036870 0x7fae84038d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:50:11.493 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:11.487+0000 7fae8e7fc700 1 --2- 172.21.15.183:0/2942458102 >> [v2:172.21.15.133:6800/3944459195,v1:172.21.15.133:6801/3944459195] conn(0x7fae84036870 0x7fae84038d20 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fae80009990 tx=0x7fae80006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:50:11.494 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:11.492+0000 7fae8ffff700 1 -- 172.21.15.183:0/2942458102 <== mon.0 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7fae8801a6b0 con 0x7fae980fcbb0 2024-02-20T07:50:11.748 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:11.746+0000 7fae9d7e8700 1 -- 172.21.15.183:0/2942458102 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fae78005590 con 0x7fae980fcbb0 2024-02-20T07:50:11.749 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:11.747+0000 7fae8ffff700 1 -- 172.21.15.183:0/2942458102 <== mon.0 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+743 (secure 0 0 0) 0x7fae8801a2f0 con 0x7fae980fcbb0 2024-02-20T07:50:11.749 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-02-20T07:50:11.749 INFO:teuthology.orchestra.run.smithi183.stdout:{"epoch":1,"fsid":"4537327c-cfc4-11ee-95bb-87774f69a715","modified":"2024-02-20T07:48:10.704626Z","created":"2024-02-20T07:48:10.704626Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi133","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:3300","nonce":0},{"type":"v1","addr":"172.21.15.133:6789","nonce":0}]},"addr":"172.21.15.133:6789/0","public_addr":"172.21.15.133:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T07:50:11.751 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:11.749+0000 7fae9d7e8700 1 -- 172.21.15.183:0/2942458102 >> [v2:172.21.15.133:6800/3944459195,v1:172.21.15.133:6801/3944459195] conn(0x7fae84036870 msgr2=0x7fae84038d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:50:11.751 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:11.749+0000 7fae9d7e8700 1 --2- 172.21.15.183:0/2942458102 >> [v2:172.21.15.133:6800/3944459195,v1:172.21.15.133:6801/3944459195] conn(0x7fae84036870 0x7fae84038d20 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fae80009990 tx=0x7fae80006e40 comp rx=0 tx=0).stop 2024-02-20T07:50:11.751 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:11.749+0000 7fae9d7e8700 1 -- 172.21.15.183:0/2942458102 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fae980fcbb0 msgr2=0x7fae9810cec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:50:11.751 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:11.749+0000 7fae9d7e8700 1 --2- 172.21.15.183:0/2942458102 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fae980fcbb0 0x7fae9810cec0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fae88009680 tx=0x7fae88009e30 comp rx=0 tx=0).stop 2024-02-20T07:50:11.751 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:11.749+0000 7fae9d7e8700 1 -- 172.21.15.183:0/2942458102 shutdown_connections 2024-02-20T07:50:11.751 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:11.749+0000 7fae9d7e8700 1 --2- 172.21.15.183:0/2942458102 >> [v2:172.21.15.133:6800/3944459195,v1:172.21.15.133:6801/3944459195] conn(0x7fae84036870 0x7fae84038d20 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:50:11.751 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:11.749+0000 7fae9d7e8700 1 --2- 172.21.15.183:0/2942458102 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fae980fcbb0 0x7fae9810cec0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:50:11.751 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:11.749+0000 7fae9d7e8700 1 -- 172.21.15.183:0/2942458102 >> 172.21.15.183:0/2942458102 conn(0x7fae980f8330 msgr2=0x7fae980f90b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:50:11.751 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:11.749+0000 7fae9d7e8700 1 -- 172.21.15.183:0/2942458102 shutdown_connections 2024-02-20T07:50:11.751 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:11.749+0000 7fae9d7e8700 1 -- 172.21.15.183:0/2942458102 wait complete. 2024-02-20T07:50:11.752 INFO:teuthology.orchestra.run.smithi183.stderr:dumped monmap epoch 1 2024-02-20T07:50:12.864 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:50:12 smithi133 ceph-mon[29117]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T07:50:12.865 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:50:12 smithi133 ceph-mon[29117]: from='client.? 172.21.15.183:0/2942458102' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T07:50:13.335 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T07:50:13.336 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4537327c-cfc4-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T07:50:13.585 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-20T07:50:13.631 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-20T07:50:14.980 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:14.977+0000 7fd35d505700 1 -- 172.21.15.183:0/492793390 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fd3580fcbc0 msgr2=0x7fd3580fcfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:50:14.981 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:14.977+0000 7fd35d505700 1 --2- 172.21.15.183:0/492793390 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fd3580fcbc0 0x7fd3580fcfd0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fd3480048c0 tx=0x7fd348009e30 comp rx=0 tx=0).stop 2024-02-20T07:50:14.981 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:14.977+0000 7fd35d505700 1 -- 172.21.15.183:0/492793390 shutdown_connections 2024-02-20T07:50:14.981 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:14.977+0000 7fd35d505700 1 --2- 172.21.15.183:0/492793390 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fd3580fcbc0 0x7fd3580fcfd0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:50:14.981 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:14.977+0000 7fd35d505700 1 -- 172.21.15.183:0/492793390 >> 172.21.15.183:0/492793390 conn(0x7fd3580f82a0 msgr2=0x7fd3580fa6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:50:14.981 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:14.977+0000 7fd35d505700 1 -- 172.21.15.183:0/492793390 shutdown_connections 2024-02-20T07:50:14.981 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:14.977+0000 7fd35d505700 1 -- 172.21.15.183:0/492793390 wait complete. 2024-02-20T07:50:14.981 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:14.978+0000 7fd35d505700 1 Processor -- start 2024-02-20T07:50:14.981 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:14.978+0000 7fd35d505700 1 -- start start 2024-02-20T07:50:14.981 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:14.978+0000 7fd35d505700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fd3580fcbc0 0x7fd35810ce90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:50:14.981 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:14.978+0000 7fd35d505700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd35810d3d0 con 0x7fd3580fcbc0 2024-02-20T07:50:14.981 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:14.978+0000 7fd356ffd700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fd3580fcbc0 0x7fd35810ce90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:50:14.981 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:14.978+0000 7fd356ffd700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fd3580fcbc0 0x7fd35810ce90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.183:44006/0 (socket says 172.21.15.183:44006) 2024-02-20T07:50:14.981 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:14.978+0000 7fd356ffd700 1 -- 172.21.15.183:0/866860728 learned_addr learned my addr 172.21.15.183:0/866860728 (peer_addr_for_me v2:172.21.15.183:0/0) 2024-02-20T07:50:14.983 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:14.978+0000 7fd356ffd700 1 -- 172.21.15.183:0/866860728 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd348005040 con 0x7fd3580fcbc0 2024-02-20T07:50:14.983 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:14.979+0000 7fd356ffd700 1 --2- 172.21.15.183:0/866860728 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fd3580fcbc0 0x7fd35810ce90 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fd348009680 tx=0x7fd348009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:50:14.983 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:14.979+0000 7fd34ffff700 1 -- 172.21.15.183:0/866860728 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd34800bed0 con 0x7fd3580fcbc0 2024-02-20T07:50:14.983 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:14.979+0000 7fd35d505700 1 -- 172.21.15.183:0/866860728 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd35810d5d0 con 0x7fd3580fcbc0 2024-02-20T07:50:14.983 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:14.979+0000 7fd34ffff700 1 -- 172.21.15.183:0/866860728 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd34800b5c0 con 0x7fd3580fcbc0 2024-02-20T07:50:14.983 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:14.979+0000 7fd34ffff700 1 -- 172.21.15.183:0/866860728 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd34801c5b0 con 0x7fd3580fcbc0 2024-02-20T07:50:14.983 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:14.979+0000 7fd35d505700 1 -- 172.21.15.183:0/866860728 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd35810e270 con 0x7fd3580fcbc0 2024-02-20T07:50:14.983 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:14.980+0000 7fd34ffff700 1 -- 172.21.15.183:0/866860728 <== mon.0 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 13) v1 ==== 43425+0+0 (secure 0 0 0) 0x7fd348013420 con 0x7fd3580fcbc0 2024-02-20T07:50:14.983 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:14.980+0000 7fd34ffff700 1 --2- 172.21.15.183:0/866860728 >> [v2:172.21.15.133:6800/3944459195,v1:172.21.15.133:6801/3944459195] conn(0x7fd344036820 0x7fd344038cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:50:14.983 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:14.980+0000 7fd34ffff700 1 -- 172.21.15.183:0/866860728 <== mon.0 v2:172.21.15.133:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7fd348049ba0 con 0x7fd3580fcbc0 2024-02-20T07:50:14.984 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:14.980+0000 7fd35d505700 1 -- 172.21.15.183:0/866860728 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd35805f5c0 con 0x7fd3580fcbc0 2024-02-20T07:50:14.984 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:14.980+0000 7fd34e7fc700 1 --2- 172.21.15.183:0/866860728 >> [v2:172.21.15.133:6800/3944459195,v1:172.21.15.133:6801/3944459195] conn(0x7fd344036820 0x7fd344038cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:50:14.984 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:14.981+0000 7fd34e7fc700 1 --2- 172.21.15.183:0/866860728 >> [v2:172.21.15.133:6800/3944459195,v1:172.21.15.133:6801/3944459195] conn(0x7fd344036820 0x7fd344038cd0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fd340009990 tx=0x7fd340006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:50:14.987 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:14.985+0000 7fd34ffff700 1 -- 172.21.15.183:0/866860728 <== mon.0 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7fd348025400 con 0x7fd3580fcbc0 2024-02-20T07:50:15.185 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:50:14 smithi133 ceph-mon[29117]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T07:50:15.242 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:15.240+0000 7fd35d505700 1 -- 172.21.15.183:0/866860728 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fd35804bed0 con 0x7fd3580fcbc0 2024-02-20T07:50:15.243 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:15.240+0000 7fd34ffff700 1 -- 172.21.15.183:0/866860728 <== mon.0 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+743 (secure 0 0 0) 0x7fd348004d30 con 0x7fd3580fcbc0 2024-02-20T07:50:15.243 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-02-20T07:50:15.243 INFO:teuthology.orchestra.run.smithi183.stdout:{"epoch":1,"fsid":"4537327c-cfc4-11ee-95bb-87774f69a715","modified":"2024-02-20T07:48:10.704626Z","created":"2024-02-20T07:48:10.704626Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi133","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:3300","nonce":0},{"type":"v1","addr":"172.21.15.133:6789","nonce":0}]},"addr":"172.21.15.133:6789/0","public_addr":"172.21.15.133:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T07:50:15.245 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:15.242+0000 7fd35d505700 1 -- 172.21.15.183:0/866860728 >> [v2:172.21.15.133:6800/3944459195,v1:172.21.15.133:6801/3944459195] conn(0x7fd344036820 msgr2=0x7fd344038cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:50:15.245 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:15.242+0000 7fd35d505700 1 --2- 172.21.15.183:0/866860728 >> [v2:172.21.15.133:6800/3944459195,v1:172.21.15.133:6801/3944459195] conn(0x7fd344036820 0x7fd344038cd0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fd340009990 tx=0x7fd340006e40 comp rx=0 tx=0).stop 2024-02-20T07:50:15.245 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:15.242+0000 7fd35d505700 1 -- 172.21.15.183:0/866860728 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fd3580fcbc0 msgr2=0x7fd35810ce90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:50:15.245 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:15.242+0000 7fd35d505700 1 --2- 172.21.15.183:0/866860728 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fd3580fcbc0 0x7fd35810ce90 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fd348009680 tx=0x7fd348009e30 comp rx=0 tx=0).stop 2024-02-20T07:50:15.245 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:15.242+0000 7fd35d505700 1 -- 172.21.15.183:0/866860728 shutdown_connections 2024-02-20T07:50:15.245 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:15.242+0000 7fd35d505700 1 --2- 172.21.15.183:0/866860728 >> [v2:172.21.15.133:6800/3944459195,v1:172.21.15.133:6801/3944459195] conn(0x7fd344036820 0x7fd344038cd0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:50:15.245 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:15.242+0000 7fd35d505700 1 --2- 172.21.15.183:0/866860728 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fd3580fcbc0 0x7fd35810ce90 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:50:15.246 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:15.242+0000 7fd35d505700 1 -- 172.21.15.183:0/866860728 >> 172.21.15.183:0/866860728 conn(0x7fd3580f82a0 msgr2=0x7fd3580f8fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:50:15.246 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:15.242+0000 7fd35d505700 1 -- 172.21.15.183:0/866860728 shutdown_connections 2024-02-20T07:50:15.246 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:15.242+0000 7fd35d505700 1 -- 172.21.15.183:0/866860728 wait complete. 2024-02-20T07:50:15.246 INFO:teuthology.orchestra.run.smithi183.stderr:dumped monmap epoch 1 2024-02-20T07:50:15.828 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:50:15 smithi133 ceph-mon[29117]: from='client.? 172.21.15.183:0/866860728' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T07:50:16.836 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T07:50:16.837 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4537327c-cfc4-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T07:50:17.091 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-20T07:50:17.138 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-20T07:50:17.242 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:50:16 smithi133 ceph-mon[29117]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T07:50:17.242 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:50:16 smithi133 ceph-mon[29117]: from='mgr.14162 172.21.15.133:0/3642562278' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:50:17.242 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:50:16 smithi133 ceph-mon[29117]: from='mgr.14162 172.21.15.133:0/3642562278' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:50:17.242 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:50:16 smithi133 ceph-mon[29117]: from='mgr.14162 172.21.15.133:0/3642562278' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:50:17.242 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:50:16 smithi133 ceph-mon[29117]: from='mgr.14162 172.21.15.133:0/3642562278' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-02-20T07:50:18.225 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:50:17 smithi133 ceph-mon[29117]: from='mgr.14162 172.21.15.133:0/3642562278' entity='mgr.smithi133.mrvcbp' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-02-20T07:50:18.226 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:50:17 smithi133 ceph-mon[29117]: mgrmap e14: smithi133.mrvcbp(active, since 57s) 2024-02-20T07:50:18.408 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:18.404+0000 7fb6b4583700 1 -- 172.21.15.183:0/2407337605 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb6ac0fbad0 msgr2=0x7fb6ac0fbee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:50:18.409 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:18.404+0000 7fb6b4583700 1 --2- 172.21.15.183:0/2407337605 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb6ac0fbad0 0x7fb6ac0fbee0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fb6a4009c60 tx=0x7fb6a4004d10 comp rx=0 tx=0).stop 2024-02-20T07:50:18.409 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:18.404+0000 7fb6b4583700 1 -- 172.21.15.183:0/2407337605 shutdown_connections 2024-02-20T07:50:18.409 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:18.404+0000 7fb6b4583700 1 --2- 172.21.15.183:0/2407337605 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb6ac0fbad0 0x7fb6ac0fbee0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:50:18.409 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:18.404+0000 7fb6b4583700 1 -- 172.21.15.183:0/2407337605 >> 172.21.15.183:0/2407337605 conn(0x7fb6ac0f7190 msgr2=0x7fb6ac0f95c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:50:18.409 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:18.404+0000 7fb6b4583700 1 -- 172.21.15.183:0/2407337605 shutdown_connections 2024-02-20T07:50:18.409 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:18.404+0000 7fb6b4583700 1 -- 172.21.15.183:0/2407337605 wait complete. 2024-02-20T07:50:18.409 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:18.405+0000 7fb6b4583700 1 Processor -- start 2024-02-20T07:50:18.409 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:18.405+0000 7fb6b4583700 1 -- start start 2024-02-20T07:50:18.409 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:18.405+0000 7fb6b4583700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb6ac0fbad0 0x7fb6ac10fb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:50:18.409 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:18.405+0000 7fb6b4583700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6ac065dc0 con 0x7fb6ac0fbad0 2024-02-20T07:50:18.410 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:18.405+0000 7fb6b231f700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb6ac0fbad0 0x7fb6ac10fb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:50:18.410 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:18.405+0000 7fb6b231f700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb6ac0fbad0 0x7fb6ac10fb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.183:43488/0 (socket says 172.21.15.183:43488) 2024-02-20T07:50:18.410 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:18.405+0000 7fb6b231f700 1 -- 172.21.15.183:0/840586671 learned_addr learned my addr 172.21.15.183:0/840586671 (peer_addr_for_me v2:172.21.15.183:0/0) 2024-02-20T07:50:18.410 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:18.405+0000 7fb6b231f700 1 -- 172.21.15.183:0/840586671 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6a4009940 con 0x7fb6ac0fbad0 2024-02-20T07:50:18.410 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:18.406+0000 7fb6b231f700 1 --2- 172.21.15.183:0/840586671 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb6ac0fbad0 0x7fb6ac10fb70 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fb6a4015040 tx=0x7fb6a4004d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:50:18.410 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:18.406+0000 7fb6aaffd700 1 -- 172.21.15.183:0/840586671 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb6a4007c80 con 0x7fb6ac0fbad0 2024-02-20T07:50:18.410 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:18.406+0000 7fb6b4583700 1 -- 172.21.15.183:0/840586671 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb6ac065fc0 con 0x7fb6ac0fbad0 2024-02-20T07:50:18.410 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:18.406+0000 7fb6aaffd700 1 -- 172.21.15.183:0/840586671 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb6a401d910 con 0x7fb6ac0fbad0 2024-02-20T07:50:18.410 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:18.406+0000 7fb6aaffd700 1 -- 172.21.15.183:0/840586671 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb6a40129f0 con 0x7fb6ac0fbad0 2024-02-20T07:50:18.410 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:18.406+0000 7fb6b4583700 1 -- 172.21.15.183:0/840586671 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb6ac1102f0 con 0x7fb6ac0fbad0 2024-02-20T07:50:18.412 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:18.407+0000 7fb6aaffd700 1 -- 172.21.15.183:0/840586671 <== mon.0 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 14) v1 ==== 43439+0+0 (secure 0 0 0) 0x7fb6a4019040 con 0x7fb6ac0fbad0 2024-02-20T07:50:18.412 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:18.407+0000 7fb6b4583700 1 -- 172.21.15.183:0/840586671 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb6ac1058a0 con 0x7fb6ac0fbad0 2024-02-20T07:50:18.412 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:18.407+0000 7fb6aaffd700 1 --2- 172.21.15.183:0/840586671 >> [v2:172.21.15.133:6800/3944459195,v1:172.21.15.133:6801/3944459195] conn(0x7fb69c0368c0 0x7fb69c038d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:50:18.412 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:18.407+0000 7fb6aaffd700 1 -- 172.21.15.183:0/840586671 <== mon.0 v2:172.21.15.133:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7fb6a404a4f0 con 0x7fb6ac0fbad0 2024-02-20T07:50:18.412 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:18.407+0000 7fb6b1b1e700 1 -- 172.21.15.183:0/840586671 >> [v2:172.21.15.133:6800/3944459195,v1:172.21.15.133:6801/3944459195] conn(0x7fb69c0368c0 msgr2=0x7fb69c038d70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.133:6800/3944459195 2024-02-20T07:50:18.412 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:18.407+0000 7fb6b1b1e700 1 --2- 172.21.15.183:0/840586671 >> [v2:172.21.15.133:6800/3944459195,v1:172.21.15.133:6801/3944459195] conn(0x7fb69c0368c0 0x7fb69c038d70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-02-20T07:50:18.418 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:18.416+0000 7fb6aaffd700 1 -- 172.21.15.183:0/840586671 <== mon.0 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7fb6a402eb40 con 0x7fb6ac0fbad0 2024-02-20T07:50:18.610 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:18.608+0000 7fb6b1b1e700 1 -- 172.21.15.183:0/840586671 >> [v2:172.21.15.133:6800/3944459195,v1:172.21.15.133:6801/3944459195] conn(0x7fb69c0368c0 msgr2=0x7fb69c038d70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.133:6800/3944459195 2024-02-20T07:50:18.611 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:18.608+0000 7fb6b1b1e700 1 --2- 172.21.15.183:0/840586671 >> [v2:172.21.15.133:6800/3944459195,v1:172.21.15.133:6801/3944459195] conn(0x7fb69c0368c0 0x7fb69c038d70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-02-20T07:50:18.673 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:18.670+0000 7fb6b4583700 1 -- 172.21.15.183:0/840586671 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb6ac04bed0 con 0x7fb6ac0fbad0 2024-02-20T07:50:18.673 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:18.671+0000 7fb6aaffd700 1 -- 172.21.15.183:0/840586671 <== mon.0 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+743 (secure 0 0 0) 0x7fb6a4017070 con 0x7fb6ac0fbad0 2024-02-20T07:50:18.674 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-02-20T07:50:18.674 INFO:teuthology.orchestra.run.smithi183.stdout:{"epoch":1,"fsid":"4537327c-cfc4-11ee-95bb-87774f69a715","modified":"2024-02-20T07:48:10.704626Z","created":"2024-02-20T07:48:10.704626Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi133","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:3300","nonce":0},{"type":"v1","addr":"172.21.15.133:6789","nonce":0}]},"addr":"172.21.15.133:6789/0","public_addr":"172.21.15.133:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T07:50:18.675 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:18.674+0000 7fb6b4583700 1 -- 172.21.15.183:0/840586671 >> [v2:172.21.15.133:6800/3944459195,v1:172.21.15.133:6801/3944459195] conn(0x7fb69c0368c0 msgr2=0x7fb69c038d70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-02-20T07:50:18.675 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:18.674+0000 7fb6b4583700 1 --2- 172.21.15.183:0/840586671 >> [v2:172.21.15.133:6800/3944459195,v1:172.21.15.133:6801/3944459195] conn(0x7fb69c0368c0 0x7fb69c038d70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:50:18.676 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:18.674+0000 7fb6b4583700 1 -- 172.21.15.183:0/840586671 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb6ac0fbad0 msgr2=0x7fb6ac10fb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:50:18.676 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:18.674+0000 7fb6b4583700 1 --2- 172.21.15.183:0/840586671 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb6ac0fbad0 0x7fb6ac10fb70 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fb6a4015040 tx=0x7fb6a4004d10 comp rx=0 tx=0).stop 2024-02-20T07:50:18.676 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:18.674+0000 7fb6b4583700 1 -- 172.21.15.183:0/840586671 shutdown_connections 2024-02-20T07:50:18.676 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:18.674+0000 7fb6b4583700 1 --2- 172.21.15.183:0/840586671 >> [v2:172.21.15.133:6800/3944459195,v1:172.21.15.133:6801/3944459195] conn(0x7fb69c0368c0 0x7fb69c038d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:50:18.676 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:18.674+0000 7fb6b4583700 1 --2- 172.21.15.183:0/840586671 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb6ac0fbad0 0x7fb6ac10fb70 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:50:18.676 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:18.674+0000 7fb6b4583700 1 -- 172.21.15.183:0/840586671 >> 172.21.15.183:0/840586671 conn(0x7fb6ac0f7190 msgr2=0x7fb6ac0f7ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:50:18.676 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:18.674+0000 7fb6b4583700 1 -- 172.21.15.183:0/840586671 shutdown_connections 2024-02-20T07:50:18.676 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:18.674+0000 7fb6b4583700 1 -- 172.21.15.183:0/840586671 wait complete. 2024-02-20T07:50:18.676 INFO:teuthology.orchestra.run.smithi183.stderr:dumped monmap epoch 1 2024-02-20T07:50:19.073 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:50:18 smithi133 ceph-mon[29117]: from='client.? 172.21.15.183:0/840586671' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T07:50:20.257 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T07:50:20.257 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4537327c-cfc4-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T07:50:20.507 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-20T07:50:20.553 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-20T07:50:21.896 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:21.892+0000 7f2eab3de700 1 -- 172.21.15.183:0/4063473649 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f2ea40fcbd0 msgr2=0x7f2ea40fcfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:50:21.896 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:21.892+0000 7f2eab3de700 1 --2- 172.21.15.183:0/4063473649 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f2ea40fcbd0 0x7f2ea40fcfe0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f2e940048c0 tx=0x7f2e94009e30 comp rx=0 tx=0).stop 2024-02-20T07:50:21.896 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:21.892+0000 7f2eab3de700 1 -- 172.21.15.183:0/4063473649 shutdown_connections 2024-02-20T07:50:21.896 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:21.892+0000 7f2eab3de700 1 --2- 172.21.15.183:0/4063473649 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f2ea40fcbd0 0x7f2ea40fcfe0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:50:21.896 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:21.892+0000 7f2eab3de700 1 -- 172.21.15.183:0/4063473649 >> 172.21.15.183:0/4063473649 conn(0x7f2ea40f8270 msgr2=0x7f2ea40fa6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:50:21.896 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:21.892+0000 7f2eab3de700 1 -- 172.21.15.183:0/4063473649 shutdown_connections 2024-02-20T07:50:21.896 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:21.892+0000 7f2eab3de700 1 -- 172.21.15.183:0/4063473649 wait complete. 2024-02-20T07:50:21.896 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:21.893+0000 7f2eab3de700 1 Processor -- start 2024-02-20T07:50:21.896 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:21.893+0000 7f2eab3de700 1 -- start start 2024-02-20T07:50:21.897 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:21.893+0000 7f2eab3de700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f2ea40fcbd0 0x7f2ea410cec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:50:21.897 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:21.893+0000 7f2eab3de700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ea410d400 con 0x7f2ea40fcbd0 2024-02-20T07:50:21.897 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:21.893+0000 7f2ea917a700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f2ea40fcbd0 0x7f2ea410cec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:50:21.897 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:21.893+0000 7f2ea917a700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f2ea40fcbd0 0x7f2ea410cec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.183:43498/0 (socket says 172.21.15.183:43498) 2024-02-20T07:50:21.897 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:21.893+0000 7f2ea917a700 1 -- 172.21.15.183:0/2174839758 learned_addr learned my addr 172.21.15.183:0/2174839758 (peer_addr_for_me v2:172.21.15.183:0/0) 2024-02-20T07:50:21.897 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:21.894+0000 7f2ea917a700 1 -- 172.21.15.183:0/2174839758 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e94005040 con 0x7f2ea40fcbd0 2024-02-20T07:50:21.897 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:21.894+0000 7f2ea917a700 1 --2- 172.21.15.183:0/2174839758 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f2ea40fcbd0 0x7f2ea410cec0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f2e94009680 tx=0x7f2e94009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:50:21.897 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:21.894+0000 7f2e99ffb700 1 -- 172.21.15.183:0/2174839758 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2e9400bed0 con 0x7f2ea40fcbd0 2024-02-20T07:50:21.897 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:21.894+0000 7f2e99ffb700 1 -- 172.21.15.183:0/2174839758 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2e9400b5c0 con 0x7f2ea40fcbd0 2024-02-20T07:50:21.897 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:21.894+0000 7f2eab3de700 1 -- 172.21.15.183:0/2174839758 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ea410d600 con 0x7f2ea40fcbd0 2024-02-20T07:50:21.897 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:21.894+0000 7f2e99ffb700 1 -- 172.21.15.183:0/2174839758 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2e9401c5b0 con 0x7f2ea40fcbd0 2024-02-20T07:50:21.897 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:21.894+0000 7f2eab3de700 1 -- 172.21.15.183:0/2174839758 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ea410e2a0 con 0x7f2ea40fcbd0 2024-02-20T07:50:21.898 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:21.895+0000 7f2e99ffb700 1 -- 172.21.15.183:0/2174839758 <== mon.0 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 14) v1 ==== 43439+0+0 (secure 0 0 0) 0x7f2e94013420 con 0x7f2ea40fcbd0 2024-02-20T07:50:21.898 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:21.895+0000 7f2eab3de700 1 -- 172.21.15.183:0/2174839758 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ea405f5c0 con 0x7f2ea40fcbd0 2024-02-20T07:50:21.899 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:21.895+0000 7f2e99ffb700 1 --2- 172.21.15.183:0/2174839758 >> [v2:172.21.15.133:6800/3944459195,v1:172.21.15.133:6801/3944459195] conn(0x7f2e90036870 0x7f2e90038d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:50:21.899 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:21.895+0000 7f2e99ffb700 1 -- 172.21.15.183:0/2174839758 <== mon.0 v2:172.21.15.133:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f2e94049bd0 con 0x7f2ea40fcbd0 2024-02-20T07:50:21.899 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:21.895+0000 7f2ea8979700 1 -- 172.21.15.183:0/2174839758 >> [v2:172.21.15.133:6800/3944459195,v1:172.21.15.133:6801/3944459195] conn(0x7f2e90036870 msgr2=0x7f2e90038d20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.133:6800/3944459195 2024-02-20T07:50:21.899 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:21.895+0000 7f2ea8979700 1 --2- 172.21.15.183:0/2174839758 >> [v2:172.21.15.133:6800/3944459195,v1:172.21.15.133:6801/3944459195] conn(0x7f2e90036870 0x7f2e90038d20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-02-20T07:50:21.901 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:21.900+0000 7f2e99ffb700 1 -- 172.21.15.183:0/2174839758 <== mon.0 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f2e94025400 con 0x7f2ea40fcbd0 2024-02-20T07:50:22.097 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:22.096+0000 7f2ea8979700 1 -- 172.21.15.183:0/2174839758 >> [v2:172.21.15.133:6800/3944459195,v1:172.21.15.133:6801/3944459195] conn(0x7f2e90036870 msgr2=0x7f2e90038d20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.133:6800/3944459195 2024-02-20T07:50:22.097 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:22.096+0000 7f2ea8979700 1 --2- 172.21.15.183:0/2174839758 >> [v2:172.21.15.133:6800/3944459195,v1:172.21.15.133:6801/3944459195] conn(0x7f2e90036870 0x7f2e90038d20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-02-20T07:50:22.156 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:22.155+0000 7f2eab3de700 1 -- 172.21.15.183:0/2174839758 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f2ea410edf0 con 0x7f2ea40fcbd0 2024-02-20T07:50:22.157 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:22.155+0000 7f2e99ffb700 1 -- 172.21.15.183:0/2174839758 <== mon.0 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+743 (secure 0 0 0) 0x7f2e94004d30 con 0x7f2ea40fcbd0 2024-02-20T07:50:22.157 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-02-20T07:50:22.157 INFO:teuthology.orchestra.run.smithi183.stdout:{"epoch":1,"fsid":"4537327c-cfc4-11ee-95bb-87774f69a715","modified":"2024-02-20T07:48:10.704626Z","created":"2024-02-20T07:48:10.704626Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi133","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:3300","nonce":0},{"type":"v1","addr":"172.21.15.133:6789","nonce":0}]},"addr":"172.21.15.133:6789/0","public_addr":"172.21.15.133:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T07:50:22.159 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:22.157+0000 7f2eab3de700 1 -- 172.21.15.183:0/2174839758 >> [v2:172.21.15.133:6800/3944459195,v1:172.21.15.133:6801/3944459195] conn(0x7f2e90036870 msgr2=0x7f2e90038d20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-02-20T07:50:22.159 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:22.157+0000 7f2eab3de700 1 --2- 172.21.15.183:0/2174839758 >> [v2:172.21.15.133:6800/3944459195,v1:172.21.15.133:6801/3944459195] conn(0x7f2e90036870 0x7f2e90038d20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:50:22.159 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:22.157+0000 7f2eab3de700 1 -- 172.21.15.183:0/2174839758 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f2ea40fcbd0 msgr2=0x7f2ea410cec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:50:22.159 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:22.157+0000 7f2eab3de700 1 --2- 172.21.15.183:0/2174839758 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f2ea40fcbd0 0x7f2ea410cec0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f2e94009680 tx=0x7f2e94009e30 comp rx=0 tx=0).stop 2024-02-20T07:50:22.159 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:22.157+0000 7f2eab3de700 1 -- 172.21.15.183:0/2174839758 shutdown_connections 2024-02-20T07:50:22.159 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:22.157+0000 7f2eab3de700 1 --2- 172.21.15.183:0/2174839758 >> [v2:172.21.15.133:6800/3944459195,v1:172.21.15.133:6801/3944459195] conn(0x7f2e90036870 0x7f2e90038d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:50:22.159 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:22.157+0000 7f2eab3de700 1 --2- 172.21.15.183:0/2174839758 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f2ea40fcbd0 0x7f2ea410cec0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:50:22.159 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:22.157+0000 7f2eab3de700 1 -- 172.21.15.183:0/2174839758 >> 172.21.15.183:0/2174839758 conn(0x7f2ea40f8270 msgr2=0x7f2ea40f9000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:50:22.159 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:22.157+0000 7f2eab3de700 1 -- 172.21.15.183:0/2174839758 shutdown_connections 2024-02-20T07:50:22.159 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:22.157+0000 7f2eab3de700 1 -- 172.21.15.183:0/2174839758 wait complete. 2024-02-20T07:50:22.160 INFO:teuthology.orchestra.run.smithi183.stderr:dumped monmap epoch 1 2024-02-20T07:50:22.992 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:50:22 smithi133 ceph-mon[29117]: from='client.? 172.21.15.183:0/2174839758' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T07:50:23.750 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T07:50:23.751 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4537327c-cfc4-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T07:50:23.992 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:50:23 smithi133 ceph-mon[29117]: Active manager daemon smithi133.mrvcbp restarted 2024-02-20T07:50:23.992 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:50:23 smithi133 ceph-mon[29117]: Activating manager daemon smithi133.mrvcbp 2024-02-20T07:50:23.992 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:50:23 smithi133 ceph-mon[29117]: osdmap e5: 0 total, 0 up, 0 in 2024-02-20T07:50:23.992 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:50:23 smithi133 ceph-mon[29117]: mgrmap e15: smithi133.mrvcbp(active, starting, since 0.00310102s) 2024-02-20T07:50:23.992 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:50:23 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "mon metadata", "id": "smithi133"}]: dispatch 2024-02-20T07:50:23.993 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:50:23 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "mgr metadata", "who": "smithi133.mrvcbp", "id": "smithi133.mrvcbp"}]: dispatch 2024-02-20T07:50:23.993 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:50:23 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-02-20T07:50:23.993 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:50:23 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-02-20T07:50:23.993 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:50:23 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-02-20T07:50:23.993 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:50:23 smithi133 ceph-mon[29117]: Manager daemon smithi133.mrvcbp is now available 2024-02-20T07:50:23.993 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:50:23 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-20T07:50:23.993 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:50:23 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-20T07:50:23.993 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:50:23 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:50:23.993 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:50:23 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-20T07:50:23.993 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:50:23 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi133.mrvcbp/mirror_snapshot_schedule"}]: dispatch 2024-02-20T07:50:24.008 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-20T07:50:24.058 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-20T07:50:24.742 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:50:24 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi133.mrvcbp/trash_purge_schedule"}]: dispatch 2024-02-20T07:50:24.742 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:50:24 smithi133 ceph-mon[29117]: [20/Feb/2024:07:50:23] ENGINE Bus STARTING 2024-02-20T07:50:24.742 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:50:24 smithi133 ceph-mon[29117]: [20/Feb/2024:07:50:23] ENGINE Serving on https://172.21.15.133:7150 2024-02-20T07:50:24.742 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:50:24 smithi133 ceph-mon[29117]: [20/Feb/2024:07:50:23] ENGINE Bus STARTED 2024-02-20T07:50:24.742 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:50:24 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:50:24.742 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:50:24 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:50:24.743 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:50:24 smithi133 ceph-mon[29117]: mgrmap e16: smithi133.mrvcbp(active, since 1.00544s) 2024-02-20T07:50:25.390 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:25.387+0000 7f38d7131700 1 -- 172.21.15.183:0/3694879447 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f38d00fd710 msgr2=0x7f38d00ffaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:50:25.390 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:25.387+0000 7f38d7131700 1 --2- 172.21.15.183:0/3694879447 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f38d00fd710 0x7f38d00ffaf0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f38c40048c0 tx=0x7f38c4009e30 comp rx=0 tx=0).stop 2024-02-20T07:50:25.390 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:25.387+0000 7f38d7131700 1 -- 172.21.15.183:0/3694879447 shutdown_connections 2024-02-20T07:50:25.390 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:25.387+0000 7f38d7131700 1 --2- 172.21.15.183:0/3694879447 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f38d00fd710 0x7f38d00ffaf0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:50:25.390 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:25.387+0000 7f38d7131700 1 -- 172.21.15.183:0/3694879447 >> 172.21.15.183:0/3694879447 conn(0x7f38d00f7190 msgr2=0x7f38d00f95c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:50:25.390 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:25.387+0000 7f38d7131700 1 -- 172.21.15.183:0/3694879447 shutdown_connections 2024-02-20T07:50:25.390 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:25.388+0000 7f38d7131700 1 -- 172.21.15.183:0/3694879447 wait complete. 2024-02-20T07:50:25.390 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:25.388+0000 7f38d7131700 1 Processor -- start 2024-02-20T07:50:25.390 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:25.388+0000 7f38d7131700 1 -- start start 2024-02-20T07:50:25.391 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:25.388+0000 7f38d7131700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f38d00fd710 0x7f38d010cea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:50:25.391 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:25.388+0000 7f38d7131700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38d010d3e0 con 0x7f38d00fd710 2024-02-20T07:50:25.391 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:25.389+0000 7f38d4ecd700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f38d00fd710 0x7f38d010cea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:50:25.391 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:25.389+0000 7f38d4ecd700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f38d00fd710 0x7f38d010cea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.183:43520/0 (socket says 172.21.15.183:43520) 2024-02-20T07:50:25.391 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:25.389+0000 7f38d4ecd700 1 -- 172.21.15.183:0/1525964628 learned_addr learned my addr 172.21.15.183:0/1525964628 (peer_addr_for_me v2:172.21.15.183:0/0) 2024-02-20T07:50:25.392 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:25.389+0000 7f38d4ecd700 1 -- 172.21.15.183:0/1525964628 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38c4005040 con 0x7f38d00fd710 2024-02-20T07:50:25.392 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:25.389+0000 7f38d4ecd700 1 --2- 172.21.15.183:0/1525964628 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f38d00fd710 0x7f38d010cea0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f38c4009680 tx=0x7f38c4009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:50:25.392 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:25.389+0000 7f38cd7fa700 1 -- 172.21.15.183:0/1525964628 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f38c400b5c0 con 0x7f38d00fd710 2024-02-20T07:50:25.393 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:25.389+0000 7f38cd7fa700 1 -- 172.21.15.183:0/1525964628 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f38c4004e60 con 0x7f38d00fd710 2024-02-20T07:50:25.393 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:25.389+0000 7f38d7131700 1 -- 172.21.15.183:0/1525964628 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f38d010d5e0 con 0x7f38d00fd710 2024-02-20T07:50:25.393 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:25.389+0000 7f38cd7fa700 1 -- 172.21.15.183:0/1525964628 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f38c401c590 con 0x7f38d00fd710 2024-02-20T07:50:25.393 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:25.389+0000 7f38d7131700 1 -- 172.21.15.183:0/1525964628 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f38d010e280 con 0x7f38d00fd710 2024-02-20T07:50:25.393 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:25.390+0000 7f38cd7fa700 1 -- 172.21.15.183:0/1525964628 <== mon.0 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 16) v1 ==== 43289+0+0 (secure 0 0 0) 0x7f38c401c810 con 0x7f38d00fd710 2024-02-20T07:50:25.393 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:25.390+0000 7f38d7131700 1 -- 172.21.15.183:0/1525964628 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f38d005f5c0 con 0x7f38d00fd710 2024-02-20T07:50:25.393 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:25.390+0000 7f38cd7fa700 1 --2- 172.21.15.183:0/1525964628 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7f38c0036700 0x7f38c0038bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:50:25.393 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:25.390+0000 7f38cd7fa700 1 -- 172.21.15.183:0/1525964628 <== mon.0 v2:172.21.15.133:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1366+0+0 (secure 0 0 0) 0x7f38c4049c80 con 0x7f38d00fd710 2024-02-20T07:50:25.393 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:25.390+0000 7f38cffff700 1 --2- 172.21.15.183:0/1525964628 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7f38c0036700 0x7f38c0038bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:50:25.393 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:25.391+0000 7f38cffff700 1 --2- 172.21.15.183:0/1525964628 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7f38c0036700 0x7f38c0038bb0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f38bc009990 tx=0x7f38bc006e40 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:50:25.396 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:25.395+0000 7f38cd7fa700 1 -- 172.21.15.183:0/1525964628 <== mon.0 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f38c4023300 con 0x7f38d00fd710 2024-02-20T07:50:25.685 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:25.683+0000 7f38d7131700 1 -- 172.21.15.183:0/1525964628 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f38d010edd0 con 0x7f38d00fd710 2024-02-20T07:50:25.685 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:25.683+0000 7f38cd7fa700 1 -- 172.21.15.183:0/1525964628 <== mon.0 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+743 (secure 0 0 0) 0x7f38c4011b60 con 0x7f38d00fd710 2024-02-20T07:50:25.685 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-02-20T07:50:25.686 INFO:teuthology.orchestra.run.smithi183.stdout:{"epoch":1,"fsid":"4537327c-cfc4-11ee-95bb-87774f69a715","modified":"2024-02-20T07:48:10.704626Z","created":"2024-02-20T07:48:10.704626Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi133","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:3300","nonce":0},{"type":"v1","addr":"172.21.15.133:6789","nonce":0}]},"addr":"172.21.15.133:6789/0","public_addr":"172.21.15.133:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T07:50:25.687 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:25.686+0000 7f38d7131700 1 -- 172.21.15.183:0/1525964628 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7f38c0036700 msgr2=0x7f38c0038bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:50:25.687 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:25.686+0000 7f38d7131700 1 --2- 172.21.15.183:0/1525964628 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7f38c0036700 0x7f38c0038bb0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f38bc009990 tx=0x7f38bc006e40 comp rx=0 tx=0).stop 2024-02-20T07:50:25.687 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:25.686+0000 7f38d7131700 1 -- 172.21.15.183:0/1525964628 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f38d00fd710 msgr2=0x7f38d010cea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:50:25.687 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:25.686+0000 7f38d7131700 1 --2- 172.21.15.183:0/1525964628 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f38d00fd710 0x7f38d010cea0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f38c4009680 tx=0x7f38c4009e30 comp rx=0 tx=0).stop 2024-02-20T07:50:25.687 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:25.686+0000 7f38d7131700 1 -- 172.21.15.183:0/1525964628 shutdown_connections 2024-02-20T07:50:25.688 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:25.686+0000 7f38d7131700 1 --2- 172.21.15.183:0/1525964628 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7f38c0036700 0x7f38c0038bb0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:50:25.688 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:25.686+0000 7f38d7131700 1 --2- 172.21.15.183:0/1525964628 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f38d00fd710 0x7f38d010cea0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:50:25.688 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:25.686+0000 7f38d7131700 1 -- 172.21.15.183:0/1525964628 >> 172.21.15.183:0/1525964628 conn(0x7f38d00f7190 msgr2=0x7f38d00f7e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:50:25.688 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:25.686+0000 7f38d7131700 1 -- 172.21.15.183:0/1525964628 shutdown_connections 2024-02-20T07:50:25.688 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:25.686+0000 7f38d7131700 1 -- 172.21.15.183:0/1525964628 wait complete. 2024-02-20T07:50:25.688 INFO:teuthology.orchestra.run.smithi183.stderr:dumped monmap epoch 1 2024-02-20T07:50:26.381 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:50:26 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:50:26.381 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:50:26 smithi133 ceph-mon[29117]: from='client.? 172.21.15.183:0/1525964628' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T07:50:26.381 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:50:26 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:50:27.296 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T07:50:27.296 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4537327c-cfc4-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T07:50:27.492 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:50:27 smithi133 ceph-mon[29117]: mgrmap e17: smithi133.mrvcbp(active, since 2s) 2024-02-20T07:50:27.838 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-20T07:50:27.897 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-20T07:50:28.742 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:50:28 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:50:28.742 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:50:28 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:50:28.742 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:50:28 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:50:28.742 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:50:28 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:50:28.742 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:50:28 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "config rm", "who": "osd/host:smithi133", "name": "osd_memory_target"}]: dispatch 2024-02-20T07:50:29.478 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:29.476+0000 7f7b7dd34700 1 -- 172.21.15.183:0/2988772145 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f7b780fe2c0 msgr2=0x7f7b780fe6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:50:29.479 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:29.476+0000 7f7b7dd34700 1 --2- 172.21.15.183:0/2988772145 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f7b780fe2c0 0x7f7b780fe6d0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f7b600048c0 tx=0x7f7b60009e30 comp rx=0 tx=0).stop 2024-02-20T07:50:29.479 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:29.476+0000 7f7b7dd34700 1 -- 172.21.15.183:0/2988772145 shutdown_connections 2024-02-20T07:50:29.479 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:29.476+0000 7f7b7dd34700 1 --2- 172.21.15.183:0/2988772145 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f7b780fe2c0 0x7f7b780fe6d0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:50:29.479 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:29.476+0000 7f7b7dd34700 1 -- 172.21.15.183:0/2988772145 >> 172.21.15.183:0/2988772145 conn(0x7f7b780f69c0 msgr2=0x7f7b780f8dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:50:29.479 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:29.476+0000 7f7b7dd34700 1 -- 172.21.15.183:0/2988772145 shutdown_connections 2024-02-20T07:50:29.479 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:29.476+0000 7f7b7dd34700 1 -- 172.21.15.183:0/2988772145 wait complete. 2024-02-20T07:50:29.479 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:29.476+0000 7f7b7dd34700 1 Processor -- start 2024-02-20T07:50:29.479 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:29.476+0000 7f7b7dd34700 1 -- start start 2024-02-20T07:50:29.479 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:29.476+0000 7f7b7dd34700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f7b780fe2c0 0x7f7b780fdb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:50:29.479 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:29.476+0000 7f7b7dd34700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b780fbf80 con 0x7f7b780fe2c0 2024-02-20T07:50:29.479 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:29.477+0000 7f7b777fe700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f7b780fe2c0 0x7f7b780fdb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:50:29.479 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:29.477+0000 7f7b777fe700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f7b780fe2c0 0x7f7b780fdb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.183:47282/0 (socket says 172.21.15.183:47282) 2024-02-20T07:50:29.479 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:29.477+0000 7f7b777fe700 1 -- 172.21.15.183:0/550154681 learned_addr learned my addr 172.21.15.183:0/550154681 (peer_addr_for_me v2:172.21.15.183:0/0) 2024-02-20T07:50:29.479 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:29.477+0000 7f7b777fe700 1 -- 172.21.15.183:0/550154681 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b60005040 con 0x7f7b780fe2c0 2024-02-20T07:50:29.479 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:29.477+0000 7f7b777fe700 1 --2- 172.21.15.183:0/550154681 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f7b780fe2c0 0x7f7b780fdb00 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f7b60009300 tx=0x7f7b60009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:50:29.481 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:29.477+0000 7f7b6ffff700 1 -- 172.21.15.183:0/550154681 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7b60004e60 con 0x7f7b780fe2c0 2024-02-20T07:50:29.481 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:29.477+0000 7f7b7dd34700 1 -- 172.21.15.183:0/550154681 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b780fc180 con 0x7f7b780fe2c0 2024-02-20T07:50:29.481 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:29.477+0000 7f7b6ffff700 1 -- 172.21.15.183:0/550154681 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7b60009680 con 0x7f7b780fe2c0 2024-02-20T07:50:29.481 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:29.477+0000 7f7b6ffff700 1 -- 172.21.15.183:0/550154681 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7b6001c5b0 con 0x7f7b780fe2c0 2024-02-20T07:50:29.481 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:29.477+0000 7f7b7dd34700 1 -- 172.21.15.183:0/550154681 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b780fc670 con 0x7f7b780fe2c0 2024-02-20T07:50:29.481 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:29.478+0000 7f7b7dd34700 1 -- 172.21.15.183:0/550154681 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7b78106910 con 0x7f7b780fe2c0 2024-02-20T07:50:29.481 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:29.478+0000 7f7b6ffff700 1 -- 172.21.15.183:0/550154681 <== mon.0 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 17) v1 ==== 43483+0+0 (secure 0 0 0) 0x7f7b600097f0 con 0x7f7b780fe2c0 2024-02-20T07:50:29.481 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:29.479+0000 7f7b6ffff700 1 --2- 172.21.15.183:0/550154681 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7f7b64036960 0x7f7b64038e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:50:29.481 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:29.479+0000 7f7b6ffff700 1 -- 172.21.15.183:0/550154681 <== mon.0 v2:172.21.15.133:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1366+0+0 (secure 0 0 0) 0x7f7b60049d20 con 0x7f7b780fe2c0 2024-02-20T07:50:29.481 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:29.479+0000 7f7b76ffd700 1 --2- 172.21.15.183:0/550154681 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7f7b64036960 0x7f7b64038e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:50:29.481 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:29.479+0000 7f7b76ffd700 1 --2- 172.21.15.183:0/550154681 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7f7b64036960 0x7f7b64038e10 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f7b68009990 tx=0x7f7b68006e40 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:50:29.485 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:29.484+0000 7f7b6ffff700 1 -- 172.21.15.183:0/550154681 <== mon.0 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f7b60011550 con 0x7f7b780fe2c0 2024-02-20T07:50:29.750 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:29.748+0000 7f7b7dd34700 1 -- 172.21.15.183:0/550154681 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f7b7804bed0 con 0x7f7b780fe2c0 2024-02-20T07:50:29.750 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:29.749+0000 7f7b6ffff700 1 -- 172.21.15.183:0/550154681 <== mon.0 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+743 (secure 0 0 0) 0x7f7b60011370 con 0x7f7b780fe2c0 2024-02-20T07:50:29.750 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-02-20T07:50:29.751 INFO:teuthology.orchestra.run.smithi183.stdout:{"epoch":1,"fsid":"4537327c-cfc4-11ee-95bb-87774f69a715","modified":"2024-02-20T07:48:10.704626Z","created":"2024-02-20T07:48:10.704626Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi133","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:3300","nonce":0},{"type":"v1","addr":"172.21.15.133:6789","nonce":0}]},"addr":"172.21.15.133:6789/0","public_addr":"172.21.15.133:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T07:50:29.753 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:29.751+0000 7f7b7dd34700 1 -- 172.21.15.183:0/550154681 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7f7b64036960 msgr2=0x7f7b64038e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:50:29.753 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:29.751+0000 7f7b7dd34700 1 --2- 172.21.15.183:0/550154681 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7f7b64036960 0x7f7b64038e10 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f7b68009990 tx=0x7f7b68006e40 comp rx=0 tx=0).stop 2024-02-20T07:50:29.753 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:29.751+0000 7f7b7dd34700 1 -- 172.21.15.183:0/550154681 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f7b780fe2c0 msgr2=0x7f7b780fdb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:50:29.753 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:29.751+0000 7f7b7dd34700 1 --2- 172.21.15.183:0/550154681 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f7b780fe2c0 0x7f7b780fdb00 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f7b60009300 tx=0x7f7b60009e30 comp rx=0 tx=0).stop 2024-02-20T07:50:29.753 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:29.751+0000 7f7b7dd34700 1 -- 172.21.15.183:0/550154681 shutdown_connections 2024-02-20T07:50:29.753 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:29.751+0000 7f7b7dd34700 1 --2- 172.21.15.183:0/550154681 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7f7b64036960 0x7f7b64038e10 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:50:29.753 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:29.751+0000 7f7b7dd34700 1 --2- 172.21.15.183:0/550154681 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f7b780fe2c0 0x7f7b780fdb00 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:50:29.753 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:29.751+0000 7f7b7dd34700 1 -- 172.21.15.183:0/550154681 >> 172.21.15.183:0/550154681 conn(0x7f7b780f69c0 msgr2=0x7f7b780f7810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:50:29.753 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:29.751+0000 7f7b7dd34700 1 -- 172.21.15.183:0/550154681 shutdown_connections 2024-02-20T07:50:29.753 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:29.751+0000 7f7b7dd34700 1 -- 172.21.15.183:0/550154681 wait complete. 2024-02-20T07:50:29.754 INFO:teuthology.orchestra.run.smithi183.stderr:dumped monmap epoch 1 2024-02-20T07:50:30.242 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:50:29 smithi133 ceph-mon[29117]: from='client.? 172.21.15.183:0/550154681' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T07:50:31.242 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:50:30 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:50:31.242 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:50:30 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:50:31.242 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:50:30 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:50:31.242 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:50:30 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:50:31.242 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:50:30 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "config rm", "who": "osd/host:smithi183", "name": "osd_memory_target"}]: dispatch 2024-02-20T07:50:31.242 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:50:30 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-20T07:50:31.243 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:50:30 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-20T07:50:31.243 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:50:30 smithi133 ceph-mon[29117]: Updating smithi133:/etc/ceph/ceph.conf 2024-02-20T07:50:31.243 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:50:30 smithi133 ceph-mon[29117]: Updating smithi183:/etc/ceph/ceph.conf 2024-02-20T07:50:31.243 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:50:30 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:50:31.243 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:50:30 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:50:31.243 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:50:30 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:50:31.243 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:50:30 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:50:31.243 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:50:30 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:50:31.243 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:50:30 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi183", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-02-20T07:50:31.243 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:50:30 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd='[{"prefix": "auth get-or-create", "entity": "client.crash.smithi183", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]': finished 2024-02-20T07:50:31.243 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:50:30 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-20T07:50:31.533 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T07:50:31.533 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4537327c-cfc4-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T07:50:32.102 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/4537327c-cfc4-11ee-95bb-87774f69a715/config/ceph.conf 2024-02-20T07:50:32.242 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:50:31 smithi133 ceph-mon[29117]: Updating smithi133:/var/lib/ceph/4537327c-cfc4-11ee-95bb-87774f69a715/config/ceph.conf 2024-02-20T07:50:32.242 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:50:31 smithi133 ceph-mon[29117]: Updating smithi183:/var/lib/ceph/4537327c-cfc4-11ee-95bb-87774f69a715/config/ceph.conf 2024-02-20T07:50:32.243 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:50:31 smithi133 ceph-mon[29117]: Updating smithi133:/etc/ceph/ceph.client.admin.keyring 2024-02-20T07:50:32.243 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:50:31 smithi133 ceph-mon[29117]: Updating smithi183:/etc/ceph/ceph.client.admin.keyring 2024-02-20T07:50:32.243 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:50:31 smithi133 ceph-mon[29117]: Updating smithi133:/var/lib/ceph/4537327c-cfc4-11ee-95bb-87774f69a715/config/ceph.client.admin.keyring 2024-02-20T07:50:32.243 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:50:31 smithi133 ceph-mon[29117]: Updating smithi183:/var/lib/ceph/4537327c-cfc4-11ee-95bb-87774f69a715/config/ceph.client.admin.keyring 2024-02-20T07:50:32.243 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:50:31 smithi133 ceph-mon[29117]: Deploying daemon crash.smithi183 on smithi183 2024-02-20T07:50:33.762 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:33.758+0000 7f75e7cff700 1 -- 172.21.15.183:0/952888540 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f75e00fb3a0 msgr2=0x7f75e00fb770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:50:33.762 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:33.758+0000 7f75e7cff700 1 --2- 172.21.15.183:0/952888540 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f75e00fb3a0 0x7f75e00fb770 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f75d00048c0 tx=0x7f75d0009e30 comp rx=0 tx=0).stop 2024-02-20T07:50:33.762 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:33.758+0000 7f75e7cff700 1 -- 172.21.15.183:0/952888540 shutdown_connections 2024-02-20T07:50:33.762 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:33.758+0000 7f75e7cff700 1 --2- 172.21.15.183:0/952888540 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f75e00fb3a0 0x7f75e00fb770 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:50:33.762 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:33.758+0000 7f75e7cff700 1 -- 172.21.15.183:0/952888540 >> 172.21.15.183:0/952888540 conn(0x7f75e00720b0 msgr2=0x7f75e00724b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:50:33.762 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:33.759+0000 7f75e7cff700 1 -- 172.21.15.183:0/952888540 shutdown_connections 2024-02-20T07:50:33.762 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:33.759+0000 7f75e7cff700 1 -- 172.21.15.183:0/952888540 wait complete. 2024-02-20T07:50:33.762 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:33.759+0000 7f75e7cff700 1 Processor -- start 2024-02-20T07:50:33.762 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:33.759+0000 7f75e7cff700 1 -- start start 2024-02-20T07:50:33.762 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:33.759+0000 7f75e7cff700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f75e00fb3a0 0x7f75e0112ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:50:33.763 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:33.759+0000 7f75e7cff700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75e0113530 con 0x7f75e00fb3a0 2024-02-20T07:50:33.763 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:33.760+0000 7f75e5a9b700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f75e00fb3a0 0x7f75e0112ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:50:33.763 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:33.760+0000 7f75e5a9b700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f75e00fb3a0 0x7f75e0112ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.183:47298/0 (socket says 172.21.15.183:47298) 2024-02-20T07:50:33.763 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:33.760+0000 7f75e5a9b700 1 -- 172.21.15.183:0/4280831597 learned_addr learned my addr 172.21.15.183:0/4280831597 (peer_addr_for_me v2:172.21.15.183:0/0) 2024-02-20T07:50:33.763 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:33.760+0000 7f75e5a9b700 1 -- 172.21.15.183:0/4280831597 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75d0005040 con 0x7f75e00fb3a0 2024-02-20T07:50:33.763 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:33.760+0000 7f75e5a9b700 1 --2- 172.21.15.183:0/4280831597 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f75e00fb3a0 0x7f75e0112ff0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f75d0009330 tx=0x7f75d0009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:50:33.763 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:33.760+0000 7f75d67fc700 1 -- 172.21.15.183:0/4280831597 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f75d000be50 con 0x7f75e00fb3a0 2024-02-20T07:50:33.763 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:33.760+0000 7f75e7cff700 1 -- 172.21.15.183:0/4280831597 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f75e0113730 con 0x7f75e00fb3a0 2024-02-20T07:50:33.763 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:33.760+0000 7f75d67fc700 1 -- 172.21.15.183:0/4280831597 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f75d0009630 con 0x7f75e00fb3a0 2024-02-20T07:50:33.763 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:33.760+0000 7f75d67fc700 1 -- 172.21.15.183:0/4280831597 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f75d001c3f0 con 0x7f75e00fb3a0 2024-02-20T07:50:33.763 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:33.760+0000 7f75e7cff700 1 -- 172.21.15.183:0/4280831597 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f75e0113ba0 con 0x7f75e00fb3a0 2024-02-20T07:50:33.764 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:33.761+0000 7f75d67fc700 1 -- 172.21.15.183:0/4280831597 <== mon.0 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 17) v1 ==== 43483+0+0 (secure 0 0 0) 0x7f75d00097a0 con 0x7f75e00fb3a0 2024-02-20T07:50:33.764 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:33.761+0000 7f75e7cff700 1 -- 172.21.15.183:0/4280831597 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f75e005f5c0 con 0x7f75e00fb3a0 2024-02-20T07:50:33.764 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:33.761+0000 7f75d67fc700 1 --2- 172.21.15.183:0/4280831597 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7f75cc036960 0x7f75cc038e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:50:33.764 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:33.761+0000 7f75d67fc700 1 -- 172.21.15.183:0/4280831597 <== mon.0 v2:172.21.15.133:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1366+0+0 (secure 0 0 0) 0x7f75d0049b30 con 0x7f75e00fb3a0 2024-02-20T07:50:33.764 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:33.761+0000 7f75e529a700 1 --2- 172.21.15.183:0/4280831597 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7f75cc036960 0x7f75cc038e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:50:33.765 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:33.762+0000 7f75e529a700 1 --2- 172.21.15.183:0/4280831597 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7f75cc036960 0x7f75cc038e10 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f75dc009990 tx=0x7f75dc006e40 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:50:33.767 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:33.766+0000 7f75d67fc700 1 -- 172.21.15.183:0/4280831597 <== mon.0 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f75d0011940 con 0x7f75e00fb3a0 2024-02-20T07:50:34.032 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:34.030+0000 7f75e7cff700 1 -- 172.21.15.183:0/4280831597 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f75e0106f10 con 0x7f75e00fb3a0 2024-02-20T07:50:34.032 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:34.030+0000 7f75d67fc700 1 -- 172.21.15.183:0/4280831597 <== mon.0 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+743 (secure 0 0 0) 0x7f75d0011370 con 0x7f75e00fb3a0 2024-02-20T07:50:34.032 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-02-20T07:50:34.032 INFO:teuthology.orchestra.run.smithi183.stdout:{"epoch":1,"fsid":"4537327c-cfc4-11ee-95bb-87774f69a715","modified":"2024-02-20T07:48:10.704626Z","created":"2024-02-20T07:48:10.704626Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi133","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:3300","nonce":0},{"type":"v1","addr":"172.21.15.133:6789","nonce":0}]},"addr":"172.21.15.133:6789/0","public_addr":"172.21.15.133:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T07:50:34.034 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:34.032+0000 7f75e7cff700 1 -- 172.21.15.183:0/4280831597 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7f75cc036960 msgr2=0x7f75cc038e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:50:34.034 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:34.032+0000 7f75e7cff700 1 --2- 172.21.15.183:0/4280831597 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7f75cc036960 0x7f75cc038e10 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f75dc009990 tx=0x7f75dc006e40 comp rx=0 tx=0).stop 2024-02-20T07:50:34.034 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:34.032+0000 7f75e7cff700 1 -- 172.21.15.183:0/4280831597 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f75e00fb3a0 msgr2=0x7f75e0112ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:50:34.034 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:34.032+0000 7f75e7cff700 1 --2- 172.21.15.183:0/4280831597 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f75e00fb3a0 0x7f75e0112ff0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f75d0009330 tx=0x7f75d0009e30 comp rx=0 tx=0).stop 2024-02-20T07:50:34.034 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:34.032+0000 7f75e7cff700 1 -- 172.21.15.183:0/4280831597 shutdown_connections 2024-02-20T07:50:34.034 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:34.032+0000 7f75e7cff700 1 --2- 172.21.15.183:0/4280831597 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7f75cc036960 0x7f75cc038e10 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:50:34.034 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:34.032+0000 7f75e7cff700 1 --2- 172.21.15.183:0/4280831597 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f75e00fb3a0 0x7f75e0112ff0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:50:34.034 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:34.032+0000 7f75e7cff700 1 -- 172.21.15.183:0/4280831597 >> 172.21.15.183:0/4280831597 conn(0x7f75e00720b0 msgr2=0x7f75e01010a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:50:34.035 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:34.032+0000 7f75e7cff700 1 -- 172.21.15.183:0/4280831597 shutdown_connections 2024-02-20T07:50:34.035 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:34.032+0000 7f75e7cff700 1 -- 172.21.15.183:0/4280831597 wait complete. 2024-02-20T07:50:34.035 INFO:teuthology.orchestra.run.smithi183.stderr:dumped monmap epoch 1 2024-02-20T07:50:34.492 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:50:34 smithi133 ceph-mon[29117]: from='client.? 172.21.15.183:0/4280831597' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T07:50:35.492 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:50:35 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:50:35.492 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:50:35 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:50:35.492 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:50:35 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:50:35.492 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:50:35 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:50:35.492 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:50:35 smithi133 ceph-mon[29117]: Deploying daemon node-exporter.smithi183 on smithi183 2024-02-20T07:50:35.840 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T07:50:35.841 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4537327c-cfc4-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T07:50:36.090 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/4537327c-cfc4-11ee-95bb-87774f69a715/config/ceph.conf 2024-02-20T07:50:38.269 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:38.197+0000 7f16fdaf5700 1 -- 172.21.15.183:0/3995152805 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f16f8102bb0 msgr2=0x7f16f8102f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:50:38.269 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:38.197+0000 7f16fdaf5700 1 --2- 172.21.15.183:0/3995152805 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f16f8102bb0 0x7f16f8102f80 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f16ec0048c0 tx=0x7f16ec009e30 comp rx=0 tx=0).stop 2024-02-20T07:50:38.269 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:38.197+0000 7f16fdaf5700 1 -- 172.21.15.183:0/3995152805 shutdown_connections 2024-02-20T07:50:38.269 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:38.197+0000 7f16fdaf5700 1 --2- 172.21.15.183:0/3995152805 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f16f8102bb0 0x7f16f8102f80 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:50:38.269 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:38.197+0000 7f16fdaf5700 1 -- 172.21.15.183:0/3995152805 >> 172.21.15.183:0/3995152805 conn(0x7f16f80f8760 msgr2=0x7f16f80fab70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:50:38.269 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:38.197+0000 7f16fdaf5700 1 -- 172.21.15.183:0/3995152805 shutdown_connections 2024-02-20T07:50:38.269 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:38.197+0000 7f16fdaf5700 1 -- 172.21.15.183:0/3995152805 wait complete. 2024-02-20T07:50:38.269 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:38.198+0000 7f16fdaf5700 1 Processor -- start 2024-02-20T07:50:38.270 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:38.198+0000 7f16fdaf5700 1 -- start start 2024-02-20T07:50:38.270 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:38.198+0000 7f16fdaf5700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f16f8102bb0 0x7f16f8070290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:50:38.270 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:38.198+0000 7f16fdaf5700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16f80707d0 con 0x7f16f8102bb0 2024-02-20T07:50:38.270 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:38.198+0000 7f16f77fe700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f16f8102bb0 0x7f16f8070290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:50:38.270 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:38.198+0000 7f16f77fe700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f16f8102bb0 0x7f16f8070290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.183:47334/0 (socket says 172.21.15.183:47334) 2024-02-20T07:50:38.270 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:38.198+0000 7f16f77fe700 1 -- 172.21.15.183:0/849486943 learned_addr learned my addr 172.21.15.183:0/849486943 (peer_addr_for_me v2:172.21.15.183:0/0) 2024-02-20T07:50:38.270 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:38.199+0000 7f16f77fe700 1 -- 172.21.15.183:0/849486943 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16ec005040 con 0x7f16f8102bb0 2024-02-20T07:50:38.270 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:38.199+0000 7f16f77fe700 1 --2- 172.21.15.183:0/849486943 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f16f8102bb0 0x7f16f8070290 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f16ec009300 tx=0x7f16ec009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:50:38.270 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:38.199+0000 7f16dffff700 1 -- 172.21.15.183:0/849486943 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f16ec004e60 con 0x7f16f8102bb0 2024-02-20T07:50:38.270 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:38.199+0000 7f16fdaf5700 1 -- 172.21.15.183:0/849486943 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f16f80709d0 con 0x7f16f8102bb0 2024-02-20T07:50:38.270 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:38.199+0000 7f16dffff700 1 -- 172.21.15.183:0/849486943 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f16ec013420 con 0x7f16f8102bb0 2024-02-20T07:50:38.270 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:38.199+0000 7f16dffff700 1 -- 172.21.15.183:0/849486943 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f16ec01c5b0 con 0x7f16f8102bb0 2024-02-20T07:50:38.270 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:38.199+0000 7f16fdaf5700 1 -- 172.21.15.183:0/849486943 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f16f8070e40 con 0x7f16f8102bb0 2024-02-20T07:50:38.271 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:38.200+0000 7f16fdaf5700 1 -- 172.21.15.183:0/849486943 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f16f8101ed0 con 0x7f16f8102bb0 2024-02-20T07:50:38.271 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:38.200+0000 7f16dffff700 1 -- 172.21.15.183:0/849486943 <== mon.0 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 17) v1 ==== 43483+0+0 (secure 0 0 0) 0x7f16ec013a90 con 0x7f16f8102bb0 2024-02-20T07:50:38.271 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:38.200+0000 7f16dffff700 1 --2- 172.21.15.183:0/849486943 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7f16e0036960 0x7f16e0038e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:50:38.271 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:38.200+0000 7f16dffff700 1 -- 172.21.15.183:0/849486943 <== mon.0 v2:172.21.15.133:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1366+0+0 (secure 0 0 0) 0x7f16ec049be0 con 0x7f16f8102bb0 2024-02-20T07:50:38.271 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:38.200+0000 7f16f6ffd700 1 --2- 172.21.15.183:0/849486943 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7f16e0036960 0x7f16e0038e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:50:38.271 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:38.201+0000 7f16f6ffd700 1 --2- 172.21.15.183:0/849486943 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7f16e0036960 0x7f16e0038e10 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f16e8009990 tx=0x7f16e8006e40 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:50:38.271 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:38.205+0000 7f16dffff700 1 -- 172.21.15.183:0/849486943 <== mon.0 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f16ec011960 con 0x7f16f8102bb0 2024-02-20T07:50:38.473 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:38.471+0000 7f16fdaf5700 1 -- 172.21.15.183:0/849486943 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f16f804bed0 con 0x7f16f8102bb0 2024-02-20T07:50:38.473 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:38.472+0000 7f16dffff700 1 -- 172.21.15.183:0/849486943 <== mon.0 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+743 (secure 0 0 0) 0x7f16ec017200 con 0x7f16f8102bb0 2024-02-20T07:50:38.474 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-02-20T07:50:38.474 INFO:teuthology.orchestra.run.smithi183.stdout:{"epoch":1,"fsid":"4537327c-cfc4-11ee-95bb-87774f69a715","modified":"2024-02-20T07:48:10.704626Z","created":"2024-02-20T07:48:10.704626Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi133","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:3300","nonce":0},{"type":"v1","addr":"172.21.15.133:6789","nonce":0}]},"addr":"172.21.15.133:6789/0","public_addr":"172.21.15.133:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T07:50:38.476 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:38.474+0000 7f16fdaf5700 1 -- 172.21.15.183:0/849486943 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7f16e0036960 msgr2=0x7f16e0038e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:50:38.476 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:38.474+0000 7f16fdaf5700 1 --2- 172.21.15.183:0/849486943 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7f16e0036960 0x7f16e0038e10 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f16e8009990 tx=0x7f16e8006e40 comp rx=0 tx=0).stop 2024-02-20T07:50:38.476 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:38.474+0000 7f16fdaf5700 1 -- 172.21.15.183:0/849486943 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f16f8102bb0 msgr2=0x7f16f8070290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:50:38.476 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:38.474+0000 7f16fdaf5700 1 --2- 172.21.15.183:0/849486943 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f16f8102bb0 0x7f16f8070290 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f16ec009300 tx=0x7f16ec009e30 comp rx=0 tx=0).stop 2024-02-20T07:50:38.477 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:38.474+0000 7f16fdaf5700 1 -- 172.21.15.183:0/849486943 shutdown_connections 2024-02-20T07:50:38.477 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:38.474+0000 7f16fdaf5700 1 --2- 172.21.15.183:0/849486943 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7f16e0036960 0x7f16e0038e10 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:50:38.477 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:38.474+0000 7f16fdaf5700 1 --2- 172.21.15.183:0/849486943 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f16f8102bb0 0x7f16f8070290 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:50:38.477 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:38.474+0000 7f16fdaf5700 1 -- 172.21.15.183:0/849486943 >> 172.21.15.183:0/849486943 conn(0x7f16f80f8760 msgr2=0x7f16f80fa650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:50:38.477 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:38.474+0000 7f16fdaf5700 1 -- 172.21.15.183:0/849486943 shutdown_connections 2024-02-20T07:50:38.477 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:38.474+0000 7f16fdaf5700 1 -- 172.21.15.183:0/849486943 wait complete. 2024-02-20T07:50:38.477 INFO:teuthology.orchestra.run.smithi183.stderr:dumped monmap epoch 1 2024-02-20T07:50:38.992 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:50:38 smithi133 ceph-mon[29117]: from='client.? 172.21.15.183:0/849486943' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T07:50:39.993 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:50:39 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:50:40.045 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:50:39 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-20T07:50:40.352 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T07:50:40.353 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4537327c-cfc4-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T07:50:40.679 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/4537327c-cfc4-11ee-95bb-87774f69a715/config/ceph.conf 2024-02-20T07:50:40.992 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:50:40 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:50:40.992 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:50:40 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:50:40.992 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:50:40 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:50:40.992 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:50:40 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:50:40.993 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:50:40 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi183.hebfng", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-20T07:50:40.993 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:50:40 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.smithi183.hebfng", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-02-20T07:50:40.993 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:50:40 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-20T07:50:40.993 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:50:40 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-20T07:50:40.993 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:50:40 smithi133 ceph-mon[29117]: Deploying daemon mgr.smithi183.hebfng on smithi183 2024-02-20T07:50:43.021 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:43.017+0000 7f71800ac700 1 -- 172.21.15.183:0/1327636329 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f7178102c70 msgr2=0x7f7178103040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:50:43.021 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:43.017+0000 7f71800ac700 1 --2- 172.21.15.183:0/1327636329 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f7178102c70 0x7f7178103040 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f71740048c0 tx=0x7f7174009e30 comp rx=0 tx=0).stop 2024-02-20T07:50:43.021 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:43.017+0000 7f71800ac700 1 -- 172.21.15.183:0/1327636329 shutdown_connections 2024-02-20T07:50:43.021 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:43.017+0000 7f71800ac700 1 --2- 172.21.15.183:0/1327636329 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f7178102c70 0x7f7178103040 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:50:43.021 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:43.017+0000 7f71800ac700 1 -- 172.21.15.183:0/1327636329 >> 172.21.15.183:0/1327636329 conn(0x7f71780720e0 msgr2=0x7f71780724e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:50:43.021 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:43.017+0000 7f71800ac700 1 -- 172.21.15.183:0/1327636329 shutdown_connections 2024-02-20T07:50:43.021 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:43.017+0000 7f71800ac700 1 -- 172.21.15.183:0/1327636329 wait complete. 2024-02-20T07:50:43.021 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:43.018+0000 7f71800ac700 1 Processor -- start 2024-02-20T07:50:43.021 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:43.018+0000 7f71800ac700 1 -- start start 2024-02-20T07:50:43.021 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:43.018+0000 7f71800ac700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f7178102c70 0x7f71780fc2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:50:43.021 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:43.018+0000 7f71800ac700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71780fc830 con 0x7f7178102c70 2024-02-20T07:50:43.022 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:43.018+0000 7f717de48700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f7178102c70 0x7f71780fc2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:50:43.022 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:43.018+0000 7f717de48700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f7178102c70 0x7f71780fc2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.183:45432/0 (socket says 172.21.15.183:45432) 2024-02-20T07:50:43.022 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:43.018+0000 7f717de48700 1 -- 172.21.15.183:0/3384053953 learned_addr learned my addr 172.21.15.183:0/3384053953 (peer_addr_for_me v2:172.21.15.183:0/0) 2024-02-20T07:50:43.022 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:43.019+0000 7f717de48700 1 -- 172.21.15.183:0/3384053953 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7174005040 con 0x7f7178102c70 2024-02-20T07:50:43.022 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:43.019+0000 7f717de48700 1 --2- 172.21.15.183:0/3384053953 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f7178102c70 0x7f71780fc2f0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f7174009300 tx=0x7f7174009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:50:43.022 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:43.019+0000 7f716e7fc700 1 -- 172.21.15.183:0/3384053953 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7174004e60 con 0x7f7178102c70 2024-02-20T07:50:43.022 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:43.019+0000 7f71800ac700 1 -- 172.21.15.183:0/3384053953 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f71780fca30 con 0x7f7178102c70 2024-02-20T07:50:43.022 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:43.019+0000 7f71800ac700 1 -- 172.21.15.183:0/3384053953 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f71780fcea0 con 0x7f7178102c70 2024-02-20T07:50:43.022 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:43.019+0000 7f716e7fc700 1 -- 172.21.15.183:0/3384053953 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7174013420 con 0x7f7178102c70 2024-02-20T07:50:43.022 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:43.019+0000 7f716e7fc700 1 -- 172.21.15.183:0/3384053953 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f71740255a0 con 0x7f7178102c70 2024-02-20T07:50:43.023 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:43.020+0000 7f71800ac700 1 -- 172.21.15.183:0/3384053953 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f715c0052e0 con 0x7f7178102c70 2024-02-20T07:50:43.023 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:43.020+0000 7f716e7fc700 1 -- 172.21.15.183:0/3384053953 <== mon.0 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 17) v1 ==== 43483+0+0 (secure 0 0 0) 0x7f7174009600 con 0x7f7178102c70 2024-02-20T07:50:43.023 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:43.020+0000 7f716e7fc700 1 --2- 172.21.15.183:0/3384053953 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7f7164036910 0x7f7164038dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:50:43.023 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:43.020+0000 7f716e7fc700 1 -- 172.21.15.183:0/3384053953 <== mon.0 v2:172.21.15.133:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1366+0+0 (secure 0 0 0) 0x7f7174049c40 con 0x7f7178102c70 2024-02-20T07:50:43.023 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:43.020+0000 7f717d647700 1 --2- 172.21.15.183:0/3384053953 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7f7164036910 0x7f7164038dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:50:43.023 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:43.021+0000 7f717d647700 1 --2- 172.21.15.183:0/3384053953 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7f7164036910 0x7f7164038dc0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f7168009990 tx=0x7f7168006e40 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:50:43.027 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:43.025+0000 7f716e7fc700 1 -- 172.21.15.183:0/3384053953 <== mon.0 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f7174013a90 con 0x7f7178102c70 2024-02-20T07:50:43.290 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:43.288+0000 7f71800ac700 1 -- 172.21.15.183:0/3384053953 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f715c005590 con 0x7f7178102c70 2024-02-20T07:50:43.291 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:43.289+0000 7f716e7fc700 1 -- 172.21.15.183:0/3384053953 <== mon.0 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+743 (secure 0 0 0) 0x7f7174024360 con 0x7f7178102c70 2024-02-20T07:50:43.291 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-02-20T07:50:43.291 INFO:teuthology.orchestra.run.smithi183.stdout:{"epoch":1,"fsid":"4537327c-cfc4-11ee-95bb-87774f69a715","modified":"2024-02-20T07:48:10.704626Z","created":"2024-02-20T07:48:10.704626Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi133","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:3300","nonce":0},{"type":"v1","addr":"172.21.15.133:6789","nonce":0}]},"addr":"172.21.15.133:6789/0","public_addr":"172.21.15.133:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T07:50:43.294 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:43.292+0000 7f71800ac700 1 -- 172.21.15.183:0/3384053953 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7f7164036910 msgr2=0x7f7164038dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:50:43.294 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:43.292+0000 7f71800ac700 1 --2- 172.21.15.183:0/3384053953 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7f7164036910 0x7f7164038dc0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f7168009990 tx=0x7f7168006e40 comp rx=0 tx=0).stop 2024-02-20T07:50:43.294 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:43.292+0000 7f71800ac700 1 -- 172.21.15.183:0/3384053953 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f7178102c70 msgr2=0x7f71780fc2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:50:43.294 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:43.292+0000 7f71800ac700 1 --2- 172.21.15.183:0/3384053953 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f7178102c70 0x7f71780fc2f0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f7174009300 tx=0x7f7174009e30 comp rx=0 tx=0).stop 2024-02-20T07:50:43.294 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:43.292+0000 7f71800ac700 1 -- 172.21.15.183:0/3384053953 shutdown_connections 2024-02-20T07:50:43.294 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:43.292+0000 7f71800ac700 1 --2- 172.21.15.183:0/3384053953 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7f7164036910 0x7f7164038dc0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:50:43.294 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:43.292+0000 7f71800ac700 1 --2- 172.21.15.183:0/3384053953 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f7178102c70 0x7f71780fc2f0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:50:43.294 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:43.292+0000 7f71800ac700 1 -- 172.21.15.183:0/3384053953 >> 172.21.15.183:0/3384053953 conn(0x7f71780720e0 msgr2=0x7f7178104270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:50:43.294 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:43.292+0000 7f71800ac700 1 -- 172.21.15.183:0/3384053953 shutdown_connections 2024-02-20T07:50:43.294 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:43.292+0000 7f71800ac700 1 -- 172.21.15.183:0/3384053953 wait complete. 2024-02-20T07:50:43.294 INFO:teuthology.orchestra.run.smithi183.stderr:dumped monmap epoch 1 2024-02-20T07:50:44.742 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:50:44 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:50:44.742 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:50:44 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:50:44.743 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:50:44 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:50:44.743 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:50:44 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:50:44.743 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:50:44 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-20T07:50:44.743 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:50:44 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-20T07:50:44.743 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:50:44 smithi133 ceph-mon[29117]: Deploying daemon mon.smithi183 on smithi183 2024-02-20T07:50:44.743 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:50:44 smithi133 ceph-mon[29117]: from='client.? 172.21.15.183:0/3384053953' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T07:50:44.743 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:50:44 smithi133 ceph-mon[29117]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T07:50:44.743 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:50:44 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:50:44.895 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T07:50:44.895 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4537327c-cfc4-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T07:50:45.939 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/4537327c-cfc4-11ee-95bb-87774f69a715/mon.smithi183/config 2024-02-20T07:50:46.992 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:50:46 smithi133 ceph-mon[29117]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T07:50:47.565 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:47.562+0000 7fc44029b700 1 -- 172.21.15.183:0/85429754 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fc438102d70 msgr2=0x7fc438103140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:50:47.566 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:47.562+0000 7fc44029b700 1 --2- 172.21.15.183:0/85429754 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fc438102d70 0x7fc438103140 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fc4340048c0 tx=0x7fc434009e30 comp rx=0 tx=0).stop 2024-02-20T07:50:47.566 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:47.562+0000 7fc44029b700 1 -- 172.21.15.183:0/85429754 shutdown_connections 2024-02-20T07:50:47.566 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:47.562+0000 7fc44029b700 1 --2- 172.21.15.183:0/85429754 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fc438102d70 0x7fc438103140 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:50:47.566 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:47.562+0000 7fc44029b700 1 -- 172.21.15.183:0/85429754 >> 172.21.15.183:0/85429754 conn(0x7fc438072180 msgr2=0x7fc438072580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:50:47.566 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:47.562+0000 7fc44029b700 1 -- 172.21.15.183:0/85429754 shutdown_connections 2024-02-20T07:50:47.566 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:47.562+0000 7fc44029b700 1 -- 172.21.15.183:0/85429754 wait complete. 2024-02-20T07:50:47.566 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:47.563+0000 7fc44029b700 1 Processor -- start 2024-02-20T07:50:47.566 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:47.563+0000 7fc44029b700 1 -- start start 2024-02-20T07:50:47.566 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:47.563+0000 7fc44029b700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fc438102d70 0x7fc438113060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:50:47.566 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:47.563+0000 7fc44029b700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4381135a0 con 0x7fc438102d70 2024-02-20T07:50:47.566 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:47.563+0000 7fc43e037700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fc438102d70 0x7fc438113060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:50:47.566 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:47.563+0000 7fc43e037700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fc438102d70 0x7fc438113060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.183:45464/0 (socket says 172.21.15.183:45464) 2024-02-20T07:50:47.567 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:47.563+0000 7fc43e037700 1 -- 172.21.15.183:0/3658686874 learned_addr learned my addr 172.21.15.183:0/3658686874 (peer_addr_for_me v2:172.21.15.183:0/0) 2024-02-20T07:50:47.567 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:47.564+0000 7fc43e037700 1 -- 172.21.15.183:0/3658686874 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc434005040 con 0x7fc438102d70 2024-02-20T07:50:47.567 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:47.564+0000 7fc43e037700 1 --2- 172.21.15.183:0/3658686874 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fc438102d70 0x7fc438113060 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fc434009330 tx=0x7fc434009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:50:47.567 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:47.564+0000 7fc42effd700 1 -- 172.21.15.183:0/3658686874 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc434004e60 con 0x7fc438102d70 2024-02-20T07:50:47.567 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:47.564+0000 7fc44029b700 1 -- 172.21.15.183:0/3658686874 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc438113800 con 0x7fc438102d70 2024-02-20T07:50:47.567 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:47.564+0000 7fc42effd700 1 -- 172.21.15.183:0/3658686874 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc434009630 con 0x7fc438102d70 2024-02-20T07:50:47.567 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:47.564+0000 7fc42effd700 1 -- 172.21.15.183:0/3658686874 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc43401c5b0 con 0x7fc438102d70 2024-02-20T07:50:47.567 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:47.564+0000 7fc44029b700 1 -- 172.21.15.183:0/3658686874 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc438113c70 con 0x7fc438102d70 2024-02-20T07:50:47.568 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:47.565+0000 7fc42effd700 1 -- 172.21.15.183:0/3658686874 <== mon.0 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 17) v1 ==== 43483+0+0 (secure 0 0 0) 0x7fc434013420 con 0x7fc438102d70 2024-02-20T07:50:47.568 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:47.565+0000 7fc44029b700 1 -- 172.21.15.183:0/3658686874 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc43805f5c0 con 0x7fc438102d70 2024-02-20T07:50:47.568 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:47.565+0000 7fc42effd700 1 --2- 172.21.15.183:0/3658686874 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7fc424036970 0x7fc424038e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:50:47.568 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:47.565+0000 7fc42effd700 1 -- 172.21.15.183:0/3658686874 <== mon.0 v2:172.21.15.133:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1366+0+0 (secure 0 0 0) 0x7fc434049e10 con 0x7fc438102d70 2024-02-20T07:50:47.569 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:47.565+0000 7fc43d836700 1 --2- 172.21.15.183:0/3658686874 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7fc424036970 0x7fc424038e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:50:47.569 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:47.566+0000 7fc43d836700 1 --2- 172.21.15.183:0/3658686874 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7fc424036970 0x7fc424038e20 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fc428009990 tx=0x7fc428006e40 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:50:47.571 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:47.570+0000 7fc42effd700 1 -- 172.21.15.183:0/3658686874 <== mon.0 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7fc434023300 con 0x7fc438102d70 2024-02-20T07:50:47.845 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:47.844+0000 7fc44029b700 1 -- 172.21.15.183:0/3658686874 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc43810e7c0 con 0x7fc438102d70 2024-02-20T07:50:47.846 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:47.845+0000 7fc42effd700 1 -- 172.21.15.183:0/3658686874 <== mon.0 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+743 (secure 0 0 0) 0x7fc4340116f0 con 0x7fc438102d70 2024-02-20T07:50:47.846 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-02-20T07:50:47.846 INFO:teuthology.orchestra.run.smithi183.stdout:{"epoch":1,"fsid":"4537327c-cfc4-11ee-95bb-87774f69a715","modified":"2024-02-20T07:48:10.704626Z","created":"2024-02-20T07:48:10.704626Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi133","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:3300","nonce":0},{"type":"v1","addr":"172.21.15.133:6789","nonce":0}]},"addr":"172.21.15.133:6789/0","public_addr":"172.21.15.133:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-20T07:50:47.850 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:47.848+0000 7fc44029b700 1 -- 172.21.15.183:0/3658686874 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7fc424036970 msgr2=0x7fc424038e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:50:47.850 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:47.848+0000 7fc44029b700 1 --2- 172.21.15.183:0/3658686874 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7fc424036970 0x7fc424038e20 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fc428009990 tx=0x7fc428006e40 comp rx=0 tx=0).stop 2024-02-20T07:50:47.850 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:47.848+0000 7fc44029b700 1 -- 172.21.15.183:0/3658686874 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fc438102d70 msgr2=0x7fc438113060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:50:47.850 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:47.848+0000 7fc44029b700 1 --2- 172.21.15.183:0/3658686874 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fc438102d70 0x7fc438113060 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fc434009330 tx=0x7fc434009e30 comp rx=0 tx=0).stop 2024-02-20T07:50:47.850 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:47.848+0000 7fc44029b700 1 -- 172.21.15.183:0/3658686874 shutdown_connections 2024-02-20T07:50:47.850 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:47.848+0000 7fc44029b700 1 --2- 172.21.15.183:0/3658686874 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7fc424036970 0x7fc424038e20 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:50:47.850 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:47.848+0000 7fc44029b700 1 --2- 172.21.15.183:0/3658686874 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fc438102d70 0x7fc438113060 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:50:47.850 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:47.848+0000 7fc44029b700 1 -- 172.21.15.183:0/3658686874 >> 172.21.15.183:0/3658686874 conn(0x7fc438072180 msgr2=0x7fc438104300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:50:47.850 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:47.848+0000 7fc44029b700 1 -- 172.21.15.183:0/3658686874 shutdown_connections 2024-02-20T07:50:47.850 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:47.848+0000 7fc44029b700 1 -- 172.21.15.183:0/3658686874 wait complete. 2024-02-20T07:50:47.851 INFO:teuthology.orchestra.run.smithi183.stderr:dumped monmap epoch 1 2024-02-20T07:50:48.548 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:50:48 smithi183 systemd[1]: Started Ceph mon.smithi183 for 4537327c-cfc4-11ee-95bb-87774f69a715. 2024-02-20T07:50:49.701 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-20T07:50:49.701 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4537327c-cfc4-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-20T07:50:49.956 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/4537327c-cfc4-11ee-95bb-87774f69a715/mon.smithi183/config 2024-02-20T07:50:53.742 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:50:53 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "mon metadata", "id": "smithi133"}]: dispatch 2024-02-20T07:50:53.743 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:50:53 smithi133 ceph-mon[29117]: mon.smithi133 calling monitor election 2024-02-20T07:50:53.743 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:50:53 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "mon metadata", "id": "smithi183"}]: dispatch 2024-02-20T07:50:53.743 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:50:53 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "mon metadata", "id": "smithi183"}]: dispatch 2024-02-20T07:50:53.743 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:50:53 smithi133 ceph-mon[29117]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T07:50:53.743 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:50:53 smithi133 ceph-mon[29117]: from='mgr.? 172.21.15.183:0/3676183933' entity='mgr.smithi183.hebfng' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi183.hebfng/crt"}]: dispatch 2024-02-20T07:50:53.743 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:50:53 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "mon metadata", "id": "smithi183"}]: dispatch 2024-02-20T07:50:53.743 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:50:53 smithi133 ceph-mon[29117]: mon.smithi183 calling monitor election 2024-02-20T07:50:53.743 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:50:53 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "mon metadata", "id": "smithi183"}]: dispatch 2024-02-20T07:50:53.743 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:50:53 smithi133 ceph-mon[29117]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T07:50:53.744 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:50:53 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "mon metadata", "id": "smithi183"}]: dispatch 2024-02-20T07:50:53.744 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:50:53 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "mon metadata", "id": "smithi183"}]: dispatch 2024-02-20T07:50:53.744 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:50:53 smithi133 ceph-mon[29117]: mon.smithi133 is new leader, mons smithi133,smithi183 in quorum (ranks 0,1) 2024-02-20T07:50:53.744 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:50:53 smithi133 ceph-mon[29117]: monmap e2: 2 mons at {smithi133=[v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0],smithi183=[v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-02-20T07:50:53.744 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:50:53 smithi133 ceph-mon[29117]: fsmap 2024-02-20T07:50:53.744 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:50:53 smithi133 ceph-mon[29117]: osdmap e5: 0 total, 0 up, 0 in 2024-02-20T07:50:53.744 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:50:53 smithi133 ceph-mon[29117]: mgrmap e17: smithi133.mrvcbp(active, since 29s) 2024-02-20T07:50:53.744 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:50:53 smithi133 ceph-mon[29117]: Standby manager daemon smithi183.hebfng started 2024-02-20T07:50:53.744 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:50:53 smithi133 ceph-mon[29117]: from='mgr.? 172.21.15.183:0/3676183933' entity='mgr.smithi183.hebfng' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-20T07:50:53.744 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:50:53 smithi133 ceph-mon[29117]: overall HEALTH_OK 2024-02-20T07:50:53.744 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:50:53 smithi133 ceph-mon[29117]: from='mgr.? 172.21.15.183:0/3676183933' entity='mgr.smithi183.hebfng' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi183.hebfng/key"}]: dispatch 2024-02-20T07:50:53.744 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:50:53 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:50:53.745 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:50:53 smithi133 ceph-mon[29117]: from='mgr.? 172.21.15.183:0/3676183933' entity='mgr.smithi183.hebfng' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-20T07:50:53.745 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:50:53 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:50:53.745 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:50:53 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:50:53.745 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:50:53 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:50:53.745 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:50:53 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:50:53.798 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:50:53 smithi183 ceph-mon[35832]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "mon metadata", "id": "smithi133"}]: dispatch 2024-02-20T07:50:53.798 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:50:53 smithi183 ceph-mon[35832]: mon.smithi133 calling monitor election 2024-02-20T07:50:53.798 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:50:53 smithi183 ceph-mon[35832]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "mon metadata", "id": "smithi183"}]: dispatch 2024-02-20T07:50:53.798 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:50:53 smithi183 ceph-mon[35832]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "mon metadata", "id": "smithi183"}]: dispatch 2024-02-20T07:50:53.798 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:50:53 smithi183 ceph-mon[35832]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T07:50:53.798 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:50:53 smithi183 ceph-mon[35832]: from='mgr.? 172.21.15.183:0/3676183933' entity='mgr.smithi183.hebfng' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi183.hebfng/crt"}]: dispatch 2024-02-20T07:50:53.798 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:50:53 smithi183 ceph-mon[35832]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "mon metadata", "id": "smithi183"}]: dispatch 2024-02-20T07:50:53.798 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:50:53 smithi183 ceph-mon[35832]: mon.smithi183 calling monitor election 2024-02-20T07:50:53.798 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:50:53 smithi183 ceph-mon[35832]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "mon metadata", "id": "smithi183"}]: dispatch 2024-02-20T07:50:53.799 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:50:53 smithi183 ceph-mon[35832]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T07:50:53.799 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:50:53 smithi183 ceph-mon[35832]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "mon metadata", "id": "smithi183"}]: dispatch 2024-02-20T07:50:53.799 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:50:53 smithi183 ceph-mon[35832]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "mon metadata", "id": "smithi183"}]: dispatch 2024-02-20T07:50:53.799 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:50:53 smithi183 ceph-mon[35832]: mon.smithi133 is new leader, mons smithi133,smithi183 in quorum (ranks 0,1) 2024-02-20T07:50:53.799 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:50:53 smithi183 ceph-mon[35832]: monmap e2: 2 mons at {smithi133=[v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0],smithi183=[v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-02-20T07:50:53.799 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:50:53 smithi183 ceph-mon[35832]: fsmap 2024-02-20T07:50:53.799 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:50:53 smithi183 ceph-mon[35832]: osdmap e5: 0 total, 0 up, 0 in 2024-02-20T07:50:53.799 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:50:53 smithi183 ceph-mon[35832]: mgrmap e17: smithi133.mrvcbp(active, since 29s) 2024-02-20T07:50:53.799 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:50:53 smithi183 ceph-mon[35832]: Standby manager daemon smithi183.hebfng started 2024-02-20T07:50:53.799 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:50:53 smithi183 ceph-mon[35832]: from='mgr.? 172.21.15.183:0/3676183933' entity='mgr.smithi183.hebfng' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-20T07:50:53.799 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:50:53 smithi183 ceph-mon[35832]: overall HEALTH_OK 2024-02-20T07:50:53.799 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:50:53 smithi183 ceph-mon[35832]: from='mgr.? 172.21.15.183:0/3676183933' entity='mgr.smithi183.hebfng' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi183.hebfng/key"}]: dispatch 2024-02-20T07:50:53.799 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:50:53 smithi183 ceph-mon[35832]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:50:53.799 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:50:53 smithi183 ceph-mon[35832]: from='mgr.? 172.21.15.183:0/3676183933' entity='mgr.smithi183.hebfng' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-20T07:50:53.799 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:50:53 smithi183 ceph-mon[35832]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:50:53.800 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:50:53 smithi183 ceph-mon[35832]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:50:53.800 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:50:53 smithi183 ceph-mon[35832]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:50:53.800 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:50:53 smithi183 ceph-mon[35832]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:50:54.262 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:54.259+0000 7f8d1fe4c700 1 -- 172.21.15.183:0/4241754217 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f8d04003580 msgr2=0x7f8d04005a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:50:54.262 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:54.259+0000 7f8d1fe4c700 1 --2- 172.21.15.183:0/4241754217 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f8d04003580 0x7f8d04005a00 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f8d180fcd10 tx=0x7f8d1400b040 comp rx=0 tx=0).stop 2024-02-20T07:50:54.262 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:54.259+0000 7f8d1fe4c700 1 -- 172.21.15.183:0/4241754217 shutdown_connections 2024-02-20T07:50:54.262 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:54.259+0000 7f8d1fe4c700 1 --2- 172.21.15.183:0/4241754217 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f8d04003580 0x7f8d04005a00 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:50:54.263 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:54.259+0000 7f8d1fe4c700 1 -- 172.21.15.183:0/4241754217 >> 172.21.15.183:0/4241754217 conn(0x7f8d180f8980 msgr2=0x7f8d180fad90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:50:54.263 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:54.259+0000 7f8d1fe4c700 1 -- 172.21.15.183:0/4241754217 shutdown_connections 2024-02-20T07:50:54.263 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:54.259+0000 7f8d1fe4c700 1 -- 172.21.15.183:0/4241754217 wait complete. 2024-02-20T07:50:54.265 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:54.260+0000 7f8d1fe4c700 1 Processor -- start 2024-02-20T07:50:54.265 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:54.260+0000 7f8d1fe4c700 1 -- start start 2024-02-20T07:50:54.265 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:54.261+0000 7f8d1fe4c700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f8d04003580 0x7f8d181131f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:50:54.265 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:54.261+0000 7f8d1fe4c700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f8d18113730 0x7f8d1810d0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:50:54.265 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:54.261+0000 7f8d1fe4c700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d18113d40 con 0x7f8d18113730 2024-02-20T07:50:54.265 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:54.261+0000 7f8d1fe4c700 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d18113eb0 con 0x7f8d04003580 2024-02-20T07:50:54.265 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:54.261+0000 7f8d1dbe8700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f8d04003580 0x7f8d181131f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:50:54.265 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:54.261+0000 7f8d1dbe8700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f8d04003580 0x7f8d181131f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.183:56276/0 (socket says 172.21.15.183:56276) 2024-02-20T07:50:54.265 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:54.261+0000 7f8d1dbe8700 1 -- 172.21.15.183:0/240667099 learned_addr learned my addr 172.21.15.183:0/240667099 (peer_addr_for_me v2:172.21.15.183:0/0) 2024-02-20T07:50:54.265 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:54.261+0000 7f8d1dbe8700 1 -- 172.21.15.183:0/240667099 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f8d04003580 msgr2=0x7f8d181131f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-02-20T07:50:54.266 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:54.261+0000 7f8d1dbe8700 1 -- 172.21.15.183:0/240667099 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f8d04003580 msgr2=0x7f8d181131f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-02-20T07:50:54.266 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:54.261+0000 7f8d1dbe8700 1 --2- 172.21.15.183:0/240667099 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f8d04003580 0x7f8d181131f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-02-20T07:50:54.266 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:54.261+0000 7f8d1dbe8700 1 --2- 172.21.15.183:0/240667099 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f8d04003580 0x7f8d181131f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-02-20T07:50:54.266 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:54.261+0000 7f8d1d3e7700 1 --2- 172.21.15.183:0/240667099 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f8d18113730 0x7f8d1810d0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:50:54.266 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:54.261+0000 7f8d1d3e7700 1 -- 172.21.15.183:0/240667099 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f8d04003580 msgr2=0x7f8d181131f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-02-20T07:50:54.266 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:54.261+0000 7f8d1d3e7700 1 --2- 172.21.15.183:0/240667099 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f8d04003580 0x7f8d181131f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:50:54.266 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:54.261+0000 7f8d1d3e7700 1 -- 172.21.15.183:0/240667099 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d14005040 con 0x7f8d18113730 2024-02-20T07:50:54.266 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:54.262+0000 7f8d1d3e7700 1 --2- 172.21.15.183:0/240667099 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f8d18113730 0x7f8d1810d0d0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f8d0800b790 tx=0x7f8d080091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:50:54.266 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:54.262+0000 7f8d0e7fc700 1 -- 172.21.15.183:0/240667099 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8d08011810 con 0x7f8d18113730 2024-02-20T07:50:54.266 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:54.262+0000 7f8d1fe4c700 1 -- 172.21.15.183:0/240667099 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d1810d730 con 0x7f8d18113730 2024-02-20T07:50:54.266 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:54.262+0000 7f8d0e7fc700 1 -- 172.21.15.183:0/240667099 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8d08011e40 con 0x7f8d18113730 2024-02-20T07:50:54.266 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:54.262+0000 7f8d0e7fc700 1 -- 172.21.15.183:0/240667099 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8d0800f2e0 con 0x7f8d18113730 2024-02-20T07:50:54.266 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:54.262+0000 7f8d1fe4c700 1 -- 172.21.15.183:0/240667099 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d1810dc70 con 0x7f8d18113730 2024-02-20T07:50:54.266 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:54.263+0000 7f8d1fe4c700 1 -- 172.21.15.183:0/240667099 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8d1810def0 con 0x7f8d18113730 2024-02-20T07:50:54.267 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:54.263+0000 7f8d0e7fc700 1 -- 172.21.15.183:0/240667099 <== mon.0 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 18) v1 ==== 86567+0+0 (secure 0 0 0) 0x7f8d08010380 con 0x7f8d18113730 2024-02-20T07:50:54.267 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:54.264+0000 7f8d0e7fc700 1 --2- 172.21.15.183:0/240667099 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7f8cfc069360 0x7f8cfc06b810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:50:54.267 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:54.264+0000 7f8d0e7fc700 1 -- 172.21.15.183:0/240667099 <== mon.0 v2:172.21.15.133:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1366+0+0 (secure 0 0 0) 0x7f8d08086400 con 0x7f8d18113730 2024-02-20T07:50:54.267 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:54.264+0000 7f8d1dbe8700 1 --2- 172.21.15.183:0/240667099 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7f8cfc069360 0x7f8cfc06b810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:50:54.268 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:54.264+0000 7f8d1dbe8700 1 --2- 172.21.15.183:0/240667099 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7f8cfc069360 0x7f8cfc06b810 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f8d140040d0 tx=0x7f8d14018040 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:50:54.271 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:54.268+0000 7f8d0e7fc700 1 -- 172.21.15.183:0/240667099 <== mon.0 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f8d080536f0 con 0x7f8d18113730 2024-02-20T07:50:54.559 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:54.556+0000 7f8d1fe4c700 1 -- 172.21.15.183:0/240667099 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f8d1805f5c0 con 0x7f8d18113730 2024-02-20T07:50:54.559 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:54.556+0000 7f8d0e7fc700 1 -- 172.21.15.183:0/240667099 <== mon.0 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 2 v2) v1 ==== 95+0+1021 (secure 0 0 0) 0x7f8d08015090 con 0x7f8d18113730 2024-02-20T07:50:54.559 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-02-20T07:50:54.559 INFO:teuthology.orchestra.run.smithi183.stdout:{"epoch":2,"fsid":"4537327c-cfc4-11ee-95bb-87774f69a715","modified":"2024-02-20T07:50:48.294246Z","created":"2024-02-20T07:48:10.704626Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi133","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:3300","nonce":0},{"type":"v1","addr":"172.21.15.133:6789","nonce":0}]},"addr":"172.21.15.133:6789/0","public_addr":"172.21.15.133:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"smithi183","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:3300","nonce":0},{"type":"v1","addr":"172.21.15.183:6789","nonce":0}]},"addr":"172.21.15.183:6789/0","public_addr":"172.21.15.183:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-02-20T07:50:54.561 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:54.558+0000 7f8d1fe4c700 1 -- 172.21.15.183:0/240667099 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7f8cfc069360 msgr2=0x7f8cfc06b810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:50:54.561 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:54.558+0000 7f8d1fe4c700 1 --2- 172.21.15.183:0/240667099 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7f8cfc069360 0x7f8cfc06b810 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f8d140040d0 tx=0x7f8d14018040 comp rx=0 tx=0).stop 2024-02-20T07:50:54.561 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:54.559+0000 7f8d1fe4c700 1 -- 172.21.15.183:0/240667099 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f8d18113730 msgr2=0x7f8d1810d0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:50:54.561 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:54.559+0000 7f8d1fe4c700 1 --2- 172.21.15.183:0/240667099 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f8d18113730 0x7f8d1810d0d0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f8d0800b790 tx=0x7f8d080091a0 comp rx=0 tx=0).stop 2024-02-20T07:50:54.561 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:54.559+0000 7f8d1fe4c700 1 -- 172.21.15.183:0/240667099 shutdown_connections 2024-02-20T07:50:54.562 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:54.559+0000 7f8d1fe4c700 1 --2- 172.21.15.183:0/240667099 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7f8cfc069360 0x7f8cfc06b810 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:50:54.562 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:54.559+0000 7f8d1fe4c700 1 --2- 172.21.15.183:0/240667099 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f8d04003580 0x7f8d181131f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:50:54.562 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:54.559+0000 7f8d1fe4c700 1 --2- 172.21.15.183:0/240667099 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f8d18113730 0x7f8d1810d0d0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:50:54.562 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:54.559+0000 7f8d1fe4c700 1 -- 172.21.15.183:0/240667099 >> 172.21.15.183:0/240667099 conn(0x7f8d180f8980 msgr2=0x7f8d180fe980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:50:54.562 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:54.559+0000 7f8d1fe4c700 1 -- 172.21.15.183:0/240667099 shutdown_connections 2024-02-20T07:50:54.562 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:50:54.559+0000 7f8d1fe4c700 1 -- 172.21.15.183:0/240667099 wait complete. 2024-02-20T07:50:54.562 INFO:teuthology.orchestra.run.smithi183.stderr:dumped monmap epoch 2 2024-02-20T07:50:54.694 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:50:54 smithi183 ceph-mon[35832]: mgrmap e18: smithi133.mrvcbp(active, since 29s), standbys: smithi183.hebfng 2024-02-20T07:50:54.694 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:50:54 smithi183 ceph-mon[35832]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "mgr metadata", "who": "smithi183.hebfng", "id": "smithi183.hebfng"}]: dispatch 2024-02-20T07:50:54.694 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:50:54 smithi183 ceph-mon[35832]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-20T07:50:54.694 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:50:54 smithi183 ceph-mon[35832]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T07:50:54.695 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:50:54 smithi183 ceph-mon[35832]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-20T07:50:54.695 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:50:54 smithi183 ceph-mon[35832]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:50:54.695 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:50:54 smithi183 ceph-mon[35832]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "mon metadata", "id": "smithi183"}]: dispatch 2024-02-20T07:50:54.742 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:50:54 smithi133 ceph-mon[29117]: mgrmap e18: smithi133.mrvcbp(active, since 29s), standbys: smithi183.hebfng 2024-02-20T07:50:54.742 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:50:54 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "mgr metadata", "who": "smithi183.hebfng", "id": "smithi183.hebfng"}]: dispatch 2024-02-20T07:50:54.742 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:50:54 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-20T07:50:54.742 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:50:54 smithi133 ceph-mon[29117]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T07:50:54.743 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:50:54 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-20T07:50:54.743 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:50:54 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:50:54.743 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:50:54 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "mon metadata", "id": "smithi183"}]: dispatch 2024-02-20T07:50:55.506 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-02-20T07:50:55.506 DEBUG:teuthology.orchestra.run.smithi133:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4537327c-cfc4-11ee-95bb-87774f69a715 -- ceph config generate-minimal-conf 2024-02-20T07:50:55.742 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:50:55 smithi133 ceph-mon[29117]: from='client.? 172.21.15.183:0/240667099' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T07:50:55.770 INFO:teuthology.orchestra.run.smithi133.stderr:Inferring config /var/lib/ceph/4537327c-cfc4-11ee-95bb-87774f69a715/mon.smithi133/config 2024-02-20T07:50:55.797 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:50:55 smithi183 ceph-mon[35832]: from='client.? 172.21.15.183:0/240667099' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-20T07:50:57.136 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:50:56 smithi133 ceph-mon[29117]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T07:50:57.137 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:50:56 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:50:57.137 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:50:56 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:50:57.137 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:50:56 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-20T07:50:57.137 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:50:56 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-20T07:50:57.137 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:50:56 smithi133 ceph-mon[29117]: Updating smithi133:/etc/ceph/ceph.conf 2024-02-20T07:50:57.137 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:50:56 smithi133 ceph-mon[29117]: Updating smithi183:/etc/ceph/ceph.conf 2024-02-20T07:50:57.137 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:50:56 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:50:57.137 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:50:56 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:50:57.137 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:50:56 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:50:57.137 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:50:56 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:50:57.137 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:50:56 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:50:57.137 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:50:56 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-20T07:50:57.137 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:50:56 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-20T07:50:57.138 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:50:56 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-20T07:50:57.298 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:50:56 smithi183 ceph-mon[35832]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T07:50:57.298 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:50:56 smithi183 ceph-mon[35832]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:50:57.298 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:50:56 smithi183 ceph-mon[35832]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:50:57.298 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:50:56 smithi183 ceph-mon[35832]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-20T07:50:57.298 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:50:56 smithi183 ceph-mon[35832]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-20T07:50:57.298 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:50:56 smithi183 ceph-mon[35832]: Updating smithi133:/etc/ceph/ceph.conf 2024-02-20T07:50:57.298 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:50:56 smithi183 ceph-mon[35832]: Updating smithi183:/etc/ceph/ceph.conf 2024-02-20T07:50:57.298 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:50:56 smithi183 ceph-mon[35832]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:50:57.298 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:50:56 smithi183 ceph-mon[35832]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:50:57.298 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:50:56 smithi183 ceph-mon[35832]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:50:57.298 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:50:56 smithi183 ceph-mon[35832]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:50:57.299 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:50:56 smithi183 ceph-mon[35832]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:50:57.299 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:50:56 smithi183 ceph-mon[35832]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-20T07:50:57.299 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:50:56 smithi183 ceph-mon[35832]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-20T07:50:57.299 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:50:56 smithi183 ceph-mon[35832]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-20T07:50:57.972 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:50:57.969+0000 7f5524028700 1 -- 172.21.15.133:0/1034475149 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f551c102bd0 msgr2=0x7f551c102fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:50:57.972 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:50:57.969+0000 7f5524028700 1 --2- 172.21.15.133:0/1034475149 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f551c102bd0 0x7f551c102fa0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f550c0048c0 tx=0x7f550c009e30 comp rx=0 tx=0).stop 2024-02-20T07:50:57.972 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:50:57.969+0000 7f5524028700 1 -- 172.21.15.133:0/1034475149 shutdown_connections 2024-02-20T07:50:57.972 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:50:57.969+0000 7f5524028700 1 --2- 172.21.15.133:0/1034475149 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f551c102bd0 0x7f551c102fa0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:50:57.972 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:50:57.969+0000 7f5524028700 1 -- 172.21.15.133:0/1034475149 >> 172.21.15.133:0/1034475149 conn(0x7f551c0f8760 msgr2=0x7f551c0fab70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:50:57.972 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:50:57.969+0000 7f5524028700 1 -- 172.21.15.133:0/1034475149 shutdown_connections 2024-02-20T07:50:57.972 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:50:57.969+0000 7f5524028700 1 -- 172.21.15.133:0/1034475149 wait complete. 2024-02-20T07:50:57.972 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:50:57.970+0000 7f5524028700 1 Processor -- start 2024-02-20T07:50:57.973 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:50:57.970+0000 7f5524028700 1 -- start start 2024-02-20T07:50:57.973 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:50:57.970+0000 7f5524028700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f551c102bd0 0x7f551c070290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:50:57.973 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:50:57.970+0000 7f5524028700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f551c0707d0 0x7f551c070c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:50:57.973 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:50:57.970+0000 7f5524028700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f551c1193a0 con 0x7f551c0707d0 2024-02-20T07:50:57.973 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:50:57.970+0000 7f5524028700 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f551c1194e0 con 0x7f551c102bd0 2024-02-20T07:50:57.973 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:50:57.970+0000 7f55215c3700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f551c0707d0 0x7f551c070c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:50:57.973 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:50:57.970+0000 7f55215c3700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f551c0707d0 0x7f551c070c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.133:55596/0 (socket says 172.21.15.133:55596) 2024-02-20T07:50:57.973 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:50:57.970+0000 7f55215c3700 1 -- 172.21.15.133:0/4176269015 learned_addr learned my addr 172.21.15.133:0/4176269015 (peer_addr_for_me v2:172.21.15.133:0/0) 2024-02-20T07:50:57.973 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:50:57.970+0000 7f55215c3700 1 -- 172.21.15.133:0/4176269015 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f551c102bd0 msgr2=0x7f551c070290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:50:57.973 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:50:57.970+0000 7f55215c3700 1 --2- 172.21.15.133:0/4176269015 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f551c102bd0 0x7f551c070290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:50:57.973 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:50:57.970+0000 7f55215c3700 1 -- 172.21.15.133:0/4176269015 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f550c005040 con 0x7f551c0707d0 2024-02-20T07:50:57.973 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:50:57.970+0000 7f55215c3700 1 --2- 172.21.15.133:0/4176269015 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f551c0707d0 0x7f551c070c40 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f551800b790 tx=0x7f5518009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:50:57.973 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:50:57.970+0000 7f55127fc700 1 -- 172.21.15.133:0/4176269015 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5518010870 con 0x7f551c0707d0 2024-02-20T07:50:57.974 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:50:57.970+0000 7f5524028700 1 -- 172.21.15.133:0/4176269015 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f551c119740 con 0x7f551c0707d0 2024-02-20T07:50:57.974 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:50:57.971+0000 7f55127fc700 1 -- 172.21.15.133:0/4176269015 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f551800f3e0 con 0x7f551c0707d0 2024-02-20T07:50:57.974 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:50:57.971+0000 7f55127fc700 1 -- 172.21.15.133:0/4176269015 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5518010ea0 con 0x7f551c0707d0 2024-02-20T07:50:57.974 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:50:57.971+0000 7f5524028700 1 -- 172.21.15.133:0/4176269015 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f551c119cd0 con 0x7f551c0707d0 2024-02-20T07:50:57.974 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:50:57.972+0000 7f5524028700 1 -- 172.21.15.133:0/4176269015 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f551c101ef0 con 0x7f551c0707d0 2024-02-20T07:50:57.975 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:50:57.972+0000 7f55127fc700 1 -- 172.21.15.133:0/4176269015 <== mon.0 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 18) v1 ==== 86567+0+0 (secure 0 0 0) 0x7f551800f550 con 0x7f551c0707d0 2024-02-20T07:50:57.975 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:50:57.972+0000 7f55127fc700 1 --2- 172.21.15.133:0/4176269015 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7f5508069370 0x7f550806b820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:50:57.975 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:50:57.973+0000 7f55127fc700 1 -- 172.21.15.133:0/4176269015 <== mon.0 v2:172.21.15.133:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1366+0+0 (secure 0 0 0) 0x7f55180866d0 con 0x7f551c0707d0 2024-02-20T07:50:57.975 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:50:57.973+0000 7f5521dc4700 1 --2- 172.21.15.133:0/4176269015 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7f5508069370 0x7f550806b820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:50:57.975 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:50:57.973+0000 7f5521dc4700 1 --2- 172.21.15.133:0/4176269015 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7f5508069370 0x7f550806b820 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f550c009b80 tx=0x7f550c009e30 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:50:57.978 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:50:57.977+0000 7f55127fc700 1 -- 172.21.15.133:0/4176269015 <== mon.0 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f5518053a60 con 0x7f551c0707d0 2024-02-20T07:50:58.165 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:50:58.164+0000 7f5524028700 1 -- 172.21.15.133:0/4176269015 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) v1 -- 0x7f551c05f5c0 con 0x7f551c0707d0 2024-02-20T07:50:58.165 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:50:57 smithi133 ceph-mon[29117]: Updating smithi133:/var/lib/ceph/4537327c-cfc4-11ee-95bb-87774f69a715/config/ceph.conf 2024-02-20T07:50:58.165 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:50:57 smithi133 ceph-mon[29117]: Updating smithi183:/var/lib/ceph/4537327c-cfc4-11ee-95bb-87774f69a715/config/ceph.conf 2024-02-20T07:50:58.165 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:50:57 smithi133 ceph-mon[29117]: Reconfiguring mon.smithi133 (unknown last config time)... 2024-02-20T07:50:58.165 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:50:57 smithi133 ceph-mon[29117]: Reconfiguring daemon mon.smithi133 on smithi133 2024-02-20T07:50:58.166 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:50:58.164+0000 7f55127fc700 1 -- 172.21.15.133:0/4176269015 <== mon.0 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v10) v1 ==== 76+0+227 (secure 0 0 0) 0x7f5518014100 con 0x7f551c0707d0 2024-02-20T07:50:58.166 INFO:teuthology.orchestra.run.smithi133.stdout:# minimal ceph.conf for 4537327c-cfc4-11ee-95bb-87774f69a715 2024-02-20T07:50:58.166 INFO:teuthology.orchestra.run.smithi133.stdout:[global] 2024-02-20T07:50:58.166 INFO:teuthology.orchestra.run.smithi133.stdout: fsid = 4537327c-cfc4-11ee-95bb-87774f69a715 2024-02-20T07:50:58.166 INFO:teuthology.orchestra.run.smithi133.stdout: mon_host = [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] 2024-02-20T07:50:58.167 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:50:58.166+0000 7f5524028700 1 -- 172.21.15.133:0/4176269015 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7f5508069370 msgr2=0x7f550806b820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:50:58.167 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:50:58.166+0000 7f5524028700 1 --2- 172.21.15.133:0/4176269015 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7f5508069370 0x7f550806b820 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f550c009b80 tx=0x7f550c009e30 comp rx=0 tx=0).stop 2024-02-20T07:50:58.167 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:50:58.166+0000 7f5524028700 1 -- 172.21.15.133:0/4176269015 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f551c0707d0 msgr2=0x7f551c070c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:50:58.167 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:50:58.166+0000 7f5524028700 1 --2- 172.21.15.133:0/4176269015 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f551c0707d0 0x7f551c070c40 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f551800b790 tx=0x7f5518009250 comp rx=0 tx=0).stop 2024-02-20T07:50:58.168 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:50:58.166+0000 7f5524028700 1 -- 172.21.15.133:0/4176269015 shutdown_connections 2024-02-20T07:50:58.168 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:50:58.166+0000 7f5524028700 1 --2- 172.21.15.133:0/4176269015 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7f5508069370 0x7f550806b820 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:50:58.168 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:50:58.166+0000 7f5524028700 1 --2- 172.21.15.133:0/4176269015 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f551c102bd0 0x7f551c070290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:50:58.168 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:50:58.166+0000 7f5524028700 1 --2- 172.21.15.133:0/4176269015 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f551c0707d0 0x7f551c070c40 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:50:58.168 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:50:58.166+0000 7f5524028700 1 -- 172.21.15.133:0/4176269015 >> 172.21.15.133:0/4176269015 conn(0x7f551c0f8760 msgr2=0x7f551c0fa6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:50:58.168 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:50:58.166+0000 7f5524028700 1 -- 172.21.15.133:0/4176269015 shutdown_connections 2024-02-20T07:50:58.168 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:50:58.166+0000 7f5524028700 1 -- 172.21.15.133:0/4176269015 wait complete. 2024-02-20T07:50:58.298 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:50:57 smithi183 ceph-mon[35832]: Updating smithi133:/var/lib/ceph/4537327c-cfc4-11ee-95bb-87774f69a715/config/ceph.conf 2024-02-20T07:50:58.298 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:50:57 smithi183 ceph-mon[35832]: Updating smithi183:/var/lib/ceph/4537327c-cfc4-11ee-95bb-87774f69a715/config/ceph.conf 2024-02-20T07:50:58.298 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:50:57 smithi183 ceph-mon[35832]: Reconfiguring mon.smithi133 (unknown last config time)... 2024-02-20T07:50:58.298 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:50:57 smithi183 ceph-mon[35832]: Reconfiguring daemon mon.smithi133 on smithi133 2024-02-20T07:50:59.009 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-02-20T07:50:59.010 DEBUG:teuthology.orchestra.run.smithi133:> set -ex 2024-02-20T07:50:59.010 DEBUG:teuthology.orchestra.run.smithi133:> sudo dd of=/etc/ceph/ceph.conf 2024-02-20T07:50:59.040 DEBUG:teuthology.orchestra.run.smithi133:> set -ex 2024-02-20T07:50:59.041 DEBUG:teuthology.orchestra.run.smithi133:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-02-20T07:50:59.108 DEBUG:teuthology.orchestra.run.smithi183:> set -ex 2024-02-20T07:50:59.108 DEBUG:teuthology.orchestra.run.smithi183:> sudo dd of=/etc/ceph/ceph.conf 2024-02-20T07:50:59.143 DEBUG:teuthology.orchestra.run.smithi183:> set -ex 2024-02-20T07:50:59.143 DEBUG:teuthology.orchestra.run.smithi183:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-02-20T07:50:59.159 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:50:58 smithi133 ceph-mon[29117]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T07:50:59.159 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:50:58 smithi133 ceph-mon[29117]: from='client.? 172.21.15.133:0/4176269015' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-20T07:50:59.159 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:50:58 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:50:59.159 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:50:58 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:50:59.159 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:50:58 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi133.mrvcbp", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-20T07:50:59.159 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:50:58 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-20T07:50:59.159 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:50:58 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-20T07:50:59.198 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:50:58 smithi183 ceph-mon[35832]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T07:50:59.198 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:50:58 smithi183 ceph-mon[35832]: from='client.? 172.21.15.133:0/4176269015' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-20T07:50:59.199 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:50:58 smithi183 ceph-mon[35832]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:50:59.199 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:50:58 smithi183 ceph-mon[35832]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:50:59.199 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:50:58 smithi183 ceph-mon[35832]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi133.mrvcbp", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-20T07:50:59.199 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:50:58 smithi183 ceph-mon[35832]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-20T07:50:59.199 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:50:58 smithi183 ceph-mon[35832]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-20T07:50:59.218 INFO:tasks.cephadm:Deploying OSDs... 2024-02-20T07:50:59.219 DEBUG:teuthology.orchestra.run.smithi133:> set -ex 2024-02-20T07:50:59.219 DEBUG:teuthology.orchestra.run.smithi133:> dd if=/scratch_devs of=/dev/stdout 2024-02-20T07:50:59.235 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-02-20T07:50:59.235 DEBUG:teuthology.orchestra.run.smithi133:> stat /dev/vg_nvme/lv_1 2024-02-20T07:50:59.289 INFO:teuthology.orchestra.run.smithi133.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-02-20T07:50:59.290 INFO:teuthology.orchestra.run.smithi133.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-20T07:50:59.290 INFO:teuthology.orchestra.run.smithi133.stdout:Device: 5h/5d Inode: 665 Links: 1 2024-02-20T07:50:59.290 INFO:teuthology.orchestra.run.smithi133.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-20T07:50:59.290 INFO:teuthology.orchestra.run.smithi133.stdout:Context: system_u:object_r:device_t:s0 2024-02-20T07:50:59.290 INFO:teuthology.orchestra.run.smithi133.stdout:Access: 2024-02-20 07:50:15.292844355 +0000 2024-02-20T07:50:59.290 INFO:teuthology.orchestra.run.smithi133.stdout:Modify: 2024-02-20 07:44:34.960577334 +0000 2024-02-20T07:50:59.290 INFO:teuthology.orchestra.run.smithi133.stdout:Change: 2024-02-20 07:44:34.960577334 +0000 2024-02-20T07:50:59.290 INFO:teuthology.orchestra.run.smithi133.stdout: Birth: 2024-02-20 07:44:34.960577334 +0000 2024-02-20T07:50:59.290 DEBUG:teuthology.orchestra.run.smithi133:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-02-20T07:50:59.356 INFO:teuthology.orchestra.run.smithi133.stderr:1+0 records in 2024-02-20T07:50:59.356 INFO:teuthology.orchestra.run.smithi133.stderr:1+0 records out 2024-02-20T07:50:59.356 INFO:teuthology.orchestra.run.smithi133.stderr:512 bytes copied, 0.000230214 s, 2.2 MB/s 2024-02-20T07:50:59.357 DEBUG:teuthology.orchestra.run.smithi133:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-02-20T07:50:59.413 DEBUG:teuthology.orchestra.run.smithi133:> stat /dev/vg_nvme/lv_2 2024-02-20T07:50:59.469 INFO:teuthology.orchestra.run.smithi133.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-02-20T07:50:59.469 INFO:teuthology.orchestra.run.smithi133.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-20T07:50:59.470 INFO:teuthology.orchestra.run.smithi133.stdout:Device: 5h/5d Inode: 685 Links: 1 2024-02-20T07:50:59.470 INFO:teuthology.orchestra.run.smithi133.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-20T07:50:59.470 INFO:teuthology.orchestra.run.smithi133.stdout:Context: system_u:object_r:device_t:s0 2024-02-20T07:50:59.470 INFO:teuthology.orchestra.run.smithi133.stdout:Access: 2024-02-20 07:50:15.293844336 +0000 2024-02-20T07:50:59.470 INFO:teuthology.orchestra.run.smithi133.stdout:Modify: 2024-02-20 07:44:35.314570761 +0000 2024-02-20T07:50:59.470 INFO:teuthology.orchestra.run.smithi133.stdout:Change: 2024-02-20 07:44:35.314570761 +0000 2024-02-20T07:50:59.470 INFO:teuthology.orchestra.run.smithi133.stdout: Birth: 2024-02-20 07:44:35.314570761 +0000 2024-02-20T07:50:59.470 DEBUG:teuthology.orchestra.run.smithi133:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-02-20T07:50:59.535 INFO:teuthology.orchestra.run.smithi133.stderr:1+0 records in 2024-02-20T07:50:59.535 INFO:teuthology.orchestra.run.smithi133.stderr:1+0 records out 2024-02-20T07:50:59.535 INFO:teuthology.orchestra.run.smithi133.stderr:512 bytes copied, 0.000239308 s, 2.1 MB/s 2024-02-20T07:50:59.536 DEBUG:teuthology.orchestra.run.smithi133:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-02-20T07:50:59.594 DEBUG:teuthology.orchestra.run.smithi133:> stat /dev/vg_nvme/lv_3 2024-02-20T07:50:59.648 INFO:teuthology.orchestra.run.smithi133.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-02-20T07:50:59.649 INFO:teuthology.orchestra.run.smithi133.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-20T07:50:59.649 INFO:teuthology.orchestra.run.smithi133.stdout:Device: 5h/5d Inode: 704 Links: 1 2024-02-20T07:50:59.649 INFO:teuthology.orchestra.run.smithi133.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-20T07:50:59.649 INFO:teuthology.orchestra.run.smithi133.stdout:Context: system_u:object_r:device_t:s0 2024-02-20T07:50:59.649 INFO:teuthology.orchestra.run.smithi133.stdout:Access: 2024-02-20 07:50:15.293844336 +0000 2024-02-20T07:50:59.649 INFO:teuthology.orchestra.run.smithi133.stdout:Modify: 2024-02-20 07:44:35.664564262 +0000 2024-02-20T07:50:59.649 INFO:teuthology.orchestra.run.smithi133.stdout:Change: 2024-02-20 07:44:35.664564262 +0000 2024-02-20T07:50:59.649 INFO:teuthology.orchestra.run.smithi133.stdout: Birth: 2024-02-20 07:44:35.664564262 +0000 2024-02-20T07:50:59.649 DEBUG:teuthology.orchestra.run.smithi133:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-02-20T07:50:59.714 INFO:teuthology.orchestra.run.smithi133.stderr:1+0 records in 2024-02-20T07:50:59.714 INFO:teuthology.orchestra.run.smithi133.stderr:1+0 records out 2024-02-20T07:50:59.714 INFO:teuthology.orchestra.run.smithi133.stderr:512 bytes copied, 0.000213884 s, 2.4 MB/s 2024-02-20T07:50:59.715 DEBUG:teuthology.orchestra.run.smithi133:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-02-20T07:50:59.771 DEBUG:teuthology.orchestra.run.smithi133:> stat /dev/vg_nvme/lv_4 2024-02-20T07:50:59.827 INFO:teuthology.orchestra.run.smithi133.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-02-20T07:50:59.827 INFO:teuthology.orchestra.run.smithi133.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-20T07:50:59.828 INFO:teuthology.orchestra.run.smithi133.stdout:Device: 5h/5d Inode: 723 Links: 1 2024-02-20T07:50:59.828 INFO:teuthology.orchestra.run.smithi133.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-20T07:50:59.828 INFO:teuthology.orchestra.run.smithi133.stdout:Context: system_u:object_r:device_t:s0 2024-02-20T07:50:59.829 INFO:teuthology.orchestra.run.smithi133.stdout:Access: 2024-02-20 07:50:15.293844336 +0000 2024-02-20T07:50:59.829 INFO:teuthology.orchestra.run.smithi133.stdout:Modify: 2024-02-20 07:44:36.018557690 +0000 2024-02-20T07:50:59.829 INFO:teuthology.orchestra.run.smithi133.stdout:Change: 2024-02-20 07:44:36.018557690 +0000 2024-02-20T07:50:59.829 INFO:teuthology.orchestra.run.smithi133.stdout: Birth: 2024-02-20 07:44:36.018557690 +0000 2024-02-20T07:50:59.830 DEBUG:teuthology.orchestra.run.smithi133:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-02-20T07:50:59.894 INFO:teuthology.orchestra.run.smithi133.stderr:1+0 records in 2024-02-20T07:50:59.894 INFO:teuthology.orchestra.run.smithi133.stderr:1+0 records out 2024-02-20T07:50:59.895 INFO:teuthology.orchestra.run.smithi133.stderr:512 bytes copied, 0.000208323 s, 2.5 MB/s 2024-02-20T07:50:59.896 DEBUG:teuthology.orchestra.run.smithi133:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-02-20T07:50:59.951 DEBUG:teuthology.orchestra.run.smithi183:> set -ex 2024-02-20T07:50:59.952 DEBUG:teuthology.orchestra.run.smithi183:> dd if=/scratch_devs of=/dev/stdout 2024-02-20T07:50:59.969 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-02-20T07:50:59.969 DEBUG:teuthology.orchestra.run.smithi183:> stat /dev/vg_nvme/lv_1 2024-02-20T07:50:59.992 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:50:59 smithi133 ceph-mon[29117]: Reconfiguring mgr.smithi133.mrvcbp (unknown last config time)... 2024-02-20T07:50:59.992 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:50:59 smithi133 ceph-mon[29117]: Reconfiguring daemon mgr.smithi133.mrvcbp on smithi133 2024-02-20T07:51:00.024 INFO:teuthology.orchestra.run.smithi183.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-02-20T07:51:00.024 INFO:teuthology.orchestra.run.smithi183.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-20T07:51:00.024 INFO:teuthology.orchestra.run.smithi183.stdout:Device: 5h/5d Inode: 665 Links: 1 2024-02-20T07:51:00.025 INFO:teuthology.orchestra.run.smithi183.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-20T07:51:00.025 INFO:teuthology.orchestra.run.smithi183.stdout:Context: system_u:object_r:device_t:s0 2024-02-20T07:51:00.025 INFO:teuthology.orchestra.run.smithi183.stdout:Access: 2024-02-20 07:50:46.199100995 +0000 2024-02-20T07:51:00.025 INFO:teuthology.orchestra.run.smithi183.stdout:Modify: 2024-02-20 07:44:35.000318224 +0000 2024-02-20T07:51:00.025 INFO:teuthology.orchestra.run.smithi183.stdout:Change: 2024-02-20 07:44:35.000318224 +0000 2024-02-20T07:51:00.025 INFO:teuthology.orchestra.run.smithi183.stdout: Birth: 2024-02-20 07:44:35.000318224 +0000 2024-02-20T07:51:00.025 DEBUG:teuthology.orchestra.run.smithi183:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-02-20T07:51:00.090 INFO:teuthology.orchestra.run.smithi183.stderr:1+0 records in 2024-02-20T07:51:00.090 INFO:teuthology.orchestra.run.smithi183.stderr:1+0 records out 2024-02-20T07:51:00.090 INFO:teuthology.orchestra.run.smithi183.stderr:512 bytes copied, 0.000252598 s, 2.0 MB/s 2024-02-20T07:51:00.091 DEBUG:teuthology.orchestra.run.smithi183:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-02-20T07:51:00.148 DEBUG:teuthology.orchestra.run.smithi183:> stat /dev/vg_nvme/lv_2 2024-02-20T07:51:00.207 INFO:teuthology.orchestra.run.smithi183.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-02-20T07:51:00.207 INFO:teuthology.orchestra.run.smithi183.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-20T07:51:00.207 INFO:teuthology.orchestra.run.smithi183.stdout:Device: 5h/5d Inode: 685 Links: 1 2024-02-20T07:51:00.207 INFO:teuthology.orchestra.run.smithi183.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-20T07:51:00.207 INFO:teuthology.orchestra.run.smithi183.stdout:Context: system_u:object_r:device_t:s0 2024-02-20T07:51:00.207 INFO:teuthology.orchestra.run.smithi183.stdout:Access: 2024-02-20 07:50:46.199100995 +0000 2024-02-20T07:51:00.207 INFO:teuthology.orchestra.run.smithi183.stdout:Modify: 2024-02-20 07:44:35.331309275 +0000 2024-02-20T07:51:00.208 INFO:teuthology.orchestra.run.smithi183.stdout:Change: 2024-02-20 07:44:35.331309275 +0000 2024-02-20T07:51:00.208 INFO:teuthology.orchestra.run.smithi183.stdout: Birth: 2024-02-20 07:44:35.331309275 +0000 2024-02-20T07:51:00.208 DEBUG:teuthology.orchestra.run.smithi183:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-02-20T07:51:00.270 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:50:59 smithi183 ceph-mon[35832]: Reconfiguring mgr.smithi133.mrvcbp (unknown last config time)... 2024-02-20T07:51:00.270 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:50:59 smithi183 ceph-mon[35832]: Reconfiguring daemon mgr.smithi133.mrvcbp on smithi133 2024-02-20T07:51:00.273 INFO:teuthology.orchestra.run.smithi183.stderr:1+0 records in 2024-02-20T07:51:00.273 INFO:teuthology.orchestra.run.smithi183.stderr:1+0 records out 2024-02-20T07:51:00.273 INFO:teuthology.orchestra.run.smithi183.stderr:512 bytes copied, 0.000175181 s, 2.9 MB/s 2024-02-20T07:51:00.274 DEBUG:teuthology.orchestra.run.smithi183:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-02-20T07:51:00.330 DEBUG:teuthology.orchestra.run.smithi183:> stat /dev/vg_nvme/lv_3 2024-02-20T07:51:00.385 INFO:teuthology.orchestra.run.smithi183.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-02-20T07:51:00.386 INFO:teuthology.orchestra.run.smithi183.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-20T07:51:00.386 INFO:teuthology.orchestra.run.smithi183.stdout:Device: 5h/5d Inode: 706 Links: 1 2024-02-20T07:51:00.386 INFO:teuthology.orchestra.run.smithi183.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-20T07:51:00.386 INFO:teuthology.orchestra.run.smithi183.stdout:Context: system_u:object_r:device_t:s0 2024-02-20T07:51:00.386 INFO:teuthology.orchestra.run.smithi183.stdout:Access: 2024-02-20 07:50:46.199100995 +0000 2024-02-20T07:51:00.386 INFO:teuthology.orchestra.run.smithi183.stdout:Modify: 2024-02-20 07:44:35.667300192 +0000 2024-02-20T07:51:00.387 INFO:teuthology.orchestra.run.smithi183.stdout:Change: 2024-02-20 07:44:35.667300192 +0000 2024-02-20T07:51:00.387 INFO:teuthology.orchestra.run.smithi183.stdout: Birth: 2024-02-20 07:44:35.667300192 +0000 2024-02-20T07:51:00.387 DEBUG:teuthology.orchestra.run.smithi183:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-02-20T07:51:00.452 INFO:teuthology.orchestra.run.smithi183.stderr:1+0 records in 2024-02-20T07:51:00.453 INFO:teuthology.orchestra.run.smithi183.stderr:1+0 records out 2024-02-20T07:51:00.453 INFO:teuthology.orchestra.run.smithi183.stderr:512 bytes copied, 0.00023324 s, 2.2 MB/s 2024-02-20T07:51:00.454 DEBUG:teuthology.orchestra.run.smithi183:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-02-20T07:51:00.510 DEBUG:teuthology.orchestra.run.smithi183:> stat /dev/vg_nvme/lv_4 2024-02-20T07:51:00.565 INFO:teuthology.orchestra.run.smithi183.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-02-20T07:51:00.565 INFO:teuthology.orchestra.run.smithi183.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-20T07:51:00.565 INFO:teuthology.orchestra.run.smithi183.stdout:Device: 5h/5d Inode: 722 Links: 1 2024-02-20T07:51:00.565 INFO:teuthology.orchestra.run.smithi183.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-20T07:51:00.565 INFO:teuthology.orchestra.run.smithi183.stdout:Context: system_u:object_r:device_t:s0 2024-02-20T07:51:00.565 INFO:teuthology.orchestra.run.smithi183.stdout:Access: 2024-02-20 07:50:46.200100968 +0000 2024-02-20T07:51:00.565 INFO:teuthology.orchestra.run.smithi183.stdout:Modify: 2024-02-20 07:44:36.012290865 +0000 2024-02-20T07:51:00.565 INFO:teuthology.orchestra.run.smithi183.stdout:Change: 2024-02-20 07:44:36.012290865 +0000 2024-02-20T07:51:00.565 INFO:teuthology.orchestra.run.smithi183.stdout: Birth: 2024-02-20 07:44:36.012290865 +0000 2024-02-20T07:51:00.566 DEBUG:teuthology.orchestra.run.smithi183:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-02-20T07:51:00.631 INFO:teuthology.orchestra.run.smithi183.stderr:1+0 records in 2024-02-20T07:51:00.631 INFO:teuthology.orchestra.run.smithi183.stderr:1+0 records out 2024-02-20T07:51:00.631 INFO:teuthology.orchestra.run.smithi183.stderr:512 bytes copied, 0.000184877 s, 2.8 MB/s 2024-02-20T07:51:00.632 DEBUG:teuthology.orchestra.run.smithi183:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-02-20T07:51:00.687 INFO:tasks.cephadm:Deploying osd.0 on smithi133 with /dev/vg_nvme/lv_4... 2024-02-20T07:51:00.687 DEBUG:teuthology.orchestra.run.smithi133:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4537327c-cfc4-11ee-95bb-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-02-20T07:51:01.242 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:51:00 smithi133 ceph-mon[29117]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T07:51:01.242 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:51:00 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:51:01.242 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:51:00 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:51:01.243 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:51:00 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi133", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-02-20T07:51:01.243 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:51:00 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-20T07:51:01.297 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:51:00 smithi183 ceph-mon[35832]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T07:51:01.298 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:51:00 smithi183 ceph-mon[35832]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:51:01.298 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:51:00 smithi183 ceph-mon[35832]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:51:01.298 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:51:00 smithi183 ceph-mon[35832]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi133", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-02-20T07:51:01.298 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:51:00 smithi183 ceph-mon[35832]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-20T07:51:01.561 INFO:teuthology.orchestra.run.smithi133.stderr:Inferring config /var/lib/ceph/4537327c-cfc4-11ee-95bb-87774f69a715/mon.smithi133/config 2024-02-20T07:51:01.982 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:51:01 smithi133 ceph-mon[29117]: Reconfiguring crash.smithi133 (monmap changed)... 2024-02-20T07:51:01.983 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:51:01 smithi133 ceph-mon[29117]: Reconfiguring daemon crash.smithi133 on smithi133 2024-02-20T07:51:01.983 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:51:01 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:51:01.983 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:51:01 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:51:02.298 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:51:01 smithi183 ceph-mon[35832]: Reconfiguring crash.smithi133 (monmap changed)... 2024-02-20T07:51:02.298 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:51:01 smithi183 ceph-mon[35832]: Reconfiguring daemon crash.smithi133 on smithi133 2024-02-20T07:51:02.298 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:51:01 smithi183 ceph-mon[35832]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:51:02.298 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:51:01 smithi183 ceph-mon[35832]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:51:03.242 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:51:02 smithi133 ceph-mon[29117]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T07:51:03.242 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:51:02 smithi133 ceph-mon[29117]: Reconfiguring alertmanager.smithi133 (dependencies changed)... 2024-02-20T07:51:03.242 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:51:02 smithi133 ceph-mon[29117]: Reconfiguring daemon alertmanager.smithi133 on smithi133 2024-02-20T07:51:03.298 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:51:02 smithi183 ceph-mon[35832]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T07:51:03.298 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:51:02 smithi183 ceph-mon[35832]: Reconfiguring alertmanager.smithi133 (dependencies changed)... 2024-02-20T07:51:03.298 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:51:02 smithi183 ceph-mon[35832]: Reconfiguring daemon alertmanager.smithi133 on smithi133 2024-02-20T07:51:05.218 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:51:04 smithi133 ceph-mon[29117]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T07:51:05.297 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:51:04 smithi183 ceph-mon[35832]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T07:51:06.063 INFO:teuthology.orchestra.run.smithi133.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-02-20T07:51:06.064 INFO:teuthology.orchestra.run.smithi133.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-02-20T07:51:06.064 INFO:teuthology.orchestra.run.smithi133.stderr: stderr: 10+0 records in 2024-02-20T07:51:06.064 INFO:teuthology.orchestra.run.smithi133.stderr:10+0 records out 2024-02-20T07:51:06.064 INFO:teuthology.orchestra.run.smithi133.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0115315 s, 909 MB/s 2024-02-20T07:51:06.064 INFO:teuthology.orchestra.run.smithi133.stderr:--> Zapping successful for: 2024-02-20T07:51:06.992 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:51:06 smithi133 ceph-mon[29117]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T07:51:07.126 DEBUG:teuthology.orchestra.run.smithi133:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4537327c-cfc4-11ee-95bb-87774f69a715 -- ceph orch daemon add osd smithi133:vg_nvme/lv_4 2024-02-20T07:51:07.297 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:51:06 smithi183 ceph-mon[35832]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T07:51:07.776 INFO:teuthology.orchestra.run.smithi133.stderr:Inferring config /var/lib/ceph/4537327c-cfc4-11ee-95bb-87774f69a715/mon.smithi133/config 2024-02-20T07:51:09.104 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:51:09 smithi133 ceph-mon[29117]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T07:51:09.104 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:51:09 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:51:09.104 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:51:09 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:51:09.104 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:51:09 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-20T07:51:09.298 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:51:09 smithi183 ceph-mon[35832]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T07:51:09.298 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:51:09 smithi183 ceph-mon[35832]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:51:09.298 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:51:09 smithi183 ceph-mon[35832]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:51:09.298 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:51:09 smithi183 ceph-mon[35832]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-20T07:51:10.242 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:51:10 smithi133 ceph-mon[29117]: Reconfiguring grafana.smithi133 (dependencies changed)... 2024-02-20T07:51:10.242 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:51:10 smithi133 ceph-mon[29117]: Reconfiguring daemon grafana.smithi133 on smithi133 2024-02-20T07:51:10.298 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:51:10 smithi183 ceph-mon[35832]: Reconfiguring grafana.smithi133 (dependencies changed)... 2024-02-20T07:51:10.298 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:51:10 smithi183 ceph-mon[35832]: Reconfiguring daemon grafana.smithi133 on smithi133 2024-02-20T07:51:11.086 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:51:11.082+0000 7ff12da67700 1 -- 172.21.15.133:0/3325882590 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ff1280fcfb0 msgr2=0x7ff1280fd420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:51:11.086 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:51:11.082+0000 7ff12da67700 1 --2- 172.21.15.133:0/3325882590 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ff1280fcfb0 0x7ff1280fd420 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7ff11c004940 tx=0x7ff11c009e30 comp rx=0 tx=0).stop 2024-02-20T07:51:11.086 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:51:11.082+0000 7ff12da67700 1 -- 172.21.15.133:0/3325882590 shutdown_connections 2024-02-20T07:51:11.086 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:51:11.082+0000 7ff12da67700 1 --2- 172.21.15.133:0/3325882590 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ff1280fcfb0 0x7ff1280fd420 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:51:11.086 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:51:11.082+0000 7ff12da67700 1 --2- 172.21.15.133:0/3325882590 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff1280fbe40 0x7ff1280fc250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:51:11.086 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:51:11.082+0000 7ff12da67700 1 -- 172.21.15.133:0/3325882590 >> 172.21.15.133:0/3325882590 conn(0x7ff1280f7500 msgr2=0x7ff1280f9930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:51:11.087 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:51:11.082+0000 7ff12da67700 1 -- 172.21.15.133:0/3325882590 shutdown_connections 2024-02-20T07:51:11.087 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:51:11.082+0000 7ff12da67700 1 -- 172.21.15.133:0/3325882590 wait complete. 2024-02-20T07:51:11.087 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:51:11.083+0000 7ff12da67700 1 Processor -- start 2024-02-20T07:51:11.087 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:51:11.083+0000 7ff12da67700 1 -- start start 2024-02-20T07:51:11.087 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:51:11.083+0000 7ff12da67700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ff1280fbe40 0x7ff12810d940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:51:11.087 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:51:11.083+0000 7ff12da67700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff1280fcfb0 0x7ff12810de80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:51:11.087 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:51:11.083+0000 7ff12da67700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff12810e480 con 0x7ff1280fbe40 2024-02-20T07:51:11.087 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:51:11.083+0000 7ff12da67700 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff12810ed20 con 0x7ff1280fcfb0 2024-02-20T07:51:11.087 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:51:11.083+0000 7ff1277fe700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ff1280fbe40 0x7ff12810d940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:51:11.087 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:51:11.083+0000 7ff1277fe700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ff1280fbe40 0x7ff12810d940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.133:48232/0 (socket says 172.21.15.133:48232) 2024-02-20T07:51:11.087 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:51:11.083+0000 7ff1277fe700 1 -- 172.21.15.133:0/3673408949 learned_addr learned my addr 172.21.15.133:0/3673408949 (peer_addr_for_me v2:172.21.15.133:0/0) 2024-02-20T07:51:11.087 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:51:11.083+0000 7ff126ffd700 1 --2- 172.21.15.133:0/3673408949 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff1280fcfb0 0x7ff12810de80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:51:11.087 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:51:11.083+0000 7ff1277fe700 1 -- 172.21.15.133:0/3673408949 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff1280fcfb0 msgr2=0x7ff12810de80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:51:11.087 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:51:11.083+0000 7ff1277fe700 1 --2- 172.21.15.133:0/3673408949 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff1280fcfb0 0x7ff12810de80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:51:11.088 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:51:11.083+0000 7ff1277fe700 1 -- 172.21.15.133:0/3673408949 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff11c005040 con 0x7ff1280fbe40 2024-02-20T07:51:11.088 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:51:11.084+0000 7ff1277fe700 1 --2- 172.21.15.133:0/3673408949 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ff1280fbe40 0x7ff12810d940 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7ff118002a50 tx=0x7ff1180046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:51:11.088 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:51:11.084+0000 7ff10ffff700 1 -- 172.21.15.133:0/3673408949 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff118007cc0 con 0x7ff1280fbe40 2024-02-20T07:51:11.088 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:51:11.084+0000 7ff12da67700 1 -- 172.21.15.133:0/3673408949 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff12810f000 con 0x7ff1280fbe40 2024-02-20T07:51:11.088 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:51:11.084+0000 7ff10ffff700 1 -- 172.21.15.133:0/3673408949 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff11800f040 con 0x7ff1280fbe40 2024-02-20T07:51:11.088 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:51:11.084+0000 7ff10ffff700 1 -- 172.21.15.133:0/3673408949 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff11800da30 con 0x7ff1280fbe40 2024-02-20T07:51:11.088 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:51:11.084+0000 7ff12da67700 1 -- 172.21.15.133:0/3673408949 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff12810f540 con 0x7ff1280fbe40 2024-02-20T07:51:11.089 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:51:11.085+0000 7ff12da67700 1 -- 172.21.15.133:0/3673408949 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff12804bed0 con 0x7ff1280fbe40 2024-02-20T07:51:11.089 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:51:11.085+0000 7ff10ffff700 1 -- 172.21.15.133:0/3673408949 <== mon.0 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 18) v1 ==== 86567+0+0 (secure 0 0 0) 0x7ff11800db90 con 0x7ff1280fbe40 2024-02-20T07:51:11.089 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:51:11.085+0000 7ff10ffff700 1 --2- 172.21.15.133:0/3673408949 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7ff110069210 0x7ff11006b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:51:11.089 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:51:11.086+0000 7ff10ffff700 1 -- 172.21.15.133:0/3673408949 <== mon.0 v2:172.21.15.133:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1366+0+0 (secure 0 0 0) 0x7ff118085e30 con 0x7ff1280fbe40 2024-02-20T07:51:11.089 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:51:11.086+0000 7ff126ffd700 1 --2- 172.21.15.133:0/3673408949 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7ff110069210 0x7ff11006b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:51:11.089 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:51:11.086+0000 7ff126ffd700 1 --2- 172.21.15.133:0/3673408949 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7ff110069210 0x7ff11006b6c0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7ff11c00b3b0 tx=0x7ff11c009e30 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:51:11.092 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:51:11.090+0000 7ff10ffff700 1 -- 172.21.15.133:0/3673408949 <== mon.0 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7ff118056c50 con 0x7ff1280fbe40 2024-02-20T07:51:11.242 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:51:11 smithi133 ceph-mon[29117]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T07:51:11.287 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:51:11.285+0000 7ff12da67700 1 -- 172.21.15.133:0/3673408949 --> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi133:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) v1 -- 0x7ff12810f960 con 0x7ff110069210 2024-02-20T07:51:11.297 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:51:11 smithi183 ceph-mon[35832]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T07:51:12.242 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:51:12 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-20T07:51:12.242 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:51:12 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-20T07:51:12.242 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:51:12 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-20T07:51:12.298 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:51:12 smithi183 ceph-mon[35832]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-20T07:51:12.298 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:51:12 smithi183 ceph-mon[35832]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-20T07:51:12.298 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:51:12 smithi183 ceph-mon[35832]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-20T07:51:13.296 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:51:13 smithi133 ceph-mon[29117]: from='client.14244 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi133:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-02-20T07:51:13.296 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:51:13 smithi133 ceph-mon[29117]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T07:51:13.297 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:51:13 smithi183 ceph-mon[35832]: from='client.14244 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi133:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-02-20T07:51:13.297 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:51:13 smithi183 ceph-mon[35832]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T07:51:15.242 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:51:15 smithi133 ceph-mon[29117]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T07:51:15.242 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:51:15 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:51:15.242 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:51:15 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:51:15.298 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:51:15 smithi183 ceph-mon[35832]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T07:51:15.298 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:51:15 smithi183 ceph-mon[35832]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:51:15.298 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:51:15 smithi183 ceph-mon[35832]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:51:16.492 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:51:16 smithi133 ceph-mon[29117]: Reconfiguring prometheus.smithi133 (dependencies changed)... 2024-02-20T07:51:16.492 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:51:16 smithi133 ceph-mon[29117]: Reconfiguring daemon prometheus.smithi133 on smithi133 2024-02-20T07:51:16.492 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:51:16 smithi133 ceph-mon[29117]: from='client.? 172.21.15.133:0/3423964852' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0a20b241-99e8-4e0b-ba5e-be07c8fdff94"}]: dispatch 2024-02-20T07:51:16.492 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:51:16 smithi133 ceph-mon[29117]: from='client.? 172.21.15.133:0/3423964852' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0a20b241-99e8-4e0b-ba5e-be07c8fdff94"}]': finished 2024-02-20T07:51:16.492 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:51:16 smithi133 ceph-mon[29117]: osdmap e6: 1 total, 0 up, 1 in 2024-02-20T07:51:16.492 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:51:16 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-20T07:51:16.493 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:51:16 smithi133 ceph-mon[29117]: from='client.? 172.21.15.133:0/3072758546' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-20T07:51:16.548 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:51:16 smithi183 ceph-mon[35832]: Reconfiguring prometheus.smithi133 (dependencies changed)... 2024-02-20T07:51:16.548 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:51:16 smithi183 ceph-mon[35832]: Reconfiguring daemon prometheus.smithi133 on smithi133 2024-02-20T07:51:16.548 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:51:16 smithi183 ceph-mon[35832]: from='client.? 172.21.15.133:0/3423964852' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0a20b241-99e8-4e0b-ba5e-be07c8fdff94"}]: dispatch 2024-02-20T07:51:16.548 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:51:16 smithi183 ceph-mon[35832]: from='client.? 172.21.15.133:0/3423964852' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0a20b241-99e8-4e0b-ba5e-be07c8fdff94"}]': finished 2024-02-20T07:51:16.548 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:51:16 smithi183 ceph-mon[35832]: osdmap e6: 1 total, 0 up, 1 in 2024-02-20T07:51:16.548 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:51:16 smithi183 ceph-mon[35832]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-20T07:51:16.548 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:51:16 smithi183 ceph-mon[35832]: from='client.? 172.21.15.133:0/3072758546' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-20T07:51:17.492 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:51:17 smithi133 ceph-mon[29117]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T07:51:17.547 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:51:17 smithi183 ceph-mon[35832]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T07:51:18.242 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:51:18 smithi133 ceph-mon[29117]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T07:51:18.547 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:51:18 smithi183 ceph-mon[35832]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T07:51:20.841 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:51:20 smithi133 ceph-mon[29117]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T07:51:21.048 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:51:20 smithi183 ceph-mon[35832]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T07:51:22.969 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:51:22 smithi133 ceph-mon[29117]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T07:51:23.047 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:51:22 smithi183 ceph-mon[35832]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T07:51:23.742 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:51:23 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-20T07:51:24.048 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:51:23 smithi183 ceph-mon[35832]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-20T07:51:24.742 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:51:24 smithi133 ceph-mon[29117]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T07:51:25.047 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:51:24 smithi183 ceph-mon[35832]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T07:51:26.155 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:51:26 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:51:26.155 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:51:26 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:51:26.155 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:51:26 smithi133 ceph-mon[29117]: Reconfiguring crash.smithi183 (monmap changed)... 2024-02-20T07:51:26.155 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:51:26 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi183", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-02-20T07:51:26.155 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:51:26 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-20T07:51:26.156 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:51:26 smithi133 ceph-mon[29117]: Reconfiguring daemon crash.smithi183 on smithi183 2024-02-20T07:51:26.156 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:51:26 smithi133 ceph-mon[29117]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T07:51:26.346 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:51:26 smithi183 ceph-mon[35832]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:51:26.346 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:51:26 smithi183 ceph-mon[35832]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:51:26.346 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:51:26 smithi183 ceph-mon[35832]: Reconfiguring crash.smithi183 (monmap changed)... 2024-02-20T07:51:26.346 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:51:26 smithi183 ceph-mon[35832]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi183", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-02-20T07:51:26.346 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:51:26 smithi183 ceph-mon[35832]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-20T07:51:26.346 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:51:26 smithi183 ceph-mon[35832]: Reconfiguring daemon crash.smithi183 on smithi183 2024-02-20T07:51:26.346 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:51:26 smithi183 ceph-mon[35832]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T07:51:27.492 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:51:27 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:51:27.492 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:51:27 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:51:27.492 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:51:27 smithi133 ceph-mon[29117]: Reconfiguring mgr.smithi183.hebfng (monmap changed)... 2024-02-20T07:51:27.492 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:51:27 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi183.hebfng", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-20T07:51:27.492 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:51:27 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-20T07:51:27.492 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:51:27 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-20T07:51:27.493 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:51:27 smithi133 ceph-mon[29117]: Reconfiguring daemon mgr.smithi183.hebfng on smithi183 2024-02-20T07:51:27.493 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:51:27 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-02-20T07:51:27.493 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:51:27 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-20T07:51:27.493 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:51:27 smithi133 ceph-mon[29117]: Deploying daemon osd.0 on smithi133 2024-02-20T07:51:27.798 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:51:27 smithi183 ceph-mon[35832]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:51:27.798 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:51:27 smithi183 ceph-mon[35832]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:51:27.798 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:51:27 smithi183 ceph-mon[35832]: Reconfiguring mgr.smithi183.hebfng (monmap changed)... 2024-02-20T07:51:27.798 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:51:27 smithi183 ceph-mon[35832]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi183.hebfng", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-20T07:51:27.798 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:51:27 smithi183 ceph-mon[35832]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-20T07:51:27.798 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:51:27 smithi183 ceph-mon[35832]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-20T07:51:27.798 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:51:27 smithi183 ceph-mon[35832]: Reconfiguring daemon mgr.smithi183.hebfng on smithi183 2024-02-20T07:51:27.798 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:51:27 smithi183 ceph-mon[35832]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-02-20T07:51:27.798 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:51:27 smithi183 ceph-mon[35832]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-20T07:51:27.798 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:51:27 smithi183 ceph-mon[35832]: Deploying daemon osd.0 on smithi133 2024-02-20T07:51:28.883 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:51:28 smithi183 ceph-mon[35832]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T07:51:28.883 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:51:28 smithi183 ceph-mon[35832]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:51:28.883 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:51:28 smithi183 ceph-mon[35832]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:51:28.883 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:51:28 smithi183 ceph-mon[35832]: Reconfiguring mon.smithi183 (monmap changed)... 2024-02-20T07:51:28.883 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:51:28 smithi183 ceph-mon[35832]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-20T07:51:28.883 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:51:28 smithi183 ceph-mon[35832]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-20T07:51:28.883 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:51:28 smithi183 ceph-mon[35832]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-20T07:51:28.884 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:51:28 smithi183 ceph-mon[35832]: Reconfiguring daemon mon.smithi183 on smithi183 2024-02-20T07:51:28.925 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:51:28 smithi133 ceph-mon[29117]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T07:51:28.925 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:51:28 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:51:28.925 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:51:28 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:51:28.925 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:51:28 smithi133 ceph-mon[29117]: Reconfiguring mon.smithi183 (monmap changed)... 2024-02-20T07:51:28.925 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:51:28 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-20T07:51:28.925 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:51:28 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-20T07:51:28.925 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:51:28 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-20T07:51:28.926 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:51:28 smithi133 ceph-mon[29117]: Reconfiguring daemon mon.smithi183 on smithi183 2024-02-20T07:51:30.242 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:51:29 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:51:30.242 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:51:29 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:51:30.242 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:51:29 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-02-20T07:51:30.243 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:51:29 smithi133 ceph-mon[29117]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-02-20T07:51:30.243 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:51:29 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi133.front.sepia.ceph.com:9093"}]: dispatch 2024-02-20T07:51:30.243 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:51:29 smithi133 ceph-mon[29117]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi133.front.sepia.ceph.com:9093"}]: dispatch 2024-02-20T07:51:30.243 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:51:29 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:51:30.243 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:51:29 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-02-20T07:51:30.243 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:51:29 smithi133 ceph-mon[29117]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-02-20T07:51:30.243 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:51:29 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi133.front.sepia.ceph.com:3000"}]: dispatch 2024-02-20T07:51:30.243 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:51:29 smithi133 ceph-mon[29117]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi133.front.sepia.ceph.com:3000"}]: dispatch 2024-02-20T07:51:30.243 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:51:29 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:51:30.243 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:51:29 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-02-20T07:51:30.243 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:51:29 smithi133 ceph-mon[29117]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-02-20T07:51:30.243 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:51:29 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi133.front.sepia.ceph.com:9095"}]: dispatch 2024-02-20T07:51:30.243 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:51:29 smithi133 ceph-mon[29117]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi133.front.sepia.ceph.com:9095"}]: dispatch 2024-02-20T07:51:30.243 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:51:29 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:51:30.243 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:51:29 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-20T07:51:30.279 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:51:29 smithi183 ceph-mon[35832]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:51:30.279 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:51:29 smithi183 ceph-mon[35832]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:51:30.279 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:51:29 smithi183 ceph-mon[35832]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-02-20T07:51:30.279 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:51:29 smithi183 ceph-mon[35832]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-02-20T07:51:30.279 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:51:29 smithi183 ceph-mon[35832]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi133.front.sepia.ceph.com:9093"}]: dispatch 2024-02-20T07:51:30.279 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:51:29 smithi183 ceph-mon[35832]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi133.front.sepia.ceph.com:9093"}]: dispatch 2024-02-20T07:51:30.279 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:51:29 smithi183 ceph-mon[35832]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:51:30.279 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:51:29 smithi183 ceph-mon[35832]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-02-20T07:51:30.279 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:51:29 smithi183 ceph-mon[35832]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-02-20T07:51:30.279 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:51:29 smithi183 ceph-mon[35832]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi133.front.sepia.ceph.com:3000"}]: dispatch 2024-02-20T07:51:30.279 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:51:29 smithi183 ceph-mon[35832]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi133.front.sepia.ceph.com:3000"}]: dispatch 2024-02-20T07:51:30.279 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:51:29 smithi183 ceph-mon[35832]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:51:30.280 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:51:29 smithi183 ceph-mon[35832]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-02-20T07:51:30.280 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:51:29 smithi183 ceph-mon[35832]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-02-20T07:51:30.280 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:51:29 smithi183 ceph-mon[35832]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi133.front.sepia.ceph.com:9095"}]: dispatch 2024-02-20T07:51:30.280 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:51:29 smithi183 ceph-mon[35832]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi133.front.sepia.ceph.com:9095"}]: dispatch 2024-02-20T07:51:30.280 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:51:29 smithi183 ceph-mon[35832]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:51:30.280 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:51:29 smithi183 ceph-mon[35832]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-20T07:51:31.242 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:51:31 smithi133 ceph-mon[29117]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T07:51:31.242 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:51:31 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:51:31.242 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:51:31 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:51:31.297 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:51:31 smithi183 ceph-mon[35832]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T07:51:31.297 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:51:31 smithi183 ceph-mon[35832]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:51:31.298 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:51:31 smithi183 ceph-mon[35832]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:51:33.123 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:51:33 smithi133 ceph-mon[29117]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T07:51:33.298 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:51:33 smithi183 ceph-mon[35832]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T07:51:34.992 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:51:34 smithi133 ceph-mon[29117]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T07:51:34.992 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:51:34 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:51:34.992 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:51:34 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:51:34.992 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:51:34 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:51:34.992 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:51:34 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:51:35.047 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:51:34 smithi183 ceph-mon[35832]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T07:51:35.048 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:51:34 smithi183 ceph-mon[35832]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:51:35.048 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:51:34 smithi183 ceph-mon[35832]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:51:35.048 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:51:34 smithi183 ceph-mon[35832]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:51:35.048 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:51:34 smithi183 ceph-mon[35832]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:51:35.830 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:51:35 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-20T07:51:35.830 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:51:35 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-20T07:51:35.830 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:51:35 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:51:35.830 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:51:35 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-20T07:51:36.048 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:51:35 smithi183 ceph-mon[35832]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-20T07:51:36.048 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:51:35 smithi183 ceph-mon[35832]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-20T07:51:36.048 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:51:35 smithi183 ceph-mon[35832]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:51:36.048 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:51:35 smithi183 ceph-mon[35832]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-20T07:51:36.909 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:51:36 smithi133 ceph-mon[29117]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T07:51:36.948 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:51:36.947+0000 7ff10ffff700 1 -- 172.21.15.133:0/3673408949 <== mgr.14208 v2:172.21.15.133:6800/2703586938 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7ff12810f960 con 0x7ff110069210 2024-02-20T07:51:36.949 INFO:teuthology.orchestra.run.smithi133.stdout:Created osd(s) 0 on host 'smithi133' 2024-02-20T07:51:36.951 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:51:36.949+0000 7ff12da67700 1 -- 172.21.15.133:0/3673408949 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7ff110069210 msgr2=0x7ff11006b6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:51:36.951 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:51:36.949+0000 7ff12da67700 1 --2- 172.21.15.133:0/3673408949 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7ff110069210 0x7ff11006b6c0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7ff11c00b3b0 tx=0x7ff11c009e30 comp rx=0 tx=0).stop 2024-02-20T07:51:36.951 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:51:36.950+0000 7ff12da67700 1 -- 172.21.15.133:0/3673408949 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ff1280fbe40 msgr2=0x7ff12810d940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:51:36.951 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:51:36.950+0000 7ff12da67700 1 --2- 172.21.15.133:0/3673408949 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ff1280fbe40 0x7ff12810d940 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7ff118002a50 tx=0x7ff1180046b0 comp rx=0 tx=0).stop 2024-02-20T07:51:36.951 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:51:36.950+0000 7ff12da67700 1 -- 172.21.15.133:0/3673408949 shutdown_connections 2024-02-20T07:51:36.951 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:51:36.950+0000 7ff12da67700 1 --2- 172.21.15.133:0/3673408949 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ff1280fbe40 0x7ff12810d940 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:51:36.951 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:51:36.950+0000 7ff12da67700 1 --2- 172.21.15.133:0/3673408949 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7ff110069210 0x7ff11006b6c0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:51:36.951 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:51:36.950+0000 7ff12da67700 1 --2- 172.21.15.133:0/3673408949 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff1280fcfb0 0x7ff12810de80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:51:36.951 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:51:36.950+0000 7ff12da67700 1 -- 172.21.15.133:0/3673408949 >> 172.21.15.133:0/3673408949 conn(0x7ff1280f7500 msgr2=0x7ff128065f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:51:36.951 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:51:36.950+0000 7ff12da67700 1 -- 172.21.15.133:0/3673408949 shutdown_connections 2024-02-20T07:51:36.951 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:51:36.950+0000 7ff12da67700 1 -- 172.21.15.133:0/3673408949 wait complete. 2024-02-20T07:51:37.047 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:51:36 smithi183 ceph-mon[35832]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T07:51:38.049 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:51:37 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:51:38.049 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:51:37 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:51:38.049 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:51:37 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:51:38.049 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:51:37 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:51:38.050 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:51:37 smithi133 ceph-mon[29117]: from='osd.0 [v2:172.21.15.133:6802/261892236,v1:172.21.15.133:6803/261892236]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-02-20T07:51:38.073 DEBUG:teuthology.orchestra.run.smithi133:osd.0> sudo journalctl -f -n 0 -u ceph-4537327c-cfc4-11ee-95bb-87774f69a715@osd.0.service 2024-02-20T07:51:38.075 INFO:tasks.cephadm:Deploying osd.1 on smithi133 with /dev/vg_nvme/lv_3... 2024-02-20T07:51:38.076 DEBUG:teuthology.orchestra.run.smithi133:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4537327c-cfc4-11ee-95bb-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-02-20T07:51:38.298 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:51:37 smithi183 ceph-mon[35832]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:51:38.298 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:51:37 smithi183 ceph-mon[35832]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:51:38.298 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:51:37 smithi183 ceph-mon[35832]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:51:38.298 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:51:37 smithi183 ceph-mon[35832]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:51:38.298 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:51:37 smithi183 ceph-mon[35832]: from='osd.0 [v2:172.21.15.133:6802/261892236,v1:172.21.15.133:6803/261892236]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-02-20T07:51:38.951 INFO:teuthology.orchestra.run.smithi133.stderr:Inferring config /var/lib/ceph/4537327c-cfc4-11ee-95bb-87774f69a715/mon.smithi133/config 2024-02-20T07:51:39.297 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:51:39 smithi183 ceph-mon[35832]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T07:51:39.298 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:51:39 smithi183 ceph-mon[35832]: from='osd.0 [v2:172.21.15.133:6802/261892236,v1:172.21.15.133:6803/261892236]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-02-20T07:51:39.298 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:51:39 smithi183 ceph-mon[35832]: osdmap e7: 1 total, 0 up, 1 in 2024-02-20T07:51:39.298 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:51:39 smithi183 ceph-mon[35832]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-20T07:51:39.298 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:51:39 smithi183 ceph-mon[35832]: from='osd.0 [v2:172.21.15.133:6802/261892236,v1:172.21.15.133:6803/261892236]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi133", "root=default"]}]: dispatch 2024-02-20T07:51:39.298 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:51:39 smithi183 ceph-mon[35832]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-20T07:51:39.406 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:51:39 smithi133 ceph-mon[29117]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T07:51:39.406 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:51:39 smithi133 ceph-mon[29117]: from='osd.0 [v2:172.21.15.133:6802/261892236,v1:172.21.15.133:6803/261892236]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-02-20T07:51:39.406 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:51:39 smithi133 ceph-mon[29117]: osdmap e7: 1 total, 0 up, 1 in 2024-02-20T07:51:39.406 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:51:39 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-20T07:51:39.406 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:51:39 smithi133 ceph-mon[29117]: from='osd.0 [v2:172.21.15.133:6802/261892236,v1:172.21.15.133:6803/261892236]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi133", "root=default"]}]: dispatch 2024-02-20T07:51:39.406 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:51:39 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-20T07:51:39.406 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:51:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:51:39.013+0000 7f8a5e86c700 -1 osd.0 0 waiting for initial osdmap 2024-02-20T07:51:39.407 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:51:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:51:39.016+0000 7f8a55e60700 -1 osd.0 8 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-20T07:51:40.242 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:51:40 smithi133 ceph-mon[29117]: from='osd.0 [v2:172.21.15.133:6802/261892236,v1:172.21.15.133:6803/261892236]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi133", "root=default"]}]': finished 2024-02-20T07:51:40.242 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:51:40 smithi133 ceph-mon[29117]: osdmap e8: 1 total, 0 up, 1 in 2024-02-20T07:51:40.242 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:51:40 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-20T07:51:40.242 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:51:40 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-20T07:51:40.297 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:51:40 smithi183 ceph-mon[35832]: from='osd.0 [v2:172.21.15.133:6802/261892236,v1:172.21.15.133:6803/261892236]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi133", "root=default"]}]': finished 2024-02-20T07:51:40.297 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:51:40 smithi183 ceph-mon[35832]: osdmap e8: 1 total, 0 up, 1 in 2024-02-20T07:51:40.298 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:51:40 smithi183 ceph-mon[35832]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-20T07:51:40.298 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:51:40 smithi183 ceph-mon[35832]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-20T07:51:41.242 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:51:41 smithi133 ceph-mon[29117]: purged_snaps scrub starts 2024-02-20T07:51:41.242 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:51:41 smithi133 ceph-mon[29117]: purged_snaps scrub ok 2024-02-20T07:51:41.242 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:51:41 smithi133 ceph-mon[29117]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T07:51:41.242 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:51:41 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-20T07:51:41.242 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:51:41 smithi133 ceph-mon[29117]: osd.0 [v2:172.21.15.133:6802/261892236,v1:172.21.15.133:6803/261892236] boot 2024-02-20T07:51:41.242 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:51:41 smithi133 ceph-mon[29117]: osdmap e9: 1 total, 1 up, 1 in 2024-02-20T07:51:41.242 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:51:41 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-20T07:51:41.298 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:51:41 smithi183 ceph-mon[35832]: purged_snaps scrub starts 2024-02-20T07:51:41.298 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:51:41 smithi183 ceph-mon[35832]: purged_snaps scrub ok 2024-02-20T07:51:41.298 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:51:41 smithi183 ceph-mon[35832]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-20T07:51:41.298 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:51:41 smithi183 ceph-mon[35832]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-20T07:51:41.298 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:51:41 smithi183 ceph-mon[35832]: osd.0 [v2:172.21.15.133:6802/261892236,v1:172.21.15.133:6803/261892236] boot 2024-02-20T07:51:41.298 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:51:41 smithi183 ceph-mon[35832]: osdmap e9: 1 total, 1 up, 1 in 2024-02-20T07:51:41.298 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:51:41 smithi183 ceph-mon[35832]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-20T07:51:42.127 INFO:teuthology.orchestra.run.smithi133.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-02-20T07:51:42.127 INFO:teuthology.orchestra.run.smithi133.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-02-20T07:51:42.128 INFO:teuthology.orchestra.run.smithi133.stderr: stderr: 10+0 records in 2024-02-20T07:51:42.128 INFO:teuthology.orchestra.run.smithi133.stderr:10+0 records out 2024-02-20T07:51:42.128 INFO:teuthology.orchestra.run.smithi133.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0149968 s, 699 MB/s 2024-02-20T07:51:42.128 INFO:teuthology.orchestra.run.smithi133.stderr:--> Zapping successful for: 2024-02-20T07:51:42.297 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:51:42 smithi183 ceph-mon[35832]: osdmap e10: 1 total, 1 up, 1 in 2024-02-20T07:51:42.457 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:51:42 smithi133 ceph-mon[29117]: osdmap e10: 1 total, 1 up, 1 in 2024-02-20T07:51:42.766 DEBUG:teuthology.orchestra.run.smithi133:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4537327c-cfc4-11ee-95bb-87774f69a715 -- ceph orch daemon add osd smithi133:vg_nvme/lv_3 2024-02-20T07:51:43.029 INFO:teuthology.orchestra.run.smithi133.stderr:Inferring config /var/lib/ceph/4537327c-cfc4-11ee-95bb-87774f69a715/mon.smithi133/config 2024-02-20T07:51:43.282 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:51:43 smithi133 ceph-mon[29117]: pgmap v37: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-20T07:51:43.297 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:51:43 smithi183 ceph-mon[35832]: pgmap v37: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-20T07:51:44.285 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:51:44 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:51:44.285 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:51:44 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:51:44.285 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:51:44 smithi133 ceph-mon[29117]: pgmap v38: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-20T07:51:44.547 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:51:44 smithi183 ceph-mon[35832]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:51:44.548 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:51:44 smithi183 ceph-mon[35832]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:51:44.548 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:51:44 smithi183 ceph-mon[35832]: pgmap v38: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-20T07:51:44.717 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:51:44.714+0000 7fb5bfed5700 1 -- 172.21.15.133:0/3415102537 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb5b8066c70 msgr2=0x7fb5b80ffaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:51:44.718 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:51:44.714+0000 7fb5bfed5700 1 --2- 172.21.15.133:0/3415102537 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb5b8066c70 0x7fb5b80ffaf0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fb5b4004940 tx=0x7fb5b4009e30 comp rx=0 tx=0).stop 2024-02-20T07:51:44.718 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:51:44.714+0000 7fb5bfed5700 1 -- 172.21.15.133:0/3415102537 shutdown_connections 2024-02-20T07:51:44.718 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:51:44.714+0000 7fb5bfed5700 1 --2- 172.21.15.133:0/3415102537 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb5b81000c0 0x7fb5b8102520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:51:44.718 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:51:44.714+0000 7fb5bfed5700 1 --2- 172.21.15.133:0/3415102537 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb5b8066c70 0x7fb5b80ffaf0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:51:44.718 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:51:44.715+0000 7fb5bfed5700 1 -- 172.21.15.133:0/3415102537 >> 172.21.15.133:0/3415102537 conn(0x7fb5b80f7520 msgr2=0x7fb5b80f9930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:51:44.718 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:51:44.715+0000 7fb5bfed5700 1 -- 172.21.15.133:0/3415102537 shutdown_connections 2024-02-20T07:51:44.718 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:51:44.715+0000 7fb5bfed5700 1 -- 172.21.15.133:0/3415102537 wait complete. 2024-02-20T07:51:44.718 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:51:44.715+0000 7fb5bfed5700 1 Processor -- start 2024-02-20T07:51:44.718 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:51:44.715+0000 7fb5bfed5700 1 -- start start 2024-02-20T07:51:44.718 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:51:44.715+0000 7fb5bfed5700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb5b8066c70 0x7fb5b80fd570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:51:44.718 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:51:44.715+0000 7fb5bfed5700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb5b81000c0 0x7fb5b80fb9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:51:44.718 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:51:44.715+0000 7fb5bfed5700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5b80fbff0 con 0x7fb5b81000c0 2024-02-20T07:51:44.718 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:51:44.715+0000 7fb5bfed5700 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5b80fc160 con 0x7fb5b8066c70 2024-02-20T07:51:44.719 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:51:44.715+0000 7fb5bd470700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb5b81000c0 0x7fb5b80fb9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:51:44.719 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:51:44.715+0000 7fb5bd470700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb5b81000c0 0x7fb5b80fb9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.133:56944/0 (socket says 172.21.15.133:56944) 2024-02-20T07:51:44.719 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:51:44.715+0000 7fb5bd470700 1 -- 172.21.15.133:0/3242105735 learned_addr learned my addr 172.21.15.133:0/3242105735 (peer_addr_for_me v2:172.21.15.133:0/0) 2024-02-20T07:51:44.719 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:51:44.716+0000 7fb5bdc71700 1 --2- 172.21.15.133:0/3242105735 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb5b8066c70 0x7fb5b80fd570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:51:44.719 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:51:44.716+0000 7fb5bd470700 1 -- 172.21.15.133:0/3242105735 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb5b8066c70 msgr2=0x7fb5b80fd570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:51:44.719 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:51:44.716+0000 7fb5bd470700 1 --2- 172.21.15.133:0/3242105735 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb5b8066c70 0x7fb5b80fd570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:51:44.719 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:51:44.716+0000 7fb5bd470700 1 -- 172.21.15.133:0/3242105735 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5b4005040 con 0x7fb5b81000c0 2024-02-20T07:51:44.719 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:51:44.716+0000 7fb5bdc71700 1 --2- 172.21.15.133:0/3242105735 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb5b8066c70 0x7fb5b80fd570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-20T07:51:44.719 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:51:44.716+0000 7fb5bd470700 1 --2- 172.21.15.133:0/3242105735 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb5b81000c0 0x7fb5b80fb9f0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fb5a800c7e0 tx=0x7fb5a8007f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:51:44.719 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:51:44.716+0000 7fb5ae7fc700 1 -- 172.21.15.133:0/3242105735 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb5a8012810 con 0x7fb5b81000c0 2024-02-20T07:51:44.719 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:51:44.716+0000 7fb5ae7fc700 1 -- 172.21.15.133:0/3242105735 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb5a8012e40 con 0x7fb5b81000c0 2024-02-20T07:51:44.719 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:51:44.716+0000 7fb5bfed5700 1 -- 172.21.15.133:0/3242105735 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb5b80fc440 con 0x7fb5b81000c0 2024-02-20T07:51:44.719 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:51:44.716+0000 7fb5ae7fc700 1 -- 172.21.15.133:0/3242105735 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb5a800a550 con 0x7fb5b81000c0 2024-02-20T07:51:44.719 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:51:44.716+0000 7fb5bfed5700 1 -- 172.21.15.133:0/3242105735 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb5b81159d0 con 0x7fb5b81000c0 2024-02-20T07:51:44.720 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:51:44.717+0000 7fb5bfed5700 1 -- 172.21.15.133:0/3242105735 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb5b8107a90 con 0x7fb5b81000c0 2024-02-20T07:51:44.720 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:51:44.717+0000 7fb5ae7fc700 1 -- 172.21.15.133:0/3242105735 <== mon.0 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 18) v1 ==== 86567+0+0 (secure 0 0 0) 0x7fb5a8011380 con 0x7fb5b81000c0 2024-02-20T07:51:44.720 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:51:44.718+0000 7fb5ae7fc700 1 --2- 172.21.15.133:0/3242105735 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7fb5a4069210 0x7fb5a406b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:51:44.720 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:51:44.718+0000 7fb5ae7fc700 1 -- 172.21.15.133:0/3242105735 <== mon.0 v2:172.21.15.133:3300/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1978+0+0 (secure 0 0 0) 0x7fb5a8086640 con 0x7fb5b81000c0 2024-02-20T07:51:44.721 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:51:44.718+0000 7fb5bdc71700 1 --2- 172.21.15.133:0/3242105735 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7fb5a4069210 0x7fb5a406b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:51:44.721 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:51:44.718+0000 7fb5bdc71700 1 --2- 172.21.15.133:0/3242105735 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7fb5a4069210 0x7fb5a406b6c0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fb5b4009fd0 tx=0x7fb5b4009e30 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:51:44.723 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:51:44.722+0000 7fb5ae7fc700 1 -- 172.21.15.133:0/3242105735 <== mon.0 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7fb5a8057280 con 0x7fb5b81000c0 2024-02-20T07:51:44.920 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:51:44.919+0000 7fb5bfed5700 1 -- 172.21.15.133:0/3242105735 --> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi133:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) v1 -- 0x7fb5b802b3b0 con 0x7fb5a4069210 2024-02-20T07:51:45.242 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:51:45 smithi133 ceph-mon[29117]: from='client.14262 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi133:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-02-20T07:51:45.242 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:51:45 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-20T07:51:45.242 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:51:45 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-20T07:51:45.242 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:51:45 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-20T07:51:45.547 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:51:45 smithi183 ceph-mon[35832]: from='client.14262 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi133:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-02-20T07:51:45.548 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:51:45 smithi183 ceph-mon[35832]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-20T07:51:45.548 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:51:45 smithi183 ceph-mon[35832]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-20T07:51:45.548 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:51:45 smithi183 ceph-mon[35832]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-20T07:51:46.242 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:51:46 smithi133 ceph-mon[29117]: pgmap v39: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-20T07:51:46.547 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:51:46 smithi183 ceph-mon[35832]: pgmap v39: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-20T07:51:48.547 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:51:48 smithi183 ceph-mon[35832]: Detected new or changed devices on smithi133 2024-02-20T07:51:48.547 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:51:48 smithi183 ceph-mon[35832]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:51:48.547 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:51:48 smithi183 ceph-mon[35832]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:51:48.548 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:51:48 smithi183 ceph-mon[35832]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-20T07:51:48.548 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:51:48 smithi183 ceph-mon[35832]: Adjusting osd_memory_target on smithi133 to 12864M 2024-02-20T07:51:48.548 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:51:48 smithi183 ceph-mon[35832]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:51:48.548 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:51:48 smithi183 ceph-mon[35832]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-20T07:51:48.548 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:51:48 smithi183 ceph-mon[35832]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-20T07:51:48.548 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:51:48 smithi183 ceph-mon[35832]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:51:48.548 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:51:48 smithi183 ceph-mon[35832]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-20T07:51:48.548 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:51:48 smithi183 ceph-mon[35832]: pgmap v40: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-20T07:51:48.666 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:51:48 smithi133 ceph-mon[29117]: Detected new or changed devices on smithi133 2024-02-20T07:51:48.666 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:51:48 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:51:48.666 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:51:48 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:51:48.666 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:51:48 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-20T07:51:48.667 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:51:48 smithi133 ceph-mon[29117]: Adjusting osd_memory_target on smithi133 to 12864M 2024-02-20T07:51:48.667 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:51:48 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:51:48.667 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:51:48 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-20T07:51:48.667 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:51:48 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-20T07:51:48.667 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:51:48 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:51:48.667 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:51:48 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-20T07:51:48.667 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:51:48 smithi133 ceph-mon[29117]: pgmap v40: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-20T07:51:49.492 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:51:49 smithi133 ceph-mon[29117]: from='client.? 172.21.15.133:0/662576770' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2b1b8e8f-74b4-4d41-8753-502296eb87df"}]: dispatch 2024-02-20T07:51:49.492 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:51:49 smithi133 ceph-mon[29117]: from='client.? 172.21.15.133:0/662576770' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "2b1b8e8f-74b4-4d41-8753-502296eb87df"}]': finished 2024-02-20T07:51:49.492 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:51:49 smithi133 ceph-mon[29117]: osdmap e11: 2 total, 1 up, 2 in 2024-02-20T07:51:49.492 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:51:49 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-20T07:51:49.492 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:51:49 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:51:49.492 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:51:49 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:51:49.797 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:51:49 smithi183 ceph-mon[35832]: from='client.? 172.21.15.133:0/662576770' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2b1b8e8f-74b4-4d41-8753-502296eb87df"}]: dispatch 2024-02-20T07:51:49.797 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:51:49 smithi183 ceph-mon[35832]: from='client.? 172.21.15.133:0/662576770' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "2b1b8e8f-74b4-4d41-8753-502296eb87df"}]': finished 2024-02-20T07:51:49.798 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:51:49 smithi183 ceph-mon[35832]: osdmap e11: 2 total, 1 up, 2 in 2024-02-20T07:51:49.798 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:51:49 smithi183 ceph-mon[35832]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-20T07:51:49.798 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:51:49 smithi183 ceph-mon[35832]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:51:49.798 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:51:49 smithi183 ceph-mon[35832]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:51:50.492 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:51:50 smithi133 ceph-mon[29117]: from='client.? 172.21.15.133:0/3929459177' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-20T07:51:50.492 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:51:50 smithi133 ceph-mon[29117]: pgmap v42: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-20T07:51:50.798 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:51:50 smithi183 ceph-mon[35832]: from='client.? 172.21.15.133:0/3929459177' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-20T07:51:50.798 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:51:50 smithi183 ceph-mon[35832]: pgmap v42: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-20T07:51:52.992 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:51:52 smithi133 ceph-mon[29117]: pgmap v43: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-20T07:51:52.992 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:51:52 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:51:52.992 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:51:52 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:51:52.992 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:51:52 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-20T07:51:52.992 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:51:52 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-20T07:51:52.992 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:51:52 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:51:53.298 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:51:52 smithi183 ceph-mon[35832]: pgmap v43: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-20T07:51:53.298 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:51:52 smithi183 ceph-mon[35832]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:51:53.298 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:51:52 smithi183 ceph-mon[35832]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:51:53.298 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:51:52 smithi183 ceph-mon[35832]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-20T07:51:53.298 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:51:52 smithi183 ceph-mon[35832]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-20T07:51:53.298 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:51:52 smithi183 ceph-mon[35832]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:51:54.041 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:51:53 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-20T07:51:54.297 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:51:53 smithi183 ceph-mon[35832]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-20T07:51:55.098 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:51:54 smithi133 ceph-mon[29117]: pgmap v44: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-20T07:51:55.298 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:51:54 smithi183 ceph-mon[35832]: pgmap v44: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-20T07:51:57.242 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:51:56 smithi133 ceph-mon[29117]: pgmap v45: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-20T07:51:57.242 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:51:56 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-02-20T07:51:57.242 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:51:56 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-20T07:51:57.242 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:51:56 smithi133 ceph-mon[29117]: Deploying daemon osd.1 on smithi133 2024-02-20T07:51:57.298 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:51:56 smithi183 ceph-mon[35832]: pgmap v45: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-20T07:51:57.298 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:51:56 smithi183 ceph-mon[35832]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-02-20T07:51:57.298 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:51:56 smithi183 ceph-mon[35832]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-20T07:51:57.298 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:51:56 smithi183 ceph-mon[35832]: Deploying daemon osd.1 on smithi133 2024-02-20T07:51:59.242 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:51:58 smithi133 ceph-mon[29117]: pgmap v46: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-20T07:51:59.297 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:51:58 smithi183 ceph-mon[35832]: pgmap v46: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-20T07:52:01.242 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:52:00 smithi133 ceph-mon[29117]: pgmap v47: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-20T07:52:01.297 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:52:00 smithi183 ceph-mon[35832]: pgmap v47: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-20T07:52:02.295 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:52:02 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:52:02.295 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:52:02 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:52:02.295 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:52:02 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-20T07:52:02.297 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:52:02 smithi183 ceph-mon[35832]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:52:02.297 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:52:02 smithi183 ceph-mon[35832]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:52:02.297 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:52:02 smithi183 ceph-mon[35832]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-20T07:52:03.242 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:52:03 smithi133 ceph-mon[29117]: pgmap v48: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-20T07:52:03.297 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:52:03 smithi183 ceph-mon[35832]: pgmap v48: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-20T07:52:04.293 INFO:teuthology.orchestra.run.smithi133.stdout:Created osd(s) 1 on host 'smithi133' 2024-02-20T07:52:04.293 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:52:04.290+0000 7fb5ae7fc700 1 -- 172.21.15.133:0/3242105735 <== mgr.14208 v2:172.21.15.133:6800/2703586938 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7fb5b802b3b0 con 0x7fb5a4069210 2024-02-20T07:52:04.295 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:52:04.292+0000 7fb5bfed5700 1 -- 172.21.15.133:0/3242105735 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7fb5a4069210 msgr2=0x7fb5a406b6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:52:04.295 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:52:04.292+0000 7fb5bfed5700 1 --2- 172.21.15.133:0/3242105735 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7fb5a4069210 0x7fb5a406b6c0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fb5b4009fd0 tx=0x7fb5b4009e30 comp rx=0 tx=0).stop 2024-02-20T07:52:04.295 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:52:04.293+0000 7fb5bfed5700 1 -- 172.21.15.133:0/3242105735 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb5b81000c0 msgr2=0x7fb5b80fb9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:52:04.295 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:52:04.293+0000 7fb5bfed5700 1 --2- 172.21.15.133:0/3242105735 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb5b81000c0 0x7fb5b80fb9f0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fb5a800c7e0 tx=0x7fb5a8007f90 comp rx=0 tx=0).stop 2024-02-20T07:52:04.295 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:52:04.293+0000 7fb5bfed5700 1 -- 172.21.15.133:0/3242105735 shutdown_connections 2024-02-20T07:52:04.295 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:52:04.293+0000 7fb5bfed5700 1 --2- 172.21.15.133:0/3242105735 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7fb5a4069210 0x7fb5a406b6c0 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:52:04.295 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:52:04.293+0000 7fb5bfed5700 1 --2- 172.21.15.133:0/3242105735 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb5b8066c70 0x7fb5b80fd570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:52:04.295 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:52:04.293+0000 7fb5bfed5700 1 --2- 172.21.15.133:0/3242105735 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb5b81000c0 0x7fb5b80fb9f0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:52:04.295 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:52:04.293+0000 7fb5bfed5700 1 -- 172.21.15.133:0/3242105735 >> 172.21.15.133:0/3242105735 conn(0x7fb5b80f7520 msgr2=0x7fb5b80f9900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:52:04.295 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:52:04.293+0000 7fb5bfed5700 1 -- 172.21.15.133:0/3242105735 shutdown_connections 2024-02-20T07:52:04.295 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:52:04.293+0000 7fb5bfed5700 1 -- 172.21.15.133:0/3242105735 wait complete. 2024-02-20T07:52:04.898 DEBUG:teuthology.orchestra.run.smithi133:osd.1> sudo journalctl -f -n 0 -u ceph-4537327c-cfc4-11ee-95bb-87774f69a715@osd.1.service 2024-02-20T07:52:04.901 INFO:tasks.cephadm:Deploying osd.2 on smithi133 with /dev/vg_nvme/lv_2... 2024-02-20T07:52:04.901 DEBUG:teuthology.orchestra.run.smithi133:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4537327c-cfc4-11ee-95bb-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-02-20T07:52:05.150 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:52:05 smithi133 ceph-mon[29117]: pgmap v49: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-20T07:52:05.151 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:52:05 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:52:05.151 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:52:05 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:52:05.151 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:52:05 smithi133 ceph-mon[29117]: from='osd.1 [v2:172.21.15.133:6810/1652695599,v1:172.21.15.133:6811/1652695599]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-02-20T07:52:05.151 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:52:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:52:04.940+0000 7f010ae87540 -1 osd.1 0 log_to_monitors true 2024-02-20T07:52:05.298 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:52:05 smithi183 ceph-mon[35832]: pgmap v49: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-20T07:52:05.298 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:52:05 smithi183 ceph-mon[35832]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:52:05.298 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:52:05 smithi183 ceph-mon[35832]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:52:05.298 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:52:05 smithi183 ceph-mon[35832]: from='osd.1 [v2:172.21.15.133:6810/1652695599,v1:172.21.15.133:6811/1652695599]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-02-20T07:52:05.654 INFO:teuthology.orchestra.run.smithi133.stderr:Inferring config /var/lib/ceph/4537327c-cfc4-11ee-95bb-87774f69a715/mon.smithi133/config 2024-02-20T07:52:06.331 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:52:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:52:06.280+0000 7f0101659700 -1 osd.1 0 waiting for initial osdmap 2024-02-20T07:52:06.331 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:52:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:52:06.286+0000 7f00fc454700 -1 osd.1 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-20T07:52:06.331 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:52:06 smithi133 ceph-mon[29117]: from='osd.1 [v2:172.21.15.133:6810/1652695599,v1:172.21.15.133:6811/1652695599]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-02-20T07:52:06.331 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:52:06 smithi133 ceph-mon[29117]: osdmap e12: 2 total, 1 up, 2 in 2024-02-20T07:52:06.331 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:52:06 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-20T07:52:06.331 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:52:06 smithi133 ceph-mon[29117]: from='osd.1 [v2:172.21.15.133:6810/1652695599,v1:172.21.15.133:6811/1652695599]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi133", "root=default"]}]: dispatch 2024-02-20T07:52:06.331 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:52:06 smithi133 ceph-mon[29117]: pgmap v51: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-20T07:52:06.548 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:52:06 smithi183 ceph-mon[35832]: from='osd.1 [v2:172.21.15.133:6810/1652695599,v1:172.21.15.133:6811/1652695599]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-02-20T07:52:06.548 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:52:06 smithi183 ceph-mon[35832]: osdmap e12: 2 total, 1 up, 2 in 2024-02-20T07:52:06.548 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:52:06 smithi183 ceph-mon[35832]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-20T07:52:06.548 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:52:06 smithi183 ceph-mon[35832]: from='osd.1 [v2:172.21.15.133:6810/1652695599,v1:172.21.15.133:6811/1652695599]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi133", "root=default"]}]: dispatch 2024-02-20T07:52:06.548 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:52:06 smithi183 ceph-mon[35832]: pgmap v51: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-20T07:52:07.547 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:52:07 smithi183 ceph-mon[35832]: from='osd.1 [v2:172.21.15.133:6810/1652695599,v1:172.21.15.133:6811/1652695599]' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi133", "root=default"]}]': finished 2024-02-20T07:52:07.548 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:52:07 smithi183 ceph-mon[35832]: osdmap e13: 2 total, 1 up, 2 in 2024-02-20T07:52:07.548 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:52:07 smithi183 ceph-mon[35832]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-20T07:52:07.548 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:52:07 smithi183 ceph-mon[35832]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-20T07:52:07.612 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:52:07 smithi133 ceph-mon[29117]: from='osd.1 [v2:172.21.15.133:6810/1652695599,v1:172.21.15.133:6811/1652695599]' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi133", "root=default"]}]': finished 2024-02-20T07:52:07.612 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:52:07 smithi133 ceph-mon[29117]: osdmap e13: 2 total, 1 up, 2 in 2024-02-20T07:52:07.613 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:52:07 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-20T07:52:07.613 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:52:07 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-20T07:52:08.371 INFO:teuthology.orchestra.run.smithi133.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-02-20T07:52:08.371 INFO:teuthology.orchestra.run.smithi133.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-02-20T07:52:08.372 INFO:teuthology.orchestra.run.smithi133.stderr: stderr: 10+0 records in 2024-02-20T07:52:08.372 INFO:teuthology.orchestra.run.smithi133.stderr:10+0 records out 2024-02-20T07:52:08.372 INFO:teuthology.orchestra.run.smithi133.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0124752 s, 841 MB/s 2024-02-20T07:52:08.372 INFO:teuthology.orchestra.run.smithi133.stderr:--> Zapping successful for: 2024-02-20T07:52:08.548 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:52:08 smithi183 ceph-mon[35832]: purged_snaps scrub starts 2024-02-20T07:52:08.548 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:52:08 smithi183 ceph-mon[35832]: purged_snaps scrub ok 2024-02-20T07:52:08.548 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:52:08 smithi183 ceph-mon[35832]: osd.1 [v2:172.21.15.133:6810/1652695599,v1:172.21.15.133:6811/1652695599] boot 2024-02-20T07:52:08.548 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:52:08 smithi183 ceph-mon[35832]: osdmap e14: 2 total, 2 up, 2 in 2024-02-20T07:52:08.548 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:52:08 smithi183 ceph-mon[35832]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-20T07:52:08.548 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:52:08 smithi183 ceph-mon[35832]: pgmap v54: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-20T07:52:08.687 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:52:08 smithi133 ceph-mon[29117]: purged_snaps scrub starts 2024-02-20T07:52:08.687 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:52:08 smithi133 ceph-mon[29117]: purged_snaps scrub ok 2024-02-20T07:52:08.687 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:52:08 smithi133 ceph-mon[29117]: osd.1 [v2:172.21.15.133:6810/1652695599,v1:172.21.15.133:6811/1652695599] boot 2024-02-20T07:52:08.687 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:52:08 smithi133 ceph-mon[29117]: osdmap e14: 2 total, 2 up, 2 in 2024-02-20T07:52:08.688 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:52:08 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-20T07:52:08.688 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:52:08 smithi133 ceph-mon[29117]: pgmap v54: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-20T07:52:08.983 DEBUG:teuthology.orchestra.run.smithi133:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4537327c-cfc4-11ee-95bb-87774f69a715 -- ceph orch daemon add osd smithi133:vg_nvme/lv_2 2024-02-20T07:52:09.547 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:52:09 smithi183 ceph-mon[35832]: osdmap e15: 2 total, 2 up, 2 in 2024-02-20T07:52:09.548 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:52:09 smithi183 ceph-mon[35832]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-20T07:52:09.572 INFO:teuthology.orchestra.run.smithi133.stderr:Inferring config /var/lib/ceph/4537327c-cfc4-11ee-95bb-87774f69a715/mon.smithi133/config 2024-02-20T07:52:09.691 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:52:09 smithi133 ceph-mon[29117]: osdmap e15: 2 total, 2 up, 2 in 2024-02-20T07:52:09.691 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:52:09 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-20T07:52:10.548 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:52:10 smithi183 ceph-mon[35832]: pgmap v56: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-20T07:52:10.548 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:52:10 smithi183 ceph-mon[35832]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:52:10.548 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:52:10 smithi183 ceph-mon[35832]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:52:10.602 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:52:10 smithi133 ceph-mon[29117]: pgmap v56: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-20T07:52:10.625 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:52:10 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:52:10.625 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:52:10 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:52:11.128 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:52:11.126+0000 7f00df595700 1 -- 172.21.15.133:0/1906843669 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f00d80fe1f0 msgr2=0x7f00d80fe660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:52:11.129 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:52:11.126+0000 7f00df595700 1 --2- 172.21.15.133:0/1906843669 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f00d80fe1f0 0x7f00d80fe660 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f00c8004940 tx=0x7f00c8009e30 comp rx=0 tx=0).stop 2024-02-20T07:52:11.129 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:52:11.126+0000 7f00df595700 1 -- 172.21.15.133:0/1906843669 shutdown_connections 2024-02-20T07:52:11.129 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:52:11.126+0000 7f00df595700 1 --2- 172.21.15.133:0/1906843669 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f00d80fe1f0 0x7f00d80fe660 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:52:11.129 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:52:11.126+0000 7f00df595700 1 --2- 172.21.15.133:0/1906843669 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f00d80fcf40 0x7f00d80fd350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:52:11.129 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:52:11.126+0000 7f00df595700 1 -- 172.21.15.133:0/1906843669 >> 172.21.15.133:0/1906843669 conn(0x7f00d80f8680 msgr2=0x7f00d80faad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:52:11.129 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:52:11.126+0000 7f00df595700 1 -- 172.21.15.133:0/1906843669 shutdown_connections 2024-02-20T07:52:11.129 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:52:11.126+0000 7f00df595700 1 -- 172.21.15.133:0/1906843669 wait complete. 2024-02-20T07:52:11.129 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:52:11.127+0000 7f00df595700 1 Processor -- start 2024-02-20T07:52:11.129 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:52:11.127+0000 7f00df595700 1 -- start start 2024-02-20T07:52:11.131 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:52:11.127+0000 7f00df595700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f00d80fcf40 0x7f00d810d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:52:11.131 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:52:11.127+0000 7f00df595700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f00d80fe1f0 0x7f00d810ded0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:52:11.131 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:52:11.127+0000 7f00df595700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00d810e4d0 con 0x7f00d80fe1f0 2024-02-20T07:52:11.131 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:52:11.127+0000 7f00df595700 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00d810ed70 con 0x7f00d80fcf40 2024-02-20T07:52:11.131 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:52:11.127+0000 7f00dcb30700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f00d80fe1f0 0x7f00d810ded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:52:11.131 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:52:11.127+0000 7f00dcb30700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f00d80fe1f0 0x7f00d810ded0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.133:57154/0 (socket says 172.21.15.133:57154) 2024-02-20T07:52:11.132 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:52:11.127+0000 7f00dcb30700 1 -- 172.21.15.133:0/3486414639 learned_addr learned my addr 172.21.15.133:0/3486414639 (peer_addr_for_me v2:172.21.15.133:0/0) 2024-02-20T07:52:11.132 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:52:11.127+0000 7f00dd331700 1 --2- 172.21.15.133:0/3486414639 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f00d80fcf40 0x7f00d810d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:52:11.132 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:52:11.127+0000 7f00dcb30700 1 -- 172.21.15.133:0/3486414639 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f00d80fcf40 msgr2=0x7f00d810d990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:52:11.132 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:52:11.127+0000 7f00dcb30700 1 --2- 172.21.15.133:0/3486414639 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f00d80fcf40 0x7f00d810d990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:52:11.132 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:52:11.127+0000 7f00dcb30700 1 -- 172.21.15.133:0/3486414639 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00c8005040 con 0x7f00d80fe1f0 2024-02-20T07:52:11.132 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:52:11.127+0000 7f00dd331700 1 --2- 172.21.15.133:0/3486414639 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f00d80fcf40 0x7f00d810d990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-20T07:52:11.132 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:52:11.128+0000 7f00dcb30700 1 --2- 172.21.15.133:0/3486414639 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f00d80fe1f0 0x7f00d810ded0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f00c80093b0 tx=0x7f00c8009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:52:11.132 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:52:11.128+0000 7f00cdffb700 1 -- 172.21.15.133:0/3486414639 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f00c800e040 con 0x7f00d80fe1f0 2024-02-20T07:52:11.132 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:52:11.128+0000 7f00cdffb700 1 -- 172.21.15.133:0/3486414639 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f00c800bcc0 con 0x7f00d80fe1f0 2024-02-20T07:52:11.132 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:52:11.128+0000 7f00df595700 1 -- 172.21.15.133:0/3486414639 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f00d810eff0 con 0x7f00d80fe1f0 2024-02-20T07:52:11.132 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:52:11.128+0000 7f00cdffb700 1 -- 172.21.15.133:0/3486414639 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f00c801c630 con 0x7f00d80fe1f0 2024-02-20T07:52:11.132 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:52:11.128+0000 7f00df595700 1 -- 172.21.15.133:0/3486414639 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f00d810f530 con 0x7f00d80fe1f0 2024-02-20T07:52:11.132 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:52:11.129+0000 7f00df595700 1 -- 172.21.15.133:0/3486414639 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f00d804bed0 con 0x7f00d80fe1f0 2024-02-20T07:52:11.132 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:52:11.129+0000 7f00cdffb700 1 -- 172.21.15.133:0/3486414639 <== mon.0 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 18) v1 ==== 86567+0+0 (secure 0 0 0) 0x7f00c80139f0 con 0x7f00d80fe1f0 2024-02-20T07:52:11.133 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:52:11.130+0000 7f00cdffb700 1 --2- 172.21.15.133:0/3486414639 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7f00c4069210 0x7f00c406b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:52:11.133 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:52:11.130+0000 7f00cdffb700 1 -- 172.21.15.133:0/3486414639 <== mon.0 v2:172.21.15.133:3300/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 2410+0+0 (secure 0 0 0) 0x7f00c80877c0 con 0x7f00d80fe1f0 2024-02-20T07:52:11.133 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:52:11.130+0000 7f00dd331700 1 --2- 172.21.15.133:0/3486414639 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7f00c4069210 0x7f00c406b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:52:11.133 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:52:11.130+0000 7f00dd331700 1 --2- 172.21.15.133:0/3486414639 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7f00c4069210 0x7f00c406b6c0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f00d40041d0 tx=0x7f00d4009040 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:52:11.136 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:52:11.134+0000 7f00cdffb700 1 -- 172.21.15.133:0/3486414639 <== mon.0 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f00c80582f0 con 0x7f00d80fe1f0 2024-02-20T07:52:11.331 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:52:11.329+0000 7f00df595700 1 -- 172.21.15.133:0/3486414639 --> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi133:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) v1 -- 0x7f00d810f990 con 0x7f00c4069210 2024-02-20T07:52:11.480 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:52:11 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-20T07:52:11.480 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:52:11 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-20T07:52:11.481 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:52:11 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-20T07:52:11.797 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:52:11 smithi183 ceph-mon[35832]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-20T07:52:11.798 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:52:11 smithi183 ceph-mon[35832]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-20T07:52:11.798 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:52:11 smithi183 ceph-mon[35832]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-20T07:52:12.992 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:52:12 smithi133 ceph-mon[29117]: from='client.14280 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi133:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-02-20T07:52:12.992 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:52:12 smithi133 ceph-mon[29117]: pgmap v57: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-20T07:52:13.047 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:52:12 smithi183 ceph-mon[35832]: from='client.14280 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi133:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-02-20T07:52:13.048 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:52:12 smithi183 ceph-mon[35832]: pgmap v57: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-20T07:52:14.742 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:52:14 smithi133 ceph-mon[29117]: pgmap v58: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-20T07:52:15.047 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:52:14 smithi183 ceph-mon[35832]: pgmap v58: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-20T07:52:16.162 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:52:16 smithi133 ceph-mon[29117]: Detected new or changed devices on smithi133 2024-02-20T07:52:16.163 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:52:16 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:52:16.163 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:52:16 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:52:16.163 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:52:16 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-20T07:52:16.163 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:52:16 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-20T07:52:16.163 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:52:16 smithi133 ceph-mon[29117]: Adjusting osd_memory_target on smithi133 to 6432M 2024-02-20T07:52:16.163 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:52:16 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:52:16.163 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:52:16 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-20T07:52:16.163 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:52:16 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-20T07:52:16.163 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:52:16 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:52:16.163 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:52:16 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-20T07:52:16.163 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:52:16 smithi133 ceph-mon[29117]: pgmap v59: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-20T07:52:16.548 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:52:16 smithi183 ceph-mon[35832]: Detected new or changed devices on smithi133 2024-02-20T07:52:16.548 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:52:16 smithi183 ceph-mon[35832]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:52:16.548 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:52:16 smithi183 ceph-mon[35832]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:52:16.548 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:52:16 smithi183 ceph-mon[35832]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-20T07:52:16.548 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:52:16 smithi183 ceph-mon[35832]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-20T07:52:16.548 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:52:16 smithi183 ceph-mon[35832]: Adjusting osd_memory_target on smithi133 to 6432M 2024-02-20T07:52:16.548 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:52:16 smithi183 ceph-mon[35832]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:52:16.548 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:52:16 smithi183 ceph-mon[35832]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-20T07:52:16.548 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:52:16 smithi183 ceph-mon[35832]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-20T07:52:16.548 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:52:16 smithi183 ceph-mon[35832]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:52:16.549 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:52:16 smithi183 ceph-mon[35832]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-20T07:52:16.549 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:52:16 smithi183 ceph-mon[35832]: pgmap v59: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-20T07:52:17.492 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:52:17 smithi133 ceph-mon[29117]: from='client.? 172.21.15.133:0/140851075' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "95848108-ed6d-4030-9534-d687a769350f"}]: dispatch 2024-02-20T07:52:17.492 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:52:17 smithi133 ceph-mon[29117]: from='client.? 172.21.15.133:0/140851075' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "95848108-ed6d-4030-9534-d687a769350f"}]': finished 2024-02-20T07:52:17.492 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:52:17 smithi133 ceph-mon[29117]: osdmap e16: 3 total, 2 up, 3 in 2024-02-20T07:52:17.492 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:52:17 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-20T07:52:17.492 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:52:17 smithi133 ceph-mon[29117]: from='client.? 172.21.15.133:0/1306204479' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-20T07:52:17.547 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:52:17 smithi183 ceph-mon[35832]: from='client.? 172.21.15.133:0/140851075' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "95848108-ed6d-4030-9534-d687a769350f"}]: dispatch 2024-02-20T07:52:17.548 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:52:17 smithi183 ceph-mon[35832]: from='client.? 172.21.15.133:0/140851075' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "95848108-ed6d-4030-9534-d687a769350f"}]': finished 2024-02-20T07:52:17.548 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:52:17 smithi183 ceph-mon[35832]: osdmap e16: 3 total, 2 up, 3 in 2024-02-20T07:52:17.548 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:52:17 smithi183 ceph-mon[35832]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-20T07:52:17.548 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:52:17 smithi183 ceph-mon[35832]: from='client.? 172.21.15.133:0/1306204479' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-20T07:52:18.492 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:52:18 smithi133 ceph-mon[29117]: pgmap v61: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-20T07:52:18.547 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:52:18 smithi183 ceph-mon[35832]: pgmap v61: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-20T07:52:20.882 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:52:20 smithi133 ceph-mon[29117]: pgmap v62: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-20T07:52:20.882 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:52:20 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:52:20.883 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:52:20 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:52:20.883 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:52:20 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-20T07:52:20.883 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:52:20 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-20T07:52:20.883 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:52:20 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:52:21.047 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:52:20 smithi183 ceph-mon[35832]: pgmap v62: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-20T07:52:21.048 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:52:20 smithi183 ceph-mon[35832]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:52:21.048 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:52:20 smithi183 ceph-mon[35832]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:52:21.048 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:52:20 smithi183 ceph-mon[35832]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-20T07:52:21.048 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:52:20 smithi183 ceph-mon[35832]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-20T07:52:21.048 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:52:20 smithi183 ceph-mon[35832]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:52:22.944 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:52:22 smithi133 ceph-mon[29117]: pgmap v63: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-20T07:52:23.047 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:52:22 smithi183 ceph-mon[35832]: pgmap v63: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-20T07:52:23.992 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:52:23 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-02-20T07:52:23.992 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:52:23 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-20T07:52:23.992 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:52:23 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-20T07:52:24.047 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:52:23 smithi183 ceph-mon[35832]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-02-20T07:52:24.047 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:52:23 smithi183 ceph-mon[35832]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-20T07:52:24.048 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:52:23 smithi183 ceph-mon[35832]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-20T07:52:24.992 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:52:24 smithi133 ceph-mon[29117]: pgmap v64: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-20T07:52:24.992 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:52:24 smithi133 ceph-mon[29117]: Deploying daemon osd.2 on smithi133 2024-02-20T07:52:25.047 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:52:24 smithi183 ceph-mon[35832]: pgmap v64: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-20T07:52:25.047 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:52:24 smithi183 ceph-mon[35832]: Deploying daemon osd.2 on smithi133 2024-02-20T07:52:26.902 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:52:26 smithi133 ceph-mon[29117]: pgmap v65: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-20T07:52:27.047 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:52:26 smithi183 ceph-mon[35832]: pgmap v65: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-20T07:52:28.742 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:52:28 smithi133 ceph-mon[29117]: pgmap v66: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-20T07:52:28.742 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:52:28 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:52:28.742 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:52:28 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:52:28.742 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:52:28 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-20T07:52:29.047 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:52:28 smithi183 ceph-mon[35832]: pgmap v66: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-20T07:52:29.047 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:52:28 smithi183 ceph-mon[35832]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:52:29.048 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:52:28 smithi183 ceph-mon[35832]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:52:29.048 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:52:28 smithi183 ceph-mon[35832]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-20T07:52:30.742 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:52:30 smithi133 ceph-mon[29117]: pgmap v67: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-20T07:52:31.047 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:52:30 smithi183 ceph-mon[35832]: pgmap v67: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-20T07:52:31.900 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:52:31.898+0000 7f00cdffb700 1 -- 172.21.15.133:0/3486414639 <== mgr.14208 v2:172.21.15.133:6800/2703586938 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f00d810f990 con 0x7f00c4069210 2024-02-20T07:52:31.900 INFO:teuthology.orchestra.run.smithi133.stdout:Created osd(s) 2 on host 'smithi133' 2024-02-20T07:52:31.902 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:52:31.901+0000 7f00df595700 1 -- 172.21.15.133:0/3486414639 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7f00c4069210 msgr2=0x7f00c406b6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:52:31.902 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:52:31.901+0000 7f00df595700 1 --2- 172.21.15.133:0/3486414639 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7f00c4069210 0x7f00c406b6c0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f00d40041d0 tx=0x7f00d4009040 comp rx=0 tx=0).stop 2024-02-20T07:52:31.902 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:52:31.901+0000 7f00df595700 1 -- 172.21.15.133:0/3486414639 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f00d80fe1f0 msgr2=0x7f00d810ded0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:52:31.902 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:52:31.901+0000 7f00df595700 1 --2- 172.21.15.133:0/3486414639 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f00d80fe1f0 0x7f00d810ded0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f00c80093b0 tx=0x7f00c8009e30 comp rx=0 tx=0).stop 2024-02-20T07:52:31.902 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:52:31.901+0000 7f00df595700 1 -- 172.21.15.133:0/3486414639 shutdown_connections 2024-02-20T07:52:31.902 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:52:31.901+0000 7f00df595700 1 --2- 172.21.15.133:0/3486414639 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7f00c4069210 0x7f00c406b6c0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:52:31.902 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:52:31.901+0000 7f00df595700 1 --2- 172.21.15.133:0/3486414639 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f00d80fcf40 0x7f00d810d990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:52:31.902 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:52:31.901+0000 7f00df595700 1 --2- 172.21.15.133:0/3486414639 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f00d80fe1f0 0x7f00d810ded0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:52:31.902 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:52:31.901+0000 7f00df595700 1 -- 172.21.15.133:0/3486414639 >> 172.21.15.133:0/3486414639 conn(0x7f00d80f8680 msgr2=0x7f00d8101420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:52:31.902 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:52:31.901+0000 7f00df595700 1 -- 172.21.15.133:0/3486414639 shutdown_connections 2024-02-20T07:52:31.903 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:52:31.901+0000 7f00df595700 1 -- 172.21.15.133:0/3486414639 wait complete. 2024-02-20T07:52:32.549 DEBUG:teuthology.orchestra.run.smithi133:osd.2> sudo journalctl -f -n 0 -u ceph-4537327c-cfc4-11ee-95bb-87774f69a715@osd.2.service 2024-02-20T07:52:32.551 INFO:tasks.cephadm:Deploying osd.3 on smithi183 with /dev/vg_nvme/lv_4... 2024-02-20T07:52:32.552 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4537327c-cfc4-11ee-95bb-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-02-20T07:52:32.810 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/4537327c-cfc4-11ee-95bb-87774f69a715/mon.smithi183/config 2024-02-20T07:52:33.132 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:52:32 smithi183 ceph-mon[35832]: pgmap v68: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-20T07:52:33.133 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:52:32 smithi183 ceph-mon[35832]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:52:33.133 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:52:32 smithi183 ceph-mon[35832]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:52:33.133 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:52:32 smithi183 ceph-mon[35832]: from='osd.2 [v2:172.21.15.133:6818/3838005412,v1:172.21.15.133:6819/3838005412]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-02-20T07:52:33.133 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:52:32 smithi133 ceph-mon[29117]: pgmap v68: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-20T07:52:33.133 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:52:32 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:52:33.133 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:52:32 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:52:33.134 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:52:32 smithi133 ceph-mon[29117]: from='osd.2 [v2:172.21.15.133:6818/3838005412,v1:172.21.15.133:6819/3838005412]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-02-20T07:52:33.992 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:52:33 smithi133 ceph-mon[29117]: from='osd.2 [v2:172.21.15.133:6818/3838005412,v1:172.21.15.133:6819/3838005412]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-02-20T07:52:33.992 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:52:33 smithi133 ceph-mon[29117]: osdmap e17: 3 total, 2 up, 3 in 2024-02-20T07:52:33.992 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:52:33 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-20T07:52:33.992 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:52:33 smithi133 ceph-mon[29117]: from='osd.2 [v2:172.21.15.133:6818/3838005412,v1:172.21.15.133:6819/3838005412]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi133", "root=default"]}]: dispatch 2024-02-20T07:52:33.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:52:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:52:33.892+0000 7f8e5b288700 -1 osd.2 0 waiting for initial osdmap 2024-02-20T07:52:33.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:52:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:52:33.899+0000 7f8e57085700 -1 osd.2 18 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-20T07:52:34.026 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:52:33 smithi183 ceph-mon[35832]: from='osd.2 [v2:172.21.15.133:6818/3838005412,v1:172.21.15.133:6819/3838005412]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-02-20T07:52:34.026 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:52:33 smithi183 ceph-mon[35832]: osdmap e17: 3 total, 2 up, 3 in 2024-02-20T07:52:34.026 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:52:33 smithi183 ceph-mon[35832]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-20T07:52:34.026 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:52:33 smithi183 ceph-mon[35832]: from='osd.2 [v2:172.21.15.133:6818/3838005412,v1:172.21.15.133:6819/3838005412]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi133", "root=default"]}]: dispatch 2024-02-20T07:52:34.698 INFO:teuthology.orchestra.run.smithi183.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-02-20T07:52:34.698 INFO:teuthology.orchestra.run.smithi183.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-02-20T07:52:34.698 INFO:teuthology.orchestra.run.smithi183.stderr: stderr: 10+0 records in 2024-02-20T07:52:34.699 INFO:teuthology.orchestra.run.smithi183.stderr:10+0 records out 2024-02-20T07:52:34.699 INFO:teuthology.orchestra.run.smithi183.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0113345 s, 925 MB/s 2024-02-20T07:52:34.699 INFO:teuthology.orchestra.run.smithi183.stderr:--> Zapping successful for: 2024-02-20T07:52:35.011 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:52:34 smithi133 ceph-mon[29117]: pgmap v70: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-20T07:52:35.011 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:52:34 smithi133 ceph-mon[29117]: from='osd.2 [v2:172.21.15.133:6818/3838005412,v1:172.21.15.133:6819/3838005412]' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi133", "root=default"]}]': finished 2024-02-20T07:52:35.011 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:52:34 smithi133 ceph-mon[29117]: osdmap e18: 3 total, 2 up, 3 in 2024-02-20T07:52:35.011 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:52:34 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-20T07:52:35.011 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:52:34 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-20T07:52:35.030 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:52:34 smithi183 ceph-mon[35832]: pgmap v70: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-20T07:52:35.092 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:52:34 smithi183 ceph-mon[35832]: from='osd.2 [v2:172.21.15.133:6818/3838005412,v1:172.21.15.133:6819/3838005412]' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi133", "root=default"]}]': finished 2024-02-20T07:52:35.092 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:52:34 smithi183 ceph-mon[35832]: osdmap e18: 3 total, 2 up, 3 in 2024-02-20T07:52:35.093 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:52:34 smithi183 ceph-mon[35832]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-20T07:52:35.093 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:52:34 smithi183 ceph-mon[35832]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-20T07:52:35.324 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4537327c-cfc4-11ee-95bb-87774f69a715 -- ceph orch daemon add osd smithi183:vg_nvme/lv_4 2024-02-20T07:52:35.590 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/4537327c-cfc4-11ee-95bb-87774f69a715/mon.smithi183/config 2024-02-20T07:52:36.241 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:52:35 smithi133 ceph-mon[29117]: purged_snaps scrub starts 2024-02-20T07:52:36.242 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:52:35 smithi133 ceph-mon[29117]: purged_snaps scrub ok 2024-02-20T07:52:36.242 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:52:35 smithi133 ceph-mon[29117]: osd.2 [v2:172.21.15.133:6818/3838005412,v1:172.21.15.133:6819/3838005412] boot 2024-02-20T07:52:36.242 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:52:35 smithi133 ceph-mon[29117]: osdmap e19: 3 total, 3 up, 3 in 2024-02-20T07:52:36.242 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:52:35 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-20T07:52:36.242 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:52:35 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]: dispatch 2024-02-20T07:52:36.242 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:52:35 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:52:36.242 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:52:35 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:52:36.297 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:52:35 smithi183 ceph-mon[35832]: purged_snaps scrub starts 2024-02-20T07:52:36.297 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:52:35 smithi183 ceph-mon[35832]: purged_snaps scrub ok 2024-02-20T07:52:36.297 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:52:35 smithi183 ceph-mon[35832]: osd.2 [v2:172.21.15.133:6818/3838005412,v1:172.21.15.133:6819/3838005412] boot 2024-02-20T07:52:36.298 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:52:35 smithi183 ceph-mon[35832]: osdmap e19: 3 total, 3 up, 3 in 2024-02-20T07:52:36.298 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:52:35 smithi183 ceph-mon[35832]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-20T07:52:36.298 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:52:35 smithi183 ceph-mon[35832]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]: dispatch 2024-02-20T07:52:36.298 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:52:35 smithi183 ceph-mon[35832]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:52:36.298 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:52:35 smithi183 ceph-mon[35832]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:52:37.111 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:52:37.107+0000 7ff7ed2f7700 1 -- 172.21.15.183:0/3617362091 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff7e80fe2b0 msgr2=0x7ff7e80fe700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:52:37.111 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:52:37.107+0000 7ff7ed2f7700 1 --2- 172.21.15.183:0/3617362091 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff7e80fe2b0 0x7ff7e80fe700 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7ff7dc004940 tx=0x7ff7dc009e30 comp rx=0 tx=0).stop 2024-02-20T07:52:37.111 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:52:37.107+0000 7ff7ed2f7700 1 -- 172.21.15.183:0/3617362091 shutdown_connections 2024-02-20T07:52:37.111 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:52:37.107+0000 7ff7ed2f7700 1 --2- 172.21.15.183:0/3617362091 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff7e80fe2b0 0x7ff7e80fe700 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:52:37.111 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:52:37.107+0000 7ff7ed2f7700 1 --2- 172.21.15.183:0/3617362091 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ff7e80fb220 0x7ff7e80fb630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:52:37.111 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:52:37.107+0000 7ff7ed2f7700 1 -- 172.21.15.183:0/3617362091 >> 172.21.15.183:0/3617362091 conn(0x7ff7e80f6ca0 msgr2=0x7ff7e80f90d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:52:37.111 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:52:37.107+0000 7ff7ed2f7700 1 -- 172.21.15.183:0/3617362091 shutdown_connections 2024-02-20T07:52:37.111 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:52:37.107+0000 7ff7ed2f7700 1 -- 172.21.15.183:0/3617362091 wait complete. 2024-02-20T07:52:37.111 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:52:37.108+0000 7ff7ed2f7700 1 Processor -- start 2024-02-20T07:52:37.111 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:52:37.108+0000 7ff7ed2f7700 1 -- start start 2024-02-20T07:52:37.111 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:52:37.108+0000 7ff7ed2f7700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ff7e80fb220 0x7ff7e80fdd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:52:37.111 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:52:37.108+0000 7ff7ed2f7700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff7e80fe2b0 0x7ff7e80fc1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:52:37.112 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:52:37.108+0000 7ff7ed2f7700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7e80fc7c0 con 0x7ff7e80fb220 2024-02-20T07:52:37.112 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:52:37.108+0000 7ff7ed2f7700 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7e80fc930 con 0x7ff7e80fe2b0 2024-02-20T07:52:37.112 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:52:37.108+0000 7ff7e67fc700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff7e80fe2b0 0x7ff7e80fc1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:52:37.112 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:52:37.108+0000 7ff7e67fc700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff7e80fe2b0 0x7ff7e80fc1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.183:41966/0 (socket says 172.21.15.183:41966) 2024-02-20T07:52:37.112 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:52:37.108+0000 7ff7e67fc700 1 -- 172.21.15.183:0/2049763614 learned_addr learned my addr 172.21.15.183:0/2049763614 (peer_addr_for_me v2:172.21.15.183:0/0) 2024-02-20T07:52:37.112 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:52:37.108+0000 7ff7e6ffd700 1 --2- 172.21.15.183:0/2049763614 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ff7e80fb220 0x7ff7e80fdd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:52:37.112 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:52:37.109+0000 7ff7e67fc700 1 -- 172.21.15.183:0/2049763614 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ff7e80fb220 msgr2=0x7ff7e80fdd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:52:37.112 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:52:37.109+0000 7ff7e67fc700 1 --2- 172.21.15.183:0/2049763614 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ff7e80fb220 0x7ff7e80fdd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:52:37.112 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:52:37.109+0000 7ff7e67fc700 1 -- 172.21.15.183:0/2049763614 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7dc005040 con 0x7ff7e80fe2b0 2024-02-20T07:52:37.112 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:52:37.109+0000 7ff7e6ffd700 1 --2- 172.21.15.183:0/2049763614 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ff7e80fb220 0x7ff7e80fdd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-20T07:52:37.112 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:52:37.109+0000 7ff7e67fc700 1 --2- 172.21.15.183:0/2049763614 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff7e80fe2b0 0x7ff7e80fc1c0 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7ff7dc000c00 tx=0x7ff7dc009e30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:52:37.112 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:52:37.109+0000 7ff7cf7fe700 1 -- 172.21.15.183:0/2049763614 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff7dc021070 con 0x7ff7e80fe2b0 2024-02-20T07:52:37.112 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:52:37.109+0000 7ff7cf7fe700 1 -- 172.21.15.183:0/2049763614 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff7dc009430 con 0x7ff7e80fe2b0 2024-02-20T07:52:37.112 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:52:37.109+0000 7ff7cf7fe700 1 -- 172.21.15.183:0/2049763614 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff7dc00e070 con 0x7ff7e80fe2b0 2024-02-20T07:52:37.113 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:52:37.109+0000 7ff7ed2f7700 1 -- 172.21.15.183:0/2049763614 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff7e80fcbb0 con 0x7ff7e80fe2b0 2024-02-20T07:52:37.113 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:52:37.109+0000 7ff7ed2f7700 1 -- 172.21.15.183:0/2049763614 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff7e81159b0 con 0x7ff7e80fe2b0 2024-02-20T07:52:37.113 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:52:37.110+0000 7ff7ed2f7700 1 -- 172.21.15.183:0/2049763614 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff7e804bed0 con 0x7ff7e80fe2b0 2024-02-20T07:52:37.114 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:52:37.110+0000 7ff7cf7fe700 1 -- 172.21.15.183:0/2049763614 <== mon.1 v2:172.21.15.183:3300/0 4 ==== mgrmap(e 18) v1 ==== 86567+0+0 (secure 0 0 0) 0x7ff7dc01c400 con 0x7ff7e80fe2b0 2024-02-20T07:52:37.114 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:52:37.111+0000 7ff7cf7fe700 1 --2- 172.21.15.183:0/2049763614 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7ff7d0069210 0x7ff7d006b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:52:37.114 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:52:37.111+0000 7ff7cf7fe700 1 -- 172.21.15.183:0/2049763614 <== mon.1 v2:172.21.15.183:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 3228+0+0 (secure 0 0 0) 0x7ff7dc087920 con 0x7ff7e80fe2b0 2024-02-20T07:52:37.114 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:52:37.111+0000 7ff7e6ffd700 1 --2- 172.21.15.183:0/2049763614 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7ff7d0069210 0x7ff7d006b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:52:37.114 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:52:37.111+0000 7ff7e6ffd700 1 --2- 172.21.15.183:0/2049763614 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7ff7d0069210 0x7ff7d006b6c0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7ff7e80fdad0 tx=0x7ff7d800b040 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:52:37.116 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:52:37.115+0000 7ff7cf7fe700 1 -- 172.21.15.183:0/2049763614 <== mon.1 v2:172.21.15.183:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7ff7dc058080 con 0x7ff7e80fe2b0 2024-02-20T07:52:37.242 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:52:36 smithi133 ceph-mon[29117]: pgmap v73: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-20T07:52:37.242 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:52:36 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]': finished 2024-02-20T07:52:37.242 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:52:36 smithi133 ceph-mon[29117]: osdmap e20: 3 total, 3 up, 3 in 2024-02-20T07:52:37.242 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:52:36 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-02-20T07:52:37.297 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:52:36 smithi183 ceph-mon[35832]: pgmap v73: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-20T07:52:37.297 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:52:36 smithi183 ceph-mon[35832]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]': finished 2024-02-20T07:52:37.298 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:52:36 smithi183 ceph-mon[35832]: osdmap e20: 3 total, 3 up, 3 in 2024-02-20T07:52:37.298 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:52:36 smithi183 ceph-mon[35832]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-02-20T07:52:37.308 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:52:37.307+0000 7ff7ed2f7700 1 -- 172.21.15.183:0/2049763614 --> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi183:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) v1 -- 0x7ff7e8102c00 con 0x7ff7d0069210 2024-02-20T07:52:37.742 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:52:37 smithi133 sudo[64463]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-02-20T07:52:37.742 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:52:37 smithi133 sudo[64463]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-02-20T07:52:37.742 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:52:37 smithi133 sudo[64463]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-02-20T07:52:37.742 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:52:37 smithi133 sudo[64463]: pam_unix(sudo:session): session closed for user root 2024-02-20T07:52:37.978 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:52:37 smithi183 ceph-mon[35832]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-02-20T07:52:37.979 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:52:37 smithi183 ceph-mon[35832]: osdmap e21: 3 total, 3 up, 3 in 2024-02-20T07:52:37.979 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:52:37 smithi183 ceph-mon[35832]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-20T07:52:37.979 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:52:37 smithi183 ceph-mon[35832]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-20T07:52:37.979 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:52:37 smithi183 ceph-mon[35832]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-20T07:52:37.979 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:52:37 smithi183 ceph-mon[35832]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:52:38.018 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:52:37 smithi133 sudo[64487]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-02-20T07:52:38.018 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:52:37 smithi133 sudo[64487]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-02-20T07:52:38.018 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:52:37 smithi133 sudo[64487]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-02-20T07:52:38.018 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:52:37 smithi133 sudo[64487]: pam_unix(sudo:session): session closed for user root 2024-02-20T07:52:38.018 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:52:37 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-02-20T07:52:38.018 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:52:37 smithi133 ceph-mon[29117]: osdmap e21: 3 total, 3 up, 3 in 2024-02-20T07:52:38.018 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:52:37 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-20T07:52:38.019 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:52:37 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-20T07:52:38.019 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:52:37 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-20T07:52:38.019 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:52:37 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:52:38.279 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:52:38 smithi133 sudo[64497]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-02-20T07:52:38.279 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:52:38 smithi133 sudo[64497]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-02-20T07:52:38.279 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:52:38 smithi133 sudo[64497]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-02-20T07:52:38.279 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:52:38 smithi133 sudo[64497]: pam_unix(sudo:session): session closed for user root 2024-02-20T07:52:38.532 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:52:38 smithi133 sudo[64508]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-02-20T07:52:38.533 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:52:38 smithi133 sudo[64508]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-02-20T07:52:38.533 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:52:38 smithi133 sudo[64508]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-02-20T07:52:38.533 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:52:38 smithi133 sudo[64508]: pam_unix(sudo:session): session closed for user root 2024-02-20T07:52:38.799 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:52:38 smithi133 sudo[65524]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-02-20T07:52:38.799 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:52:38 smithi133 sudo[65524]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-02-20T07:52:38.799 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:52:38 smithi133 sudo[65524]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-02-20T07:52:38.799 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:52:38 smithi133 sudo[65524]: pam_unix(sudo:session): session closed for user root 2024-02-20T07:52:39.052 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:52:38 smithi133 sudo[67225]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-02-20T07:52:39.053 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:52:38 smithi133 sudo[67225]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-02-20T07:52:39.053 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:52:38 smithi133 sudo[67225]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-02-20T07:52:39.053 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:52:38 smithi133 sudo[67225]: pam_unix(sudo:session): session closed for user root 2024-02-20T07:52:39.053 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:52:39 smithi133 sudo[67231]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-02-20T07:52:39.053 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:52:39 smithi133 sudo[67231]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-02-20T07:52:39.053 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:52:39 smithi133 sudo[67231]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-02-20T07:52:39.305 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:52:39 smithi133 sudo[67231]: pam_unix(sudo:session): session closed for user root 2024-02-20T07:52:39.318 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:52:39.317+0000 7ff7cf7fe700 1 -- 172.21.15.183:0/2049763614 <== mon.1 v2:172.21.15.183:3300/0 7 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7ff7dc0549c0 con 0x7ff7e80fe2b0 2024-02-20T07:52:39.564 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:52:39 smithi183 ceph-mon[35832]: from='client.24131 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi183:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-02-20T07:52:39.564 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:52:39 smithi183 ceph-mon[35832]: pgmap v76: 1 pgs: 1 unknown; 0 B data, 871 MiB used, 267 GiB / 268 GiB avail 2024-02-20T07:52:39.564 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:52:39 smithi183 ceph-mon[35832]: osdmap e22: 3 total, 3 up, 3 in 2024-02-20T07:52:39.564 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:52:39 smithi183 ceph-mon[35832]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-20T07:52:39.742 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:52:39 smithi133 sudo[67235]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-02-20T07:52:39.742 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:52:39 smithi133 sudo[67235]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-02-20T07:52:39.742 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:52:39 smithi133 sudo[67235]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-02-20T07:52:39.742 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:52:39 smithi133 sudo[67235]: pam_unix(sudo:session): session closed for user root 2024-02-20T07:52:39.742 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:52:39 smithi133 ceph-mon[29117]: from='client.24131 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi183:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-02-20T07:52:39.742 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:52:39 smithi133 ceph-mon[29117]: pgmap v76: 1 pgs: 1 unknown; 0 B data, 871 MiB used, 267 GiB / 268 GiB avail 2024-02-20T07:52:39.742 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:52:39 smithi133 ceph-mon[29117]: osdmap e22: 3 total, 3 up, 3 in 2024-02-20T07:52:39.742 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:52:39 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-20T07:52:39.818 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:52:39 smithi183 sudo[39203]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-02-20T07:52:39.818 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:52:39 smithi183 sudo[39203]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-02-20T07:52:39.818 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:52:39 smithi183 sudo[39203]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-02-20T07:52:39.818 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:52:39 smithi183 sudo[39203]: pam_unix(sudo:session): session closed for user root 2024-02-20T07:52:40.206 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:52:39 smithi183 sudo[39206]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-02-20T07:52:40.207 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:52:39 smithi183 sudo[39206]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-02-20T07:52:40.207 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:52:39 smithi183 sudo[39206]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-02-20T07:52:40.207 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:52:39 smithi183 sudo[39206]: pam_unix(sudo:session): session closed for user root 2024-02-20T07:52:40.548 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:52:40 smithi183 ceph-mon[35832]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-02-20T07:52:40.548 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:52:40 smithi183 ceph-mon[35832]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-02-20T07:52:40.548 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:52:40 smithi183 ceph-mon[35832]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "mon metadata", "id": "smithi133"}]: dispatch 2024-02-20T07:52:40.548 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:52:40 smithi183 ceph-mon[35832]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "mon metadata", "id": "smithi183"}]: dispatch 2024-02-20T07:52:40.548 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:52:40 smithi183 ceph-mon[35832]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-02-20T07:52:40.548 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:52:40 smithi183 ceph-mon[35832]: mgrmap e19: smithi133.mrvcbp(active, since 2m), standbys: smithi183.hebfng 2024-02-20T07:52:40.548 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:52:40 smithi183 ceph-mon[35832]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "mon metadata", "id": "smithi133"}]: dispatch 2024-02-20T07:52:40.548 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:52:40 smithi183 ceph-mon[35832]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "mon metadata", "id": "smithi183"}]: dispatch 2024-02-20T07:52:40.548 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:52:40 smithi183 ceph-mon[35832]: Detected new or changed devices on smithi133 2024-02-20T07:52:40.548 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:52:40 smithi183 ceph-mon[35832]: pgmap v78: 1 pgs: 1 active+clean; 577 KiB data, 872 MiB used, 267 GiB / 268 GiB avail 2024-02-20T07:52:40.548 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:52:40 smithi183 ceph-mon[35832]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-02-20T07:52:40.548 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:52:40 smithi183 ceph-mon[35832]: from='client.? 172.21.15.183:0/2694469525' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c2015ce7-f7fb-428d-b42f-e5fb665933e0"}]: dispatch 2024-02-20T07:52:40.548 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:52:40 smithi183 ceph-mon[35832]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:52:40.548 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:52:40 smithi183 ceph-mon[35832]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c2015ce7-f7fb-428d-b42f-e5fb665933e0"}]: dispatch 2024-02-20T07:52:40.549 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:52:40 smithi183 ceph-mon[35832]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c2015ce7-f7fb-428d-b42f-e5fb665933e0"}]': finished 2024-02-20T07:52:40.549 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:52:40 smithi183 ceph-mon[35832]: osdmap e23: 4 total, 3 up, 4 in 2024-02-20T07:52:40.549 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:52:40 smithi183 ceph-mon[35832]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-20T07:52:40.549 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:52:40 smithi183 ceph-mon[35832]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:52:40.549 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:52:40 smithi183 ceph-mon[35832]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-20T07:52:40.549 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:52:40 smithi183 ceph-mon[35832]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-20T07:52:40.549 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:52:40 smithi183 ceph-mon[35832]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-02-20T07:52:40.549 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:52:40 smithi183 ceph-mon[35832]: Adjusting osd_memory_target on smithi133 to 4288M 2024-02-20T07:52:40.549 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:52:40 smithi183 ceph-mon[35832]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:52:40.549 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:52:40 smithi183 ceph-mon[35832]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-20T07:52:40.549 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:52:40 smithi183 ceph-mon[35832]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-20T07:52:40.549 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:52:40 smithi183 ceph-mon[35832]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:52:40.549 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:52:40 smithi183 ceph-mon[35832]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:52:40.549 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:52:40 smithi183 ceph-mon[35832]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-20T07:52:40.742 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:52:40 smithi133 ceph-mon[29117]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-02-20T07:52:40.742 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:52:40 smithi133 ceph-mon[29117]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-02-20T07:52:40.742 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:52:40 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "mon metadata", "id": "smithi133"}]: dispatch 2024-02-20T07:52:40.742 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:52:40 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "mon metadata", "id": "smithi183"}]: dispatch 2024-02-20T07:52:40.742 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:52:40 smithi133 ceph-mon[29117]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-02-20T07:52:40.742 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:52:40 smithi133 ceph-mon[29117]: mgrmap e19: smithi133.mrvcbp(active, since 2m), standbys: smithi183.hebfng 2024-02-20T07:52:40.743 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:52:40 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "mon metadata", "id": "smithi133"}]: dispatch 2024-02-20T07:52:40.743 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:52:40 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "mon metadata", "id": "smithi183"}]: dispatch 2024-02-20T07:52:40.743 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:52:40 smithi133 ceph-mon[29117]: Detected new or changed devices on smithi133 2024-02-20T07:52:40.743 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:52:40 smithi133 ceph-mon[29117]: pgmap v78: 1 pgs: 1 active+clean; 577 KiB data, 872 MiB used, 267 GiB / 268 GiB avail 2024-02-20T07:52:40.743 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:52:40 smithi133 ceph-mon[29117]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-02-20T07:52:40.743 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:52:40 smithi133 ceph-mon[29117]: from='client.? 172.21.15.183:0/2694469525' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c2015ce7-f7fb-428d-b42f-e5fb665933e0"}]: dispatch 2024-02-20T07:52:40.743 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:52:40 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:52:40.743 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:52:40 smithi133 ceph-mon[29117]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c2015ce7-f7fb-428d-b42f-e5fb665933e0"}]: dispatch 2024-02-20T07:52:40.743 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:52:40 smithi133 ceph-mon[29117]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c2015ce7-f7fb-428d-b42f-e5fb665933e0"}]': finished 2024-02-20T07:52:40.743 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:52:40 smithi133 ceph-mon[29117]: osdmap e23: 4 total, 3 up, 4 in 2024-02-20T07:52:40.743 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:52:40 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-20T07:52:40.743 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:52:40 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:52:40.743 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:52:40 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-20T07:52:40.743 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:52:40 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-20T07:52:40.743 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:52:40 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-02-20T07:52:40.744 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:52:40 smithi133 ceph-mon[29117]: Adjusting osd_memory_target on smithi133 to 4288M 2024-02-20T07:52:40.744 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:52:40 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:52:40.744 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:52:40 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-20T07:52:40.744 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:52:40 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-20T07:52:40.744 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:52:40 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:52:40.744 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:52:40 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:52:40.744 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:52:40 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-20T07:52:41.742 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:52:41 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-20T07:52:41.742 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:52:41 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-20T07:52:41.742 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:52:41 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:52:41.742 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:52:41 smithi133 ceph-mon[29117]: from='client.? 172.21.15.183:0/1443248315' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-20T07:52:41.797 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:52:41 smithi183 ceph-mon[35832]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-20T07:52:41.797 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:52:41 smithi183 ceph-mon[35832]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-20T07:52:41.797 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:52:41 smithi183 ceph-mon[35832]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:52:41.798 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:52:41 smithi183 ceph-mon[35832]: from='client.? 172.21.15.183:0/1443248315' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-20T07:52:42.992 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:52:42 smithi133 ceph-mon[29117]: pgmap v80: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-02-20T07:52:43.047 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:52:42 smithi183 ceph-mon[35832]: pgmap v80: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-02-20T07:52:44.926 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:52:44 smithi183 ceph-mon[35832]: pgmap v81: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 4.5 KiB/s rd, 196 KiB/s wr, 13 op/s 2024-02-20T07:52:44.992 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:52:44 smithi133 ceph-mon[29117]: pgmap v81: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 4.5 KiB/s rd, 196 KiB/s wr, 13 op/s 2024-02-20T07:52:46.797 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:52:46 smithi183 ceph-mon[35832]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 3.0 KiB/s rd, 131 KiB/s wr, 9 op/s 2024-02-20T07:52:46.992 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:52:46 smithi133 ceph-mon[29117]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 3.0 KiB/s rd, 131 KiB/s wr, 9 op/s 2024-02-20T07:52:47.797 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:52:47 smithi183 ceph-mon[35832]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-02-20T07:52:47.798 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:52:47 smithi183 ceph-mon[35832]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-20T07:52:47.798 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:52:47 smithi183 ceph-mon[35832]: Deploying daemon osd.3 on smithi183 2024-02-20T07:52:47.992 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:52:47 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-02-20T07:52:47.992 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:52:47 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-20T07:52:47.992 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:52:47 smithi133 ceph-mon[29117]: Deploying daemon osd.3 on smithi183 2024-02-20T07:52:48.797 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:52:48 smithi183 ceph-mon[35832]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 2.2 KiB/s rd, 98 KiB/s wr, 6 op/s 2024-02-20T07:52:48.991 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:52:48 smithi133 ceph-mon[29117]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 2.2 KiB/s rd, 98 KiB/s wr, 6 op/s 2024-02-20T07:52:50.872 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:52:50 smithi183 ceph-mon[35832]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 1.8 KiB/s rd, 78 KiB/s wr, 5 op/s 2024-02-20T07:52:50.991 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:52:50 smithi133 ceph-mon[29117]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 1.8 KiB/s rd, 78 KiB/s wr, 5 op/s 2024-02-20T07:52:52.992 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:52:52 smithi133 ceph-mon[29117]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 1.5 KiB/s rd, 67 KiB/s wr, 4 op/s 2024-02-20T07:52:52.992 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:52:52 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:52:52.992 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:52:52 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:52:52.992 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:52:52 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-20T07:52:53.008 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:52:52 smithi183 ceph-mon[35832]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 1.5 KiB/s rd, 67 KiB/s wr, 4 op/s 2024-02-20T07:52:53.008 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:52:52 smithi183 ceph-mon[35832]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:52:53.008 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:52:52 smithi183 ceph-mon[35832]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:52:53.008 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:52:52 smithi183 ceph-mon[35832]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-20T07:52:53.980 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:52:53 smithi183 ceph-mon[35832]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-20T07:52:53.991 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:52:53 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-20T07:52:54.992 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:52:54 smithi133 ceph-mon[29117]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 1.5 KiB/s rd, 65 KiB/s wr, 4 op/s 2024-02-20T07:52:55.019 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:52:54 smithi183 ceph-mon[35832]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 1.5 KiB/s rd, 65 KiB/s wr, 4 op/s 2024-02-20T07:52:55.317 INFO:teuthology.orchestra.run.smithi183.stdout:Created osd(s) 3 on host 'smithi183' 2024-02-20T07:52:55.317 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:52:55.315+0000 7ff7cf7fe700 1 -- 172.21.15.183:0/2049763614 <== mgr.14208 v2:172.21.15.133:6800/2703586938 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7ff7e8102c00 con 0x7ff7d0069210 2024-02-20T07:52:55.319 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:52:55.318+0000 7ff7ed2f7700 1 -- 172.21.15.183:0/2049763614 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7ff7d0069210 msgr2=0x7ff7d006b6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:52:55.319 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:52:55.318+0000 7ff7ed2f7700 1 --2- 172.21.15.183:0/2049763614 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7ff7d0069210 0x7ff7d006b6c0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7ff7e80fdad0 tx=0x7ff7d800b040 comp rx=0 tx=0).stop 2024-02-20T07:52:55.319 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:52:55.318+0000 7ff7ed2f7700 1 -- 172.21.15.183:0/2049763614 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff7e80fe2b0 msgr2=0x7ff7e80fc1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:52:55.319 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:52:55.318+0000 7ff7ed2f7700 1 --2- 172.21.15.183:0/2049763614 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff7e80fe2b0 0x7ff7e80fc1c0 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7ff7dc000c00 tx=0x7ff7dc009e30 comp rx=0 tx=0).stop 2024-02-20T07:52:55.319 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:52:55.318+0000 7ff7ed2f7700 1 -- 172.21.15.183:0/2049763614 shutdown_connections 2024-02-20T07:52:55.319 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:52:55.318+0000 7ff7ed2f7700 1 --2- 172.21.15.183:0/2049763614 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ff7e80fb220 0x7ff7e80fdd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:52:55.319 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:52:55.318+0000 7ff7ed2f7700 1 --2- 172.21.15.183:0/2049763614 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7ff7d0069210 0x7ff7d006b6c0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:52:55.319 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:52:55.318+0000 7ff7ed2f7700 1 --2- 172.21.15.183:0/2049763614 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff7e80fe2b0 0x7ff7e80fc1c0 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:52:55.320 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:52:55.318+0000 7ff7ed2f7700 1 -- 172.21.15.183:0/2049763614 >> 172.21.15.183:0/2049763614 conn(0x7ff7e80f6ca0 msgr2=0x7ff7e81014e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:52:55.320 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:52:55.318+0000 7ff7ed2f7700 1 -- 172.21.15.183:0/2049763614 shutdown_connections 2024-02-20T07:52:55.320 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:52:55.318+0000 7ff7ed2f7700 1 -- 172.21.15.183:0/2049763614 wait complete. 2024-02-20T07:52:55.941 DEBUG:teuthology.orchestra.run.smithi183:osd.3> sudo journalctl -f -n 0 -u ceph-4537327c-cfc4-11ee-95bb-87774f69a715@osd.3.service 2024-02-20T07:52:55.943 INFO:tasks.cephadm:Deploying osd.4 on smithi183 with /dev/vg_nvme/lv_3... 2024-02-20T07:52:55.944 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4537327c-cfc4-11ee-95bb-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-02-20T07:52:56.239 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/4537327c-cfc4-11ee-95bb-87774f69a715/mon.smithi183/config 2024-02-20T07:52:56.585 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:52:56 smithi183 ceph-mon[35832]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:52:56.585 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:52:56 smithi183 ceph-mon[35832]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:52:56.585 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:52:56 smithi183 ceph-mon[35832]: from='osd.3 [v2:172.21.15.183:6800/1053980618,v1:172.21.15.183:6801/1053980618]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-02-20T07:52:56.585 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:52:56 smithi183 ceph-mon[35832]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-02-20T07:52:56.585 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:52:56 smithi183 ceph-mon[35832]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-02-20T07:52:56.585 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:52:56 smithi183 ceph-mon[35832]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:52:56.585 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:52:56 smithi183 ceph-mon[35832]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:52:56.742 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:52:56 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:52:56.742 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:52:56 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:52:56.742 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:52:56 smithi133 ceph-mon[29117]: from='osd.3 [v2:172.21.15.183:6800/1053980618,v1:172.21.15.183:6801/1053980618]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-02-20T07:52:56.742 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:52:56 smithi133 ceph-mon[29117]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-02-20T07:52:56.742 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:52:56 smithi133 ceph-mon[29117]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-02-20T07:52:56.742 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:52:56 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:52:56.742 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:52:56 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:52:57.626 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:52:57 smithi183 ceph-mon[35832]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-02-20T07:52:57.626 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:52:57 smithi183 ceph-mon[35832]: osdmap e24: 4 total, 3 up, 4 in 2024-02-20T07:52:57.626 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:52:57 smithi183 ceph-mon[35832]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-20T07:52:57.626 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:52:57 smithi183 ceph-mon[35832]: from='osd.3 [v2:172.21.15.183:6800/1053980618,v1:172.21.15.183:6801/1053980618]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi183", "root=default"]}]: dispatch 2024-02-20T07:52:57.626 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:52:57 smithi183 ceph-mon[35832]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi183", "root=default"]}]: dispatch 2024-02-20T07:52:57.626 INFO:journalctl@ceph.osd.3.smithi183.stdout:Feb 20 07:52:57 smithi183 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-3[41430]: 2024-02-20T07:52:57.326+0000 7f5bf3961700 -1 osd.3 0 waiting for initial osdmap 2024-02-20T07:52:57.626 INFO:journalctl@ceph.osd.3.smithi183.stdout:Feb 20 07:52:57 smithi183 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-3[41430]: 2024-02-20T07:52:57.331+0000 7f5bef75e700 -1 osd.3 25 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-20T07:52:57.742 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:52:57 smithi133 ceph-mon[29117]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-02-20T07:52:57.742 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:52:57 smithi133 ceph-mon[29117]: osdmap e24: 4 total, 3 up, 4 in 2024-02-20T07:52:57.742 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:52:57 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-20T07:52:57.742 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:52:57 smithi133 ceph-mon[29117]: from='osd.3 [v2:172.21.15.183:6800/1053980618,v1:172.21.15.183:6801/1053980618]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi183", "root=default"]}]: dispatch 2024-02-20T07:52:57.742 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:52:57 smithi133 ceph-mon[29117]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi183", "root=default"]}]: dispatch 2024-02-20T07:52:58.634 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:52:58 smithi183 ceph-mon[35832]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi183", "root=default"]}]': finished 2024-02-20T07:52:58.634 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:52:58 smithi183 ceph-mon[35832]: osdmap e25: 4 total, 3 up, 4 in 2024-02-20T07:52:58.635 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:52:58 smithi183 ceph-mon[35832]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-20T07:52:58.635 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:52:58 smithi183 ceph-mon[35832]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-20T07:52:58.635 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:52:58 smithi183 ceph-mon[35832]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-02-20T07:52:58.635 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:52:58 smithi183 ceph-mon[35832]: osd.3 [v2:172.21.15.183:6800/1053980618,v1:172.21.15.183:6801/1053980618] boot 2024-02-20T07:52:58.635 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:52:58 smithi183 ceph-mon[35832]: osdmap e26: 4 total, 4 up, 4 in 2024-02-20T07:52:58.635 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:52:58 smithi183 ceph-mon[35832]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-20T07:52:58.742 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:52:58 smithi133 ceph-mon[29117]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi183", "root=default"]}]': finished 2024-02-20T07:52:58.742 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:52:58 smithi133 ceph-mon[29117]: osdmap e25: 4 total, 3 up, 4 in 2024-02-20T07:52:58.742 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:52:58 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-20T07:52:58.742 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:52:58 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-20T07:52:58.742 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:52:58 smithi133 ceph-mon[29117]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-02-20T07:52:58.742 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:52:58 smithi133 ceph-mon[29117]: osd.3 [v2:172.21.15.183:6800/1053980618,v1:172.21.15.183:6801/1053980618] boot 2024-02-20T07:52:58.742 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:52:58 smithi133 ceph-mon[29117]: osdmap e26: 4 total, 4 up, 4 in 2024-02-20T07:52:58.742 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:52:58 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-20T07:52:59.378 INFO:teuthology.orchestra.run.smithi183.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-02-20T07:52:59.379 INFO:teuthology.orchestra.run.smithi183.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-02-20T07:52:59.379 INFO:teuthology.orchestra.run.smithi183.stderr: stderr: 10+0 records in 2024-02-20T07:52:59.379 INFO:teuthology.orchestra.run.smithi183.stderr:10+0 records out 2024-02-20T07:52:59.379 INFO:teuthology.orchestra.run.smithi183.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0124732 s, 841 MB/s 2024-02-20T07:52:59.379 INFO:teuthology.orchestra.run.smithi183.stderr:--> Zapping successful for: 2024-02-20T07:52:59.547 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:52:59 smithi183 ceph-mon[35832]: purged_snaps scrub starts 2024-02-20T07:52:59.547 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:52:59 smithi183 ceph-mon[35832]: purged_snaps scrub ok 2024-02-20T07:52:59.547 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:52:59 smithi183 ceph-mon[35832]: osdmap e27: 4 total, 4 up, 4 in 2024-02-20T07:52:59.742 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:52:59 smithi133 ceph-mon[29117]: purged_snaps scrub starts 2024-02-20T07:52:59.742 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:52:59 smithi133 ceph-mon[29117]: purged_snaps scrub ok 2024-02-20T07:52:59.742 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:52:59 smithi133 ceph-mon[29117]: osdmap e27: 4 total, 4 up, 4 in 2024-02-20T07:53:00.047 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4537327c-cfc4-11ee-95bb-87774f69a715 -- ceph orch daemon add osd smithi183:vg_nvme/lv_3 2024-02-20T07:53:00.710 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/4537327c-cfc4-11ee-95bb-87774f69a715/mon.smithi183/config 2024-02-20T07:53:00.741 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:53:00 smithi133 ceph-mon[29117]: pgmap v93: 1 pgs: 1 peering; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-02-20T07:53:00.771 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:53:00 smithi133 ceph-mon[29117]: osdmap e28: 4 total, 4 up, 4 in 2024-02-20T07:53:00.800 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:53:00 smithi183 ceph-mon[35832]: pgmap v93: 1 pgs: 1 peering; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-02-20T07:53:00.800 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:53:00 smithi183 ceph-mon[35832]: osdmap e28: 4 total, 4 up, 4 in 2024-02-20T07:53:01.798 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:53:01 smithi183 ceph-mon[35832]: Detected new or changed devices on smithi183 2024-02-20T07:53:01.798 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:53:01 smithi183 ceph-mon[35832]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:53:01.798 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:53:01 smithi183 ceph-mon[35832]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:53:01.798 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:53:01 smithi183 ceph-mon[35832]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-02-20T07:53:01.798 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:53:01 smithi183 ceph-mon[35832]: Adjusting osd_memory_target on smithi183 to 15936M 2024-02-20T07:53:01.798 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:53:01 smithi183 ceph-mon[35832]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:53:01.798 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:53:01 smithi183 ceph-mon[35832]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-20T07:53:01.798 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:53:01 smithi183 ceph-mon[35832]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-20T07:53:01.798 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:53:01 smithi183 ceph-mon[35832]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:53:01.798 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:53:01 smithi183 ceph-mon[35832]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-20T07:53:01.798 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:53:01 smithi183 ceph-mon[35832]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-20T07:53:01.798 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:53:01 smithi183 ceph-mon[35832]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-20T07:53:01.799 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:53:01 smithi183 ceph-mon[35832]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:53:01.799 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:53:01 smithi183 ceph-mon[35832]: osdmap e29: 4 total, 4 up, 4 in 2024-02-20T07:53:01.992 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:53:01 smithi133 ceph-mon[29117]: Detected new or changed devices on smithi183 2024-02-20T07:53:01.992 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:53:01 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:53:01.992 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:53:01 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:53:01.992 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:53:01 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-02-20T07:53:01.992 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:53:01 smithi133 ceph-mon[29117]: Adjusting osd_memory_target on smithi183 to 15936M 2024-02-20T07:53:01.992 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:53:01 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:53:01.992 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:53:01 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-20T07:53:01.992 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:53:01 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-20T07:53:01.993 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:53:01 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:53:01.993 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:53:01 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-20T07:53:01.993 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:53:01 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-20T07:53:01.993 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:53:01 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-20T07:53:01.993 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:53:01 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:53:01.993 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:53:01 smithi133 ceph-mon[29117]: osdmap e29: 4 total, 4 up, 4 in 2024-02-20T07:53:02.345 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:53:02.343+0000 7fe6b6344700 1 -- 172.21.15.183:0/2589553980 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe6b00663c0 msgr2=0x7fe6b00ffb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:53:02.345 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:53:02.343+0000 7fe6b6344700 1 --2- 172.21.15.183:0/2589553980 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe6b00663c0 0x7fe6b00ffb10 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fe6a0004940 tx=0x7fe6a0009e30 comp rx=0 tx=0).stop 2024-02-20T07:53:02.345 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:53:02.343+0000 7fe6b6344700 1 -- 172.21.15.183:0/2589553980 shutdown_connections 2024-02-20T07:53:02.345 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:53:02.343+0000 7fe6b6344700 1 --2- 172.21.15.183:0/2589553980 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fe6b01000e0 0x7fe6b0102520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:53:02.345 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:53:02.343+0000 7fe6b6344700 1 --2- 172.21.15.183:0/2589553980 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe6b00663c0 0x7fe6b00ffb10 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:53:02.345 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:53:02.343+0000 7fe6b6344700 1 -- 172.21.15.183:0/2589553980 >> 172.21.15.183:0/2589553980 conn(0x7fe6b00f74e0 msgr2=0x7fe6b00f9930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:53:02.345 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:53:02.343+0000 7fe6b6344700 1 -- 172.21.15.183:0/2589553980 shutdown_connections 2024-02-20T07:53:02.345 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:53:02.343+0000 7fe6b6344700 1 -- 172.21.15.183:0/2589553980 wait complete. 2024-02-20T07:53:02.345 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:53:02.344+0000 7fe6b6344700 1 Processor -- start 2024-02-20T07:53:02.345 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:53:02.344+0000 7fe6b6344700 1 -- start start 2024-02-20T07:53:02.347 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:53:02.344+0000 7fe6b6344700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe6b00663c0 0x7fe6b010d910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:53:02.347 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:53:02.344+0000 7fe6b6344700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fe6b01000e0 0x7fe6b010de50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:53:02.347 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:53:02.344+0000 7fe6b6344700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6b010e450 con 0x7fe6b01000e0 2024-02-20T07:53:02.347 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:53:02.344+0000 7fe6b6344700 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6b010ecf0 con 0x7fe6b00663c0 2024-02-20T07:53:02.347 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:53:02.344+0000 7fe6affff700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe6b00663c0 0x7fe6b010d910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:53:02.347 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:53:02.344+0000 7fe6affff700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe6b00663c0 0x7fe6b010d910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.183:38964/0 (socket says 172.21.15.183:38964) 2024-02-20T07:53:02.347 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:53:02.344+0000 7fe6affff700 1 -- 172.21.15.183:0/4006557456 learned_addr learned my addr 172.21.15.183:0/4006557456 (peer_addr_for_me v2:172.21.15.183:0/0) 2024-02-20T07:53:02.347 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:53:02.344+0000 7fe6af7fe700 1 --2- 172.21.15.183:0/4006557456 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fe6b01000e0 0x7fe6b010de50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:53:02.347 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:53:02.344+0000 7fe6affff700 1 -- 172.21.15.183:0/4006557456 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fe6b01000e0 msgr2=0x7fe6b010de50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:53:02.347 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:53:02.344+0000 7fe6affff700 1 --2- 172.21.15.183:0/4006557456 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fe6b01000e0 0x7fe6b010de50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:53:02.347 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:53:02.344+0000 7fe6affff700 1 -- 172.21.15.183:0/4006557456 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6a0005040 con 0x7fe6b00663c0 2024-02-20T07:53:02.347 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:53:02.344+0000 7fe6af7fe700 1 --2- 172.21.15.183:0/4006557456 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fe6b01000e0 0x7fe6b010de50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-20T07:53:02.347 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:53:02.345+0000 7fe6affff700 1 --2- 172.21.15.183:0/4006557456 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe6b00663c0 0x7fe6b010d910 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fe6a000b3b0 tx=0x7fe6a0009e30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:53:02.348 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:53:02.345+0000 7fe6acff9700 1 -- 172.21.15.183:0/4006557456 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe6a000e040 con 0x7fe6b00663c0 2024-02-20T07:53:02.348 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:53:02.345+0000 7fe6b6344700 1 -- 172.21.15.183:0/4006557456 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe6b010ef70 con 0x7fe6b00663c0 2024-02-20T07:53:02.348 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:53:02.345+0000 7fe6acff9700 1 -- 172.21.15.183:0/4006557456 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe6a000bd90 con 0x7fe6b00663c0 2024-02-20T07:53:02.348 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:53:02.345+0000 7fe6acff9700 1 -- 172.21.15.183:0/4006557456 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe6a001c630 con 0x7fe6b00663c0 2024-02-20T07:53:02.348 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:53:02.345+0000 7fe6b6344700 1 -- 172.21.15.183:0/4006557456 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe6b010f4b0 con 0x7fe6b00663c0 2024-02-20T07:53:02.348 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:53:02.346+0000 7fe6b6344700 1 -- 172.21.15.183:0/4006557456 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe6b004bed0 con 0x7fe6b00663c0 2024-02-20T07:53:02.349 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:53:02.346+0000 7fe6acff9700 1 -- 172.21.15.183:0/4006557456 <== mon.1 v2:172.21.15.183:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7fe6a0013420 con 0x7fe6b00663c0 2024-02-20T07:53:02.349 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:53:02.347+0000 7fe6acff9700 1 --2- 172.21.15.183:0/4006557456 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7fe6980692b0 0x7fe69806b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:53:02.349 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:53:02.347+0000 7fe6acff9700 1 -- 172.21.15.183:0/4006557456 <== mon.1 v2:172.21.15.183:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 3770+0+0 (secure 0 0 0) 0x7fe6a0088000 con 0x7fe6b00663c0 2024-02-20T07:53:02.349 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:53:02.347+0000 7fe6af7fe700 1 --2- 172.21.15.183:0/4006557456 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7fe6980692b0 0x7fe69806b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:53:02.349 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:53:02.347+0000 7fe6af7fe700 1 --2- 172.21.15.183:0/4006557456 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7fe6980692b0 0x7fe69806b760 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fe6a4007fd0 tx=0x7fe6a4009f60 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:53:02.352 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:53:02.351+0000 7fe6acff9700 1 -- 172.21.15.183:0/4006557456 <== mon.1 v2:172.21.15.183:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7fe6a0058460 con 0x7fe6b00663c0 2024-02-20T07:53:02.542 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:53:02.541+0000 7fe6b6344700 1 -- 172.21.15.183:0/4006557456 --> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi183:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) v1 -- 0x7fe6b010f100 con 0x7fe6980692b0 2024-02-20T07:53:02.797 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:53:02 smithi183 ceph-mon[35832]: pgmap v96: 1 pgs: 1 peering; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-02-20T07:53:02.797 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:53:02 smithi183 ceph-mon[35832]: osdmap e30: 4 total, 4 up, 4 in 2024-02-20T07:53:02.798 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:53:02 smithi183 ceph-mon[35832]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-20T07:53:02.798 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:53:02 smithi183 ceph-mon[35832]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-20T07:53:02.798 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:53:02 smithi183 ceph-mon[35832]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-20T07:53:02.992 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:53:02 smithi133 ceph-mon[29117]: pgmap v96: 1 pgs: 1 peering; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-02-20T07:53:02.992 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:53:02 smithi133 ceph-mon[29117]: osdmap e30: 4 total, 4 up, 4 in 2024-02-20T07:53:02.992 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:53:02 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-20T07:53:02.992 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:53:02 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-20T07:53:02.992 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:53:02 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-20T07:53:03.991 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:53:03 smithi133 ceph-mon[29117]: from='client.24153 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi183:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-02-20T07:53:04.025 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:53:03 smithi183 ceph-mon[35832]: from='client.24153 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi183:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-02-20T07:53:04.992 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:53:04 smithi133 ceph-mon[29117]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail; 136 KiB/s, 0 objects/s recovering 2024-02-20T07:53:05.047 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:53:04 smithi183 ceph-mon[35832]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail; 136 KiB/s, 0 objects/s recovering 2024-02-20T07:53:05.797 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:53:05 smithi183 ceph-mon[35832]: from='client.? 172.21.15.183:0/3962116825' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "55148881-b92f-40e6-bfe8-b8320406c96a"}]: dispatch 2024-02-20T07:53:05.797 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:53:05 smithi183 ceph-mon[35832]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "55148881-b92f-40e6-bfe8-b8320406c96a"}]: dispatch 2024-02-20T07:53:05.798 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:53:05 smithi183 ceph-mon[35832]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "55148881-b92f-40e6-bfe8-b8320406c96a"}]': finished 2024-02-20T07:53:05.798 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:53:05 smithi183 ceph-mon[35832]: osdmap e31: 5 total, 4 up, 5 in 2024-02-20T07:53:05.798 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:53:05 smithi183 ceph-mon[35832]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-20T07:53:05.992 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:53:05 smithi133 ceph-mon[29117]: from='client.? 172.21.15.183:0/3962116825' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "55148881-b92f-40e6-bfe8-b8320406c96a"}]: dispatch 2024-02-20T07:53:05.992 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:53:05 smithi133 ceph-mon[29117]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "55148881-b92f-40e6-bfe8-b8320406c96a"}]: dispatch 2024-02-20T07:53:05.992 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:53:05 smithi133 ceph-mon[29117]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "55148881-b92f-40e6-bfe8-b8320406c96a"}]': finished 2024-02-20T07:53:05.992 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:53:05 smithi133 ceph-mon[29117]: osdmap e31: 5 total, 4 up, 5 in 2024-02-20T07:53:05.992 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:53:05 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-20T07:53:06.992 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:53:06 smithi133 ceph-mon[29117]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail; 110 KiB/s, 0 objects/s recovering 2024-02-20T07:53:06.992 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:53:06 smithi133 ceph-mon[29117]: from='client.? 172.21.15.183:0/2778297322' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-20T07:53:07.047 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:53:06 smithi183 ceph-mon[35832]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail; 110 KiB/s, 0 objects/s recovering 2024-02-20T07:53:07.047 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:53:06 smithi183 ceph-mon[35832]: from='client.? 172.21.15.183:0/2778297322' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-20T07:53:08.985 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:53:08 smithi183 ceph-mon[35832]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail; 93 KiB/s, 0 objects/s recovering 2024-02-20T07:53:08.985 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:53:08 smithi183 ceph-mon[35832]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-20T07:53:08.992 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:53:08 smithi133 ceph-mon[29117]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail; 93 KiB/s, 0 objects/s recovering 2024-02-20T07:53:08.992 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:53:08 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-20T07:53:10.992 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:53:10 smithi133 ceph-mon[29117]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-02-20T07:53:11.022 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:53:10 smithi183 ceph-mon[35832]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-02-20T07:53:12.992 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:53:12 smithi133 ceph-mon[29117]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail; 63 KiB/s, 0 objects/s recovering 2024-02-20T07:53:12.992 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:53:12 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-02-20T07:53:12.992 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:53:12 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-20T07:53:13.007 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:53:12 smithi183 ceph-mon[35832]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail; 63 KiB/s, 0 objects/s recovering 2024-02-20T07:53:13.007 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:53:12 smithi183 ceph-mon[35832]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-02-20T07:53:13.007 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:53:12 smithi183 ceph-mon[35832]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-20T07:53:13.797 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:53:13 smithi183 ceph-mon[35832]: Deploying daemon osd.4 on smithi183 2024-02-20T07:53:13.992 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:53:13 smithi133 ceph-mon[29117]: Deploying daemon osd.4 on smithi183 2024-02-20T07:53:14.984 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:53:14 smithi183 ceph-mon[35832]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-02-20T07:53:14.991 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:53:14 smithi133 ceph-mon[29117]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-02-20T07:53:16.934 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:53:16 smithi183 ceph-mon[35832]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-02-20T07:53:16.992 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:53:16 smithi133 ceph-mon[29117]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-02-20T07:53:18.797 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:53:18 smithi183 ceph-mon[35832]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:53:18.797 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:53:18 smithi183 ceph-mon[35832]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:53:18.798 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:53:18 smithi183 ceph-mon[35832]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-02-20T07:53:18.798 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:53:18 smithi183 ceph-mon[35832]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-20T07:53:18.992 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:53:18 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:53:18.992 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:53:18 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:53:18.992 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:53:18 smithi133 ceph-mon[29117]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-02-20T07:53:18.992 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:53:18 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-20T07:53:20.706 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:53:20 smithi183 ceph-mon[35832]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-02-20T07:53:20.707 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:53:20 smithi183 ceph-mon[35832]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:53:20.707 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:53:20 smithi183 ceph-mon[35832]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:53:20.707 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:53:20 smithi183 ceph-mon[35832]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-20T07:53:20.707 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:53:20 smithi183 ceph-mon[35832]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-20T07:53:20.707 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:53:20 smithi183 ceph-mon[35832]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:53:20.992 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:53:20 smithi133 ceph-mon[29117]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-02-20T07:53:20.992 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:53:20 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:53:20.992 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:53:20 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:53:20.992 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:53:20 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-20T07:53:20.992 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:53:20 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-20T07:53:20.992 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:53:20 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:53:21.406 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:53:21.404+0000 7fe6acff9700 1 -- 172.21.15.183:0/4006557456 <== mgr.14208 v2:172.21.15.133:6800/2703586938 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7fe6b010f100 con 0x7fe6980692b0 2024-02-20T07:53:21.406 INFO:teuthology.orchestra.run.smithi183.stdout:Created osd(s) 4 on host 'smithi183' 2024-02-20T07:53:21.408 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:53:21.406+0000 7fe6b6344700 1 -- 172.21.15.183:0/4006557456 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7fe6980692b0 msgr2=0x7fe69806b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:53:21.408 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:53:21.406+0000 7fe6b6344700 1 --2- 172.21.15.183:0/4006557456 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7fe6980692b0 0x7fe69806b760 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fe6a4007fd0 tx=0x7fe6a4009f60 comp rx=0 tx=0).stop 2024-02-20T07:53:21.408 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:53:21.406+0000 7fe6b6344700 1 -- 172.21.15.183:0/4006557456 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe6b00663c0 msgr2=0x7fe6b010d910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:53:21.409 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:53:21.406+0000 7fe6b6344700 1 --2- 172.21.15.183:0/4006557456 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe6b00663c0 0x7fe6b010d910 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fe6a000b3b0 tx=0x7fe6a0009e30 comp rx=0 tx=0).stop 2024-02-20T07:53:21.409 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:53:21.406+0000 7fe6b6344700 1 -- 172.21.15.183:0/4006557456 shutdown_connections 2024-02-20T07:53:21.409 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:53:21.406+0000 7fe6b6344700 1 --2- 172.21.15.183:0/4006557456 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7fe6980692b0 0x7fe69806b760 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:53:21.409 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:53:21.406+0000 7fe6b6344700 1 --2- 172.21.15.183:0/4006557456 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe6b00663c0 0x7fe6b010d910 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:53:21.409 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:53:21.406+0000 7fe6b6344700 1 --2- 172.21.15.183:0/4006557456 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fe6b01000e0 0x7fe6b010de50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:53:21.409 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:53:21.406+0000 7fe6b6344700 1 -- 172.21.15.183:0/4006557456 >> 172.21.15.183:0/4006557456 conn(0x7fe6b00f74e0 msgr2=0x7fe6b00f9930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:53:21.409 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:53:21.406+0000 7fe6b6344700 1 -- 172.21.15.183:0/4006557456 shutdown_connections 2024-02-20T07:53:21.409 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:53:21.406+0000 7fe6b6344700 1 -- 172.21.15.183:0/4006557456 wait complete. 2024-02-20T07:53:22.035 DEBUG:teuthology.orchestra.run.smithi183:osd.4> sudo journalctl -f -n 0 -u ceph-4537327c-cfc4-11ee-95bb-87774f69a715@osd.4.service 2024-02-20T07:53:22.037 INFO:tasks.cephadm:Deploying osd.5 on smithi183 with /dev/vg_nvme/lv_2... 2024-02-20T07:53:22.037 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4537327c-cfc4-11ee-95bb-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-02-20T07:53:22.742 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:53:22 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:53:22.742 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:53:22 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-20T07:53:22.742 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:53:22 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:53:22.742 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:53:22 smithi133 ceph-mon[29117]: from='osd.4 [v2:172.21.15.183:6808/3697465174,v1:172.21.15.183:6809/3697465174]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-02-20T07:53:22.742 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:53:22 smithi133 ceph-mon[29117]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-02-20T07:53:22.742 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:53:22 smithi133 ceph-mon[29117]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-02-20T07:53:22.747 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/4537327c-cfc4-11ee-95bb-87774f69a715/mon.smithi183/config 2024-02-20T07:53:22.797 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:53:22 smithi183 ceph-mon[35832]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:53:22.797 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:53:22 smithi183 ceph-mon[35832]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-20T07:53:22.798 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:53:22 smithi183 ceph-mon[35832]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:53:22.798 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:53:22 smithi183 ceph-mon[35832]: from='osd.4 [v2:172.21.15.183:6808/3697465174,v1:172.21.15.183:6809/3697465174]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-02-20T07:53:22.798 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:53:22 smithi183 ceph-mon[35832]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-02-20T07:53:22.798 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:53:22 smithi183 ceph-mon[35832]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-02-20T07:53:23.547 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:53:23 smithi183 ceph-mon[35832]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-02-20T07:53:23.548 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:53:23 smithi183 ceph-mon[35832]: osdmap e32: 5 total, 4 up, 5 in 2024-02-20T07:53:23.548 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:53:23 smithi183 ceph-mon[35832]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-20T07:53:23.548 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:53:23 smithi183 ceph-mon[35832]: from='osd.4 [v2:172.21.15.183:6808/3697465174,v1:172.21.15.183:6809/3697465174]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi183", "root=default"]}]: dispatch 2024-02-20T07:53:23.548 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:53:23 smithi183 ceph-mon[35832]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi183", "root=default"]}]: dispatch 2024-02-20T07:53:23.548 INFO:journalctl@ceph.osd.4.smithi183.stdout:Feb 20 07:53:23 smithi183 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-4[47858]: 2024-02-20T07:53:23.377+0000 7f515a724700 -1 osd.4 0 waiting for initial osdmap 2024-02-20T07:53:23.548 INFO:journalctl@ceph.osd.4.smithi183.stdout:Feb 20 07:53:23 smithi183 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-4[47858]: 2024-02-20T07:53:23.383+0000 7f5151d18700 -1 osd.4 33 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-20T07:53:23.742 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:53:23 smithi133 ceph-mon[29117]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-02-20T07:53:23.742 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:53:23 smithi133 ceph-mon[29117]: osdmap e32: 5 total, 4 up, 5 in 2024-02-20T07:53:23.742 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:53:23 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-20T07:53:23.742 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:53:23 smithi133 ceph-mon[29117]: from='osd.4 [v2:172.21.15.183:6808/3697465174,v1:172.21.15.183:6809/3697465174]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi183", "root=default"]}]: dispatch 2024-02-20T07:53:23.742 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:53:23 smithi133 ceph-mon[29117]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi183", "root=default"]}]: dispatch 2024-02-20T07:53:24.547 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:53:24 smithi183 ceph-mon[35832]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi183", "root=default"]}]': finished 2024-02-20T07:53:24.547 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:53:24 smithi183 ceph-mon[35832]: osdmap e33: 5 total, 4 up, 5 in 2024-02-20T07:53:24.547 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:53:24 smithi183 ceph-mon[35832]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-20T07:53:24.548 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:53:24 smithi183 ceph-mon[35832]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-20T07:53:24.548 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:53:24 smithi183 ceph-mon[35832]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-02-20T07:53:24.548 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:53:24 smithi183 ceph-mon[35832]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-20T07:53:24.742 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:53:24 smithi133 ceph-mon[29117]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi183", "root=default"]}]': finished 2024-02-20T07:53:24.742 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:53:24 smithi133 ceph-mon[29117]: osdmap e33: 5 total, 4 up, 5 in 2024-02-20T07:53:24.742 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:53:24 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-20T07:53:24.742 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:53:24 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-20T07:53:24.742 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:53:24 smithi133 ceph-mon[29117]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-02-20T07:53:24.742 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:53:24 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-20T07:53:25.548 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:53:25 smithi183 ceph-mon[35832]: purged_snaps scrub starts 2024-02-20T07:53:25.548 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:53:25 smithi183 ceph-mon[35832]: purged_snaps scrub ok 2024-02-20T07:53:25.548 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:53:25 smithi183 ceph-mon[35832]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-20T07:53:25.548 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:53:25 smithi183 ceph-mon[35832]: osd.4 [v2:172.21.15.183:6808/3697465174,v1:172.21.15.183:6809/3697465174] boot 2024-02-20T07:53:25.548 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:53:25 smithi183 ceph-mon[35832]: osdmap e34: 5 total, 5 up, 5 in 2024-02-20T07:53:25.548 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:53:25 smithi183 ceph-mon[35832]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-20T07:53:25.598 INFO:teuthology.orchestra.run.smithi183.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-02-20T07:53:25.599 INFO:teuthology.orchestra.run.smithi183.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-02-20T07:53:25.599 INFO:teuthology.orchestra.run.smithi183.stderr: stderr: 10+0 records in 2024-02-20T07:53:25.599 INFO:teuthology.orchestra.run.smithi183.stderr:10+0 records out 2024-02-20T07:53:25.599 INFO:teuthology.orchestra.run.smithi183.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0123281 s, 851 MB/s 2024-02-20T07:53:25.599 INFO:teuthology.orchestra.run.smithi183.stderr:--> Zapping successful for: 2024-02-20T07:53:25.742 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:53:25 smithi133 ceph-mon[29117]: purged_snaps scrub starts 2024-02-20T07:53:25.742 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:53:25 smithi133 ceph-mon[29117]: purged_snaps scrub ok 2024-02-20T07:53:25.742 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:53:25 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-20T07:53:25.742 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:53:25 smithi133 ceph-mon[29117]: osd.4 [v2:172.21.15.183:6808/3697465174,v1:172.21.15.183:6809/3697465174] boot 2024-02-20T07:53:25.742 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:53:25 smithi133 ceph-mon[29117]: osdmap e34: 5 total, 5 up, 5 in 2024-02-20T07:53:25.742 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:53:25 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-20T07:53:26.383 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4537327c-cfc4-11ee-95bb-87774f69a715 -- ceph orch daemon add osd smithi183:vg_nvme/lv_2 2024-02-20T07:53:26.633 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/4537327c-cfc4-11ee-95bb-87774f69a715/mon.smithi183/config 2024-02-20T07:53:27.048 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:53:26 smithi183 ceph-mon[35832]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-20T07:53:27.048 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:53:26 smithi183 ceph-mon[35832]: Detected new or changed devices on smithi183 2024-02-20T07:53:27.048 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:53:26 smithi183 ceph-mon[35832]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:53:27.048 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:53:26 smithi183 ceph-mon[35832]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:53:27.048 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:53:26 smithi183 ceph-mon[35832]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-02-20T07:53:27.048 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:53:26 smithi183 ceph-mon[35832]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-20T07:53:27.048 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:53:26 smithi183 ceph-mon[35832]: Adjusting osd_memory_target on smithi183 to 7968M 2024-02-20T07:53:27.048 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:53:26 smithi183 ceph-mon[35832]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:53:27.048 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:53:26 smithi183 ceph-mon[35832]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-20T07:53:27.048 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:53:26 smithi183 ceph-mon[35832]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-20T07:53:27.049 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:53:26 smithi183 ceph-mon[35832]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:53:27.049 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:53:26 smithi183 ceph-mon[35832]: osdmap e35: 5 total, 5 up, 5 in 2024-02-20T07:53:27.242 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:53:26 smithi133 ceph-mon[29117]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-20T07:53:27.242 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:53:26 smithi133 ceph-mon[29117]: Detected new or changed devices on smithi183 2024-02-20T07:53:27.242 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:53:26 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:53:27.242 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:53:26 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:53:27.242 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:53:26 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-02-20T07:53:27.242 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:53:26 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-20T07:53:27.242 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:53:26 smithi133 ceph-mon[29117]: Adjusting osd_memory_target on smithi183 to 7968M 2024-02-20T07:53:27.242 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:53:26 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:53:27.243 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:53:26 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-20T07:53:27.243 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:53:26 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-20T07:53:27.243 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:53:26 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:53:27.243 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:53:26 smithi133 ceph-mon[29117]: osdmap e35: 5 total, 5 up, 5 in 2024-02-20T07:53:28.073 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:53:28.072+0000 7f3151695700 1 -- 172.21.15.183:0/4034952656 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f314c0fda80 msgr2=0x7f314c0ffe60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:53:28.073 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:53:28.072+0000 7f3151695700 1 --2- 172.21.15.183:0/4034952656 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f314c0fda80 0x7f314c0ffe60 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f313c009ce0 tx=0x7f313c004d10 comp rx=0 tx=0).stop 2024-02-20T07:53:28.074 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:53:28.072+0000 7f3151695700 1 -- 172.21.15.183:0/4034952656 shutdown_connections 2024-02-20T07:53:28.074 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:53:28.072+0000 7f3151695700 1 --2- 172.21.15.183:0/4034952656 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f314c1003a0 0x7f314c102780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:53:28.074 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:53:28.072+0000 7f3151695700 1 --2- 172.21.15.183:0/4034952656 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f314c0fda80 0x7f314c0ffe60 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:53:28.074 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:53:28.072+0000 7f3151695700 1 -- 172.21.15.183:0/4034952656 >> 172.21.15.183:0/4034952656 conn(0x7f314c0f74e0 msgr2=0x7f314c0f9930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:53:28.074 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:53:28.072+0000 7f3151695700 1 -- 172.21.15.183:0/4034952656 shutdown_connections 2024-02-20T07:53:28.074 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:53:28.072+0000 7f3151695700 1 -- 172.21.15.183:0/4034952656 wait complete. 2024-02-20T07:53:28.078 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:53:28.073+0000 7f3151695700 1 Processor -- start 2024-02-20T07:53:28.078 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:53:28.073+0000 7f3151695700 1 -- start start 2024-02-20T07:53:28.078 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:53:28.073+0000 7f3151695700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f314c0fda80 0x7f314c10dad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:53:28.078 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:53:28.073+0000 7f3151695700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f314c1003a0 0x7f314c10e010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:53:28.078 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:53:28.073+0000 7f3151695700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f314c10e610 con 0x7f314c0fda80 2024-02-20T07:53:28.078 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:53:28.073+0000 7f3151695700 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f314c10eeb0 con 0x7f314c1003a0 2024-02-20T07:53:28.078 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:53:28.073+0000 7f314a7fc700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f314c1003a0 0x7f314c10e010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:53:28.078 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:53:28.073+0000 7f314a7fc700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f314c1003a0 0x7f314c10e010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.183:54448/0 (socket says 172.21.15.183:54448) 2024-02-20T07:53:28.078 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:53:28.073+0000 7f314a7fc700 1 -- 172.21.15.183:0/3031751078 learned_addr learned my addr 172.21.15.183:0/3031751078 (peer_addr_for_me v2:172.21.15.183:0/0) 2024-02-20T07:53:28.078 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:53:28.073+0000 7f314affd700 1 --2- 172.21.15.183:0/3031751078 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f314c0fda80 0x7f314c10dad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:53:28.078 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:53:28.073+0000 7f314a7fc700 1 -- 172.21.15.183:0/3031751078 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f314c0fda80 msgr2=0x7f314c10dad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:53:28.078 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:53:28.073+0000 7f314a7fc700 1 --2- 172.21.15.183:0/3031751078 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f314c0fda80 0x7f314c10dad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:53:28.078 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:53:28.073+0000 7f314a7fc700 1 -- 172.21.15.183:0/3031751078 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f313c009940 con 0x7f314c1003a0 2024-02-20T07:53:28.079 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:53:28.073+0000 7f314affd700 1 --2- 172.21.15.183:0/3031751078 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f314c0fda80 0x7f314c10dad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-20T07:53:28.079 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:53:28.074+0000 7f314a7fc700 1 --2- 172.21.15.183:0/3031751078 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f314c1003a0 0x7f314c10e010 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f3134002a50 tx=0x7f31340046b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:53:28.079 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:53:28.074+0000 7f31437fe700 1 -- 172.21.15.183:0/3031751078 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3134007cc0 con 0x7f314c1003a0 2024-02-20T07:53:28.079 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:53:28.074+0000 7f3151695700 1 -- 172.21.15.183:0/3031751078 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f314c10f190 con 0x7f314c1003a0 2024-02-20T07:53:28.079 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:53:28.074+0000 7f31437fe700 1 -- 172.21.15.183:0/3031751078 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f313400f040 con 0x7f314c1003a0 2024-02-20T07:53:28.079 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:53:28.074+0000 7f3151695700 1 -- 172.21.15.183:0/3031751078 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f314c10f6d0 con 0x7f314c1003a0 2024-02-20T07:53:28.079 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:53:28.074+0000 7f31437fe700 1 -- 172.21.15.183:0/3031751078 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f313400db70 con 0x7f314c1003a0 2024-02-20T07:53:28.079 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:53:28.075+0000 7f3151695700 1 -- 172.21.15.183:0/3031751078 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f312c0052e0 con 0x7f314c1003a0 2024-02-20T07:53:28.079 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:53:28.075+0000 7f31437fe700 1 -- 172.21.15.183:0/3031751078 <== mon.1 v2:172.21.15.183:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7f313401c400 con 0x7f314c1003a0 2024-02-20T07:53:28.079 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:53:28.075+0000 7f31437fe700 1 --2- 172.21.15.183:0/3031751078 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7f31380692a0 0x7f313806b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:53:28.079 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:53:28.076+0000 7f31437fe700 1 -- 172.21.15.183:0/3031751078 <== mon.1 v2:172.21.15.183:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 4202+0+0 (secure 0 0 0) 0x7f3134086320 con 0x7f314c1003a0 2024-02-20T07:53:28.079 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:53:28.076+0000 7f314affd700 1 --2- 172.21.15.183:0/3031751078 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7f31380692a0 0x7f313806b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:53:28.079 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:53:28.076+0000 7f314affd700 1 --2- 172.21.15.183:0/3031751078 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7f31380692a0 0x7f313806b750 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f313c00b740 tx=0x7f313c004d10 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:53:28.080 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:53:28.080+0000 7f31437fe700 1 -- 172.21.15.183:0/3031751078 <== mon.1 v2:172.21.15.183:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f3134056550 con 0x7f314c1003a0 2024-02-20T07:53:28.268 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:53:28.267+0000 7f3151695700 1 -- 172.21.15.183:0/3031751078 --> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi183:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) v1 -- 0x7f312c000bf0 con 0x7f31380692a0 2024-02-20T07:53:28.967 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:53:28 smithi183 ceph-mon[35832]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-20T07:53:28.967 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:53:28 smithi183 ceph-mon[35832]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-20T07:53:28.968 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:53:28 smithi183 ceph-mon[35832]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-20T07:53:28.968 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:53:28 smithi183 ceph-mon[35832]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-20T07:53:29.242 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:53:28 smithi133 ceph-mon[29117]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-20T07:53:29.242 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:53:28 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-20T07:53:29.242 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:53:28 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-20T07:53:29.242 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:53:28 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-20T07:53:30.241 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:53:29 smithi133 ceph-mon[29117]: from='client.24171 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi183:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-02-20T07:53:30.297 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:53:29 smithi183 ceph-mon[35832]: from='client.24171 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi183:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-02-20T07:53:31.242 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:53:30 smithi133 ceph-mon[29117]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-20T07:53:31.242 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:53:30 smithi133 ceph-mon[29117]: from='client.? 172.21.15.183:0/3035853633' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ee5510f9-fab7-4aa3-8372-3856a0787e92"}]: dispatch 2024-02-20T07:53:31.242 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:53:30 smithi133 ceph-mon[29117]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ee5510f9-fab7-4aa3-8372-3856a0787e92"}]: dispatch 2024-02-20T07:53:31.242 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:53:30 smithi133 ceph-mon[29117]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ee5510f9-fab7-4aa3-8372-3856a0787e92"}]': finished 2024-02-20T07:53:31.242 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:53:30 smithi133 ceph-mon[29117]: osdmap e36: 6 total, 5 up, 6 in 2024-02-20T07:53:31.242 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:53:30 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-20T07:53:31.297 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:53:30 smithi183 ceph-mon[35832]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-20T07:53:31.298 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:53:30 smithi183 ceph-mon[35832]: from='client.? 172.21.15.183:0/3035853633' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ee5510f9-fab7-4aa3-8372-3856a0787e92"}]: dispatch 2024-02-20T07:53:31.298 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:53:30 smithi183 ceph-mon[35832]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ee5510f9-fab7-4aa3-8372-3856a0787e92"}]: dispatch 2024-02-20T07:53:31.298 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:53:30 smithi183 ceph-mon[35832]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ee5510f9-fab7-4aa3-8372-3856a0787e92"}]': finished 2024-02-20T07:53:31.298 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:53:30 smithi183 ceph-mon[35832]: osdmap e36: 6 total, 5 up, 6 in 2024-02-20T07:53:31.298 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:53:30 smithi183 ceph-mon[35832]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-20T07:53:32.241 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:53:31 smithi133 ceph-mon[29117]: from='client.? 172.21.15.183:0/383596735' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-20T07:53:32.297 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:53:31 smithi183 ceph-mon[35832]: from='client.? 172.21.15.183:0/383596735' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-20T07:53:33.242 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:53:32 smithi133 ceph-mon[29117]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-20T07:53:33.297 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:53:32 smithi183 ceph-mon[35832]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-20T07:53:35.173 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:53:34 smithi183 ceph-mon[35832]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-20T07:53:35.241 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:53:34 smithi133 ceph-mon[29117]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-20T07:53:37.233 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:53:36 smithi183 ceph-mon[35832]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-20T07:53:37.241 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:53:36 smithi133 ceph-mon[29117]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-20T07:53:39.181 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:53:38 smithi183 ceph-mon[35832]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-20T07:53:39.181 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:53:38 smithi183 ceph-mon[35832]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-02-20T07:53:39.181 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:53:38 smithi183 ceph-mon[35832]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-20T07:53:39.181 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:53:38 smithi183 ceph-mon[35832]: Deploying daemon osd.5 on smithi183 2024-02-20T07:53:39.182 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:53:38 smithi183 ceph-mon[35832]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-20T07:53:39.242 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:53:38 smithi133 ceph-mon[29117]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-20T07:53:39.242 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:53:38 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-02-20T07:53:39.242 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:53:38 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-20T07:53:39.242 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:53:38 smithi133 ceph-mon[29117]: Deploying daemon osd.5 on smithi183 2024-02-20T07:53:39.242 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:53:38 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-20T07:53:41.047 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:53:40 smithi183 ceph-mon[35832]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-20T07:53:41.242 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:53:40 smithi133 ceph-mon[29117]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-20T07:53:43.047 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:53:42 smithi183 ceph-mon[35832]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-20T07:53:43.047 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:53:42 smithi183 ceph-mon[35832]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:53:43.047 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:53:42 smithi183 ceph-mon[35832]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:53:43.242 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:53:42 smithi133 ceph-mon[29117]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-20T07:53:43.242 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:53:42 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:53:43.242 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:53:42 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:53:44.163 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:53:43 smithi183 ceph-mon[35832]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-20T07:53:44.241 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:53:43 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-20T07:53:45.173 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:53:44 smithi183 ceph-mon[35832]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-20T07:53:45.242 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:53:44 smithi133 ceph-mon[29117]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-20T07:53:45.726 INFO:teuthology.orchestra.run.smithi183.stdout:Created osd(s) 5 on host 'smithi183' 2024-02-20T07:53:45.727 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:53:45.725+0000 7f31437fe700 1 -- 172.21.15.183:0/3031751078 <== mgr.14208 v2:172.21.15.133:6800/2703586938 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f312c000bf0 con 0x7f31380692a0 2024-02-20T07:53:45.728 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:53:45.727+0000 7f3151695700 1 -- 172.21.15.183:0/3031751078 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7f31380692a0 msgr2=0x7f313806b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:53:45.728 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:53:45.727+0000 7f3151695700 1 --2- 172.21.15.183:0/3031751078 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7f31380692a0 0x7f313806b750 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f313c00b740 tx=0x7f313c004d10 comp rx=0 tx=0).stop 2024-02-20T07:53:45.728 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:53:45.727+0000 7f3151695700 1 -- 172.21.15.183:0/3031751078 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f314c1003a0 msgr2=0x7f314c10e010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:53:45.729 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:53:45.727+0000 7f3151695700 1 --2- 172.21.15.183:0/3031751078 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f314c1003a0 0x7f314c10e010 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f3134002a50 tx=0x7f31340046b0 comp rx=0 tx=0).stop 2024-02-20T07:53:45.729 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:53:45.727+0000 7f3151695700 1 -- 172.21.15.183:0/3031751078 shutdown_connections 2024-02-20T07:53:45.729 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:53:45.727+0000 7f3151695700 1 --2- 172.21.15.183:0/3031751078 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f314c0fda80 0x7f314c10dad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:53:45.729 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:53:45.727+0000 7f3151695700 1 --2- 172.21.15.183:0/3031751078 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7f31380692a0 0x7f313806b750 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:53:45.729 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:53:45.727+0000 7f3151695700 1 --2- 172.21.15.183:0/3031751078 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f314c1003a0 0x7f314c10e010 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:53:45.729 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:53:45.727+0000 7f3151695700 1 -- 172.21.15.183:0/3031751078 >> 172.21.15.183:0/3031751078 conn(0x7f314c0f74e0 msgr2=0x7f314c0f9930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:53:45.729 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:53:45.727+0000 7f3151695700 1 -- 172.21.15.183:0/3031751078 shutdown_connections 2024-02-20T07:53:45.729 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:53:45.727+0000 7f3151695700 1 -- 172.21.15.183:0/3031751078 wait complete. 2024-02-20T07:53:46.560 DEBUG:teuthology.orchestra.run.smithi183:osd.5> sudo journalctl -f -n 0 -u ceph-4537327c-cfc4-11ee-95bb-87774f69a715@osd.5.service 2024-02-20T07:53:46.562 INFO:tasks.cephadm:Waiting for 6 OSDs to come up... 2024-02-20T07:53:46.563 DEBUG:teuthology.orchestra.run.smithi133:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4537327c-cfc4-11ee-95bb-87774f69a715 -- ceph osd stat -f json 2024-02-20T07:53:46.797 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:53:46 smithi183 ceph-mon[35832]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-20T07:53:46.797 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:53:46 smithi183 ceph-mon[35832]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:53:46.797 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:53:46 smithi183 ceph-mon[35832]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:53:46.798 INFO:journalctl@ceph.osd.5.smithi183.stdout:Feb 20 07:53:46 smithi183 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-5[54334]: 2024-02-20T07:53:46.721+0000 7fee78383540 -1 osd.5 0 log_to_monitors true 2024-02-20T07:53:46.828 INFO:teuthology.orchestra.run.smithi133.stderr:Inferring config /var/lib/ceph/4537327c-cfc4-11ee-95bb-87774f69a715/mon.smithi133/config 2024-02-20T07:53:46.950 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:53:46 smithi133 ceph-mon[29117]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-20T07:53:46.950 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:53:46 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:53:46.950 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:53:46 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:53:47.977 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:53:47 smithi133 ceph-mon[29117]: from='osd.5 [v2:172.21.15.183:6816/81997900,v1:172.21.15.183:6817/81997900]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-02-20T07:53:47.977 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:53:47 smithi133 ceph-mon[29117]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-02-20T07:53:47.977 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:53:47 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:53:47.977 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:53:47 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:53:48.047 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:53:47 smithi183 ceph-mon[35832]: from='osd.5 [v2:172.21.15.183:6816/81997900,v1:172.21.15.183:6817/81997900]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-02-20T07:53:48.047 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:53:47 smithi183 ceph-mon[35832]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-02-20T07:53:48.047 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:53:47 smithi183 ceph-mon[35832]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:53:48.047 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:53:47 smithi183 ceph-mon[35832]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:53:48.566 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:53:48.562+0000 7fd5cce0d700 1 -- 172.21.15.133:0/2131882635 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fd5c80fcfb0 msgr2=0x7fd5c80fd420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:53:48.567 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:53:48.562+0000 7fd5cce0d700 1 --2- 172.21.15.133:0/2131882635 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fd5c80fcfb0 0x7fd5c80fd420 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fd5b8004940 tx=0x7fd5b8009e30 comp rx=0 tx=0).stop 2024-02-20T07:53:48.567 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:53:48.562+0000 7fd5cce0d700 1 -- 172.21.15.133:0/2131882635 shutdown_connections 2024-02-20T07:53:48.567 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:53:48.562+0000 7fd5cce0d700 1 --2- 172.21.15.133:0/2131882635 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fd5c80fcfb0 0x7fd5c80fd420 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:53:48.567 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:53:48.562+0000 7fd5cce0d700 1 --2- 172.21.15.133:0/2131882635 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd5c80fbe40 0x7fd5c80fc250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:53:48.567 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:53:48.562+0000 7fd5cce0d700 1 -- 172.21.15.133:0/2131882635 >> 172.21.15.133:0/2131882635 conn(0x7fd5c80f74e0 msgr2=0x7fd5c80f9930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:53:48.567 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:53:48.562+0000 7fd5cce0d700 1 -- 172.21.15.133:0/2131882635 shutdown_connections 2024-02-20T07:53:48.567 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:53:48.562+0000 7fd5cce0d700 1 -- 172.21.15.133:0/2131882635 wait complete. 2024-02-20T07:53:48.567 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:53:48.563+0000 7fd5cce0d700 1 Processor -- start 2024-02-20T07:53:48.567 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:53:48.563+0000 7fd5cce0d700 1 -- start start 2024-02-20T07:53:48.567 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:53:48.563+0000 7fd5cce0d700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd5c80fbe40 0x7fd5c8110670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:53:48.567 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:53:48.563+0000 7fd5cce0d700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fd5c80fcfb0 0x7fd5c8112bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:53:48.568 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:53:48.563+0000 7fd5cce0d700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5c80664f0 con 0x7fd5c80fcfb0 2024-02-20T07:53:48.568 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:53:48.563+0000 7fd5cce0d700 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5c8066630 con 0x7fd5c80fbe40 2024-02-20T07:53:48.568 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:53:48.563+0000 7fd5c5d9b700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fd5c80fcfb0 0x7fd5c8112bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:53:48.568 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:53:48.563+0000 7fd5c5d9b700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fd5c80fcfb0 0x7fd5c8112bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.133:56830/0 (socket says 172.21.15.133:56830) 2024-02-20T07:53:48.568 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:53:48.563+0000 7fd5c5d9b700 1 -- 172.21.15.133:0/1551979798 learned_addr learned my addr 172.21.15.133:0/1551979798 (peer_addr_for_me v2:172.21.15.133:0/0) 2024-02-20T07:53:48.568 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:53:48.563+0000 7fd5c659c700 1 --2- 172.21.15.133:0/1551979798 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd5c80fbe40 0x7fd5c8110670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:53:48.568 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:53:48.563+0000 7fd5c5d9b700 1 -- 172.21.15.133:0/1551979798 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd5c80fbe40 msgr2=0x7fd5c8110670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:53:48.568 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:53:48.563+0000 7fd5c5d9b700 1 --2- 172.21.15.133:0/1551979798 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd5c80fbe40 0x7fd5c8110670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:53:48.568 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:53:48.563+0000 7fd5c5d9b700 1 -- 172.21.15.133:0/1551979798 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5b8005040 con 0x7fd5c80fcfb0 2024-02-20T07:53:48.568 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:53:48.563+0000 7fd5c5d9b700 1 --2- 172.21.15.133:0/1551979798 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fd5c80fcfb0 0x7fd5c8112bc0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fd5b80093b0 tx=0x7fd5b8009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:53:48.568 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:53:48.563+0000 7fd5b6ffd700 1 -- 172.21.15.133:0/1551979798 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd5b800e040 con 0x7fd5c80fcfb0 2024-02-20T07:53:48.568 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:53:48.563+0000 7fd5b6ffd700 1 -- 172.21.15.133:0/1551979798 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd5b800bcc0 con 0x7fd5c80fcfb0 2024-02-20T07:53:48.569 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:53:48.563+0000 7fd5b6ffd700 1 -- 172.21.15.133:0/1551979798 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd5b801c630 con 0x7fd5c80fcfb0 2024-02-20T07:53:48.569 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:53:48.563+0000 7fd5cce0d700 1 -- 172.21.15.133:0/1551979798 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd5c81131f0 con 0x7fd5c80fcfb0 2024-02-20T07:53:48.569 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:53:48.564+0000 7fd5cce0d700 1 -- 172.21.15.133:0/1551979798 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd5c81136d0 con 0x7fd5c80fcfb0 2024-02-20T07:53:48.569 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:53:48.565+0000 7fd5cce0d700 1 -- 172.21.15.133:0/1551979798 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd5c804bed0 con 0x7fd5c80fcfb0 2024-02-20T07:53:48.569 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:53:48.565+0000 7fd5b6ffd700 1 -- 172.21.15.133:0/1551979798 <== mon.0 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7fd5b8013980 con 0x7fd5c80fcfb0 2024-02-20T07:53:48.569 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:53:48.565+0000 7fd5b6ffd700 1 --2- 172.21.15.133:0/1551979798 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7fd5b00692a0 0x7fd5b006b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:53:48.570 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:53:48.565+0000 7fd5b6ffd700 1 -- 172.21.15.133:0/1551979798 <== mon.0 v2:172.21.15.133:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 4338+0+0 (secure 0 0 0) 0x7fd5b8088070 con 0x7fd5c80fcfb0 2024-02-20T07:53:48.570 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:53:48.565+0000 7fd5c659c700 1 --2- 172.21.15.133:0/1551979798 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7fd5b00692a0 0x7fd5b006b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:53:48.570 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:53:48.566+0000 7fd5c659c700 1 --2- 172.21.15.133:0/1551979798 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7fd5b00692a0 0x7fd5b006b750 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fd5c80660e0 tx=0x7fd5bc0091a0 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:53:48.571 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:53:48.570+0000 7fd5b6ffd700 1 -- 172.21.15.133:0/1551979798 <== mon.0 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7fd5b80548a0 con 0x7fd5c80fcfb0 2024-02-20T07:53:48.760 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:53:48.758+0000 7fd5cce0d700 1 -- 172.21.15.133:0/1551979798 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) v1 -- 0x7fd5c81049b0 con 0x7fd5c80fcfb0 2024-02-20T07:53:48.760 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:53:48.758+0000 7fd5b6ffd700 1 -- 172.21.15.133:0/1551979798 <== mon.0 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v38) v1 ==== 74+0+130 (secure 0 0 0) 0x7fd5b8054020 con 0x7fd5c80fcfb0 2024-02-20T07:53:48.760 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-02-20T07:53:48.762 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:53:48.761+0000 7fd5cce0d700 1 -- 172.21.15.133:0/1551979798 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7fd5b00692a0 msgr2=0x7fd5b006b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:53:48.763 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:53:48.761+0000 7fd5cce0d700 1 --2- 172.21.15.133:0/1551979798 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7fd5b00692a0 0x7fd5b006b750 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fd5c80660e0 tx=0x7fd5bc0091a0 comp rx=0 tx=0).stop 2024-02-20T07:53:48.763 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:53:48.761+0000 7fd5cce0d700 1 -- 172.21.15.133:0/1551979798 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fd5c80fcfb0 msgr2=0x7fd5c8112bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:53:48.763 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:53:48.761+0000 7fd5cce0d700 1 --2- 172.21.15.133:0/1551979798 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fd5c80fcfb0 0x7fd5c8112bc0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fd5b80093b0 tx=0x7fd5b8009e30 comp rx=0 tx=0).stop 2024-02-20T07:53:48.763 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:53:48.761+0000 7fd5cce0d700 1 -- 172.21.15.133:0/1551979798 shutdown_connections 2024-02-20T07:53:48.763 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:53:48.761+0000 7fd5cce0d700 1 --2- 172.21.15.133:0/1551979798 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7fd5b00692a0 0x7fd5b006b750 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:53:48.763 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:53:48.761+0000 7fd5cce0d700 1 --2- 172.21.15.133:0/1551979798 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd5c80fbe40 0x7fd5c8110670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:53:48.763 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:53:48.761+0000 7fd5cce0d700 1 --2- 172.21.15.133:0/1551979798 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fd5c80fcfb0 0x7fd5c8112bc0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:53:48.763 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:53:48.761+0000 7fd5cce0d700 1 -- 172.21.15.133:0/1551979798 >> 172.21.15.133:0/1551979798 conn(0x7fd5c80f74e0 msgr2=0x7fd5c8100270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:53:48.763 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:53:48.761+0000 7fd5cce0d700 1 -- 172.21.15.133:0/1551979798 shutdown_connections 2024-02-20T07:53:48.763 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:53:48.761+0000 7fd5cce0d700 1 -- 172.21.15.133:0/1551979798 wait complete. 2024-02-20T07:53:48.797 INFO:journalctl@ceph.osd.5.smithi183.stdout:Feb 20 07:53:48 smithi183 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-5[54334]: 2024-02-20T07:53:48.688+0000 7fee6eb55700 -1 osd.5 0 waiting for initial osdmap 2024-02-20T07:53:48.797 INFO:journalctl@ceph.osd.5.smithi183.stdout:Feb 20 07:53:48 smithi183 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-5[54334]: 2024-02-20T07:53:48.696+0000 7fee6814d700 -1 osd.5 38 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-20T07:53:48.797 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:53:48 smithi183 ceph-mon[35832]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-20T07:53:48.798 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:53:48 smithi183 ceph-mon[35832]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-02-20T07:53:48.798 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:53:48 smithi183 ceph-mon[35832]: osdmap e37: 6 total, 5 up, 6 in 2024-02-20T07:53:48.798 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:53:48 smithi183 ceph-mon[35832]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-20T07:53:48.798 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:53:48 smithi183 ceph-mon[35832]: from='osd.5 [v2:172.21.15.183:6816/81997900,v1:172.21.15.183:6817/81997900]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi183", "root=default"]}]: dispatch 2024-02-20T07:53:48.798 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:53:48 smithi183 ceph-mon[35832]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi183", "root=default"]}]: dispatch 2024-02-20T07:53:48.991 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:53:48 smithi133 ceph-mon[29117]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-20T07:53:48.992 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:53:48 smithi133 ceph-mon[29117]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-02-20T07:53:48.992 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:53:48 smithi133 ceph-mon[29117]: osdmap e37: 6 total, 5 up, 6 in 2024-02-20T07:53:48.992 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:53:48 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-20T07:53:48.992 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:53:48 smithi133 ceph-mon[29117]: from='osd.5 [v2:172.21.15.183:6816/81997900,v1:172.21.15.183:6817/81997900]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi183", "root=default"]}]: dispatch 2024-02-20T07:53:48.992 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:53:48 smithi133 ceph-mon[29117]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi183", "root=default"]}]: dispatch 2024-02-20T07:53:49.402 INFO:teuthology.orchestra.run.smithi133.stdout:{"epoch":38,"num_osds":6,"num_up_osds":5,"osd_up_since":1708415604,"num_in_osds":6,"osd_in_since":1708415610,"num_remapped_pgs":0} 2024-02-20T07:53:49.992 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:53:49 smithi133 ceph-mon[29117]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi183", "root=default"]}]': finished 2024-02-20T07:53:49.992 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:53:49 smithi133 ceph-mon[29117]: osdmap e38: 6 total, 5 up, 6 in 2024-02-20T07:53:49.992 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:53:49 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-20T07:53:49.992 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:53:49 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-20T07:53:49.992 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:53:49 smithi133 ceph-mon[29117]: from='client.? 172.21.15.133:0/1551979798' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-02-20T07:53:50.047 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:53:49 smithi183 ceph-mon[35832]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi183", "root=default"]}]': finished 2024-02-20T07:53:50.048 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:53:49 smithi183 ceph-mon[35832]: osdmap e38: 6 total, 5 up, 6 in 2024-02-20T07:53:50.048 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:53:49 smithi183 ceph-mon[35832]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-20T07:53:50.048 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:53:49 smithi183 ceph-mon[35832]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-20T07:53:50.048 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:53:49 smithi183 ceph-mon[35832]: from='client.? 172.21.15.133:0/1551979798' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-02-20T07:53:50.403 DEBUG:teuthology.orchestra.run.smithi133:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4537327c-cfc4-11ee-95bb-87774f69a715 -- ceph osd stat -f json 2024-02-20T07:53:50.665 INFO:teuthology.orchestra.run.smithi133.stderr:Inferring config /var/lib/ceph/4537327c-cfc4-11ee-95bb-87774f69a715/mon.smithi133/config 2024-02-20T07:53:50.769 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:53:50 smithi183 ceph-mon[35832]: purged_snaps scrub starts 2024-02-20T07:53:50.769 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:53:50 smithi183 ceph-mon[35832]: purged_snaps scrub ok 2024-02-20T07:53:50.769 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:53:50 smithi183 ceph-mon[35832]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-20T07:53:50.769 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:53:50 smithi183 ceph-mon[35832]: osd.5 [v2:172.21.15.183:6816/81997900,v1:172.21.15.183:6817/81997900] boot 2024-02-20T07:53:50.769 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:53:50 smithi183 ceph-mon[35832]: osdmap e39: 6 total, 6 up, 6 in 2024-02-20T07:53:50.769 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:53:50 smithi183 ceph-mon[35832]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-20T07:53:50.981 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:53:50 smithi133 ceph-mon[29117]: purged_snaps scrub starts 2024-02-20T07:53:50.981 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:53:50 smithi133 ceph-mon[29117]: purged_snaps scrub ok 2024-02-20T07:53:50.981 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:53:50 smithi133 ceph-mon[29117]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-20T07:53:50.981 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:53:50 smithi133 ceph-mon[29117]: osd.5 [v2:172.21.15.183:6816/81997900,v1:172.21.15.183:6817/81997900] boot 2024-02-20T07:53:50.981 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:53:50 smithi133 ceph-mon[29117]: osdmap e39: 6 total, 6 up, 6 in 2024-02-20T07:53:50.981 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:53:50 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-20T07:53:51.953 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:53:51 smithi133 ceph-mon[29117]: osdmap e40: 6 total, 6 up, 6 in 2024-02-20T07:53:51.953 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:53:51 smithi133 ceph-mon[29117]: Detected new or changed devices on smithi183 2024-02-20T07:53:51.954 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:53:51 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:53:51.954 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:53:51 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:53:51.954 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:53:51 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-02-20T07:53:51.954 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:53:51 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-20T07:53:51.954 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:53:51 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-20T07:53:51.954 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:53:51 smithi133 ceph-mon[29117]: Adjusting osd_memory_target on smithi183 to 5312M 2024-02-20T07:53:51.954 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:53:51 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:53:51.954 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:53:51 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-20T07:53:51.954 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:53:51 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-20T07:53:51.954 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:53:51 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:53:51.954 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:53:51 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-20T07:53:51.955 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:53:51 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-20T07:53:51.955 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:53:51 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-20T07:53:51.955 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:53:51 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:53:52.047 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:53:51 smithi183 ceph-mon[35832]: osdmap e40: 6 total, 6 up, 6 in 2024-02-20T07:53:52.048 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:53:51 smithi183 ceph-mon[35832]: Detected new or changed devices on smithi183 2024-02-20T07:53:52.048 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:53:51 smithi183 ceph-mon[35832]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:53:52.048 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:53:51 smithi183 ceph-mon[35832]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:53:52.048 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:53:51 smithi183 ceph-mon[35832]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-02-20T07:53:52.048 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:53:51 smithi183 ceph-mon[35832]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-20T07:53:52.048 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:53:51 smithi183 ceph-mon[35832]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-20T07:53:52.048 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:53:51 smithi183 ceph-mon[35832]: Adjusting osd_memory_target on smithi183 to 5312M 2024-02-20T07:53:52.048 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:53:51 smithi183 ceph-mon[35832]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:53:52.048 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:53:51 smithi183 ceph-mon[35832]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-20T07:53:52.048 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:53:51 smithi183 ceph-mon[35832]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-20T07:53:52.048 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:53:51 smithi183 ceph-mon[35832]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:53:52.048 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:53:51 smithi183 ceph-mon[35832]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-20T07:53:52.048 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:53:51 smithi183 ceph-mon[35832]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-20T07:53:52.048 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:53:51 smithi183 ceph-mon[35832]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-20T07:53:52.049 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:53:51 smithi183 ceph-mon[35832]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:53:52.463 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:53:52.460+0000 7f642720f700 1 -- 172.21.15.133:0/4063993615 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f64200fd080 msgr2=0x7f64200fd4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:53:52.463 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:53:52.460+0000 7f642720f700 1 --2- 172.21.15.133:0/4063993615 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f64200fd080 0x7f64200fd4f0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f640c004940 tx=0x7f640c009e30 comp rx=0 tx=0).stop 2024-02-20T07:53:52.463 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:53:52.460+0000 7f642720f700 1 -- 172.21.15.133:0/4063993615 shutdown_connections 2024-02-20T07:53:52.463 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:53:52.460+0000 7f642720f700 1 --2- 172.21.15.133:0/4063993615 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f64200fd080 0x7f64200fd4f0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:53:52.463 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:53:52.460+0000 7f642720f700 1 --2- 172.21.15.133:0/4063993615 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f64200fbe80 0x7f64200fc290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:53:52.463 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:53:52.460+0000 7f642720f700 1 -- 172.21.15.133:0/4063993615 >> 172.21.15.133:0/4063993615 conn(0x7f64200f7590 msgr2=0x7f64200f99c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:53:52.463 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:53:52.460+0000 7f642720f700 1 -- 172.21.15.133:0/4063993615 shutdown_connections 2024-02-20T07:53:52.464 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:53:52.460+0000 7f642720f700 1 -- 172.21.15.133:0/4063993615 wait complete. 2024-02-20T07:53:52.464 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:53:52.460+0000 7f642720f700 1 Processor -- start 2024-02-20T07:53:52.464 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:53:52.460+0000 7f642720f700 1 -- start start 2024-02-20T07:53:52.464 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:53:52.461+0000 7f642720f700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f64200fbe80 0x7f6420110630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:53:52.464 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:53:52.461+0000 7f642720f700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f64200fd080 0x7f6420112b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:53:52.464 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:53:52.461+0000 7f642720f700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6420066e00 con 0x7f64200fbe80 2024-02-20T07:53:52.464 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:53:52.461+0000 7f642720f700 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6420066f40 con 0x7f64200fd080 2024-02-20T07:53:52.464 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:53:52.461+0000 7f6424fab700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f64200fbe80 0x7f6420110630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:53:52.464 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:53:52.461+0000 7f6424fab700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f64200fbe80 0x7f6420110630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.133:56844/0 (socket says 172.21.15.133:56844) 2024-02-20T07:53:52.464 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:53:52.461+0000 7f6424fab700 1 -- 172.21.15.133:0/1569892124 learned_addr learned my addr 172.21.15.133:0/1569892124 (peer_addr_for_me v2:172.21.15.133:0/0) 2024-02-20T07:53:52.464 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:53:52.461+0000 7f641ffff700 1 --2- 172.21.15.133:0/1569892124 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f64200fd080 0x7f6420112b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:53:52.464 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:53:52.461+0000 7f6424fab700 1 -- 172.21.15.133:0/1569892124 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f64200fd080 msgr2=0x7f6420112b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:53:52.464 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:53:52.461+0000 7f6424fab700 1 --2- 172.21.15.133:0/1569892124 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f64200fd080 0x7f6420112b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:53:52.464 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:53:52.461+0000 7f6424fab700 1 -- 172.21.15.133:0/1569892124 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f640c005040 con 0x7f64200fbe80 2024-02-20T07:53:52.465 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:53:52.461+0000 7f641ffff700 1 --2- 172.21.15.133:0/1569892124 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f64200fd080 0x7f6420112b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-20T07:53:52.465 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:53:52.461+0000 7f6424fab700 1 --2- 172.21.15.133:0/1569892124 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f64200fbe80 0x7f6420110630 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f641400b790 tx=0x7f6414009220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:53:52.466 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:53:52.461+0000 7f641d7fa700 1 -- 172.21.15.133:0/1569892124 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f641400f6f0 con 0x7f64200fbe80 2024-02-20T07:53:52.466 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:53:52.461+0000 7f641d7fa700 1 -- 172.21.15.133:0/1569892124 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f641400fd20 con 0x7f64200fbe80 2024-02-20T07:53:52.466 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:53:52.461+0000 7f641d7fa700 1 -- 172.21.15.133:0/1569892124 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f64140103a0 con 0x7f64200fbe80 2024-02-20T07:53:52.466 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:53:52.461+0000 7f642720f700 1 -- 172.21.15.133:0/1569892124 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6420113260 con 0x7f64200fbe80 2024-02-20T07:53:52.466 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:53:52.462+0000 7f642720f700 1 -- 172.21.15.133:0/1569892124 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6420113750 con 0x7f64200fbe80 2024-02-20T07:53:52.466 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:53:52.463+0000 7f642720f700 1 -- 172.21.15.133:0/1569892124 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f64040052e0 con 0x7f64200fbe80 2024-02-20T07:53:52.466 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:53:52.463+0000 7f641d7fa700 1 -- 172.21.15.133:0/1569892124 <== mon.0 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7f6414010500 con 0x7f64200fbe80 2024-02-20T07:53:52.466 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:53:52.463+0000 7f641d7fa700 1 --2- 172.21.15.133:0/1569892124 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7f64100692a0 0x7f641006b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:53:52.466 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:53:52.463+0000 7f641d7fa700 1 -- 172.21.15.133:0/1569892124 <== mon.0 v2:172.21.15.133:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f6414086cf0 con 0x7f64200fbe80 2024-02-20T07:53:52.466 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:53:52.463+0000 7f641ffff700 1 --2- 172.21.15.133:0/1569892124 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7f64100692a0 0x7f641006b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:53:52.466 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:53:52.464+0000 7f641ffff700 1 --2- 172.21.15.133:0/1569892124 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7f64100692a0 0x7f641006b750 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f640c004560 tx=0x7f640c009e30 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:53:52.469 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:53:52.468+0000 7f641d7fa700 1 -- 172.21.15.133:0/1569892124 <== mon.0 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f64140532e0 con 0x7f64200fbe80 2024-02-20T07:53:52.653 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:53:52.651+0000 7f642720f700 1 -- 172.21.15.133:0/1569892124 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) v1 -- 0x7f6404005590 con 0x7f64200fbe80 2024-02-20T07:53:52.653 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:53:52.651+0000 7f641d7fa700 1 -- 172.21.15.133:0/1569892124 <== mon.0 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v40) v1 ==== 74+0+130 (secure 0 0 0) 0x7f6414056900 con 0x7f64200fbe80 2024-02-20T07:53:52.653 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-02-20T07:53:52.655 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:53:52.654+0000 7f642720f700 1 -- 172.21.15.133:0/1569892124 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7f64100692a0 msgr2=0x7f641006b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:53:52.655 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:53:52.654+0000 7f642720f700 1 --2- 172.21.15.133:0/1569892124 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7f64100692a0 0x7f641006b750 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f640c004560 tx=0x7f640c009e30 comp rx=0 tx=0).stop 2024-02-20T07:53:52.656 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:53:52.654+0000 7f642720f700 1 -- 172.21.15.133:0/1569892124 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f64200fbe80 msgr2=0x7f6420110630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:53:52.656 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:53:52.654+0000 7f642720f700 1 --2- 172.21.15.133:0/1569892124 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f64200fbe80 0x7f6420110630 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f641400b790 tx=0x7f6414009220 comp rx=0 tx=0).stop 2024-02-20T07:53:52.656 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:53:52.654+0000 7f642720f700 1 -- 172.21.15.133:0/1569892124 shutdown_connections 2024-02-20T07:53:52.656 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:53:52.654+0000 7f642720f700 1 --2- 172.21.15.133:0/1569892124 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f64200fbe80 0x7f6420110630 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:53:52.656 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:53:52.654+0000 7f642720f700 1 --2- 172.21.15.133:0/1569892124 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7f64100692a0 0x7f641006b750 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:53:52.656 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:53:52.654+0000 7f642720f700 1 --2- 172.21.15.133:0/1569892124 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f64200fd080 0x7f6420112b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:53:52.656 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:53:52.654+0000 7f642720f700 1 -- 172.21.15.133:0/1569892124 >> 172.21.15.133:0/1569892124 conn(0x7f64200f7590 msgr2=0x7f64201002b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:53:52.656 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:53:52.654+0000 7f642720f700 1 -- 172.21.15.133:0/1569892124 shutdown_connections 2024-02-20T07:53:52.656 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:53:52.654+0000 7f642720f700 1 -- 172.21.15.133:0/1569892124 wait complete. 2024-02-20T07:53:53.241 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:53:52 smithi133 ceph-mon[29117]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-20T07:53:53.242 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:53:52 smithi133 ceph-mon[29117]: from='client.? 172.21.15.133:0/1569892124' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-02-20T07:53:53.297 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:53:52 smithi183 ceph-mon[35832]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-20T07:53:53.297 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:53:52 smithi183 ceph-mon[35832]: from='client.? 172.21.15.133:0/1569892124' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-02-20T07:53:53.352 INFO:teuthology.orchestra.run.smithi133.stdout:{"epoch":40,"num_osds":6,"num_up_osds":6,"osd_up_since":1708415629,"num_in_osds":6,"osd_in_since":1708415610,"num_remapped_pgs":0} 2024-02-20T07:53:53.353 DEBUG:teuthology.orchestra.run.smithi133:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 4537327c-cfc4-11ee-95bb-87774f69a715 -- ceph osd dump --format=json 2024-02-20T07:53:53.611 INFO:teuthology.orchestra.run.smithi133.stderr:Inferring config /var/lib/ceph/4537327c-cfc4-11ee-95bb-87774f69a715/mon.smithi133/config 2024-02-20T07:53:53.992 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:53:53 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-20T07:53:54.297 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:53:53 smithi183 ceph-mon[35832]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-20T07:53:55.094 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:53:54 smithi133 ceph-mon[29117]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-20T07:53:55.173 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:53:54 smithi183 ceph-mon[35832]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-20T07:53:55.274 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:53:55.273+0000 7fdc461ed700 1 -- 172.21.15.133:0/1207424879 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fdc40102f50 msgr2=0x7fdc40103320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:53:55.275 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:53:55.273+0000 7fdc461ed700 1 --2- 172.21.15.133:0/1207424879 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fdc40102f50 0x7fdc40103320 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fdc30004940 tx=0x7fdc30009e30 comp rx=0 tx=0).stop 2024-02-20T07:53:55.275 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:53:55.273+0000 7fdc461ed700 1 -- 172.21.15.133:0/1207424879 shutdown_connections 2024-02-20T07:53:55.275 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:53:55.273+0000 7fdc461ed700 1 --2- 172.21.15.133:0/1207424879 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fdc401038f0 0x7fdc400fcf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:53:55.275 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:53:55.273+0000 7fdc461ed700 1 --2- 172.21.15.133:0/1207424879 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fdc40102f50 0x7fdc40103320 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:53:55.275 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:53:55.273+0000 7fdc461ed700 1 -- 172.21.15.133:0/1207424879 >> 172.21.15.133:0/1207424879 conn(0x7fdc400f8ae0 msgr2=0x7fdc400faef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:53:55.275 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:53:55.273+0000 7fdc461ed700 1 -- 172.21.15.133:0/1207424879 shutdown_connections 2024-02-20T07:53:55.275 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:53:55.273+0000 7fdc461ed700 1 -- 172.21.15.133:0/1207424879 wait complete. 2024-02-20T07:53:55.277 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:53:55.273+0000 7fdc461ed700 1 Processor -- start 2024-02-20T07:53:55.277 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:53:55.273+0000 7fdc461ed700 1 -- start start 2024-02-20T07:53:55.277 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:53:55.274+0000 7fdc461ed700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fdc40102f50 0x7fdc40113a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:53:55.277 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:53:55.274+0000 7fdc461ed700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fdc401038f0 0x7fdc40113f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:53:55.277 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:53:55.274+0000 7fdc461ed700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc401145e0 con 0x7fdc401038f0 2024-02-20T07:53:55.277 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:53:55.274+0000 7fdc461ed700 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc4010d910 con 0x7fdc40102f50 2024-02-20T07:53:55.277 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:53:55.274+0000 7fdc3effd700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fdc401038f0 0x7fdc40113f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:53:55.278 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:53:55.274+0000 7fdc3effd700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fdc401038f0 0x7fdc40113f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.133:56856/0 (socket says 172.21.15.133:56856) 2024-02-20T07:53:55.278 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:53:55.274+0000 7fdc3effd700 1 -- 172.21.15.133:0/310568008 learned_addr learned my addr 172.21.15.133:0/310568008 (peer_addr_for_me v2:172.21.15.133:0/0) 2024-02-20T07:53:55.278 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:53:55.274+0000 7fdc3f7fe700 1 --2- 172.21.15.133:0/310568008 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fdc40102f50 0x7fdc40113a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:53:55.278 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:53:55.274+0000 7fdc3effd700 1 -- 172.21.15.133:0/310568008 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fdc40102f50 msgr2=0x7fdc40113a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:53:55.278 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:53:55.274+0000 7fdc3effd700 1 --2- 172.21.15.133:0/310568008 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fdc40102f50 0x7fdc40113a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:53:55.278 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:53:55.274+0000 7fdc3effd700 1 -- 172.21.15.133:0/310568008 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc30005040 con 0x7fdc401038f0 2024-02-20T07:53:55.278 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:53:55.274+0000 7fdc3f7fe700 1 --2- 172.21.15.133:0/310568008 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fdc40102f50 0x7fdc40113a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-20T07:53:55.278 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:53:55.274+0000 7fdc3effd700 1 --2- 172.21.15.133:0/310568008 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fdc401038f0 0x7fdc40113f90 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fdc3400c800 tx=0x7fdc34004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:53:55.278 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:53:55.274+0000 7fdc27fff700 1 -- 172.21.15.133:0/310568008 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdc3400a760 con 0x7fdc401038f0 2024-02-20T07:53:55.278 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:53:55.274+0000 7fdc461ed700 1 -- 172.21.15.133:0/310568008 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdc4010dbf0 con 0x7fdc401038f0 2024-02-20T07:53:55.278 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:53:55.274+0000 7fdc27fff700 1 -- 172.21.15.133:0/310568008 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fdc3400ad90 con 0x7fdc401038f0 2024-02-20T07:53:55.278 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:53:55.274+0000 7fdc27fff700 1 -- 172.21.15.133:0/310568008 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdc340113a0 con 0x7fdc401038f0 2024-02-20T07:53:55.278 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:53:55.274+0000 7fdc461ed700 1 -- 172.21.15.133:0/310568008 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdc4010e130 con 0x7fdc401038f0 2024-02-20T07:53:55.278 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:53:55.275+0000 7fdc461ed700 1 -- 172.21.15.133:0/310568008 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdc400fc6e0 con 0x7fdc401038f0 2024-02-20T07:53:55.278 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:53:55.276+0000 7fdc27fff700 1 -- 172.21.15.133:0/310568008 <== mon.0 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7fdc3400a8c0 con 0x7fdc401038f0 2024-02-20T07:53:55.279 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:53:55.276+0000 7fdc27fff700 1 --2- 172.21.15.133:0/310568008 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7fdc280692a0 0x7fdc2806b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:53:55.279 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:53:55.276+0000 7fdc27fff700 1 -- 172.21.15.133:0/310568008 <== mon.0 v2:172.21.15.133:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fdc34086ed0 con 0x7fdc401038f0 2024-02-20T07:53:55.279 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:53:55.276+0000 7fdc3f7fe700 1 --2- 172.21.15.133:0/310568008 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7fdc280692a0 0x7fdc2806b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:53:55.279 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:53:55.277+0000 7fdc3f7fe700 1 --2- 172.21.15.133:0/310568008 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7fdc280692a0 0x7fdc2806b750 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fdc30004f90 tx=0x7fdc30009e30 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:53:55.282 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:53:55.280+0000 7fdc27fff700 1 -- 172.21.15.133:0/310568008 <== mon.0 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7fdc34056fd0 con 0x7fdc401038f0 2024-02-20T07:53:55.465 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:53:55.463+0000 7fdc461ed700 1 -- 172.21.15.133:0/310568008 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fdc4010e3b0 con 0x7fdc401038f0 2024-02-20T07:53:55.466 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:53:55.464+0000 7fdc27fff700 1 -- 172.21.15.133:0/310568008 <== mon.0 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v40) v1 ==== 74+0+11111 (secure 0 0 0) 0x7fdc34056b60 con 0x7fdc401038f0 2024-02-20T07:53:55.466 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-02-20T07:53:55.466 INFO:teuthology.orchestra.run.smithi133.stdout:{"epoch":40,"fsid":"4537327c-cfc4-11ee-95bb-87774f69a715","created":"2024-02-20T07:48:13.550549+0000","modified":"2024-02-20T07:53:50.692077+0000","last_up_change":"2024-02-20T07:53:49.684637+0000","last_in_change":"2024-02-20T07:53:30.651509+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":14,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-20T07:52:35.839244+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"0a20b241-99e8-4e0b-ba5e-be07c8fdff94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6802","nonce":261892236},{"type":"v1","addr":"172.21.15.133:6803","nonce":261892236}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6804","nonce":261892236},{"type":"v1","addr":"172.21.15.133:6805","nonce":261892236}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6808","nonce":261892236},{"type":"v1","addr":"172.21.15.133:6809","nonce":261892236}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6806","nonce":261892236},{"type":"v1","addr":"172.21.15.133:6807","nonce":261892236}]},"public_addr":"172.21.15.133:6803/261892236","cluster_addr":"172.21.15.133:6805/261892236","heartbeat_back_addr":"172.21.15.133:6809/261892236","heartbeat_front_addr":"172.21.15.133:6807/261892236","state":["exists","up"]},{"osd":1,"uuid":"2b1b8e8f-74b4-4d41-8753-502296eb87df","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6810","nonce":1652695599},{"type":"v1","addr":"172.21.15.133:6811","nonce":1652695599}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6812","nonce":1652695599},{"type":"v1","addr":"172.21.15.133:6813","nonce":1652695599}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6816","nonce":1652695599},{"type":"v1","addr":"172.21.15.133:6817","nonce":1652695599}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6814","nonce":1652695599},{"type":"v1","addr":"172.21.15.133:6815","nonce":1652695599}]},"public_addr":"172.21.15.133:6811/1652695599","cluster_addr":"172.21.15.133:6813/1652695599","heartbeat_back_addr":"172.21.15.133:6817/1652695599","heartbeat_front_addr":"172.21.15.133:6815/1652695599","state":["exists","up"]},{"osd":2,"uuid":"95848108-ed6d-4030-9534-d687a769350f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6818","nonce":3838005412},{"type":"v1","addr":"172.21.15.133:6819","nonce":3838005412}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6820","nonce":3838005412},{"type":"v1","addr":"172.21.15.133:6821","nonce":3838005412}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6824","nonce":3838005412},{"type":"v1","addr":"172.21.15.133:6825","nonce":3838005412}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6822","nonce":3838005412},{"type":"v1","addr":"172.21.15.133:6823","nonce":3838005412}]},"public_addr":"172.21.15.133:6819/3838005412","cluster_addr":"172.21.15.133:6821/3838005412","heartbeat_back_addr":"172.21.15.133:6825/3838005412","heartbeat_front_addr":"172.21.15.133:6823/3838005412","state":["exists","up"]},{"osd":3,"uuid":"c2015ce7-f7fb-428d-b42f-e5fb665933e0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6800","nonce":1053980618},{"type":"v1","addr":"172.21.15.183:6801","nonce":1053980618}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6802","nonce":1053980618},{"type":"v1","addr":"172.21.15.183:6803","nonce":1053980618}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6806","nonce":1053980618},{"type":"v1","addr":"172.21.15.183:6807","nonce":1053980618}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6804","nonce":1053980618},{"type":"v1","addr":"172.21.15.183:6805","nonce":1053980618}]},"public_addr":"172.21.15.183:6801/1053980618","cluster_addr":"172.21.15.183:6803/1053980618","heartbeat_back_addr":"172.21.15.183:6807/1053980618","heartbeat_front_addr":"172.21.15.183:6805/1053980618","state":["exists","up"]},{"osd":4,"uuid":"55148881-b92f-40e6-bfe8-b8320406c96a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6808","nonce":3697465174},{"type":"v1","addr":"172.21.15.183:6809","nonce":3697465174}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6810","nonce":3697465174},{"type":"v1","addr":"172.21.15.183:6811","nonce":3697465174}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6814","nonce":3697465174},{"type":"v1","addr":"172.21.15.183:6815","nonce":3697465174}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6812","nonce":3697465174},{"type":"v1","addr":"172.21.15.183:6813","nonce":3697465174}]},"public_addr":"172.21.15.183:6809/3697465174","cluster_addr":"172.21.15.183:6811/3697465174","heartbeat_back_addr":"172.21.15.183:6815/3697465174","heartbeat_front_addr":"172.21.15.183:6813/3697465174","state":["exists","up"]},{"osd":5,"uuid":"ee5510f9-fab7-4aa3-8372-3856a0787e92","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6816","nonce":81997900},{"type":"v1","addr":"172.21.15.183:6817","nonce":81997900}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6818","nonce":81997900},{"type":"v1","addr":"172.21.15.183:6819","nonce":81997900}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6822","nonce":81997900},{"type":"v1","addr":"172.21.15.183:6823","nonce":81997900}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6820","nonce":81997900},{"type":"v1","addr":"172.21.15.183:6821","nonce":81997900}]},"public_addr":"172.21.15.183:6817/81997900","cluster_addr":"172.21.15.183:6819/81997900","heartbeat_back_addr":"172.21.15.183:6823/81997900","heartbeat_front_addr":"172.21.15.183:6821/81997900","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-02-20T07:51:38.297291+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-02-20T07:52:05.983942+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-02-20T07:52:33.478124+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-02-20T07:52:56.569795+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-02-20T07:53:22.416562+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-02-20T07:53:47.770454+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.133:6800/3944459195":"2024-02-21T07:50:23.523666+0000","172.21.15.133:0/1714017955":"2024-02-21T07:50:23.523666+0000","172.21.15.133:6801/1670376163":"2024-02-21T07:49:19.684526+0000","172.21.15.133:6800/1670376163":"2024-02-21T07:49:19.684526+0000","172.21.15.133:0/3014285121":"2024-02-21T07:50:23.523666+0000","172.21.15.133:0/3813170035":"2024-02-21T07:49:19.684526+0000","172.21.15.133:0/2942023244":"2024-02-21T07:50:23.523666+0000","172.21.15.133:6800/3480413474":"2024-02-21T07:48:49.130220+0000","172.21.15.133:0/129232921":"2024-02-21T07:48:49.130220+0000","172.21.15.133:6801/3480413474":"2024-02-21T07:48:49.130220+0000","172.21.15.133:0/3354686539":"2024-02-21T07:49:19.684526+0000","172.21.15.133:0/2763668929":"2024-02-21T07:48:49.130220+0000","172.21.15.133:6801/3944459195":"2024-02-21T07:50:23.523666+0000","172.21.15.133:0/2831283357":"2024-02-21T07:49:19.684526+0000","172.21.15.133:0/1481070311":"2024-02-21T07:48:49.130220+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-20T07:53:55.468 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:53:55.466+0000 7fdc461ed700 1 -- 172.21.15.133:0/310568008 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7fdc280692a0 msgr2=0x7fdc2806b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:53:55.468 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:53:55.466+0000 7fdc461ed700 1 --2- 172.21.15.133:0/310568008 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7fdc280692a0 0x7fdc2806b750 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fdc30004f90 tx=0x7fdc30009e30 comp rx=0 tx=0).stop 2024-02-20T07:53:55.468 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:53:55.466+0000 7fdc461ed700 1 -- 172.21.15.133:0/310568008 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fdc401038f0 msgr2=0x7fdc40113f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:53:55.469 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:53:55.466+0000 7fdc461ed700 1 --2- 172.21.15.133:0/310568008 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fdc401038f0 0x7fdc40113f90 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fdc3400c800 tx=0x7fdc34004470 comp rx=0 tx=0).stop 2024-02-20T07:53:55.469 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:53:55.466+0000 7fdc461ed700 1 -- 172.21.15.133:0/310568008 shutdown_connections 2024-02-20T07:53:55.469 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:53:55.466+0000 7fdc461ed700 1 --2- 172.21.15.133:0/310568008 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7fdc280692a0 0x7fdc2806b750 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:53:55.469 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:53:55.466+0000 7fdc461ed700 1 --2- 172.21.15.133:0/310568008 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fdc40102f50 0x7fdc40113a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:53:55.469 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:53:55.466+0000 7fdc461ed700 1 --2- 172.21.15.133:0/310568008 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fdc401038f0 0x7fdc40113f90 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:53:55.469 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:53:55.466+0000 7fdc461ed700 1 -- 172.21.15.133:0/310568008 >> 172.21.15.133:0/310568008 conn(0x7fdc400f8ae0 msgr2=0x7fdc400fa500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:53:55.469 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:53:55.467+0000 7fdc461ed700 1 -- 172.21.15.133:0/310568008 shutdown_connections 2024-02-20T07:53:55.469 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:53:55.467+0000 7fdc461ed700 1 -- 172.21.15.133:0/310568008 wait complete. 2024-02-20T07:53:56.074 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:53:55 smithi133 ceph-mon[29117]: from='client.? 172.21.15.133:0/310568008' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-20T07:53:56.097 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-02-20T07:52:35.839244+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}] 2024-02-20T07:53:56.097 DEBUG:teuthology.orchestra.run.smithi133:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 4537327c-cfc4-11ee-95bb-87774f69a715 -- ceph osd pool get .mgr pg_num 2024-02-20T07:53:56.297 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:53:55 smithi183 ceph-mon[35832]: from='client.? 172.21.15.133:0/310568008' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-20T07:53:56.349 INFO:teuthology.orchestra.run.smithi133.stderr:Inferring config /var/lib/ceph/4537327c-cfc4-11ee-95bb-87774f69a715/mon.smithi133/config 2024-02-20T07:53:57.241 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:53:56 smithi133 ceph-mon[29117]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-20T07:53:57.297 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:53:56 smithi183 ceph-mon[35832]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-20T07:53:58.038 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:53:58.035+0000 7f8fea579700 1 -- 172.21.15.133:0/932150051 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f8fe40fdfb0 msgr2=0x7f8fe40fe380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:53:58.038 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:53:58.035+0000 7f8fea579700 1 --2- 172.21.15.133:0/932150051 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f8fe40fdfb0 0x7f8fe40fe380 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f8fd4004940 tx=0x7f8fd4009e30 comp rx=0 tx=0).stop 2024-02-20T07:53:58.039 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:53:58.035+0000 7f8fea579700 1 -- 172.21.15.133:0/932150051 shutdown_connections 2024-02-20T07:53:58.039 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:53:58.035+0000 7f8fea579700 1 --2- 172.21.15.133:0/932150051 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f8fe40fe950 0x7f8fe4106f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:53:58.039 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:53:58.035+0000 7f8fea579700 1 --2- 172.21.15.133:0/932150051 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f8fe40fdfb0 0x7f8fe40fe380 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:53:58.039 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:53:58.035+0000 7f8fea579700 1 -- 172.21.15.133:0/932150051 >> 172.21.15.133:0/932150051 conn(0x7f8fe40f79e0 msgr2=0x7f8fe40f9df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:53:58.039 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:53:58.035+0000 7f8fea579700 1 -- 172.21.15.133:0/932150051 shutdown_connections 2024-02-20T07:53:58.039 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:53:58.035+0000 7f8fea579700 1 -- 172.21.15.133:0/932150051 wait complete. 2024-02-20T07:53:58.039 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:53:58.035+0000 7f8fea579700 1 Processor -- start 2024-02-20T07:53:58.039 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:53:58.036+0000 7f8fea579700 1 -- start start 2024-02-20T07:53:58.039 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:53:58.036+0000 7f8fea579700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f8fe40fdfb0 0x7f8fe4113a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:53:58.039 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:53:58.036+0000 7f8fea579700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f8fe40fe950 0x7f8fe4113f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:53:58.039 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:53:58.036+0000 7f8fea579700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fe4114590 con 0x7f8fe40fdfb0 2024-02-20T07:53:58.039 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:53:58.036+0000 7f8fea579700 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fe410d8c0 con 0x7f8fe40fe950 2024-02-20T07:53:58.039 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:53:58.036+0000 7f8fe3fff700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f8fe40fdfb0 0x7f8fe4113a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:53:58.039 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:53:58.036+0000 7f8fe3fff700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f8fe40fdfb0 0x7f8fe4113a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.133:50326/0 (socket says 172.21.15.133:50326) 2024-02-20T07:53:58.040 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:53:58.036+0000 7f8fe3fff700 1 -- 172.21.15.133:0/137263792 learned_addr learned my addr 172.21.15.133:0/137263792 (peer_addr_for_me v2:172.21.15.133:0/0) 2024-02-20T07:53:58.040 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:53:58.036+0000 7f8fe37fe700 1 --2- 172.21.15.133:0/137263792 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f8fe40fe950 0x7f8fe4113f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:53:58.040 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:53:58.036+0000 7f8fe3fff700 1 -- 172.21.15.133:0/137263792 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f8fe40fe950 msgr2=0x7f8fe4113f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:53:58.040 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:53:58.036+0000 7f8fe3fff700 1 --2- 172.21.15.133:0/137263792 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f8fe40fe950 0x7f8fe4113f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:53:58.040 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:53:58.036+0000 7f8fe3fff700 1 -- 172.21.15.133:0/137263792 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8fd4005040 con 0x7f8fe40fdfb0 2024-02-20T07:53:58.040 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:53:58.036+0000 7f8fe3fff700 1 --2- 172.21.15.133:0/137263792 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f8fe40fdfb0 0x7f8fe4113a00 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f8fd4009180 tx=0x7f8fd4009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:53:58.040 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:53:58.037+0000 7f8fe0ff9700 1 -- 172.21.15.133:0/137263792 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8fd400e040 con 0x7f8fe40fdfb0 2024-02-20T07:53:58.040 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:53:58.037+0000 7f8fea579700 1 -- 172.21.15.133:0/137263792 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8fe410dba0 con 0x7f8fe40fdfb0 2024-02-20T07:53:58.040 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:53:58.037+0000 7f8fe0ff9700 1 -- 172.21.15.133:0/137263792 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8fd4004290 con 0x7f8fe40fdfb0 2024-02-20T07:53:58.040 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:53:58.037+0000 7f8fe0ff9700 1 -- 172.21.15.133:0/137263792 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8fd401c630 con 0x7f8fe40fdfb0 2024-02-20T07:53:58.040 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:53:58.037+0000 7f8fea579700 1 -- 172.21.15.133:0/137263792 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8fe410e0e0 con 0x7f8fe40fdfb0 2024-02-20T07:53:58.041 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:53:58.038+0000 7f8fea579700 1 -- 172.21.15.133:0/137263792 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8fe41045a0 con 0x7f8fe40fdfb0 2024-02-20T07:53:58.041 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:53:58.038+0000 7f8fe0ff9700 1 -- 172.21.15.133:0/137263792 <== mon.0 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7f8fd400bbf0 con 0x7f8fe40fdfb0 2024-02-20T07:53:58.041 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:53:58.038+0000 7f8fe0ff9700 1 --2- 172.21.15.133:0/137263792 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7f8fcc0692b0 0x7f8fcc06b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:53:58.041 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:53:58.039+0000 7f8fe37fe700 1 --2- 172.21.15.133:0/137263792 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7f8fcc0692b0 0x7f8fcc06b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:53:58.041 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:53:58.039+0000 7f8fe0ff9700 1 -- 172.21.15.133:0/137263792 <== mon.0 v2:172.21.15.133:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f8fd4088110 con 0x7f8fe40fdfb0 2024-02-20T07:53:58.041 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:53:58.039+0000 7f8fe37fe700 1 --2- 172.21.15.133:0/137263792 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7f8fcc0692b0 0x7f8fcc06b760 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f8fe410f230 tx=0x7f8fd800b040 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:53:58.044 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:53:58.043+0000 7f8fe0ff9700 1 -- 172.21.15.133:0/137263792 <== mon.0 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f8fd4058210 con 0x7f8fe40fdfb0 2024-02-20T07:53:58.224 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:53:58.222+0000 7f8fea579700 1 -- 172.21.15.133:0/137263792 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f8fe40fb5e0 con 0x7f8fe40fdfb0 2024-02-20T07:53:58.225 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:53:58.223+0000 7f8fe0ff9700 1 -- 172.21.15.133:0/137263792 <== mon.0 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v40) v1 ==== 93+0+10 (secure 0 0 0) 0x7f8fd4004db0 con 0x7f8fe40fdfb0 2024-02-20T07:53:58.225 INFO:teuthology.orchestra.run.smithi133.stdout:pg_num: 1 2024-02-20T07:53:58.227 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:53:58.225+0000 7f8fea579700 1 -- 172.21.15.133:0/137263792 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7f8fcc0692b0 msgr2=0x7f8fcc06b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:53:58.227 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:53:58.225+0000 7f8fea579700 1 --2- 172.21.15.133:0/137263792 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7f8fcc0692b0 0x7f8fcc06b760 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f8fe410f230 tx=0x7f8fd800b040 comp rx=0 tx=0).stop 2024-02-20T07:53:58.227 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:53:58.225+0000 7f8fea579700 1 -- 172.21.15.133:0/137263792 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f8fe40fdfb0 msgr2=0x7f8fe4113a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:53:58.227 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:53:58.225+0000 7f8fea579700 1 --2- 172.21.15.133:0/137263792 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f8fe40fdfb0 0x7f8fe4113a00 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f8fd4009180 tx=0x7f8fd4009e30 comp rx=0 tx=0).stop 2024-02-20T07:53:58.228 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:53:58.225+0000 7f8fea579700 1 -- 172.21.15.133:0/137263792 shutdown_connections 2024-02-20T07:53:58.228 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:53:58.225+0000 7f8fea579700 1 --2- 172.21.15.133:0/137263792 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f8fe40fdfb0 0x7f8fe4113a00 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:53:58.228 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:53:58.225+0000 7f8fea579700 1 --2- 172.21.15.133:0/137263792 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7f8fcc0692b0 0x7f8fcc06b760 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:53:58.228 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:53:58.225+0000 7f8fea579700 1 --2- 172.21.15.133:0/137263792 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f8fe40fe950 0x7f8fe4113f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:53:58.228 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:53:58.225+0000 7f8fea579700 1 -- 172.21.15.133:0/137263792 >> 172.21.15.133:0/137263792 conn(0x7f8fe40f79e0 msgr2=0x7f8fe4101720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:53:58.228 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:53:58.225+0000 7f8fea579700 1 -- 172.21.15.133:0/137263792 shutdown_connections 2024-02-20T07:53:58.228 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:53:58.226+0000 7f8fea579700 1 -- 172.21.15.133:0/137263792 wait complete. 2024-02-20T07:53:58.964 INFO:tasks.cephadm:Setting up client nodes... 2024-02-20T07:53:58.965 DEBUG:teuthology.orchestra.run.smithi133:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4537327c-cfc4-11ee-95bb-87774f69a715 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-02-20T07:53:59.218 INFO:teuthology.orchestra.run.smithi133.stderr:Inferring config /var/lib/ceph/4537327c-cfc4-11ee-95bb-87774f69a715/mon.smithi133/config 2024-02-20T07:53:59.241 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:53:58 smithi133 ceph-mon[29117]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-20T07:53:59.241 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:53:58 smithi133 ceph-mon[29117]: from='client.? 172.21.15.133:0/137263792' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-02-20T07:53:59.297 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:53:58 smithi183 ceph-mon[35832]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-20T07:53:59.297 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:53:58 smithi183 ceph-mon[35832]: from='client.? 172.21.15.133:0/137263792' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-02-20T07:54:00.884 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:00.881+0000 7f87fa58d700 1 -- 172.21.15.133:0/2098247546 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f87f40fcef0 msgr2=0x7f87f40fd320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:54:00.885 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:00.881+0000 7f87fa58d700 1 --2- 172.21.15.133:0/2098247546 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f87f40fcef0 0x7f87f40fd320 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f87e4004940 tx=0x7f87e4009e30 comp rx=0 tx=0).stop 2024-02-20T07:54:00.885 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:00.881+0000 7f87fa58d700 1 -- 172.21.15.133:0/2098247546 shutdown_connections 2024-02-20T07:54:00.885 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:00.881+0000 7f87fa58d700 1 --2- 172.21.15.133:0/2098247546 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f87f40fe0e0 0x7f87f40fe550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:54:00.885 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:00.881+0000 7f87fa58d700 1 --2- 172.21.15.133:0/2098247546 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f87f40fcef0 0x7f87f40fd320 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:54:00.885 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:00.881+0000 7f87fa58d700 1 -- 172.21.15.133:0/2098247546 >> 172.21.15.133:0/2098247546 conn(0x7f87f40f8620 msgr2=0x7f87f40faa30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:54:00.885 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:00.881+0000 7f87fa58d700 1 -- 172.21.15.133:0/2098247546 shutdown_connections 2024-02-20T07:54:00.885 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:00.881+0000 7f87fa58d700 1 -- 172.21.15.133:0/2098247546 wait complete. 2024-02-20T07:54:00.885 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:00.881+0000 7f87fa58d700 1 Processor -- start 2024-02-20T07:54:00.885 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:00.881+0000 7f87fa58d700 1 -- start start 2024-02-20T07:54:00.885 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:00.882+0000 7f87fa58d700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f87f40fcef0 0x7f87f410d9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:54:00.885 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:00.882+0000 7f87fa58d700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f87f40fe0e0 0x7f87f410def0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:54:00.885 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:00.882+0000 7f87fa58d700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87f410e580 con 0x7f87f40fe0e0 2024-02-20T07:54:00.886 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:00.882+0000 7f87fa58d700 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87f410edc0 con 0x7f87f40fcef0 2024-02-20T07:54:00.886 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:00.882+0000 7f87f37fe700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f87f40fe0e0 0x7f87f410def0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:54:00.886 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:00.882+0000 7f87f37fe700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f87f40fe0e0 0x7f87f410def0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.133:50350/0 (socket says 172.21.15.133:50350) 2024-02-20T07:54:00.886 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:00.882+0000 7f87f37fe700 1 -- 172.21.15.133:0/1564676478 learned_addr learned my addr 172.21.15.133:0/1564676478 (peer_addr_for_me v2:172.21.15.133:0/0) 2024-02-20T07:54:00.886 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:00.882+0000 7f87f3fff700 1 --2- 172.21.15.133:0/1564676478 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f87f40fcef0 0x7f87f410d9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:54:00.886 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:00.882+0000 7f87f37fe700 1 -- 172.21.15.133:0/1564676478 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f87f40fcef0 msgr2=0x7f87f410d9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:54:00.886 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:00.882+0000 7f87f37fe700 1 --2- 172.21.15.133:0/1564676478 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f87f40fcef0 0x7f87f410d9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:54:00.886 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:00.882+0000 7f87f37fe700 1 -- 172.21.15.133:0/1564676478 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87e4005040 con 0x7f87f40fe0e0 2024-02-20T07:54:00.886 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:00.882+0000 7f87f37fe700 1 --2- 172.21.15.133:0/1564676478 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f87f40fe0e0 0x7f87f410def0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f87e800b790 tx=0x7f87e8009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:54:00.886 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:00.882+0000 7f87f0ff9700 1 -- 172.21.15.133:0/1564676478 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f87e800f6f0 con 0x7f87f40fe0e0 2024-02-20T07:54:00.886 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:00.882+0000 7f87fa58d700 1 -- 172.21.15.133:0/1564676478 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f87f410f0a0 con 0x7f87f40fe0e0 2024-02-20T07:54:00.886 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:00.882+0000 7f87f0ff9700 1 -- 172.21.15.133:0/1564676478 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f87e800fd20 con 0x7f87f40fe0e0 2024-02-20T07:54:00.886 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:00.882+0000 7f87f0ff9700 1 -- 172.21.15.133:0/1564676478 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f87e80103a0 con 0x7f87f40fe0e0 2024-02-20T07:54:00.886 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:00.882+0000 7f87fa58d700 1 -- 172.21.15.133:0/1564676478 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f87f410f5e0 con 0x7f87f40fe0e0 2024-02-20T07:54:00.887 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:00.884+0000 7f87fa58d700 1 -- 172.21.15.133:0/1564676478 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f87f404bed0 con 0x7f87f40fe0e0 2024-02-20T07:54:00.887 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:00.884+0000 7f87f0ff9700 1 -- 172.21.15.133:0/1564676478 <== mon.0 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7f87e8011400 con 0x7f87f40fe0e0 2024-02-20T07:54:00.888 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:00.884+0000 7f87f0ff9700 1 --2- 172.21.15.133:0/1564676478 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7f87dc0692a0 0x7f87dc06b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:54:00.888 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:00.884+0000 7f87f0ff9700 1 -- 172.21.15.133:0/1564676478 <== mon.0 v2:172.21.15.133:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f87e8086c30 con 0x7f87f40fe0e0 2024-02-20T07:54:00.888 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:00.884+0000 7f87f3fff700 1 --2- 172.21.15.133:0/1564676478 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7f87dc0692a0 0x7f87dc06b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:54:00.888 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:00.885+0000 7f87f3fff700 1 --2- 172.21.15.133:0/1564676478 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7f87dc0692a0 0x7f87dc06b750 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f87e4004890 tx=0x7f87e4009e30 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:54:00.890 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:00.889+0000 7f87f0ff9700 1 -- 172.21.15.133:0/1564676478 <== mon.0 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f87e8056d30 con 0x7f87f40fe0e0 2024-02-20T07:54:01.150 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:01.147+0000 7f87fa58d700 1 -- 172.21.15.133:0/1564676478 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} v 0) v1 -- 0x7f87f4105ab0 con 0x7f87f40fe0e0 2024-02-20T07:54:01.153 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:01.151+0000 7f87f0ff9700 1 -- 172.21.15.133:0/1564676478 <== mon.0 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]=0 v14) v1 ==== 170+0+59 (secure 0 0 0) 0x7f87e80568c0 con 0x7f87f40fe0e0 2024-02-20T07:54:01.153 INFO:teuthology.orchestra.run.smithi133.stdout:[client.0] 2024-02-20T07:54:01.153 INFO:teuthology.orchestra.run.smithi133.stdout: key = AQCZWtRldwXuCBAAqW1qDt3FlGFDSM8sQgLzHw== 2024-02-20T07:54:01.155 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:01.153+0000 7f87fa58d700 1 -- 172.21.15.133:0/1564676478 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7f87dc0692a0 msgr2=0x7f87dc06b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:54:01.155 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:01.153+0000 7f87fa58d700 1 --2- 172.21.15.133:0/1564676478 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7f87dc0692a0 0x7f87dc06b750 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f87e4004890 tx=0x7f87e4009e30 comp rx=0 tx=0).stop 2024-02-20T07:54:01.155 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:01.153+0000 7f87fa58d700 1 -- 172.21.15.133:0/1564676478 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f87f40fe0e0 msgr2=0x7f87f410def0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:54:01.156 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:01.153+0000 7f87fa58d700 1 --2- 172.21.15.133:0/1564676478 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f87f40fe0e0 0x7f87f410def0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f87e800b790 tx=0x7f87e8009250 comp rx=0 tx=0).stop 2024-02-20T07:54:01.156 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:01.153+0000 7f87fa58d700 1 -- 172.21.15.133:0/1564676478 shutdown_connections 2024-02-20T07:54:01.156 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:01.153+0000 7f87fa58d700 1 --2- 172.21.15.133:0/1564676478 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7f87dc0692a0 0x7f87dc06b750 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:54:01.156 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:01.153+0000 7f87fa58d700 1 --2- 172.21.15.133:0/1564676478 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f87f40fcef0 0x7f87f410d9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:54:01.156 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:01.153+0000 7f87fa58d700 1 --2- 172.21.15.133:0/1564676478 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f87f40fe0e0 0x7f87f410def0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:54:01.156 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:01.153+0000 7f87fa58d700 1 -- 172.21.15.133:0/1564676478 >> 172.21.15.133:0/1564676478 conn(0x7f87f40f8620 msgr2=0x7f87f41013a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:54:01.156 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:01.154+0000 7f87fa58d700 1 -- 172.21.15.133:0/1564676478 shutdown_connections 2024-02-20T07:54:01.156 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:01.154+0000 7f87fa58d700 1 -- 172.21.15.133:0/1564676478 wait complete. 2024-02-20T07:54:01.241 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:54:00 smithi133 ceph-mon[29117]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-20T07:54:01.297 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:54:00 smithi183 ceph-mon[35832]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-20T07:54:01.855 DEBUG:teuthology.orchestra.run.smithi133:> set -ex 2024-02-20T07:54:01.855 DEBUG:teuthology.orchestra.run.smithi133:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-02-20T07:54:01.855 DEBUG:teuthology.orchestra.run.smithi133:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-02-20T07:54:01.898 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4537327c-cfc4-11ee-95bb-87774f69a715 -- ceph auth get-or-create client.1 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-02-20T07:54:02.162 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/4537327c-cfc4-11ee-95bb-87774f69a715/mon.smithi183/config 2024-02-20T07:54:02.241 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:54:01 smithi133 ceph-mon[29117]: from='client.? 172.21.15.133:0/1564676478' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-02-20T07:54:02.242 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:54:01 smithi133 ceph-mon[29117]: from='client.? 172.21.15.133:0/1564676478' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-02-20T07:54:02.284 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:54:01 smithi183 ceph-mon[35832]: from='client.? 172.21.15.133:0/1564676478' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-02-20T07:54:02.284 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:54:01 smithi183 ceph-mon[35832]: from='client.? 172.21.15.133:0/1564676478' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-02-20T07:54:03.200 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:54:02 smithi183 ceph-mon[35832]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-20T07:54:03.241 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:54:02 smithi133 ceph-mon[29117]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-20T07:54:03.719 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:54:03.716+0000 7fdf096df700 1 -- 172.21.15.183:0/2144855695 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fdf040fe0e0 msgr2=0x7fdf040fe550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:54:03.720 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:54:03.716+0000 7fdf096df700 1 --2- 172.21.15.183:0/2144855695 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fdf040fe0e0 0x7fdf040fe550 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fdef8004970 tx=0x7fdef8009e30 comp rx=0 tx=0).stop 2024-02-20T07:54:03.720 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:54:03.716+0000 7fdf096df700 1 -- 172.21.15.183:0/2144855695 shutdown_connections 2024-02-20T07:54:03.720 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:54:03.716+0000 7fdf096df700 1 --2- 172.21.15.183:0/2144855695 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fdf040fe0e0 0x7fdf040fe550 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:54:03.720 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:54:03.716+0000 7fdf096df700 1 --2- 172.21.15.183:0/2144855695 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fdf040fcef0 0x7fdf040fd320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:54:03.720 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:54:03.716+0000 7fdf096df700 1 -- 172.21.15.183:0/2144855695 >> 172.21.15.183:0/2144855695 conn(0x7fdf040f8620 msgr2=0x7fdf040faa30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:54:03.720 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:54:03.716+0000 7fdf096df700 1 -- 172.21.15.183:0/2144855695 shutdown_connections 2024-02-20T07:54:03.720 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:54:03.716+0000 7fdf096df700 1 -- 172.21.15.183:0/2144855695 wait complete. 2024-02-20T07:54:03.720 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:54:03.717+0000 7fdf096df700 1 Processor -- start 2024-02-20T07:54:03.720 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:54:03.717+0000 7fdf096df700 1 -- start start 2024-02-20T07:54:03.720 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:54:03.717+0000 7fdf096df700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fdf040fcef0 0x7fdf0410d8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:54:03.720 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:54:03.717+0000 7fdf096df700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fdf040fe0e0 0x7fdf0410de30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:54:03.720 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:54:03.717+0000 7fdf096df700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf0410e430 con 0x7fdf040fcef0 2024-02-20T07:54:03.721 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:54:03.717+0000 7fdf096df700 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf0410ecd0 con 0x7fdf040fe0e0 2024-02-20T07:54:03.721 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:54:03.717+0000 7fdf027fc700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fdf040fe0e0 0x7fdf0410de30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:54:03.721 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:54:03.717+0000 7fdf027fc700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fdf040fe0e0 0x7fdf0410de30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.183:47248/0 (socket says 172.21.15.183:47248) 2024-02-20T07:54:03.721 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:54:03.717+0000 7fdf027fc700 1 -- 172.21.15.183:0/4006337034 learned_addr learned my addr 172.21.15.183:0/4006337034 (peer_addr_for_me v2:172.21.15.183:0/0) 2024-02-20T07:54:03.721 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:54:03.717+0000 7fdf02ffd700 1 --2- 172.21.15.183:0/4006337034 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fdf040fcef0 0x7fdf0410d8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:54:03.721 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:54:03.717+0000 7fdf027fc700 1 -- 172.21.15.183:0/4006337034 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fdf040fcef0 msgr2=0x7fdf0410d8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:54:03.721 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:54:03.717+0000 7fdf027fc700 1 --2- 172.21.15.183:0/4006337034 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fdf040fcef0 0x7fdf0410d8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:54:03.721 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:54:03.717+0000 7fdf027fc700 1 -- 172.21.15.183:0/4006337034 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdef8005040 con 0x7fdf040fe0e0 2024-02-20T07:54:03.721 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:54:03.718+0000 7fdf027fc700 1 --2- 172.21.15.183:0/4006337034 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fdf040fe0e0 0x7fdf0410de30 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fdef800b370 tx=0x7fdef8009e30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:54:03.721 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:54:03.718+0000 7fdeeb7fe700 1 -- 172.21.15.183:0/4006337034 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdef800e040 con 0x7fdf040fe0e0 2024-02-20T07:54:03.721 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:54:03.718+0000 7fdf096df700 1 -- 172.21.15.183:0/4006337034 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdf0410ef50 con 0x7fdf040fe0e0 2024-02-20T07:54:03.721 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:54:03.718+0000 7fdeeb7fe700 1 -- 172.21.15.183:0/4006337034 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fdef8004290 con 0x7fdf040fe0e0 2024-02-20T07:54:03.721 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:54:03.718+0000 7fdeeb7fe700 1 -- 172.21.15.183:0/4006337034 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdef801c630 con 0x7fdf040fe0e0 2024-02-20T07:54:03.721 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:54:03.718+0000 7fdf096df700 1 -- 172.21.15.183:0/4006337034 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdf0410f490 con 0x7fdf040fe0e0 2024-02-20T07:54:03.721 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:54:03.719+0000 7fdf096df700 1 -- 172.21.15.183:0/4006337034 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdf0404bed0 con 0x7fdf040fe0e0 2024-02-20T07:54:03.722 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:54:03.719+0000 7fdeeb7fe700 1 -- 172.21.15.183:0/4006337034 <== mon.1 v2:172.21.15.183:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7fdef8013980 con 0x7fdf040fe0e0 2024-02-20T07:54:03.723 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:54:03.720+0000 7fdeeb7fe700 1 --2- 172.21.15.183:0/4006337034 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7fdeec0692a0 0x7fdeec06b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:54:03.723 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:54:03.720+0000 7fdeeb7fe700 1 -- 172.21.15.183:0/4006337034 <== mon.1 v2:172.21.15.183:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fdef8088140 con 0x7fdf040fe0e0 2024-02-20T07:54:03.723 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:54:03.720+0000 7fdf02ffd700 1 --2- 172.21.15.183:0/4006337034 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7fdeec0692a0 0x7fdeec06b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:54:03.723 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:54:03.720+0000 7fdf02ffd700 1 --2- 172.21.15.183:0/4006337034 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7fdeec0692a0 0x7fdeec06b750 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fdef4009fd0 tx=0x7fdef400b040 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:54:03.725 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:54:03.724+0000 7fdeeb7fe700 1 -- 172.21.15.183:0/4006337034 <== mon.1 v2:172.21.15.183:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7fdef8058240 con 0x7fdf040fe0e0 2024-02-20T07:54:03.980 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:54:03.979+0000 7fdf096df700 1 -- 172.21.15.183:0/4006337034 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_command({"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} v 0) v1 -- 0x7fdf04105a30 con 0x7fdf040fe0e0 2024-02-20T07:54:03.984 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:54:03.983+0000 7fdeeb7fe700 1 -- 172.21.15.183:0/4006337034 <== mon.1 v2:172.21.15.183:3300/0 7 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]=0 v15) v1 ==== 170+0+59 (secure 0 0 0) 0x7fdef8057dd0 con 0x7fdf040fe0e0 2024-02-20T07:54:03.984 INFO:teuthology.orchestra.run.smithi183.stdout:[client.1] 2024-02-20T07:54:03.984 INFO:teuthology.orchestra.run.smithi183.stdout: key = AQCbWtRlogJ0OhAAF620sm5Bz3/32NKHRbNrkA== 2024-02-20T07:54:03.986 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:54:03.985+0000 7fdf096df700 1 -- 172.21.15.183:0/4006337034 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7fdeec0692a0 msgr2=0x7fdeec06b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:54:03.986 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:54:03.985+0000 7fdf096df700 1 --2- 172.21.15.183:0/4006337034 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7fdeec0692a0 0x7fdeec06b750 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fdef4009fd0 tx=0x7fdef400b040 comp rx=0 tx=0).stop 2024-02-20T07:54:03.986 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:54:03.985+0000 7fdf096df700 1 -- 172.21.15.183:0/4006337034 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fdf040fe0e0 msgr2=0x7fdf0410de30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:54:03.986 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:54:03.985+0000 7fdf096df700 1 --2- 172.21.15.183:0/4006337034 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fdf040fe0e0 0x7fdf0410de30 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fdef800b370 tx=0x7fdef8009e30 comp rx=0 tx=0).stop 2024-02-20T07:54:03.986 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:54:03.985+0000 7fdf096df700 1 -- 172.21.15.183:0/4006337034 shutdown_connections 2024-02-20T07:54:03.987 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:54:03.985+0000 7fdf096df700 1 --2- 172.21.15.183:0/4006337034 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fdf040fcef0 0x7fdf0410d8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:54:03.987 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:54:03.985+0000 7fdf096df700 1 --2- 172.21.15.183:0/4006337034 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7fdeec0692a0 0x7fdeec06b750 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:54:03.987 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:54:03.985+0000 7fdf096df700 1 --2- 172.21.15.183:0/4006337034 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fdf040fe0e0 0x7fdf0410de30 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:54:03.987 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:54:03.985+0000 7fdf096df700 1 -- 172.21.15.183:0/4006337034 >> 172.21.15.183:0/4006337034 conn(0x7fdf040f8620 msgr2=0x7fdf041013a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:54:03.987 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:54:03.985+0000 7fdf096df700 1 -- 172.21.15.183:0/4006337034 shutdown_connections 2024-02-20T07:54:03.987 INFO:teuthology.orchestra.run.smithi183.stderr:2024-02-20T07:54:03.985+0000 7fdf096df700 1 -- 172.21.15.183:0/4006337034 wait complete. 2024-02-20T07:54:04.584 DEBUG:teuthology.orchestra.run.smithi183:> set -ex 2024-02-20T07:54:04.584 DEBUG:teuthology.orchestra.run.smithi183:> sudo dd of=/etc/ceph/ceph.client.1.keyring 2024-02-20T07:54:04.584 DEBUG:teuthology.orchestra.run.smithi183:> sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-02-20T07:54:04.628 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-02-20T07:54:04.629 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-02-20T07:54:04.629 DEBUG:teuthology.orchestra.run.smithi133:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 4537327c-cfc4-11ee-95bb-87774f69a715 -- ceph mgr dump --format=json 2024-02-20T07:54:04.892 INFO:teuthology.orchestra.run.smithi133.stderr:Inferring config /var/lib/ceph/4537327c-cfc4-11ee-95bb-87774f69a715/mon.smithi133/config 2024-02-20T07:54:05.232 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:54:04 smithi133 ceph-mon[29117]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-20T07:54:05.232 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:54:04 smithi133 ceph-mon[29117]: from='client.? 172.21.15.183:0/4006337034' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-02-20T07:54:05.232 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:54:04 smithi133 ceph-mon[29117]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-02-20T07:54:05.232 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:54:04 smithi133 ceph-mon[29117]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-02-20T07:54:05.297 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:54:04 smithi183 ceph-mon[35832]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-20T07:54:05.297 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:54:04 smithi183 ceph-mon[35832]: from='client.? 172.21.15.183:0/4006337034' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-02-20T07:54:05.297 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:54:04 smithi183 ceph-mon[35832]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-02-20T07:54:05.298 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:54:04 smithi183 ceph-mon[35832]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-02-20T07:54:06.574 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:06.570+0000 7fa8ba76b700 1 -- 172.21.15.133:0/1336888651 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa8b4102f50 msgr2=0x7fa8b4103320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:54:06.574 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:06.570+0000 7fa8ba76b700 1 --2- 172.21.15.133:0/1336888651 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa8b4102f50 0x7fa8b4103320 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fa8a4004940 tx=0x7fa8a4009e30 comp rx=0 tx=0).stop 2024-02-20T07:54:06.574 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:06.570+0000 7fa8ba76b700 1 -- 172.21.15.133:0/1336888651 shutdown_connections 2024-02-20T07:54:06.574 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:06.570+0000 7fa8ba76b700 1 --2- 172.21.15.133:0/1336888651 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa8b41038f0 0x7fa8b40fcf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:54:06.574 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:06.570+0000 7fa8ba76b700 1 --2- 172.21.15.133:0/1336888651 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa8b4102f50 0x7fa8b4103320 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:54:06.574 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:06.570+0000 7fa8ba76b700 1 -- 172.21.15.133:0/1336888651 >> 172.21.15.133:0/1336888651 conn(0x7fa8b40f8ae0 msgr2=0x7fa8b40faef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:54:06.574 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:06.570+0000 7fa8ba76b700 1 -- 172.21.15.133:0/1336888651 shutdown_connections 2024-02-20T07:54:06.574 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:06.570+0000 7fa8ba76b700 1 -- 172.21.15.133:0/1336888651 wait complete. 2024-02-20T07:54:06.574 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:06.571+0000 7fa8ba76b700 1 Processor -- start 2024-02-20T07:54:06.574 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:06.571+0000 7fa8ba76b700 1 -- start start 2024-02-20T07:54:06.574 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:06.571+0000 7fa8ba76b700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa8b4102f50 0x7fa8b4113a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:54:06.574 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:06.571+0000 7fa8ba76b700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa8b41038f0 0x7fa8b4113f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:54:06.575 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:06.571+0000 7fa8ba76b700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8b4114670 con 0x7fa8b41038f0 2024-02-20T07:54:06.575 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:06.571+0000 7fa8ba76b700 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8b410d910 con 0x7fa8b4102f50 2024-02-20T07:54:06.575 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:06.571+0000 7fa8b37fe700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa8b41038f0 0x7fa8b4113f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:54:06.575 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:06.571+0000 7fa8b37fe700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa8b41038f0 0x7fa8b4113f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.133:59472/0 (socket says 172.21.15.133:59472) 2024-02-20T07:54:06.575 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:06.571+0000 7fa8b37fe700 1 -- 172.21.15.133:0/2934595865 learned_addr learned my addr 172.21.15.133:0/2934595865 (peer_addr_for_me v2:172.21.15.133:0/0) 2024-02-20T07:54:06.575 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:06.571+0000 7fa8b3fff700 1 --2- 172.21.15.133:0/2934595865 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa8b4102f50 0x7fa8b4113a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:54:06.575 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:06.571+0000 7fa8b37fe700 1 -- 172.21.15.133:0/2934595865 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa8b4102f50 msgr2=0x7fa8b4113a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:54:06.575 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:06.571+0000 7fa8b37fe700 1 --2- 172.21.15.133:0/2934595865 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa8b4102f50 0x7fa8b4113a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:54:06.575 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:06.571+0000 7fa8b37fe700 1 -- 172.21.15.133:0/2934595865 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8a4005040 con 0x7fa8b41038f0 2024-02-20T07:54:06.575 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:06.572+0000 7fa8b37fe700 1 --2- 172.21.15.133:0/2934595865 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa8b41038f0 0x7fa8b4113f90 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fa8a800b790 tx=0x7fa8a8009220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:54:06.575 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:06.572+0000 7fa8b0ff9700 1 -- 172.21.15.133:0/2934595865 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa8a800f6f0 con 0x7fa8b41038f0 2024-02-20T07:54:06.575 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:06.572+0000 7fa8b0ff9700 1 -- 172.21.15.133:0/2934595865 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa8a800fd20 con 0x7fa8b41038f0 2024-02-20T07:54:06.575 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:06.572+0000 7fa8ba76b700 1 -- 172.21.15.133:0/2934595865 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa8b410dbf0 con 0x7fa8b41038f0 2024-02-20T07:54:06.575 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:06.572+0000 7fa8b0ff9700 1 -- 172.21.15.133:0/2934595865 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa8a80103a0 con 0x7fa8b41038f0 2024-02-20T07:54:06.575 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:06.572+0000 7fa8ba76b700 1 -- 172.21.15.133:0/2934595865 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa8b410e130 con 0x7fa8b41038f0 2024-02-20T07:54:06.576 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:06.573+0000 7fa8ba76b700 1 -- 172.21.15.133:0/2934595865 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa8b40fc6e0 con 0x7fa8b41038f0 2024-02-20T07:54:06.576 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:06.573+0000 7fa8b0ff9700 1 -- 172.21.15.133:0/2934595865 <== mon.0 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7fa8a8010500 con 0x7fa8b41038f0 2024-02-20T07:54:06.577 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:06.573+0000 7fa8b0ff9700 1 --2- 172.21.15.133:0/2934595865 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7fa89c0692a0 0x7fa89c06b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:54:06.577 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:06.573+0000 7fa8b0ff9700 1 -- 172.21.15.133:0/2934595865 <== mon.0 v2:172.21.15.133:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fa8a8086cc0 con 0x7fa8b41038f0 2024-02-20T07:54:06.577 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:06.574+0000 7fa8b3fff700 1 --2- 172.21.15.133:0/2934595865 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7fa89c0692a0 0x7fa89c06b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:54:06.577 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:06.574+0000 7fa8b3fff700 1 --2- 172.21.15.133:0/2934595865 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7fa89c0692a0 0x7fa89c06b750 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fa8a4004f90 tx=0x7fa8a4009e30 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:54:06.579 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:06.578+0000 7fa8b0ff9700 1 -- 172.21.15.133:0/2934595865 <== mon.0 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7fa8a8056db0 con 0x7fa8b41038f0 2024-02-20T07:54:06.816 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:06.814+0000 7fa8ba76b700 1 -- 172.21.15.133:0/2934595865 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) v1 -- 0x7fa8b410f030 con 0x7fa8b41038f0 2024-02-20T07:54:06.819 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:06.817+0000 7fa8b0ff9700 1 -- 172.21.15.133:0/2934595865 <== mon.0 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v19) v1 ==== 74+0+167179 (secure 0 0 0) 0x7fa8a8056940 con 0x7fa8b41038f0 2024-02-20T07:54:06.821 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-02-20T07:54:06.823 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:06.819+0000 7fa8ba76b700 1 -- 172.21.15.133:0/2934595865 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7fa89c0692a0 msgr2=0x7fa89c06b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:54:06.823 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:06.819+0000 7fa8ba76b700 1 --2- 172.21.15.133:0/2934595865 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7fa89c0692a0 0x7fa89c06b750 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fa8a4004f90 tx=0x7fa8a4009e30 comp rx=0 tx=0).stop 2024-02-20T07:54:06.823 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:06.819+0000 7fa8ba76b700 1 -- 172.21.15.133:0/2934595865 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa8b41038f0 msgr2=0x7fa8b4113f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:54:06.823 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:06.819+0000 7fa8ba76b700 1 --2- 172.21.15.133:0/2934595865 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa8b41038f0 0x7fa8b4113f90 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fa8a800b790 tx=0x7fa8a8009220 comp rx=0 tx=0).stop 2024-02-20T07:54:06.823 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:06.819+0000 7fa8ba76b700 1 -- 172.21.15.133:0/2934595865 shutdown_connections 2024-02-20T07:54:06.823 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:06.819+0000 7fa8ba76b700 1 --2- 172.21.15.133:0/2934595865 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7fa89c0692a0 0x7fa89c06b750 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:54:06.823 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:06.819+0000 7fa8ba76b700 1 --2- 172.21.15.133:0/2934595865 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa8b4102f50 0x7fa8b4113a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:54:06.823 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:06.819+0000 7fa8ba76b700 1 --2- 172.21.15.133:0/2934595865 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa8b41038f0 0x7fa8b4113f90 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:54:06.823 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:06.819+0000 7fa8ba76b700 1 -- 172.21.15.133:0/2934595865 >> 172.21.15.133:0/2934595865 conn(0x7fa8b40f8ae0 msgr2=0x7fa8b40fa4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:54:06.823 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:06.820+0000 7fa8ba76b700 1 -- 172.21.15.133:0/2934595865 shutdown_connections 2024-02-20T07:54:06.823 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:06.820+0000 7fa8ba76b700 1 -- 172.21.15.133:0/2934595865 wait complete. 2024-02-20T07:54:07.177 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:54:06 smithi133 ceph-mon[29117]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-20T07:54:07.177 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:54:06 smithi133 ceph-mon[29117]: from='client.? 172.21.15.133:0/2934595865' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-02-20T07:54:07.297 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:54:06 smithi183 ceph-mon[35832]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-20T07:54:07.337 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:54:06 smithi183 ceph-mon[35832]: from='client.? 172.21.15.133:0/2934595865' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-02-20T07:54:07.594 INFO:teuthology.orchestra.run.smithi133.stdout:{"epoch":19,"active_gid":14208,"active_name":"smithi133.mrvcbp","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6800","nonce":2703586938},{"type":"v1","addr":"172.21.15.133:6801","nonce":2703586938}]},"active_addr":"172.21.15.133:6801/2703586938","active_change":"2024-02-20T07:50:23.523778+0000","active_mgr_features":4540138320759226367,"available":true,"standbys":[{"gid":14230,"name":"smithi183.hebfng","mgr_features":4540138320759226367,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.1.5","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","prometheus","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.1.5","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.133:8443/","prometheus":"http://172.21.15.133:9283/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":5,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.133:0","nonce":2052331273}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.133:0","nonce":871037013}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.133:0","nonce":3135365287}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.133:0","nonce":3532484120}]}]} 2024-02-20T07:54:07.598 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-02-20T07:54:07.598 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-02-20T07:54:07.598 DEBUG:teuthology.orchestra.run.smithi133:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 4537327c-cfc4-11ee-95bb-87774f69a715 -- ceph osd dump --format=json 2024-02-20T07:54:07.854 INFO:teuthology.orchestra.run.smithi133.stderr:Inferring config /var/lib/ceph/4537327c-cfc4-11ee-95bb-87774f69a715/mon.smithi133/config 2024-02-20T07:54:09.215 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:54:08 smithi133 ceph-mon[29117]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-20T07:54:09.216 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:54:08 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-20T07:54:09.297 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:54:08 smithi183 ceph-mon[35832]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-20T07:54:09.297 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:54:08 smithi183 ceph-mon[35832]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-20T07:54:09.545 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:09.542+0000 7f75cf8a7700 1 -- 172.21.15.133:0/1107905312 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f75c80fe8f0 msgr2=0x7f75c8106fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:54:09.546 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:09.542+0000 7f75cf8a7700 1 --2- 172.21.15.133:0/1107905312 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f75c80fe8f0 0x7f75c8106fc0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f75b8004940 tx=0x7f75b8009e30 comp rx=0 tx=0).stop 2024-02-20T07:54:09.546 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:09.542+0000 7f75cf8a7700 1 -- 172.21.15.133:0/1107905312 shutdown_connections 2024-02-20T07:54:09.546 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:09.542+0000 7f75cf8a7700 1 --2- 172.21.15.133:0/1107905312 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f75c80fe8f0 0x7f75c8106fc0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:54:09.546 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:09.542+0000 7f75cf8a7700 1 --2- 172.21.15.133:0/1107905312 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f75c80fdf50 0x7f75c80fe320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:54:09.546 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:09.542+0000 7f75cf8a7700 1 -- 172.21.15.133:0/1107905312 >> 172.21.15.133:0/1107905312 conn(0x7f75c80f7a10 msgr2=0x7f75c80f9e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:54:09.546 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:09.542+0000 7f75cf8a7700 1 -- 172.21.15.133:0/1107905312 shutdown_connections 2024-02-20T07:54:09.546 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:09.542+0000 7f75cf8a7700 1 -- 172.21.15.133:0/1107905312 wait complete. 2024-02-20T07:54:09.546 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:09.542+0000 7f75cf8a7700 1 Processor -- start 2024-02-20T07:54:09.546 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:09.542+0000 7f75cf8a7700 1 -- start start 2024-02-20T07:54:09.546 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:09.543+0000 7f75cf8a7700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f75c80fdf50 0x7f75c80fbe10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:54:09.546 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:09.543+0000 7f75cf8a7700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f75c80fe8f0 0x7f75c80fc350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:54:09.546 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:09.543+0000 7f75cf8a7700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75c80fc9e0 con 0x7f75c80fe8f0 2024-02-20T07:54:09.546 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:09.543+0000 7f75cf8a7700 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75c80fcb50 con 0x7f75c80fdf50 2024-02-20T07:54:09.547 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:09.543+0000 7f75cce42700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f75c80fe8f0 0x7f75c80fc350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:54:09.547 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:09.543+0000 7f75cce42700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f75c80fe8f0 0x7f75c80fc350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.133:59500/0 (socket says 172.21.15.133:59500) 2024-02-20T07:54:09.547 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:09.543+0000 7f75cce42700 1 -- 172.21.15.133:0/4131093408 learned_addr learned my addr 172.21.15.133:0/4131093408 (peer_addr_for_me v2:172.21.15.133:0/0) 2024-02-20T07:54:09.547 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:09.543+0000 7f75cce42700 1 -- 172.21.15.133:0/4131093408 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f75c80fdf50 msgr2=0x7f75c80fbe10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:54:09.547 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:09.543+0000 7f75cd643700 1 --2- 172.21.15.133:0/4131093408 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f75c80fdf50 0x7f75c80fbe10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:54:09.547 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:09.543+0000 7f75cce42700 1 --2- 172.21.15.133:0/4131093408 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f75c80fdf50 0x7f75c80fbe10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:54:09.547 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:09.543+0000 7f75cce42700 1 -- 172.21.15.133:0/4131093408 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75b8005040 con 0x7f75c80fe8f0 2024-02-20T07:54:09.547 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:09.543+0000 7f75cd643700 1 --2- 172.21.15.133:0/4131093408 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f75c80fdf50 0x7f75c80fbe10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-20T07:54:09.547 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:09.543+0000 7f75cce42700 1 --2- 172.21.15.133:0/4131093408 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f75c80fe8f0 0x7f75c80fc350 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f75b8004890 tx=0x7f75b8009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:54:09.547 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:09.543+0000 7f75bdffb700 1 -- 172.21.15.133:0/4131093408 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f75b800e040 con 0x7f75c80fe8f0 2024-02-20T07:54:09.547 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:09.543+0000 7f75bdffb700 1 -- 172.21.15.133:0/4131093408 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f75b8004290 con 0x7f75c80fe8f0 2024-02-20T07:54:09.547 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:09.543+0000 7f75cf8a7700 1 -- 172.21.15.133:0/4131093408 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f75c8119d80 con 0x7f75c80fe8f0 2024-02-20T07:54:09.547 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:09.543+0000 7f75bdffb700 1 -- 172.21.15.133:0/4131093408 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f75b801c630 con 0x7f75c80fe8f0 2024-02-20T07:54:09.547 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:09.544+0000 7f75cf8a7700 1 -- 172.21.15.133:0/4131093408 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f75c811a270 con 0x7f75c80fe8f0 2024-02-20T07:54:09.548 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:09.545+0000 7f75cf8a7700 1 -- 172.21.15.133:0/4131093408 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f75c8104620 con 0x7f75c80fe8f0 2024-02-20T07:54:09.548 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:09.545+0000 7f75bdffb700 1 -- 172.21.15.133:0/4131093408 <== mon.0 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7f75b800bbf0 con 0x7f75c80fe8f0 2024-02-20T07:54:09.548 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:09.545+0000 7f75bdffb700 1 --2- 172.21.15.133:0/4131093408 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7f75b40692a0 0x7f75b406b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:54:09.549 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:09.546+0000 7f75bdffb700 1 -- 172.21.15.133:0/4131093408 <== mon.0 v2:172.21.15.133:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f75b80882e0 con 0x7f75c80fe8f0 2024-02-20T07:54:09.549 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:09.546+0000 7f75cd643700 1 --2- 172.21.15.133:0/4131093408 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7f75b40692a0 0x7f75b406b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:54:09.549 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:09.546+0000 7f75cd643700 1 --2- 172.21.15.133:0/4131093408 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7f75b40692a0 0x7f75b406b750 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f75c4006fd0 tx=0x7f75c4008040 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:54:09.551 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:09.550+0000 7f75bdffb700 1 -- 172.21.15.133:0/4131093408 <== mon.0 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f75b8054a00 con 0x7f75c80fe8f0 2024-02-20T07:54:09.734 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:09.732+0000 7f75cf8a7700 1 -- 172.21.15.133:0/4131093408 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f75c804bed0 con 0x7f75c80fe8f0 2024-02-20T07:54:09.734 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:09.733+0000 7f75bdffb700 1 -- 172.21.15.133:0/4131093408 <== mon.0 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v40) v1 ==== 74+0+11111 (secure 0 0 0) 0x7f75b8058020 con 0x7f75c80fe8f0 2024-02-20T07:54:09.735 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-02-20T07:54:09.735 INFO:teuthology.orchestra.run.smithi133.stdout:{"epoch":40,"fsid":"4537327c-cfc4-11ee-95bb-87774f69a715","created":"2024-02-20T07:48:13.550549+0000","modified":"2024-02-20T07:53:50.692077+0000","last_up_change":"2024-02-20T07:53:49.684637+0000","last_in_change":"2024-02-20T07:53:30.651509+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":14,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-20T07:52:35.839244+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"0a20b241-99e8-4e0b-ba5e-be07c8fdff94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6802","nonce":261892236},{"type":"v1","addr":"172.21.15.133:6803","nonce":261892236}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6804","nonce":261892236},{"type":"v1","addr":"172.21.15.133:6805","nonce":261892236}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6808","nonce":261892236},{"type":"v1","addr":"172.21.15.133:6809","nonce":261892236}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6806","nonce":261892236},{"type":"v1","addr":"172.21.15.133:6807","nonce":261892236}]},"public_addr":"172.21.15.133:6803/261892236","cluster_addr":"172.21.15.133:6805/261892236","heartbeat_back_addr":"172.21.15.133:6809/261892236","heartbeat_front_addr":"172.21.15.133:6807/261892236","state":["exists","up"]},{"osd":1,"uuid":"2b1b8e8f-74b4-4d41-8753-502296eb87df","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6810","nonce":1652695599},{"type":"v1","addr":"172.21.15.133:6811","nonce":1652695599}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6812","nonce":1652695599},{"type":"v1","addr":"172.21.15.133:6813","nonce":1652695599}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6816","nonce":1652695599},{"type":"v1","addr":"172.21.15.133:6817","nonce":1652695599}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6814","nonce":1652695599},{"type":"v1","addr":"172.21.15.133:6815","nonce":1652695599}]},"public_addr":"172.21.15.133:6811/1652695599","cluster_addr":"172.21.15.133:6813/1652695599","heartbeat_back_addr":"172.21.15.133:6817/1652695599","heartbeat_front_addr":"172.21.15.133:6815/1652695599","state":["exists","up"]},{"osd":2,"uuid":"95848108-ed6d-4030-9534-d687a769350f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6818","nonce":3838005412},{"type":"v1","addr":"172.21.15.133:6819","nonce":3838005412}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6820","nonce":3838005412},{"type":"v1","addr":"172.21.15.133:6821","nonce":3838005412}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6824","nonce":3838005412},{"type":"v1","addr":"172.21.15.133:6825","nonce":3838005412}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6822","nonce":3838005412},{"type":"v1","addr":"172.21.15.133:6823","nonce":3838005412}]},"public_addr":"172.21.15.133:6819/3838005412","cluster_addr":"172.21.15.133:6821/3838005412","heartbeat_back_addr":"172.21.15.133:6825/3838005412","heartbeat_front_addr":"172.21.15.133:6823/3838005412","state":["exists","up"]},{"osd":3,"uuid":"c2015ce7-f7fb-428d-b42f-e5fb665933e0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6800","nonce":1053980618},{"type":"v1","addr":"172.21.15.183:6801","nonce":1053980618}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6802","nonce":1053980618},{"type":"v1","addr":"172.21.15.183:6803","nonce":1053980618}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6806","nonce":1053980618},{"type":"v1","addr":"172.21.15.183:6807","nonce":1053980618}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6804","nonce":1053980618},{"type":"v1","addr":"172.21.15.183:6805","nonce":1053980618}]},"public_addr":"172.21.15.183:6801/1053980618","cluster_addr":"172.21.15.183:6803/1053980618","heartbeat_back_addr":"172.21.15.183:6807/1053980618","heartbeat_front_addr":"172.21.15.183:6805/1053980618","state":["exists","up"]},{"osd":4,"uuid":"55148881-b92f-40e6-bfe8-b8320406c96a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6808","nonce":3697465174},{"type":"v1","addr":"172.21.15.183:6809","nonce":3697465174}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6810","nonce":3697465174},{"type":"v1","addr":"172.21.15.183:6811","nonce":3697465174}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6814","nonce":3697465174},{"type":"v1","addr":"172.21.15.183:6815","nonce":3697465174}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6812","nonce":3697465174},{"type":"v1","addr":"172.21.15.183:6813","nonce":3697465174}]},"public_addr":"172.21.15.183:6809/3697465174","cluster_addr":"172.21.15.183:6811/3697465174","heartbeat_back_addr":"172.21.15.183:6815/3697465174","heartbeat_front_addr":"172.21.15.183:6813/3697465174","state":["exists","up"]},{"osd":5,"uuid":"ee5510f9-fab7-4aa3-8372-3856a0787e92","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6816","nonce":81997900},{"type":"v1","addr":"172.21.15.183:6817","nonce":81997900}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6818","nonce":81997900},{"type":"v1","addr":"172.21.15.183:6819","nonce":81997900}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6822","nonce":81997900},{"type":"v1","addr":"172.21.15.183:6823","nonce":81997900}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6820","nonce":81997900},{"type":"v1","addr":"172.21.15.183:6821","nonce":81997900}]},"public_addr":"172.21.15.183:6817/81997900","cluster_addr":"172.21.15.183:6819/81997900","heartbeat_back_addr":"172.21.15.183:6823/81997900","heartbeat_front_addr":"172.21.15.183:6821/81997900","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-02-20T07:51:38.297291+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-02-20T07:52:05.983942+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-02-20T07:52:33.478124+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-02-20T07:52:56.569795+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-02-20T07:53:22.416562+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-02-20T07:53:47.770454+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.133:6800/3944459195":"2024-02-21T07:50:23.523666+0000","172.21.15.133:0/1714017955":"2024-02-21T07:50:23.523666+0000","172.21.15.133:6801/1670376163":"2024-02-21T07:49:19.684526+0000","172.21.15.133:6800/1670376163":"2024-02-21T07:49:19.684526+0000","172.21.15.133:0/3014285121":"2024-02-21T07:50:23.523666+0000","172.21.15.133:0/3813170035":"2024-02-21T07:49:19.684526+0000","172.21.15.133:0/2942023244":"2024-02-21T07:50:23.523666+0000","172.21.15.133:6800/3480413474":"2024-02-21T07:48:49.130220+0000","172.21.15.133:0/129232921":"2024-02-21T07:48:49.130220+0000","172.21.15.133:6801/3480413474":"2024-02-21T07:48:49.130220+0000","172.21.15.133:0/3354686539":"2024-02-21T07:49:19.684526+0000","172.21.15.133:0/2763668929":"2024-02-21T07:48:49.130220+0000","172.21.15.133:6801/3944459195":"2024-02-21T07:50:23.523666+0000","172.21.15.133:0/2831283357":"2024-02-21T07:49:19.684526+0000","172.21.15.133:0/1481070311":"2024-02-21T07:48:49.130220+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-20T07:54:09.737 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:09.735+0000 7f75cf8a7700 1 -- 172.21.15.133:0/4131093408 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7f75b40692a0 msgr2=0x7f75b406b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:54:09.737 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:09.735+0000 7f75cf8a7700 1 --2- 172.21.15.133:0/4131093408 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7f75b40692a0 0x7f75b406b750 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f75c4006fd0 tx=0x7f75c4008040 comp rx=0 tx=0).stop 2024-02-20T07:54:09.737 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:09.735+0000 7f75cf8a7700 1 -- 172.21.15.133:0/4131093408 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f75c80fe8f0 msgr2=0x7f75c80fc350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:54:09.737 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:09.735+0000 7f75cf8a7700 1 --2- 172.21.15.133:0/4131093408 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f75c80fe8f0 0x7f75c80fc350 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f75b8004890 tx=0x7f75b8009e30 comp rx=0 tx=0).stop 2024-02-20T07:54:09.737 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:09.735+0000 7f75cf8a7700 1 -- 172.21.15.133:0/4131093408 shutdown_connections 2024-02-20T07:54:09.737 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:09.735+0000 7f75cf8a7700 1 --2- 172.21.15.133:0/4131093408 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7f75b40692a0 0x7f75b406b750 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:54:09.737 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:09.735+0000 7f75cf8a7700 1 --2- 172.21.15.133:0/4131093408 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f75c80fdf50 0x7f75c80fbe10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:54:09.737 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:09.735+0000 7f75cf8a7700 1 --2- 172.21.15.133:0/4131093408 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f75c80fe8f0 0x7f75c80fc350 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:54:09.738 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:09.735+0000 7f75cf8a7700 1 -- 172.21.15.133:0/4131093408 >> 172.21.15.133:0/4131093408 conn(0x7f75c80f7a10 msgr2=0x7f75c81017a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:54:09.738 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:09.735+0000 7f75cf8a7700 1 -- 172.21.15.133:0/4131093408 shutdown_connections 2024-02-20T07:54:09.738 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:09.735+0000 7f75cf8a7700 1 -- 172.21.15.133:0/4131093408 wait complete. 2024-02-20T07:54:10.241 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:54:09 smithi133 ceph-mon[29117]: from='client.? 172.21.15.133:0/4131093408' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-20T07:54:10.297 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:54:09 smithi183 ceph-mon[35832]: from='client.? 172.21.15.133:0/4131093408' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-20T07:54:10.486 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-02-20T07:54:10.486 DEBUG:teuthology.orchestra.run.smithi133:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 4537327c-cfc4-11ee-95bb-87774f69a715 -- ceph osd dump --format=json 2024-02-20T07:54:10.742 INFO:teuthology.orchestra.run.smithi133.stderr:Inferring config /var/lib/ceph/4537327c-cfc4-11ee-95bb-87774f69a715/mon.smithi133/config 2024-02-20T07:54:11.185 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:54:10 smithi133 ceph-mon[29117]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-20T07:54:11.297 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:54:10 smithi183 ceph-mon[35832]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-20T07:54:12.525 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:12.522+0000 7fa9fb7b0700 1 -- 172.21.15.133:0/946180474 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa9f40fb570 msgr2=0x7fa9f40fb9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:54:12.526 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:12.522+0000 7fa9fb7b0700 1 --2- 172.21.15.133:0/946180474 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa9f40fb570 0x7fa9f40fb9c0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fa9e8004940 tx=0x7fa9e8009e30 comp rx=0 tx=0).stop 2024-02-20T07:54:12.526 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:12.522+0000 7fa9fb7b0700 1 -- 172.21.15.133:0/946180474 shutdown_connections 2024-02-20T07:54:12.526 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:12.522+0000 7fa9fb7b0700 1 --2- 172.21.15.133:0/946180474 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa9f40fb570 0x7fa9f40fb9c0 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:54:12.526 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:12.522+0000 7fa9fb7b0700 1 --2- 172.21.15.133:0/946180474 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa9f4103140 0x7fa9f4103510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:54:12.526 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:12.522+0000 7fa9fb7b0700 1 -- 172.21.15.133:0/946180474 >> 172.21.15.133:0/946180474 conn(0x7fa9f4071560 msgr2=0x7fa9f4071960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:54:12.526 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:12.522+0000 7fa9fb7b0700 1 -- 172.21.15.133:0/946180474 shutdown_connections 2024-02-20T07:54:12.526 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:12.522+0000 7fa9fb7b0700 1 -- 172.21.15.133:0/946180474 wait complete. 2024-02-20T07:54:12.526 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:12.522+0000 7fa9fb7b0700 1 Processor -- start 2024-02-20T07:54:12.526 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:12.522+0000 7fa9fb7b0700 1 -- start start 2024-02-20T07:54:12.526 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:12.523+0000 7fa9fb7b0700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa9f40fb570 0x7fa9f40fc750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:54:12.526 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:12.523+0000 7fa9fb7b0700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa9f4103140 0x7fa9f40fcc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:54:12.526 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:12.523+0000 7fa9fb7b0700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9f40fd290 con 0x7fa9f4103140 2024-02-20T07:54:12.527 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:12.523+0000 7fa9fb7b0700 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9f40fd400 con 0x7fa9f40fb570 2024-02-20T07:54:12.527 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:12.523+0000 7fa9f8d4b700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa9f4103140 0x7fa9f40fcc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:54:12.527 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:12.523+0000 7fa9f8d4b700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa9f4103140 0x7fa9f40fcc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.133:59518/0 (socket says 172.21.15.133:59518) 2024-02-20T07:54:12.527 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:12.523+0000 7fa9f8d4b700 1 -- 172.21.15.133:0/714993417 learned_addr learned my addr 172.21.15.133:0/714993417 (peer_addr_for_me v2:172.21.15.133:0/0) 2024-02-20T07:54:12.527 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:12.523+0000 7fa9f954c700 1 --2- 172.21.15.133:0/714993417 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa9f40fb570 0x7fa9f40fc750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:54:12.527 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:12.523+0000 7fa9f8d4b700 1 -- 172.21.15.133:0/714993417 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa9f40fb570 msgr2=0x7fa9f40fc750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:54:12.527 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:12.523+0000 7fa9f8d4b700 1 --2- 172.21.15.133:0/714993417 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa9f40fb570 0x7fa9f40fc750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:54:12.527 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:12.523+0000 7fa9f8d4b700 1 -- 172.21.15.133:0/714993417 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9e8005040 con 0x7fa9f4103140 2024-02-20T07:54:12.527 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:12.523+0000 7fa9f954c700 1 --2- 172.21.15.133:0/714993417 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa9f40fb570 0x7fa9f40fc750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-20T07:54:12.527 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:12.523+0000 7fa9f8d4b700 1 --2- 172.21.15.133:0/714993417 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa9f4103140 0x7fa9f40fcc90 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fa9e8000c00 tx=0x7fa9e8009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:54:12.527 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:12.523+0000 7fa9f1ffb700 1 -- 172.21.15.133:0/714993417 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa9e800e040 con 0x7fa9f4103140 2024-02-20T07:54:12.527 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:12.523+0000 7fa9fb7b0700 1 -- 172.21.15.133:0/714993417 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa9f4119dc0 con 0x7fa9f4103140 2024-02-20T07:54:12.527 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:12.523+0000 7fa9f1ffb700 1 -- 172.21.15.133:0/714993417 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa9e800bbf0 con 0x7fa9f4103140 2024-02-20T07:54:12.527 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:12.523+0000 7fa9f1ffb700 1 -- 172.21.15.133:0/714993417 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa9e801c630 con 0x7fa9f4103140 2024-02-20T07:54:12.527 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:12.524+0000 7fa9fb7b0700 1 -- 172.21.15.133:0/714993417 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa9f411a220 con 0x7fa9f4103140 2024-02-20T07:54:12.528 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:12.524+0000 7fa9fb7b0700 1 -- 172.21.15.133:0/714993417 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa9f41054d0 con 0x7fa9f4103140 2024-02-20T07:54:12.528 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:12.525+0000 7fa9f1ffb700 1 -- 172.21.15.133:0/714993417 <== mon.0 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7fa9e8013980 con 0x7fa9f4103140 2024-02-20T07:54:12.528 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:12.525+0000 7fa9f1ffb700 1 --2- 172.21.15.133:0/714993417 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7fa9dc0692a0 0x7fa9dc06b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:54:12.529 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:12.525+0000 7fa9f1ffb700 1 -- 172.21.15.133:0/714993417 <== mon.0 v2:172.21.15.133:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fa9e8088210 con 0x7fa9f4103140 2024-02-20T07:54:12.529 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:12.525+0000 7fa9f954c700 1 --2- 172.21.15.133:0/714993417 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7fa9dc0692a0 0x7fa9dc06b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:54:12.529 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:12.526+0000 7fa9f954c700 1 --2- 172.21.15.133:0/714993417 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7fa9dc0692a0 0x7fa9dc06b750 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fa9ec004550 tx=0x7fa9ec009040 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:54:12.531 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:12.530+0000 7fa9f1ffb700 1 -- 172.21.15.133:0/714993417 <== mon.0 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7fa9e8054820 con 0x7fa9f4103140 2024-02-20T07:54:12.714 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:12.712+0000 7fa9fb7b0700 1 -- 172.21.15.133:0/714993417 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa9f404bed0 con 0x7fa9f4103140 2024-02-20T07:54:12.715 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:12.713+0000 7fa9f1ffb700 1 -- 172.21.15.133:0/714993417 <== mon.0 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v40) v1 ==== 74+0+11111 (secure 0 0 0) 0x7fa9e8057e40 con 0x7fa9f4103140 2024-02-20T07:54:12.715 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-02-20T07:54:12.715 INFO:teuthology.orchestra.run.smithi133.stdout:{"epoch":40,"fsid":"4537327c-cfc4-11ee-95bb-87774f69a715","created":"2024-02-20T07:48:13.550549+0000","modified":"2024-02-20T07:53:50.692077+0000","last_up_change":"2024-02-20T07:53:49.684637+0000","last_in_change":"2024-02-20T07:53:30.651509+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":14,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-20T07:52:35.839244+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"0a20b241-99e8-4e0b-ba5e-be07c8fdff94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6802","nonce":261892236},{"type":"v1","addr":"172.21.15.133:6803","nonce":261892236}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6804","nonce":261892236},{"type":"v1","addr":"172.21.15.133:6805","nonce":261892236}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6808","nonce":261892236},{"type":"v1","addr":"172.21.15.133:6809","nonce":261892236}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6806","nonce":261892236},{"type":"v1","addr":"172.21.15.133:6807","nonce":261892236}]},"public_addr":"172.21.15.133:6803/261892236","cluster_addr":"172.21.15.133:6805/261892236","heartbeat_back_addr":"172.21.15.133:6809/261892236","heartbeat_front_addr":"172.21.15.133:6807/261892236","state":["exists","up"]},{"osd":1,"uuid":"2b1b8e8f-74b4-4d41-8753-502296eb87df","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6810","nonce":1652695599},{"type":"v1","addr":"172.21.15.133:6811","nonce":1652695599}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6812","nonce":1652695599},{"type":"v1","addr":"172.21.15.133:6813","nonce":1652695599}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6816","nonce":1652695599},{"type":"v1","addr":"172.21.15.133:6817","nonce":1652695599}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6814","nonce":1652695599},{"type":"v1","addr":"172.21.15.133:6815","nonce":1652695599}]},"public_addr":"172.21.15.133:6811/1652695599","cluster_addr":"172.21.15.133:6813/1652695599","heartbeat_back_addr":"172.21.15.133:6817/1652695599","heartbeat_front_addr":"172.21.15.133:6815/1652695599","state":["exists","up"]},{"osd":2,"uuid":"95848108-ed6d-4030-9534-d687a769350f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6818","nonce":3838005412},{"type":"v1","addr":"172.21.15.133:6819","nonce":3838005412}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6820","nonce":3838005412},{"type":"v1","addr":"172.21.15.133:6821","nonce":3838005412}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6824","nonce":3838005412},{"type":"v1","addr":"172.21.15.133:6825","nonce":3838005412}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6822","nonce":3838005412},{"type":"v1","addr":"172.21.15.133:6823","nonce":3838005412}]},"public_addr":"172.21.15.133:6819/3838005412","cluster_addr":"172.21.15.133:6821/3838005412","heartbeat_back_addr":"172.21.15.133:6825/3838005412","heartbeat_front_addr":"172.21.15.133:6823/3838005412","state":["exists","up"]},{"osd":3,"uuid":"c2015ce7-f7fb-428d-b42f-e5fb665933e0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6800","nonce":1053980618},{"type":"v1","addr":"172.21.15.183:6801","nonce":1053980618}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6802","nonce":1053980618},{"type":"v1","addr":"172.21.15.183:6803","nonce":1053980618}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6806","nonce":1053980618},{"type":"v1","addr":"172.21.15.183:6807","nonce":1053980618}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6804","nonce":1053980618},{"type":"v1","addr":"172.21.15.183:6805","nonce":1053980618}]},"public_addr":"172.21.15.183:6801/1053980618","cluster_addr":"172.21.15.183:6803/1053980618","heartbeat_back_addr":"172.21.15.183:6807/1053980618","heartbeat_front_addr":"172.21.15.183:6805/1053980618","state":["exists","up"]},{"osd":4,"uuid":"55148881-b92f-40e6-bfe8-b8320406c96a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6808","nonce":3697465174},{"type":"v1","addr":"172.21.15.183:6809","nonce":3697465174}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6810","nonce":3697465174},{"type":"v1","addr":"172.21.15.183:6811","nonce":3697465174}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6814","nonce":3697465174},{"type":"v1","addr":"172.21.15.183:6815","nonce":3697465174}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6812","nonce":3697465174},{"type":"v1","addr":"172.21.15.183:6813","nonce":3697465174}]},"public_addr":"172.21.15.183:6809/3697465174","cluster_addr":"172.21.15.183:6811/3697465174","heartbeat_back_addr":"172.21.15.183:6815/3697465174","heartbeat_front_addr":"172.21.15.183:6813/3697465174","state":["exists","up"]},{"osd":5,"uuid":"ee5510f9-fab7-4aa3-8372-3856a0787e92","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6816","nonce":81997900},{"type":"v1","addr":"172.21.15.183:6817","nonce":81997900}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6818","nonce":81997900},{"type":"v1","addr":"172.21.15.183:6819","nonce":81997900}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6822","nonce":81997900},{"type":"v1","addr":"172.21.15.183:6823","nonce":81997900}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6820","nonce":81997900},{"type":"v1","addr":"172.21.15.183:6821","nonce":81997900}]},"public_addr":"172.21.15.183:6817/81997900","cluster_addr":"172.21.15.183:6819/81997900","heartbeat_back_addr":"172.21.15.183:6823/81997900","heartbeat_front_addr":"172.21.15.183:6821/81997900","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-02-20T07:51:38.297291+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-02-20T07:52:05.983942+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-02-20T07:52:33.478124+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-02-20T07:52:56.569795+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-02-20T07:53:22.416562+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-02-20T07:53:47.770454+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.133:6800/3944459195":"2024-02-21T07:50:23.523666+0000","172.21.15.133:0/1714017955":"2024-02-21T07:50:23.523666+0000","172.21.15.133:6801/1670376163":"2024-02-21T07:49:19.684526+0000","172.21.15.133:6800/1670376163":"2024-02-21T07:49:19.684526+0000","172.21.15.133:0/3014285121":"2024-02-21T07:50:23.523666+0000","172.21.15.133:0/3813170035":"2024-02-21T07:49:19.684526+0000","172.21.15.133:0/2942023244":"2024-02-21T07:50:23.523666+0000","172.21.15.133:6800/3480413474":"2024-02-21T07:48:49.130220+0000","172.21.15.133:0/129232921":"2024-02-21T07:48:49.130220+0000","172.21.15.133:6801/3480413474":"2024-02-21T07:48:49.130220+0000","172.21.15.133:0/3354686539":"2024-02-21T07:49:19.684526+0000","172.21.15.133:0/2763668929":"2024-02-21T07:48:49.130220+0000","172.21.15.133:6801/3944459195":"2024-02-21T07:50:23.523666+0000","172.21.15.133:0/2831283357":"2024-02-21T07:49:19.684526+0000","172.21.15.133:0/1481070311":"2024-02-21T07:48:49.130220+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-20T07:54:12.718 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:12.716+0000 7fa9fb7b0700 1 -- 172.21.15.133:0/714993417 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7fa9dc0692a0 msgr2=0x7fa9dc06b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:54:12.718 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:12.716+0000 7fa9fb7b0700 1 --2- 172.21.15.133:0/714993417 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7fa9dc0692a0 0x7fa9dc06b750 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fa9ec004550 tx=0x7fa9ec009040 comp rx=0 tx=0).stop 2024-02-20T07:54:12.718 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:12.716+0000 7fa9fb7b0700 1 -- 172.21.15.133:0/714993417 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa9f4103140 msgr2=0x7fa9f40fcc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:54:12.718 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:12.716+0000 7fa9fb7b0700 1 --2- 172.21.15.133:0/714993417 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa9f4103140 0x7fa9f40fcc90 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fa9e8000c00 tx=0x7fa9e8009e30 comp rx=0 tx=0).stop 2024-02-20T07:54:12.718 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:12.716+0000 7fa9fb7b0700 1 -- 172.21.15.133:0/714993417 shutdown_connections 2024-02-20T07:54:12.718 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:12.716+0000 7fa9fb7b0700 1 --2- 172.21.15.133:0/714993417 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7fa9dc0692a0 0x7fa9dc06b750 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:54:12.718 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:12.716+0000 7fa9fb7b0700 1 --2- 172.21.15.133:0/714993417 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa9f40fb570 0x7fa9f40fc750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:54:12.718 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:12.716+0000 7fa9fb7b0700 1 --2- 172.21.15.133:0/714993417 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fa9f4103140 0x7fa9f40fcc90 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:54:12.718 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:12.716+0000 7fa9fb7b0700 1 -- 172.21.15.133:0/714993417 >> 172.21.15.133:0/714993417 conn(0x7fa9f4071560 msgr2=0x7fa9f40f97a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:54:12.718 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:12.716+0000 7fa9fb7b0700 1 -- 172.21.15.133:0/714993417 shutdown_connections 2024-02-20T07:54:12.718 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:12.716+0000 7fa9fb7b0700 1 -- 172.21.15.133:0/714993417 wait complete. 2024-02-20T07:54:12.991 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:54:12 smithi133 ceph-mon[29117]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-20T07:54:12.992 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:54:12 smithi133 ceph-mon[29117]: from='client.? 172.21.15.133:0/714993417' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-20T07:54:13.297 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:54:12 smithi183 ceph-mon[35832]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-20T07:54:13.297 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:54:12 smithi183 ceph-mon[35832]: from='client.? 172.21.15.133:0/714993417' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-20T07:54:13.430 DEBUG:teuthology.orchestra.run.smithi133:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 4537327c-cfc4-11ee-95bb-87774f69a715 -- ceph tell osd.0 flush_pg_stats 2024-02-20T07:54:13.431 DEBUG:teuthology.orchestra.run.smithi133:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 4537327c-cfc4-11ee-95bb-87774f69a715 -- ceph tell osd.1 flush_pg_stats 2024-02-20T07:54:13.431 DEBUG:teuthology.orchestra.run.smithi133:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 4537327c-cfc4-11ee-95bb-87774f69a715 -- ceph tell osd.2 flush_pg_stats 2024-02-20T07:54:13.432 DEBUG:teuthology.orchestra.run.smithi133:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 4537327c-cfc4-11ee-95bb-87774f69a715 -- ceph tell osd.3 flush_pg_stats 2024-02-20T07:54:13.432 DEBUG:teuthology.orchestra.run.smithi133:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 4537327c-cfc4-11ee-95bb-87774f69a715 -- ceph tell osd.4 flush_pg_stats 2024-02-20T07:54:13.433 DEBUG:teuthology.orchestra.run.smithi133:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 4537327c-cfc4-11ee-95bb-87774f69a715 -- ceph tell osd.5 flush_pg_stats 2024-02-20T07:54:13.753 INFO:teuthology.orchestra.run.smithi133.stderr:Inferring config /var/lib/ceph/4537327c-cfc4-11ee-95bb-87774f69a715/mon.smithi133/config 2024-02-20T07:54:13.753 INFO:teuthology.orchestra.run.smithi133.stderr:Inferring config /var/lib/ceph/4537327c-cfc4-11ee-95bb-87774f69a715/mon.smithi133/config 2024-02-20T07:54:14.465 INFO:teuthology.orchestra.run.smithi133.stderr:Inferring config /var/lib/ceph/4537327c-cfc4-11ee-95bb-87774f69a715/mon.smithi133/config 2024-02-20T07:54:14.465 INFO:teuthology.orchestra.run.smithi133.stderr:Inferring config /var/lib/ceph/4537327c-cfc4-11ee-95bb-87774f69a715/mon.smithi133/config 2024-02-20T07:54:14.466 INFO:teuthology.orchestra.run.smithi133.stderr:Inferring config /var/lib/ceph/4537327c-cfc4-11ee-95bb-87774f69a715/mon.smithi133/config 2024-02-20T07:54:14.500 INFO:teuthology.orchestra.run.smithi133.stderr:Inferring config /var/lib/ceph/4537327c-cfc4-11ee-95bb-87774f69a715/mon.smithi133/config 2024-02-20T07:54:15.241 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:54:14 smithi133 ceph-mon[29117]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-20T07:54:15.297 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:54:14 smithi183 ceph-mon[35832]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-20T07:54:17.242 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:54:16 smithi133 ceph-mon[29117]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-20T07:54:17.297 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:54:16 smithi183 ceph-mon[35832]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-20T07:54:18.822 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:18.820+0000 7f7dd1977700 1 -- 172.21.15.133:0/4215125000 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f7dcc102800 msgr2=0x7f7dcc0fbe70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:54:18.822 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:18.820+0000 7f7dd1977700 1 --2- 172.21.15.133:0/4215125000 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f7dcc102800 0x7f7dcc0fbe70 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f7dc0004940 tx=0x7f7dc0009e30 comp rx=0 tx=0).stop 2024-02-20T07:54:18.822 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:18.820+0000 7f7dd1977700 1 -- 172.21.15.133:0/4215125000 shutdown_connections 2024-02-20T07:54:18.822 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:18.820+0000 7f7dd1977700 1 --2- 172.21.15.133:0/4215125000 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f7dcc102800 0x7f7dcc0fbe70 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:54:18.822 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:18.820+0000 7f7dd1977700 1 --2- 172.21.15.133:0/4215125000 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f7dcc101e60 0x7f7dcc102230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:54:18.822 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:18.820+0000 7f7dd1977700 1 -- 172.21.15.133:0/4215125000 >> 172.21.15.133:0/4215125000 conn(0x7f7dcc0f7a10 msgr2=0x7f7dcc0f9e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:54:18.822 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:18.820+0000 7f7dd1977700 1 -- 172.21.15.133:0/4215125000 shutdown_connections 2024-02-20T07:54:18.822 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:18.820+0000 7f7dd1977700 1 -- 172.21.15.133:0/4215125000 wait complete. 2024-02-20T07:54:18.824 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:18.821+0000 7f7dd1977700 1 Processor -- start 2024-02-20T07:54:18.824 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:18.821+0000 7f7dd1977700 1 -- start start 2024-02-20T07:54:18.824 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:18.821+0000 7f7dd1977700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f7dcc101e60 0x7f7dcc10eb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:54:18.824 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:18.821+0000 7f7dd1977700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f7dcc102800 0x7f7dcc10f0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:54:18.824 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:18.821+0000 7f7dd1977700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7dcc10f650 con 0x7f7dcc102800 2024-02-20T07:54:18.824 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:18.821+0000 7f7dd1977700 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7dcc10f7c0 con 0x7f7dcc101e60 2024-02-20T07:54:18.825 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:18.821+0000 7f7dca7fc700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f7dcc102800 0x7f7dcc10f0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:54:18.825 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:18.821+0000 7f7dca7fc700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f7dcc102800 0x7f7dcc10f0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.133:48972/0 (socket says 172.21.15.133:48972) 2024-02-20T07:54:18.825 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:18.821+0000 7f7dca7fc700 1 -- 172.21.15.133:0/3018676726 learned_addr learned my addr 172.21.15.133:0/3018676726 (peer_addr_for_me v2:172.21.15.133:0/0) 2024-02-20T07:54:18.825 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:18.821+0000 7f7dcaffd700 1 --2- 172.21.15.133:0/3018676726 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f7dcc101e60 0x7f7dcc10eb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:54:18.825 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:18.821+0000 7f7dca7fc700 1 -- 172.21.15.133:0/3018676726 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f7dcc101e60 msgr2=0x7f7dcc10eb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:54:18.825 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:18.821+0000 7f7dca7fc700 1 --2- 172.21.15.133:0/3018676726 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f7dcc101e60 0x7f7dcc10eb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:54:18.825 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:18.821+0000 7f7dca7fc700 1 -- 172.21.15.133:0/3018676726 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7dc0005040 con 0x7f7dcc102800 2024-02-20T07:54:18.825 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:18.821+0000 7f7dca7fc700 1 --2- 172.21.15.133:0/3018676726 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f7dcc102800 0x7f7dcc10f0c0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f7dc00090d0 tx=0x7f7dc0009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:54:18.825 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:18.822+0000 7f7db37fe700 1 -- 172.21.15.133:0/3018676726 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7dc000e040 con 0x7f7dcc102800 2024-02-20T07:54:18.825 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:18.822+0000 7f7dd1977700 1 -- 172.21.15.133:0/3018676726 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7dcc117af0 con 0x7f7dcc102800 2024-02-20T07:54:18.825 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:18.822+0000 7f7db37fe700 1 -- 172.21.15.133:0/3018676726 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7dc0009430 con 0x7f7dcc102800 2024-02-20T07:54:18.825 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:18.822+0000 7f7db37fe700 1 -- 172.21.15.133:0/3018676726 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7dc001c3f0 con 0x7f7dcc102800 2024-02-20T07:54:18.825 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:18.822+0000 7f7dd1977700 1 -- 172.21.15.133:0/3018676726 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7dcc117fb0 con 0x7f7dcc102800 2024-02-20T07:54:18.825 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:18.822+0000 7f7dd1977700 1 -- 172.21.15.133:0/3018676726 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f7dcc05e690 con 0x7f7dcc102800 2024-02-20T07:54:18.826 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:18.823+0000 7f7db37fe700 1 -- 172.21.15.133:0/3018676726 <== mon.0 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7f7dc000bbf0 con 0x7f7dcc102800 2024-02-20T07:54:18.827 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:18.823+0000 7f7db37fe700 1 --2- 172.21.15.133:0/3018676726 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7f7db40692a0 0x7f7db406b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:54:18.827 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:18.824+0000 7f7db37fe700 1 -- 172.21.15.133:0/3018676726 <== mon.0 v2:172.21.15.133:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f7dc0088130 con 0x7f7dcc102800 2024-02-20T07:54:18.827 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:18.824+0000 7f7dcaffd700 1 --2- 172.21.15.133:0/3018676726 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7f7db40692a0 0x7f7db406b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:54:18.827 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:18.824+0000 7f7db37fe700 1 --2- 172.21.15.133:0/3018676726 >> [v2:172.21.15.183:6808/3697465174,v1:172.21.15.183:6809/3697465174] conn(0x7f7db406ef70 0x7f7db4071380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:54:18.827 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:18.824+0000 7f7db37fe700 1 -- 172.21.15.133:0/3018676726 --> [v2:172.21.15.183:6808/3697465174,v1:172.21.15.183:6809/3697465174] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f7db4071a30 con 0x7f7db406ef70 2024-02-20T07:54:18.827 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:18.824+0000 7f7db37fe700 1 -- 172.21.15.133:0/3018676726 <== mon.0 v2:172.21.15.133:3300/0 6 ==== mon_get_version_reply(handle=1 version=40) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7dc008a720 con 0x7f7dcc102800 2024-02-20T07:54:18.827 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:18.824+0000 7f7dcaffd700 1 --2- 172.21.15.133:0/3018676726 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7f7db40692a0 0x7f7db406b750 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f7dbc004eb0 tx=0x7f7dbc00b040 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:54:18.827 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:18.824+0000 7f7dcb7fe700 1 --2- 172.21.15.133:0/3018676726 >> [v2:172.21.15.183:6808/3697465174,v1:172.21.15.183:6809/3697465174] conn(0x7f7db406ef70 0x7f7db4071380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:54:18.827 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:18.824+0000 7f7dcb7fe700 1 --2- 172.21.15.133:0/3018676726 >> [v2:172.21.15.183:6808/3697465174,v1:172.21.15.183:6809/3697465174] conn(0x7f7db406ef70 0x7f7db4071380 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:54:18.827 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:18.825+0000 7f7db37fe700 1 -- 172.21.15.133:0/3018676726 <== osd.4 v2:172.21.15.183:6808/3697465174 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (crc 0 0 0) 0x7f7db4071a30 con 0x7f7db406ef70 2024-02-20T07:54:18.842 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:18.840+0000 7f7dd1977700 1 -- 172.21.15.133:0/3018676726 --> [v2:172.21.15.183:6808/3697465174,v1:172.21.15.183:6809/3697465174] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f7dcc05eb40 con 0x7f7db406ef70 2024-02-20T07:54:18.842 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:18.841+0000 7f7db37fe700 1 -- 172.21.15.133:0/3018676726 <== osd.4 v2:172.21.15.183:6808/3697465174 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f7dcc05eb40 con 0x7f7db406ef70 2024-02-20T07:54:18.843 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:18.841+0000 7f7dd1977700 1 -- 172.21.15.133:0/3018676726 >> [v2:172.21.15.183:6808/3697465174,v1:172.21.15.183:6809/3697465174] conn(0x7f7db406ef70 msgr2=0x7f7db4071380 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:54:18.843 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:18.841+0000 7f7dd1977700 1 --2- 172.21.15.133:0/3018676726 >> [v2:172.21.15.183:6808/3697465174,v1:172.21.15.183:6809/3697465174] conn(0x7f7db406ef70 0x7f7db4071380 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:54:18.843 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:18.841+0000 7f7dd1977700 1 -- 172.21.15.133:0/3018676726 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7f7db40692a0 msgr2=0x7f7db406b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:54:18.843 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:18.841+0000 7f7dd1977700 1 --2- 172.21.15.133:0/3018676726 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7f7db40692a0 0x7f7db406b750 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f7dbc004eb0 tx=0x7f7dbc00b040 comp rx=0 tx=0).stop 2024-02-20T07:54:18.843 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:18.841+0000 7f7dd1977700 1 -- 172.21.15.133:0/3018676726 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f7dcc102800 msgr2=0x7f7dcc10f0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:54:18.844 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:18.841+0000 7f7dd1977700 1 --2- 172.21.15.133:0/3018676726 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f7dcc102800 0x7f7dcc10f0c0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f7dc00090d0 tx=0x7f7dc0009e30 comp rx=0 tx=0).stop 2024-02-20T07:54:18.844 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:18.841+0000 7f7dd1977700 1 -- 172.21.15.133:0/3018676726 shutdown_connections 2024-02-20T07:54:18.844 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:18.841+0000 7f7dd1977700 1 --2- 172.21.15.133:0/3018676726 >> [v2:172.21.15.183:6808/3697465174,v1:172.21.15.183:6809/3697465174] conn(0x7f7db406ef70 0x7f7db4071380 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:54:18.844 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:18.841+0000 7f7dd1977700 1 --2- 172.21.15.133:0/3018676726 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7f7db40692a0 0x7f7db406b750 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:54:18.844 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:18.841+0000 7f7dd1977700 1 --2- 172.21.15.133:0/3018676726 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f7dcc101e60 0x7f7dcc10eb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:54:18.844 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:18.841+0000 7f7dd1977700 1 --2- 172.21.15.133:0/3018676726 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f7dcc102800 0x7f7dcc10f0c0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:54:18.844 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:18.841+0000 7f7dd1977700 1 -- 172.21.15.133:0/3018676726 >> 172.21.15.133:0/3018676726 conn(0x7f7dcc0f7a10 msgr2=0x7f7dcc0f9530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:54:18.844 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:18.841+0000 7f7dd1977700 1 -- 172.21.15.133:0/3018676726 shutdown_connections 2024-02-20T07:54:18.844 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:18.841+0000 7f7dd1977700 1 -- 172.21.15.133:0/3018676726 wait complete. 2024-02-20T07:54:18.991 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:54:18 smithi133 ceph-mon[29117]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-20T07:54:19.297 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:54:18 smithi183 ceph-mon[35832]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-20T07:54:20.287 INFO:teuthology.orchestra.run.smithi133.stdout:146028888078 2024-02-20T07:54:20.287 DEBUG:teuthology.orchestra.run.smithi133:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 4537327c-cfc4-11ee-95bb-87774f69a715 -- ceph osd last-stat-seq osd.4 2024-02-20T07:54:20.505 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:20.503+0000 7fccf6e84700 1 -- 172.21.15.133:0/2154461595 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fccf00fdf40 msgr2=0x7fccf00fe310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:54:20.506 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:20.504+0000 7fccf6e84700 1 --2- 172.21.15.133:0/2154461595 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fccf00fdf40 0x7fccf00fe310 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fcce4004940 tx=0x7fcce4009e30 comp rx=0 tx=0).stop 2024-02-20T07:54:20.506 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:20.504+0000 7fccf6e84700 1 -- 172.21.15.133:0/2154461595 shutdown_connections 2024-02-20T07:54:20.506 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:20.504+0000 7fccf6e84700 1 --2- 172.21.15.133:0/2154461595 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fccf00fe8e0 0x7fccf0106f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:54:20.506 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:20.504+0000 7fccf6e84700 1 --2- 172.21.15.133:0/2154461595 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fccf00fdf40 0x7fccf00fe310 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:54:20.506 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:20.504+0000 7fccf6e84700 1 -- 172.21.15.133:0/2154461595 >> 172.21.15.133:0/2154461595 conn(0x7fccf00f79d0 msgr2=0x7fccf00f9de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:54:20.506 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:20.504+0000 7fccf6e84700 1 -- 172.21.15.133:0/2154461595 shutdown_connections 2024-02-20T07:54:20.506 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:20.504+0000 7fccf6e84700 1 -- 172.21.15.133:0/2154461595 wait complete. 2024-02-20T07:54:20.507 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:20.504+0000 7fccf6e84700 1 Processor -- start 2024-02-20T07:54:20.508 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:20.504+0000 7fccf6e84700 1 -- start start 2024-02-20T07:54:20.508 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:20.505+0000 7fccf6e84700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fccf00fdf40 0x7fccf0113a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:54:20.508 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:20.505+0000 7fccf6e84700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fccf00fe8e0 0x7fccf0113fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:54:20.508 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:20.505+0000 7fccf6e84700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fccf0114680 con 0x7fccf00fe8e0 2024-02-20T07:54:20.508 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:20.505+0000 7fccf6e84700 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fccf010d920 con 0x7fccf00fdf40 2024-02-20T07:54:20.508 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:20.505+0000 7fcceffff700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fccf00fe8e0 0x7fccf0113fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:54:20.508 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:20.505+0000 7fcceffff700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fccf00fe8e0 0x7fccf0113fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.133:48988/0 (socket says 172.21.15.133:48988) 2024-02-20T07:54:20.508 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:20.505+0000 7fcceffff700 1 -- 172.21.15.133:0/2342361287 learned_addr learned my addr 172.21.15.133:0/2342361287 (peer_addr_for_me v2:172.21.15.133:0/0) 2024-02-20T07:54:20.508 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:20.505+0000 7fccf4c20700 1 --2- 172.21.15.133:0/2342361287 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fccf00fdf40 0x7fccf0113a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:54:20.508 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:20.505+0000 7fcceffff700 1 -- 172.21.15.133:0/2342361287 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fccf00fdf40 msgr2=0x7fccf0113a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:54:20.508 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:20.505+0000 7fcceffff700 1 --2- 172.21.15.133:0/2342361287 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fccf00fdf40 0x7fccf0113a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:54:20.509 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:20.505+0000 7fcceffff700 1 -- 172.21.15.133:0/2342361287 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcce4005040 con 0x7fccf00fe8e0 2024-02-20T07:54:20.509 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:20.506+0000 7fcceffff700 1 --2- 172.21.15.133:0/2342361287 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fccf00fe8e0 0x7fccf0113fa0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fccdc00b790 tx=0x7fccdc009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:54:20.509 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:20.506+0000 7fcced7fa700 1 -- 172.21.15.133:0/2342361287 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fccdc00f6f0 con 0x7fccf00fe8e0 2024-02-20T07:54:20.509 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:20.506+0000 7fccf6e84700 1 -- 172.21.15.133:0/2342361287 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fccf010dc00 con 0x7fccf00fe8e0 2024-02-20T07:54:20.509 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:20.506+0000 7fcced7fa700 1 -- 172.21.15.133:0/2342361287 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fccdc00fd20 con 0x7fccf00fe8e0 2024-02-20T07:54:20.509 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:20.506+0000 7fcced7fa700 1 -- 172.21.15.133:0/2342361287 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fccdc0103a0 con 0x7fccf00fe8e0 2024-02-20T07:54:20.509 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:20.506+0000 7fccf6e84700 1 -- 172.21.15.133:0/2342361287 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fccf010e140 con 0x7fccf00fe8e0 2024-02-20T07:54:20.510 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:20.507+0000 7fccf6e84700 1 -- 172.21.15.133:0/2342361287 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fccf010e6a0 con 0x7fccf00fe8e0 2024-02-20T07:54:20.510 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:20.507+0000 7fcced7fa700 1 -- 172.21.15.133:0/2342361287 <== mon.0 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7fccdc00f850 con 0x7fccf00fe8e0 2024-02-20T07:54:20.510 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:20.508+0000 7fcced7fa700 1 --2- 172.21.15.133:0/2342361287 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7fcce00692a0 0x7fcce006b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:54:20.510 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:20.508+0000 7fcced7fa700 1 -- 172.21.15.133:0/2342361287 <== mon.0 v2:172.21.15.133:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fccdc086c30 con 0x7fccf00fe8e0 2024-02-20T07:54:20.510 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:20.508+0000 7fccf4c20700 1 --2- 172.21.15.133:0/2342361287 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7fcce00692a0 0x7fcce006b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:54:20.510 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:20.508+0000 7fcced7fa700 1 --2- 172.21.15.133:0/2342361287 >> [v2:172.21.15.133:6810/1652695599,v1:172.21.15.133:6811/1652695599] conn(0x7fcce006ef70 0x7fcce0071380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:54:20.511 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:20.508+0000 7fcced7fa700 1 -- 172.21.15.133:0/2342361287 --> [v2:172.21.15.133:6810/1652695599,v1:172.21.15.133:6811/1652695599] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fcce0071a30 con 0x7fcce006ef70 2024-02-20T07:54:20.511 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:20.508+0000 7fcced7fa700 1 -- 172.21.15.133:0/2342361287 <== mon.0 v2:172.21.15.133:3300/0 6 ==== mon_get_version_reply(handle=1 version=40) v2 ==== 24+0+0 (secure 0 0 0) 0x7fccdc08c290 con 0x7fccf00fe8e0 2024-02-20T07:54:20.511 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:20.508+0000 7fccf5421700 1 --2- 172.21.15.133:0/2342361287 >> [v2:172.21.15.133:6810/1652695599,v1:172.21.15.133:6811/1652695599] conn(0x7fcce006ef70 0x7fcce0071380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:54:20.511 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:20.508+0000 7fccf4c20700 1 --2- 172.21.15.133:0/2342361287 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7fcce00692a0 0x7fcce006b750 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fcce4004f90 tx=0x7fcce4009e30 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:54:20.511 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:20.509+0000 7fccf5421700 1 --2- 172.21.15.133:0/2342361287 >> [v2:172.21.15.133:6810/1652695599,v1:172.21.15.133:6811/1652695599] conn(0x7fcce006ef70 0x7fcce0071380 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:54:20.511 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:20.509+0000 7fcced7fa700 1 -- 172.21.15.133:0/2342361287 <== osd.1 v2:172.21.15.133:6810/1652695599 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (crc 0 0 0) 0x7fcce0071a30 con 0x7fcce006ef70 2024-02-20T07:54:20.532 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:20.531+0000 7fccf6e84700 1 -- 172.21.15.133:0/2342361287 --> [v2:172.21.15.133:6810/1652695599,v1:172.21.15.133:6811/1652695599] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fccf005e690 con 0x7fcce006ef70 2024-02-20T07:54:20.533 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:20.531+0000 7fcced7fa700 1 -- 172.21.15.133:0/2342361287 <== osd.1 v2:172.21.15.133:6810/1652695599 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fccf005e690 con 0x7fcce006ef70 2024-02-20T07:54:20.533 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:20.531+0000 7fccf6e84700 1 -- 172.21.15.133:0/2342361287 >> [v2:172.21.15.133:6810/1652695599,v1:172.21.15.133:6811/1652695599] conn(0x7fcce006ef70 msgr2=0x7fcce0071380 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:54:20.533 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:20.532+0000 7fccf6e84700 1 --2- 172.21.15.133:0/2342361287 >> [v2:172.21.15.133:6810/1652695599,v1:172.21.15.133:6811/1652695599] conn(0x7fcce006ef70 0x7fcce0071380 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:54:20.534 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:20.532+0000 7fccf6e84700 1 -- 172.21.15.133:0/2342361287 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7fcce00692a0 msgr2=0x7fcce006b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:54:20.534 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:20.532+0000 7fccf6e84700 1 --2- 172.21.15.133:0/2342361287 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7fcce00692a0 0x7fcce006b750 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fcce4004f90 tx=0x7fcce4009e30 comp rx=0 tx=0).stop 2024-02-20T07:54:20.534 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:20.532+0000 7fccf6e84700 1 -- 172.21.15.133:0/2342361287 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fccf00fe8e0 msgr2=0x7fccf0113fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:54:20.534 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:20.532+0000 7fccf6e84700 1 --2- 172.21.15.133:0/2342361287 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fccf00fe8e0 0x7fccf0113fa0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fccdc00b790 tx=0x7fccdc009250 comp rx=0 tx=0).stop 2024-02-20T07:54:20.534 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:20.532+0000 7fccf6e84700 1 -- 172.21.15.133:0/2342361287 shutdown_connections 2024-02-20T07:54:20.534 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:20.532+0000 7fccf6e84700 1 --2- 172.21.15.133:0/2342361287 >> [v2:172.21.15.133:6810/1652695599,v1:172.21.15.133:6811/1652695599] conn(0x7fcce006ef70 0x7fcce0071380 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:54:20.534 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:20.532+0000 7fccf6e84700 1 --2- 172.21.15.133:0/2342361287 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7fcce00692a0 0x7fcce006b750 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:54:20.534 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:20.532+0000 7fccf6e84700 1 --2- 172.21.15.133:0/2342361287 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fccf00fdf40 0x7fccf0113a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:54:20.534 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:20.532+0000 7fccf6e84700 1 --2- 172.21.15.133:0/2342361287 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fccf00fe8e0 0x7fccf0113fa0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:54:20.535 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:20.532+0000 7fccf6e84700 1 -- 172.21.15.133:0/2342361287 >> 172.21.15.133:0/2342361287 conn(0x7fccf00f79d0 msgr2=0x7fccf01016b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:54:20.535 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:20.532+0000 7fccf6e84700 1 -- 172.21.15.133:0/2342361287 shutdown_connections 2024-02-20T07:54:20.535 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:20.532+0000 7fccf6e84700 1 -- 172.21.15.133:0/2342361287 wait complete. 2024-02-20T07:54:21.186 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:54:20 smithi133 ceph-mon[29117]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-20T07:54:21.297 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:54:20 smithi183 ceph-mon[35832]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-20T07:54:22.383 INFO:teuthology.orchestra.run.smithi133.stdout:60129542173 2024-02-20T07:54:22.383 DEBUG:teuthology.orchestra.run.smithi133:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 4537327c-cfc4-11ee-95bb-87774f69a715 -- ceph osd last-stat-seq osd.1 2024-02-20T07:54:23.088 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:23.086+0000 7efc1864d700 1 -- 172.21.15.133:0/3723950510 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7efc101038f0 msgr2=0x7efc100fcf60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:54:23.088 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:23.086+0000 7efc1864d700 1 --2- 172.21.15.133:0/3723950510 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7efc101038f0 0x7efc100fcf60 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7efc00004940 tx=0x7efc00009e30 comp rx=0 tx=0).stop 2024-02-20T07:54:23.088 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:23.086+0000 7efc1864d700 1 -- 172.21.15.133:0/3723950510 shutdown_connections 2024-02-20T07:54:23.088 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:23.086+0000 7efc1864d700 1 --2- 172.21.15.133:0/3723950510 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7efc101038f0 0x7efc100fcf60 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:54:23.088 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:23.086+0000 7efc1864d700 1 --2- 172.21.15.133:0/3723950510 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7efc10102f50 0x7efc10103320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:54:23.089 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:23.086+0000 7efc1864d700 1 -- 172.21.15.133:0/3723950510 >> 172.21.15.133:0/3723950510 conn(0x7efc100f8ae0 msgr2=0x7efc100faef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:54:23.089 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:23.086+0000 7efc1864d700 1 -- 172.21.15.133:0/3723950510 shutdown_connections 2024-02-20T07:54:23.089 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:23.086+0000 7efc1864d700 1 -- 172.21.15.133:0/3723950510 wait complete. 2024-02-20T07:54:23.089 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:23.086+0000 7efc1864d700 1 Processor -- start 2024-02-20T07:54:23.089 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:23.086+0000 7efc1864d700 1 -- start start 2024-02-20T07:54:23.089 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:23.087+0000 7efc1864d700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7efc10102f50 0x7efc10113ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:54:23.089 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:23.087+0000 7efc1864d700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7efc101038f0 0x7efc10114000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:54:23.089 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:23.087+0000 7efc1864d700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc101146e0 con 0x7efc10102f50 2024-02-20T07:54:23.089 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:23.087+0000 7efc1864d700 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc1010d980 con 0x7efc101038f0 2024-02-20T07:54:23.089 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:23.087+0000 7efc163e9700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7efc10102f50 0x7efc10113ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:54:23.089 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:23.087+0000 7efc163e9700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7efc10102f50 0x7efc10113ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.133:49010/0 (socket says 172.21.15.133:49010) 2024-02-20T07:54:23.089 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:23.087+0000 7efc163e9700 1 -- 172.21.15.133:0/3511672136 learned_addr learned my addr 172.21.15.133:0/3511672136 (peer_addr_for_me v2:172.21.15.133:0/0) 2024-02-20T07:54:23.089 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:23.087+0000 7efc15be8700 1 --2- 172.21.15.133:0/3511672136 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7efc101038f0 0x7efc10114000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:54:23.089 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:23.087+0000 7efc163e9700 1 -- 172.21.15.133:0/3511672136 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7efc101038f0 msgr2=0x7efc10114000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:54:23.090 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:23.087+0000 7efc163e9700 1 --2- 172.21.15.133:0/3511672136 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7efc101038f0 0x7efc10114000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:54:23.090 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:23.087+0000 7efc163e9700 1 -- 172.21.15.133:0/3511672136 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc00005040 con 0x7efc10102f50 2024-02-20T07:54:23.090 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:23.087+0000 7efc163e9700 1 --2- 172.21.15.133:0/3511672136 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7efc10102f50 0x7efc10113ac0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7efc0800bbb0 tx=0x7efc08009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:54:23.090 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:23.087+0000 7efc06ffd700 1 -- 172.21.15.133:0/3511672136 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7efc080092e0 con 0x7efc10102f50 2024-02-20T07:54:23.090 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:23.087+0000 7efc06ffd700 1 -- 172.21.15.133:0/3511672136 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7efc0800cb20 con 0x7efc10102f50 2024-02-20T07:54:23.090 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:23.087+0000 7efc06ffd700 1 -- 172.21.15.133:0/3511672136 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7efc08015d70 con 0x7efc10102f50 2024-02-20T07:54:23.090 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:23.087+0000 7efc1864d700 1 -- 172.21.15.133:0/3511672136 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efc1010dc60 con 0x7efc10102f50 2024-02-20T07:54:23.090 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:23.087+0000 7efc1864d700 1 -- 172.21.15.133:0/3511672136 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efc1010e170 con 0x7efc10102f50 2024-02-20T07:54:23.091 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:23.088+0000 7efc1864d700 1 -- 172.21.15.133:0/3511672136 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7efc1010e690 con 0x7efc10102f50 2024-02-20T07:54:23.091 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:23.089+0000 7efc06ffd700 1 -- 172.21.15.133:0/3511672136 <== mon.0 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7efc08012070 con 0x7efc10102f50 2024-02-20T07:54:23.091 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:23.089+0000 7efc06ffd700 1 --2- 172.21.15.133:0/3511672136 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7efbfc0692a0 0x7efbfc06b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:54:23.091 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:23.089+0000 7efc06ffd700 1 -- 172.21.15.133:0/3511672136 <== mon.0 v2:172.21.15.133:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7efc08086b90 con 0x7efc10102f50 2024-02-20T07:54:23.091 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:23.089+0000 7efc15be8700 1 --2- 172.21.15.133:0/3511672136 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7efbfc0692a0 0x7efbfc06b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:54:23.092 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:23.089+0000 7efc06ffd700 1 --2- 172.21.15.133:0/3511672136 >> [v2:172.21.15.133:6802/261892236,v1:172.21.15.133:6803/261892236] conn(0x7efbfc06ef70 0x7efbfc071380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:54:23.092 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:23.089+0000 7efc06ffd700 1 -- 172.21.15.133:0/3511672136 --> [v2:172.21.15.133:6802/261892236,v1:172.21.15.133:6803/261892236] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7efbfc071a30 con 0x7efbfc06ef70 2024-02-20T07:54:23.092 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:23.089+0000 7efc06ffd700 1 -- 172.21.15.133:0/3511672136 <== mon.0 v2:172.21.15.133:3300/0 6 ==== mon_get_version_reply(handle=1 version=40) v2 ==== 24+0+0 (secure 0 0 0) 0x7efc0808b1c0 con 0x7efc10102f50 2024-02-20T07:54:23.092 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:23.090+0000 7efc16bea700 1 --2- 172.21.15.133:0/3511672136 >> [v2:172.21.15.133:6802/261892236,v1:172.21.15.133:6803/261892236] conn(0x7efbfc06ef70 0x7efbfc071380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:54:23.092 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:23.090+0000 7efc15be8700 1 --2- 172.21.15.133:0/3511672136 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7efbfc0692a0 0x7efbfc06b750 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7efc1010ecb0 tx=0x7efc00009e30 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:54:23.092 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:23.090+0000 7efc16bea700 1 --2- 172.21.15.133:0/3511672136 >> [v2:172.21.15.133:6802/261892236,v1:172.21.15.133:6803/261892236] conn(0x7efbfc06ef70 0x7efbfc071380 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:54:23.092 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:23.090+0000 7efc06ffd700 1 -- 172.21.15.133:0/3511672136 <== osd.0 v2:172.21.15.133:6802/261892236 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (crc 0 0 0) 0x7efbfc071a30 con 0x7efbfc06ef70 2024-02-20T07:54:23.107 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:23.106+0000 7efc1864d700 1 -- 172.21.15.133:0/3511672136 --> [v2:172.21.15.133:6802/261892236,v1:172.21.15.133:6803/261892236] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7efc100fc6e0 con 0x7efbfc06ef70 2024-02-20T07:54:23.107 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:23.106+0000 7efc06ffd700 1 -- 172.21.15.133:0/3511672136 <== osd.0 v2:172.21.15.133:6802/261892236 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7efc100fc6e0 con 0x7efbfc06ef70 2024-02-20T07:54:23.108 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:23.107+0000 7efc1864d700 1 -- 172.21.15.133:0/3511672136 >> [v2:172.21.15.133:6802/261892236,v1:172.21.15.133:6803/261892236] conn(0x7efbfc06ef70 msgr2=0x7efbfc071380 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:54:23.108 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:23.107+0000 7efc1864d700 1 --2- 172.21.15.133:0/3511672136 >> [v2:172.21.15.133:6802/261892236,v1:172.21.15.133:6803/261892236] conn(0x7efbfc06ef70 0x7efbfc071380 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:54:23.108 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:23.107+0000 7efc1864d700 1 -- 172.21.15.133:0/3511672136 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7efbfc0692a0 msgr2=0x7efbfc06b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:54:23.108 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:23.107+0000 7efc1864d700 1 --2- 172.21.15.133:0/3511672136 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7efbfc0692a0 0x7efbfc06b750 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7efc1010ecb0 tx=0x7efc00009e30 comp rx=0 tx=0).stop 2024-02-20T07:54:23.108 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:23.107+0000 7efc1864d700 1 -- 172.21.15.133:0/3511672136 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7efc10102f50 msgr2=0x7efc10113ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:54:23.108 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:23.107+0000 7efc1864d700 1 --2- 172.21.15.133:0/3511672136 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7efc10102f50 0x7efc10113ac0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7efc0800bbb0 tx=0x7efc08009840 comp rx=0 tx=0).stop 2024-02-20T07:54:23.109 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:23.107+0000 7efc1864d700 1 -- 172.21.15.133:0/3511672136 shutdown_connections 2024-02-20T07:54:23.109 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:23.107+0000 7efc1864d700 1 --2- 172.21.15.133:0/3511672136 >> [v2:172.21.15.133:6802/261892236,v1:172.21.15.133:6803/261892236] conn(0x7efbfc06ef70 0x7efbfc071380 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:54:23.109 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:23.107+0000 7efc1864d700 1 --2- 172.21.15.133:0/3511672136 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7efc10102f50 0x7efc10113ac0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:54:23.109 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:23.107+0000 7efc1864d700 1 --2- 172.21.15.133:0/3511672136 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7efbfc0692a0 0x7efbfc06b750 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:54:23.109 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:23.107+0000 7efc1864d700 1 --2- 172.21.15.133:0/3511672136 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7efc101038f0 0x7efc10114000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:54:23.109 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:23.107+0000 7efc1864d700 1 -- 172.21.15.133:0/3511672136 >> 172.21.15.133:0/3511672136 conn(0x7efc100f8ae0 msgr2=0x7efc100fa480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:54:23.109 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:23.107+0000 7efc1864d700 1 -- 172.21.15.133:0/3511672136 shutdown_connections 2024-02-20T07:54:23.109 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:23.107+0000 7efc1864d700 1 -- 172.21.15.133:0/3511672136 wait complete. 2024-02-20T07:54:23.241 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:54:22 smithi133 ceph-mon[29117]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-20T07:54:23.297 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:54:22 smithi183 ceph-mon[35832]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-20T07:54:23.716 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:23.714+0000 7f2b24cf1700 1 -- 172.21.15.133:0/3687979777 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f2b200fe910 msgr2=0x7f2b20106fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:54:23.717 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:23.714+0000 7f2b24cf1700 1 --2- 172.21.15.133:0/3687979777 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f2b200fe910 0x7f2b20106fe0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f2b10004970 tx=0x7f2b10009e30 comp rx=0 tx=0).stop 2024-02-20T07:54:23.717 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:23.715+0000 7f2b24cf1700 1 -- 172.21.15.133:0/3687979777 shutdown_connections 2024-02-20T07:54:23.717 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:23.715+0000 7f2b24cf1700 1 --2- 172.21.15.133:0/3687979777 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f2b200fe910 0x7f2b20106fe0 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:54:23.717 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:23.715+0000 7f2b24cf1700 1 --2- 172.21.15.133:0/3687979777 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2b200fdf70 0x7f2b200fe340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:54:23.717 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:23.715+0000 7f2b24cf1700 1 -- 172.21.15.133:0/3687979777 >> 172.21.15.133:0/3687979777 conn(0x7f2b200f7a10 msgr2=0x7f2b200f9e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:54:23.717 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:23.715+0000 7f2b24cf1700 1 -- 172.21.15.133:0/3687979777 shutdown_connections 2024-02-20T07:54:23.717 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:23.715+0000 7f2b24cf1700 1 -- 172.21.15.133:0/3687979777 wait complete. 2024-02-20T07:54:23.717 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:23.715+0000 7f2b24cf1700 1 Processor -- start 2024-02-20T07:54:23.717 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:23.715+0000 7f2b24cf1700 1 -- start start 2024-02-20T07:54:23.717 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:23.715+0000 7f2b24cf1700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f2b200fdf70 0x7f2b200fbe30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:54:23.717 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:23.715+0000 7f2b24cf1700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2b200fe910 0x7f2b200fc370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:54:23.717 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:23.715+0000 7f2b24cf1700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b200fc970 con 0x7f2b200fdf70 2024-02-20T07:54:23.717 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:23.715+0000 7f2b24cf1700 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b200fcae0 con 0x7f2b200fe910 2024-02-20T07:54:23.718 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:23.715+0000 7f2b1e59c700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f2b200fdf70 0x7f2b200fbe30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:54:23.718 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:23.715+0000 7f2b1e59c700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f2b200fdf70 0x7f2b200fbe30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.133:49034/0 (socket says 172.21.15.133:49034) 2024-02-20T07:54:23.718 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:23.715+0000 7f2b1e59c700 1 -- 172.21.15.133:0/2777059178 learned_addr learned my addr 172.21.15.133:0/2777059178 (peer_addr_for_me v2:172.21.15.133:0/0) 2024-02-20T07:54:23.718 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:23.716+0000 7f2b1dd9b700 1 --2- 172.21.15.133:0/2777059178 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2b200fe910 0x7f2b200fc370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:54:23.718 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:23.716+0000 7f2b1e59c700 1 -- 172.21.15.133:0/2777059178 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2b200fe910 msgr2=0x7f2b200fc370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:54:23.718 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:23.716+0000 7f2b1e59c700 1 --2- 172.21.15.133:0/2777059178 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2b200fe910 0x7f2b200fc370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:54:23.718 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:23.716+0000 7f2b1e59c700 1 -- 172.21.15.133:0/2777059178 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b10005040 con 0x7f2b200fdf70 2024-02-20T07:54:23.718 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:23.716+0000 7f2b1e59c700 1 --2- 172.21.15.133:0/2777059178 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f2b200fdf70 0x7f2b200fbe30 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f2b1400eb40 tx=0x7f2b1400b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:54:23.718 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:23.716+0000 7f2b0effd700 1 -- 172.21.15.133:0/2777059178 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2b14007cc0 con 0x7f2b200fdf70 2024-02-20T07:54:23.718 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:23.716+0000 7f2b0effd700 1 -- 172.21.15.133:0/2777059178 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2b1400f040 con 0x7f2b200fdf70 2024-02-20T07:54:23.718 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:23.716+0000 7f2b24cf1700 1 -- 172.21.15.133:0/2777059178 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2b20119cc0 con 0x7f2b200fdf70 2024-02-20T07:54:23.718 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:23.716+0000 7f2b0effd700 1 -- 172.21.15.133:0/2777059178 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2b1400ca10 con 0x7f2b200fdf70 2024-02-20T07:54:23.718 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:23.716+0000 7f2b24cf1700 1 -- 172.21.15.133:0/2777059178 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2b2011a120 con 0x7f2b200fdf70 2024-02-20T07:54:23.720 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:23.717+0000 7f2b24cf1700 1 -- 172.21.15.133:0/2777059178 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f2b2005e610 con 0x7f2b200fdf70 2024-02-20T07:54:23.720 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:23.717+0000 7f2b0effd700 1 -- 172.21.15.133:0/2777059178 <== mon.0 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7f2b1400cc80 con 0x7f2b200fdf70 2024-02-20T07:54:23.720 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:23.718+0000 7f2b0effd700 1 --2- 172.21.15.133:0/2777059178 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7f2b080692a0 0x7f2b0806b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:54:23.720 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:23.718+0000 7f2b0effd700 1 -- 172.21.15.133:0/2777059178 <== mon.0 v2:172.21.15.133:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f2b14086a40 con 0x7f2b200fdf70 2024-02-20T07:54:23.720 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:23.718+0000 7f2b1dd9b700 1 --2- 172.21.15.133:0/2777059178 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7f2b080692a0 0x7f2b0806b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:54:23.720 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:23.718+0000 7f2b0effd700 1 --2- 172.21.15.133:0/2777059178 >> [v2:172.21.15.133:6818/3838005412,v1:172.21.15.133:6819/3838005412] conn(0x7f2b0806ef70 0x7f2b08071380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:54:23.720 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:23.718+0000 7f2b0effd700 1 -- 172.21.15.133:0/2777059178 --> [v2:172.21.15.133:6818/3838005412,v1:172.21.15.133:6819/3838005412] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f2b08071a30 con 0x7f2b0806ef70 2024-02-20T07:54:23.720 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:23.718+0000 7f2b0effd700 1 -- 172.21.15.133:0/2777059178 <== mon.0 v2:172.21.15.133:3300/0 6 ==== mon_get_version_reply(handle=1 version=40) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2b1408b220 con 0x7f2b200fdf70 2024-02-20T07:54:23.720 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:23.718+0000 7f2b1ed9d700 1 --2- 172.21.15.133:0/2777059178 >> [v2:172.21.15.133:6818/3838005412,v1:172.21.15.133:6819/3838005412] conn(0x7f2b0806ef70 0x7f2b08071380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:54:23.720 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:23.718+0000 7f2b1dd9b700 1 --2- 172.21.15.133:0/2777059178 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7f2b080692a0 0x7f2b0806b750 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f2b10000c00 tx=0x7f2b10009e30 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:54:23.720 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:23.718+0000 7f2b1ed9d700 1 --2- 172.21.15.133:0/2777059178 >> [v2:172.21.15.133:6818/3838005412,v1:172.21.15.133:6819/3838005412] conn(0x7f2b0806ef70 0x7f2b08071380 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:54:23.720 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:23.719+0000 7f2b0effd700 1 -- 172.21.15.133:0/2777059178 <== osd.2 v2:172.21.15.133:6818/3838005412 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (crc 0 0 0) 0x7f2b08071a30 con 0x7f2b0806ef70 2024-02-20T07:54:23.734 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:23.734+0000 7f2b24cf1700 1 -- 172.21.15.133:0/2777059178 --> [v2:172.21.15.133:6818/3838005412,v1:172.21.15.133:6819/3838005412] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f2b2005ead0 con 0x7f2b0806ef70 2024-02-20T07:54:23.735 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:23.734+0000 7f2b0effd700 1 -- 172.21.15.133:0/2777059178 <== osd.2 v2:172.21.15.133:6818/3838005412 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f2b2005ead0 con 0x7f2b0806ef70 2024-02-20T07:54:23.735 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:23.734+0000 7f2b24cf1700 1 -- 172.21.15.133:0/2777059178 >> [v2:172.21.15.133:6818/3838005412,v1:172.21.15.133:6819/3838005412] conn(0x7f2b0806ef70 msgr2=0x7f2b08071380 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:54:23.736 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:23.734+0000 7f2b24cf1700 1 --2- 172.21.15.133:0/2777059178 >> [v2:172.21.15.133:6818/3838005412,v1:172.21.15.133:6819/3838005412] conn(0x7f2b0806ef70 0x7f2b08071380 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:54:23.736 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:23.734+0000 7f2b24cf1700 1 -- 172.21.15.133:0/2777059178 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7f2b080692a0 msgr2=0x7f2b0806b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:54:23.736 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:23.734+0000 7f2b24cf1700 1 --2- 172.21.15.133:0/2777059178 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7f2b080692a0 0x7f2b0806b750 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f2b10000c00 tx=0x7f2b10009e30 comp rx=0 tx=0).stop 2024-02-20T07:54:23.736 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:23.734+0000 7f2b24cf1700 1 -- 172.21.15.133:0/2777059178 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f2b200fdf70 msgr2=0x7f2b200fbe30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:54:23.736 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:23.734+0000 7f2b24cf1700 1 --2- 172.21.15.133:0/2777059178 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f2b200fdf70 0x7f2b200fbe30 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f2b1400eb40 tx=0x7f2b1400b040 comp rx=0 tx=0).stop 2024-02-20T07:54:23.736 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:23.734+0000 7f2b24cf1700 1 -- 172.21.15.133:0/2777059178 shutdown_connections 2024-02-20T07:54:23.736 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:23.734+0000 7f2b24cf1700 1 --2- 172.21.15.133:0/2777059178 >> [v2:172.21.15.133:6818/3838005412,v1:172.21.15.133:6819/3838005412] conn(0x7f2b0806ef70 0x7f2b08071380 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:54:23.736 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:23.734+0000 7f2b24cf1700 1 --2- 172.21.15.133:0/2777059178 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f2b200fdf70 0x7f2b200fbe30 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:54:23.736 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:23.734+0000 7f2b24cf1700 1 --2- 172.21.15.133:0/2777059178 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7f2b080692a0 0x7f2b0806b750 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:54:23.736 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:23.734+0000 7f2b24cf1700 1 --2- 172.21.15.133:0/2777059178 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2b200fe910 0x7f2b200fc370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:54:23.736 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:23.734+0000 7f2b24cf1700 1 -- 172.21.15.133:0/2777059178 >> 172.21.15.133:0/2777059178 conn(0x7f2b200f7a10 msgr2=0x7f2b20101820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:54:23.736 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:23.734+0000 7f2b24cf1700 1 -- 172.21.15.133:0/2777059178 shutdown_connections 2024-02-20T07:54:23.736 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:23.734+0000 7f2b24cf1700 1 -- 172.21.15.133:0/2777059178 wait complete. 2024-02-20T07:54:23.952 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:23.948+0000 7f5011e4a700 1 -- 172.21.15.133:0/2672100814 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f500c102f50 msgr2=0x7f500c103320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:54:23.952 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:23.948+0000 7f5011e4a700 1 --2- 172.21.15.133:0/2672100814 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f500c102f50 0x7f500c103320 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f5004009ce0 tx=0x7f5004004d10 comp rx=0 tx=0).stop 2024-02-20T07:54:23.952 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:23.948+0000 7f5011e4a700 1 -- 172.21.15.133:0/2672100814 shutdown_connections 2024-02-20T07:54:23.952 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:23.948+0000 7f5011e4a700 1 --2- 172.21.15.133:0/2672100814 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f500c1038f0 0x7f500c0fcf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:54:23.953 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:23.948+0000 7f5011e4a700 1 --2- 172.21.15.133:0/2672100814 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f500c102f50 0x7f500c103320 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:54:23.953 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:23.948+0000 7f5011e4a700 1 -- 172.21.15.133:0/2672100814 >> 172.21.15.133:0/2672100814 conn(0x7f500c0f8ae0 msgr2=0x7f500c0faef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:54:23.953 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:23.948+0000 7f5011e4a700 1 -- 172.21.15.133:0/2672100814 shutdown_connections 2024-02-20T07:54:23.953 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:23.948+0000 7f5011e4a700 1 -- 172.21.15.133:0/2672100814 wait complete. 2024-02-20T07:54:23.953 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:23.949+0000 7f5011e4a700 1 Processor -- start 2024-02-20T07:54:23.953 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:23.949+0000 7f5011e4a700 1 -- start start 2024-02-20T07:54:23.953 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:23.949+0000 7f5011e4a700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f500c102f50 0x7f500c070290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:54:23.953 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:23.949+0000 7f5011e4a700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f500c1038f0 0x7f500c0707d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:54:23.953 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:23.949+0000 7f5011e4a700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f500c070dd0 con 0x7f500c1038f0 2024-02-20T07:54:23.953 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:23.949+0000 7f5011e4a700 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f500c070f40 con 0x7f500c102f50 2024-02-20T07:54:23.953 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:23.949+0000 7f500affd700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f500c1038f0 0x7f500c0707d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:54:23.953 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:23.949+0000 7f500affd700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f500c1038f0 0x7f500c0707d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.133:49054/0 (socket says 172.21.15.133:49054) 2024-02-20T07:54:23.953 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:23.949+0000 7f500affd700 1 -- 172.21.15.133:0/4174766909 learned_addr learned my addr 172.21.15.133:0/4174766909 (peer_addr_for_me v2:172.21.15.133:0/0) 2024-02-20T07:54:23.953 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:23.949+0000 7f500b7fe700 1 --2- 172.21.15.133:0/4174766909 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f500c102f50 0x7f500c070290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:54:23.953 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:23.950+0000 7f500affd700 1 -- 172.21.15.133:0/4174766909 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f500c102f50 msgr2=0x7f500c070290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:54:23.954 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:23.950+0000 7f500affd700 1 --2- 172.21.15.133:0/4174766909 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f500c102f50 0x7f500c070290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:54:23.954 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:23.950+0000 7f500affd700 1 -- 172.21.15.133:0/4174766909 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5004009940 con 0x7f500c1038f0 2024-02-20T07:54:23.954 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:23.950+0000 7f500affd700 1 --2- 172.21.15.133:0/4174766909 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f500c1038f0 0x7f500c0707d0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f4ff4002a50 tx=0x7f4ff4004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:54:23.954 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:23.950+0000 7f5003fff700 1 -- 172.21.15.133:0/4174766909 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4ff4007cc0 con 0x7f500c1038f0 2024-02-20T07:54:23.954 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:23.950+0000 7f5003fff700 1 -- 172.21.15.133:0/4174766909 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4ff400f040 con 0x7f500c1038f0 2024-02-20T07:54:23.954 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:23.950+0000 7f5011e4a700 1 -- 172.21.15.133:0/4174766909 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f500c119e20 con 0x7f500c1038f0 2024-02-20T07:54:23.954 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:23.950+0000 7f5003fff700 1 -- 172.21.15.133:0/4174766909 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4ff400d9c0 con 0x7f500c1038f0 2024-02-20T07:54:23.954 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:23.950+0000 7f5011e4a700 1 -- 172.21.15.133:0/4174766909 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f500c11a2d0 con 0x7f500c1038f0 2024-02-20T07:54:23.954 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:23.951+0000 7f5011e4a700 1 -- 172.21.15.133:0/4174766909 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f500c05e690 con 0x7f500c1038f0 2024-02-20T07:54:23.954 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:23.952+0000 7f5003fff700 1 -- 172.21.15.133:0/4174766909 <== mon.0 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7f4ff4009730 con 0x7f500c1038f0 2024-02-20T07:54:23.955 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:23.952+0000 7f5003fff700 1 --2- 172.21.15.133:0/4174766909 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7f4ff80692a0 0x7f4ff806b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:54:23.955 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:23.952+0000 7f5003fff700 1 -- 172.21.15.133:0/4174766909 <== mon.0 v2:172.21.15.133:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f4ff4087530 con 0x7f500c1038f0 2024-02-20T07:54:23.955 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:23.952+0000 7f500b7fe700 1 --2- 172.21.15.133:0/4174766909 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7f4ff80692a0 0x7f4ff806b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:54:23.955 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:23.952+0000 7f5003fff700 1 --2- 172.21.15.133:0/4174766909 >> [v2:172.21.15.183:6800/1053980618,v1:172.21.15.183:6801/1053980618] conn(0x7f4ff806ef70 0x7f4ff8071380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:54:23.955 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:23.952+0000 7f5003fff700 1 -- 172.21.15.133:0/4174766909 --> [v2:172.21.15.183:6800/1053980618,v1:172.21.15.183:6801/1053980618] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4ff8071a30 con 0x7f4ff806ef70 2024-02-20T07:54:23.955 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:23.952+0000 7f5003fff700 1 -- 172.21.15.133:0/4174766909 <== mon.0 v2:172.21.15.133:3300/0 6 ==== mon_get_version_reply(handle=1 version=40) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4ff4087910 con 0x7f500c1038f0 2024-02-20T07:54:23.955 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:23.952+0000 7f500b7fe700 1 --2- 172.21.15.133:0/4174766909 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7f4ff80692a0 0x7f4ff806b750 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f5004003400 tx=0x7f5004004d10 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:54:23.956 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:23.952+0000 7f500bfff700 1 --2- 172.21.15.133:0/4174766909 >> [v2:172.21.15.183:6800/1053980618,v1:172.21.15.183:6801/1053980618] conn(0x7f4ff806ef70 0x7f4ff8071380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:54:23.956 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:23.953+0000 7f500bfff700 1 --2- 172.21.15.133:0/4174766909 >> [v2:172.21.15.183:6800/1053980618,v1:172.21.15.183:6801/1053980618] conn(0x7f4ff806ef70 0x7f4ff8071380 crc :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:54:23.956 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:23.954+0000 7f5003fff700 1 -- 172.21.15.133:0/4174766909 <== osd.3 v2:172.21.15.183:6800/1053980618 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (crc 0 0 0) 0x7f4ff8071a30 con 0x7f4ff806ef70 2024-02-20T07:54:23.973 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:23.972+0000 7f5011e4a700 1 -- 172.21.15.133:0/4174766909 --> [v2:172.21.15.183:6800/1053980618,v1:172.21.15.183:6801/1053980618] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f500c05eb10 con 0x7f4ff806ef70 2024-02-20T07:54:23.973 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:23.973+0000 7f5003fff700 1 -- 172.21.15.133:0/4174766909 <== osd.3 v2:172.21.15.183:6800/1053980618 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f500c05eb10 con 0x7f4ff806ef70 2024-02-20T07:54:23.974 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:23.973+0000 7f5011e4a700 1 -- 172.21.15.133:0/4174766909 >> [v2:172.21.15.183:6800/1053980618,v1:172.21.15.183:6801/1053980618] conn(0x7f4ff806ef70 msgr2=0x7f4ff8071380 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:54:23.974 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:23.973+0000 7f5011e4a700 1 --2- 172.21.15.133:0/4174766909 >> [v2:172.21.15.183:6800/1053980618,v1:172.21.15.183:6801/1053980618] conn(0x7f4ff806ef70 0x7f4ff8071380 crc :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:54:23.974 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:23.973+0000 7f5011e4a700 1 -- 172.21.15.133:0/4174766909 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7f4ff80692a0 msgr2=0x7f4ff806b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:54:23.974 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:23.973+0000 7f5011e4a700 1 --2- 172.21.15.133:0/4174766909 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7f4ff80692a0 0x7f4ff806b750 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f5004003400 tx=0x7f5004004d10 comp rx=0 tx=0).stop 2024-02-20T07:54:23.975 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:23.973+0000 7f5011e4a700 1 -- 172.21.15.133:0/4174766909 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f500c1038f0 msgr2=0x7f500c0707d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:54:23.975 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:23.973+0000 7f5011e4a700 1 --2- 172.21.15.133:0/4174766909 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f500c1038f0 0x7f500c0707d0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f4ff4002a50 tx=0x7f4ff4004600 comp rx=0 tx=0).stop 2024-02-20T07:54:23.975 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:23.973+0000 7f5011e4a700 1 -- 172.21.15.133:0/4174766909 shutdown_connections 2024-02-20T07:54:23.975 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:23.973+0000 7f5011e4a700 1 --2- 172.21.15.133:0/4174766909 >> [v2:172.21.15.183:6800/1053980618,v1:172.21.15.183:6801/1053980618] conn(0x7f4ff806ef70 0x7f4ff8071380 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:54:23.975 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:23.973+0000 7f5011e4a700 1 --2- 172.21.15.133:0/4174766909 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7f4ff80692a0 0x7f4ff806b750 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:54:23.975 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:23.973+0000 7f5011e4a700 1 --2- 172.21.15.133:0/4174766909 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f500c102f50 0x7f500c070290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:54:23.975 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:23.973+0000 7f5011e4a700 1 --2- 172.21.15.133:0/4174766909 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f500c1038f0 0x7f500c0707d0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:54:23.975 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:23.973+0000 7f5011e4a700 1 -- 172.21.15.133:0/4174766909 >> 172.21.15.133:0/4174766909 conn(0x7f500c0f8ae0 msgr2=0x7f500c0fa4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:54:23.975 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:23.974+0000 7f5011e4a700 1 -- 172.21.15.133:0/4174766909 shutdown_connections 2024-02-20T07:54:23.975 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:23.974+0000 7f5011e4a700 1 -- 172.21.15.133:0/4174766909 wait complete. 2024-02-20T07:54:24.242 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:54:23 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-20T07:54:24.297 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:54:23 smithi183 ceph-mon[35832]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-20T07:54:24.369 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:24.367+0000 7f989d27d700 1 -- 172.21.15.133:0/2102701223 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f98980fe8f0 msgr2=0x7f9898107050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:54:24.369 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:24.367+0000 7f989d27d700 1 --2- 172.21.15.133:0/2102701223 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f98980fe8f0 0x7f9898107050 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f988c004970 tx=0x7f988c009e30 comp rx=0 tx=0).stop 2024-02-20T07:54:24.369 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:24.367+0000 7f989d27d700 1 -- 172.21.15.133:0/2102701223 shutdown_connections 2024-02-20T07:54:24.369 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:24.367+0000 7f989d27d700 1 --2- 172.21.15.133:0/2102701223 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f98980fe8f0 0x7f9898107050 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:54:24.369 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:24.367+0000 7f989d27d700 1 --2- 172.21.15.133:0/2102701223 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f98980fdfe0 0x7f98980fe3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:54:24.369 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:24.367+0000 7f989d27d700 1 -- 172.21.15.133:0/2102701223 >> 172.21.15.133:0/2102701223 conn(0x7f9898071d90 msgr2=0x7f9898072190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:54:24.369 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:24.367+0000 7f989d27d700 1 -- 172.21.15.133:0/2102701223 shutdown_connections 2024-02-20T07:54:24.370 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:24.367+0000 7f989d27d700 1 -- 172.21.15.133:0/2102701223 wait complete. 2024-02-20T07:54:24.370 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:24.368+0000 7f989d27d700 1 Processor -- start 2024-02-20T07:54:24.370 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:24.368+0000 7f989d27d700 1 -- start start 2024-02-20T07:54:24.370 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:24.368+0000 7f989d27d700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f98980fdfe0 0x7f9898113a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:54:24.370 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:24.368+0000 7f989d27d700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f98980fe8f0 0x7f9898113fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:54:24.370 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:24.368+0000 7f989d27d700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9898114690 con 0x7f98980fe8f0 2024-02-20T07:54:24.370 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:24.368+0000 7f989d27d700 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f989810d930 con 0x7f98980fdfe0 2024-02-20T07:54:24.370 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:24.368+0000 7f98967fc700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f98980fe8f0 0x7f9898113fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:54:24.370 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:24.368+0000 7f98967fc700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f98980fe8f0 0x7f9898113fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.133:49078/0 (socket says 172.21.15.133:49078) 2024-02-20T07:54:24.370 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:24.368+0000 7f98967fc700 1 -- 172.21.15.133:0/1612170909 learned_addr learned my addr 172.21.15.133:0/1612170909 (peer_addr_for_me v2:172.21.15.133:0/0) 2024-02-20T07:54:24.370 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:24.368+0000 7f9896ffd700 1 --2- 172.21.15.133:0/1612170909 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f98980fdfe0 0x7f9898113a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:54:24.370 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:24.369+0000 7f98967fc700 1 -- 172.21.15.133:0/1612170909 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f98980fdfe0 msgr2=0x7f9898113a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:54:24.370 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:24.369+0000 7f98967fc700 1 --2- 172.21.15.133:0/1612170909 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f98980fdfe0 0x7f9898113a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:54:24.370 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:24.369+0000 7f98967fc700 1 -- 172.21.15.133:0/1612170909 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f988c005040 con 0x7f98980fe8f0 2024-02-20T07:54:24.372 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:24.369+0000 7f98967fc700 1 --2- 172.21.15.133:0/1612170909 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f98980fe8f0 0x7f9898113fb0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f988c000c00 tx=0x7f988c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:54:24.372 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:24.369+0000 7f987f7fe700 1 -- 172.21.15.133:0/1612170909 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f988c00e040 con 0x7f98980fe8f0 2024-02-20T07:54:24.372 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:24.369+0000 7f987f7fe700 1 -- 172.21.15.133:0/1612170909 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f988c009af0 con 0x7f98980fe8f0 2024-02-20T07:54:24.373 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:24.369+0000 7f989d27d700 1 -- 172.21.15.133:0/1612170909 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f989810dbb0 con 0x7f98980fe8f0 2024-02-20T07:54:24.373 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:24.369+0000 7f987f7fe700 1 -- 172.21.15.133:0/1612170909 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f988c01c3f0 con 0x7f98980fe8f0 2024-02-20T07:54:24.373 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:24.369+0000 7f989d27d700 1 -- 172.21.15.133:0/1612170909 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f989810e0f0 con 0x7f98980fe8f0 2024-02-20T07:54:24.373 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:24.370+0000 7f989d27d700 1 -- 172.21.15.133:0/1612170909 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f989810e590 con 0x7f98980fe8f0 2024-02-20T07:54:24.373 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:24.370+0000 7f987f7fe700 1 -- 172.21.15.133:0/1612170909 <== mon.0 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7f988c00bc10 con 0x7f98980fe8f0 2024-02-20T07:54:24.373 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:24.371+0000 7f987f7fe700 1 --2- 172.21.15.133:0/1612170909 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7f98800692a0 0x7f988006b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:54:24.373 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:24.371+0000 7f987f7fe700 1 -- 172.21.15.133:0/1612170909 <== mon.0 v2:172.21.15.133:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f988c088b30 con 0x7f98980fe8f0 2024-02-20T07:54:24.373 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:24.371+0000 7f9896ffd700 1 --2- 172.21.15.133:0/1612170909 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7f98800692a0 0x7f988006b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:54:24.373 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:24.371+0000 7f987f7fe700 1 --2- 172.21.15.133:0/1612170909 >> [v2:172.21.15.183:6816/81997900,v1:172.21.15.183:6817/81997900] conn(0x7f988006ef70 0x7f9880071380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:54:24.373 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:24.371+0000 7f987f7fe700 1 -- 172.21.15.133:0/1612170909 --> [v2:172.21.15.183:6816/81997900,v1:172.21.15.183:6817/81997900] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f9880071a30 con 0x7f988006ef70 2024-02-20T07:54:24.373 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:24.371+0000 7f987f7fe700 1 -- 172.21.15.133:0/1612170909 <== mon.0 v2:172.21.15.133:3300/0 6 ==== mon_get_version_reply(handle=1 version=40) v2 ==== 24+0+0 (secure 0 0 0) 0x7f988c088f10 con 0x7f98980fe8f0 2024-02-20T07:54:24.373 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:24.371+0000 7f9896ffd700 1 --2- 172.21.15.133:0/1612170909 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7f98800692a0 0x7f988006b750 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f9888009bf0 tx=0x7f98880091a0 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:54:24.373 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:24.371+0000 7f98977fe700 1 --2- 172.21.15.133:0/1612170909 >> [v2:172.21.15.183:6816/81997900,v1:172.21.15.183:6817/81997900] conn(0x7f988006ef70 0x7f9880071380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:54:24.374 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:24.372+0000 7f98977fe700 1 --2- 172.21.15.133:0/1612170909 >> [v2:172.21.15.183:6816/81997900,v1:172.21.15.183:6817/81997900] conn(0x7f988006ef70 0x7f9880071380 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:54:24.374 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:24.372+0000 7f987f7fe700 1 -- 172.21.15.133:0/1612170909 <== osd.5 v2:172.21.15.183:6816/81997900 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (crc 0 0 0) 0x7f9880071a30 con 0x7f988006ef70 2024-02-20T07:54:24.387 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:24.387+0000 7f989d27d700 1 -- 172.21.15.133:0/1612170909 --> [v2:172.21.15.183:6816/81997900,v1:172.21.15.183:6817/81997900] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f989805e690 con 0x7f988006ef70 2024-02-20T07:54:24.388 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:24.387+0000 7f987f7fe700 1 -- 172.21.15.133:0/1612170909 <== osd.5 v2:172.21.15.183:6816/81997900 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f989805e690 con 0x7f988006ef70 2024-02-20T07:54:24.389 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:24.388+0000 7f989d27d700 1 -- 172.21.15.133:0/1612170909 >> [v2:172.21.15.183:6816/81997900,v1:172.21.15.183:6817/81997900] conn(0x7f988006ef70 msgr2=0x7f9880071380 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:54:24.389 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:24.388+0000 7f989d27d700 1 --2- 172.21.15.133:0/1612170909 >> [v2:172.21.15.183:6816/81997900,v1:172.21.15.183:6817/81997900] conn(0x7f988006ef70 0x7f9880071380 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:54:24.389 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:24.388+0000 7f989d27d700 1 -- 172.21.15.133:0/1612170909 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7f98800692a0 msgr2=0x7f988006b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:54:24.389 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:24.388+0000 7f989d27d700 1 --2- 172.21.15.133:0/1612170909 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7f98800692a0 0x7f988006b750 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f9888009bf0 tx=0x7f98880091a0 comp rx=0 tx=0).stop 2024-02-20T07:54:24.389 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:24.388+0000 7f989d27d700 1 -- 172.21.15.133:0/1612170909 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f98980fe8f0 msgr2=0x7f9898113fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:54:24.389 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:24.388+0000 7f989d27d700 1 --2- 172.21.15.133:0/1612170909 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f98980fe8f0 0x7f9898113fb0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f988c000c00 tx=0x7f988c009e30 comp rx=0 tx=0).stop 2024-02-20T07:54:24.390 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:24.388+0000 7f989d27d700 1 -- 172.21.15.133:0/1612170909 shutdown_connections 2024-02-20T07:54:24.390 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:24.388+0000 7f989d27d700 1 --2- 172.21.15.133:0/1612170909 >> [v2:172.21.15.183:6816/81997900,v1:172.21.15.183:6817/81997900] conn(0x7f988006ef70 0x7f9880071380 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:54:24.390 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:24.388+0000 7f989d27d700 1 --2- 172.21.15.133:0/1612170909 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7f98800692a0 0x7f988006b750 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:54:24.390 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:24.388+0000 7f989d27d700 1 --2- 172.21.15.133:0/1612170909 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f98980fdfe0 0x7f9898113a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:54:24.390 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:24.388+0000 7f989d27d700 1 --2- 172.21.15.133:0/1612170909 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f98980fe8f0 0x7f9898113fb0 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:54:24.390 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:24.388+0000 7f989d27d700 1 -- 172.21.15.133:0/1612170909 >> 172.21.15.133:0/1612170909 conn(0x7f9898071d90 msgr2=0x7f9898101840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:54:24.390 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:24.388+0000 7f989d27d700 1 -- 172.21.15.133:0/1612170909 shutdown_connections 2024-02-20T07:54:24.390 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:24.388+0000 7f989d27d700 1 -- 172.21.15.133:0/1612170909 wait complete. 2024-02-20T07:54:25.089 INFO:teuthology.orchestra.run.smithi133.stdout:167503724554 2024-02-20T07:54:25.090 DEBUG:teuthology.orchestra.run.smithi133:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 4537327c-cfc4-11ee-95bb-87774f69a715 -- ceph osd last-stat-seq osd.5 2024-02-20T07:54:25.241 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:54:24 smithi133 ceph-mon[29117]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-20T07:54:25.297 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:54:24 smithi183 ceph-mon[35832]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-20T07:54:25.761 INFO:teuthology.orchestra.run.smithi133.stdout:81604378649 2024-02-20T07:54:25.761 DEBUG:teuthology.orchestra.run.smithi133:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 4537327c-cfc4-11ee-95bb-87774f69a715 -- ceph osd last-stat-seq osd.2 2024-02-20T07:54:26.422 INFO:teuthology.orchestra.run.smithi133.stderr:Inferring config /var/lib/ceph/4537327c-cfc4-11ee-95bb-87774f69a715/mon.smithi133/config 2024-02-20T07:54:26.438 INFO:teuthology.orchestra.run.smithi133.stdout:38654705700 2024-02-20T07:54:26.438 DEBUG:teuthology.orchestra.run.smithi133:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 4537327c-cfc4-11ee-95bb-87774f69a715 -- ceph osd last-stat-seq osd.0 2024-02-20T07:54:26.869 INFO:teuthology.orchestra.run.smithi133.stdout:111669149716 2024-02-20T07:54:26.869 DEBUG:teuthology.orchestra.run.smithi133:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 4537327c-cfc4-11ee-95bb-87774f69a715 -- ceph osd last-stat-seq osd.3 2024-02-20T07:54:26.894 INFO:teuthology.orchestra.run.smithi133.stderr:Inferring config /var/lib/ceph/4537327c-cfc4-11ee-95bb-87774f69a715/mon.smithi133/config 2024-02-20T07:54:27.097 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:54:26 smithi133 ceph-mon[29117]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-20T07:54:27.297 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:54:26 smithi183 ceph-mon[35832]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-20T07:54:29.242 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:54:28 smithi133 ceph-mon[29117]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-20T07:54:29.297 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:54:28 smithi183 ceph-mon[35832]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-20T07:54:29.378 INFO:teuthology.orchestra.run.smithi133.stderr:Inferring config /var/lib/ceph/4537327c-cfc4-11ee-95bb-87774f69a715/mon.smithi133/config 2024-02-20T07:54:29.381 INFO:teuthology.orchestra.run.smithi133.stderr:Inferring config /var/lib/ceph/4537327c-cfc4-11ee-95bb-87774f69a715/mon.smithi133/config 2024-02-20T07:54:29.604 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:29.601+0000 7f3afb32f700 1 -- 172.21.15.133:0/1500245595 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f3af4102880 msgr2=0x7f3af40fc680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:54:29.605 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:29.601+0000 7f3afb32f700 1 --2- 172.21.15.133:0/1500245595 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f3af4102880 0x7f3af40fc680 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f3ae8004940 tx=0x7f3ae8009e30 comp rx=0 tx=0).stop 2024-02-20T07:54:29.605 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:29.601+0000 7f3afb32f700 1 -- 172.21.15.133:0/1500245595 shutdown_connections 2024-02-20T07:54:29.605 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:29.602+0000 7f3afb32f700 1 --2- 172.21.15.133:0/1500245595 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f3af4102880 0x7f3af40fc680 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:54:29.605 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:29.602+0000 7f3afb32f700 1 --2- 172.21.15.133:0/1500245595 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f3af4101ee0 0x7f3af41022b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:54:29.605 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:29.602+0000 7f3afb32f700 1 -- 172.21.15.133:0/1500245595 >> 172.21.15.133:0/1500245595 conn(0x7f3af4071560 msgr2=0x7f3af4071960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:54:29.605 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:29.602+0000 7f3afb32f700 1 -- 172.21.15.133:0/1500245595 shutdown_connections 2024-02-20T07:54:29.605 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:29.602+0000 7f3afb32f700 1 -- 172.21.15.133:0/1500245595 wait complete. 2024-02-20T07:54:29.605 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:29.602+0000 7f3afb32f700 1 Processor -- start 2024-02-20T07:54:29.605 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:29.602+0000 7f3afb32f700 1 -- start start 2024-02-20T07:54:29.605 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:29.602+0000 7f3afb32f700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f3af4101ee0 0x7f3af4110870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:54:29.605 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:29.602+0000 7f3afb32f700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f3af4102880 0x7f3af4110db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:54:29.605 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:29.602+0000 7f3afb32f700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3af4111400 con 0x7f3af4101ee0 2024-02-20T07:54:29.605 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:29.602+0000 7f3afb32f700 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3af410b760 con 0x7f3af4102880 2024-02-20T07:54:29.605 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:29.602+0000 7f3af90cb700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f3af4101ee0 0x7f3af4110870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:54:29.606 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:29.603+0000 7f3af90cb700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f3af4101ee0 0x7f3af4110870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.133:53792/0 (socket says 172.21.15.133:53792) 2024-02-20T07:54:29.606 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:29.603+0000 7f3af90cb700 1 -- 172.21.15.133:0/818634581 learned_addr learned my addr 172.21.15.133:0/818634581 (peer_addr_for_me v2:172.21.15.133:0/0) 2024-02-20T07:54:29.606 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:29.603+0000 7f3af88ca700 1 --2- 172.21.15.133:0/818634581 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f3af4102880 0x7f3af4110db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:54:29.606 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:29.603+0000 7f3af90cb700 1 -- 172.21.15.133:0/818634581 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f3af4102880 msgr2=0x7f3af4110db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:54:29.606 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:29.603+0000 7f3af90cb700 1 --2- 172.21.15.133:0/818634581 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f3af4102880 0x7f3af4110db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:54:29.606 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:29.603+0000 7f3af90cb700 1 -- 172.21.15.133:0/818634581 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ae8005040 con 0x7f3af4101ee0 2024-02-20T07:54:29.606 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:29.603+0000 7f3af88ca700 1 --2- 172.21.15.133:0/818634581 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f3af4102880 0x7f3af4110db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-20T07:54:29.606 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:29.603+0000 7f3af90cb700 1 --2- 172.21.15.133:0/818634581 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f3af4101ee0 0x7f3af4110870 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f3ae400cb10 tx=0x7f3ae4007f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:54:29.606 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:29.603+0000 7f3af1ffb700 1 -- 172.21.15.133:0/818634581 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3ae4004200 con 0x7f3af4101ee0 2024-02-20T07:54:29.606 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:29.603+0000 7f3afb32f700 1 -- 172.21.15.133:0/818634581 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3af410ba40 con 0x7f3af4101ee0 2024-02-20T07:54:29.606 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:29.604+0000 7f3af1ffb700 1 -- 172.21.15.133:0/818634581 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3ae400acc0 con 0x7f3af4101ee0 2024-02-20T07:54:29.606 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:29.604+0000 7f3af1ffb700 1 -- 172.21.15.133:0/818634581 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3ae4003d90 con 0x7f3af4101ee0 2024-02-20T07:54:29.606 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:29.604+0000 7f3afb32f700 1 -- 172.21.15.133:0/818634581 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3af410bf80 con 0x7f3af4101ee0 2024-02-20T07:54:29.607 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:29.605+0000 7f3afb32f700 1 -- 172.21.15.133:0/818634581 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3af4073080 con 0x7f3af4101ee0 2024-02-20T07:54:29.607 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:29.606+0000 7f3af1ffb700 1 -- 172.21.15.133:0/818634581 <== mon.0 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7f3ae4003430 con 0x7f3af4101ee0 2024-02-20T07:54:29.607 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:29.606+0000 7f3af1ffb700 1 --2- 172.21.15.133:0/818634581 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7f3adc069310 0x7f3adc06b7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:54:29.608 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:29.606+0000 7f3af1ffb700 1 -- 172.21.15.133:0/818634581 <== mon.0 v2:172.21.15.133:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f3ae4087410 con 0x7f3af4101ee0 2024-02-20T07:54:29.608 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:29.606+0000 7f3af88ca700 1 --2- 172.21.15.133:0/818634581 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7f3adc069310 0x7f3adc06b7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:54:29.608 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:29.607+0000 7f3af88ca700 1 --2- 172.21.15.133:0/818634581 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7f3adc069310 0x7f3adc06b7c0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f3af410d160 tx=0x7f3ae8009e30 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:54:29.608 INFO:teuthology.orchestra.run.smithi133.stderr:Inferring config /var/lib/ceph/4537327c-cfc4-11ee-95bb-87774f69a715/mon.smithi133/config 2024-02-20T07:54:29.611 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:29.611+0000 7f3af1ffb700 1 -- 172.21.15.133:0/818634581 <== mon.0 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f3ae4053a00 con 0x7f3af4101ee0 2024-02-20T07:54:29.812 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:29.812+0000 7f3afb32f700 1 -- 172.21.15.133:0/818634581 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7f3af410cf40 con 0x7f3af4101ee0 2024-02-20T07:54:29.813 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:29.812+0000 7f3af1ffb700 1 -- 172.21.15.133:0/818634581 <== mon.0 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f3ae4057020 con 0x7f3af4101ee0 2024-02-20T07:54:29.813 INFO:teuthology.orchestra.run.smithi133.stdout:146028888080 2024-02-20T07:54:29.815 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:29.814+0000 7f3afb32f700 1 -- 172.21.15.133:0/818634581 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7f3adc069310 msgr2=0x7f3adc06b7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:54:29.815 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:29.814+0000 7f3afb32f700 1 --2- 172.21.15.133:0/818634581 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7f3adc069310 0x7f3adc06b7c0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f3af410d160 tx=0x7f3ae8009e30 comp rx=0 tx=0).stop 2024-02-20T07:54:29.816 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:29.814+0000 7f3afb32f700 1 -- 172.21.15.133:0/818634581 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f3af4101ee0 msgr2=0x7f3af4110870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:54:29.816 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:29.814+0000 7f3afb32f700 1 --2- 172.21.15.133:0/818634581 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f3af4101ee0 0x7f3af4110870 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f3ae400cb10 tx=0x7f3ae4007f90 comp rx=0 tx=0).stop 2024-02-20T07:54:29.816 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:29.814+0000 7f3afb32f700 1 -- 172.21.15.133:0/818634581 shutdown_connections 2024-02-20T07:54:29.816 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:29.814+0000 7f3afb32f700 1 --2- 172.21.15.133:0/818634581 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f3af4101ee0 0x7f3af4110870 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:54:29.816 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:29.814+0000 7f3afb32f700 1 --2- 172.21.15.133:0/818634581 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7f3adc069310 0x7f3adc06b7c0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:54:29.816 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:29.814+0000 7f3afb32f700 1 --2- 172.21.15.133:0/818634581 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f3af4102880 0x7f3af4110db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:54:29.816 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:29.814+0000 7f3afb32f700 1 -- 172.21.15.133:0/818634581 >> 172.21.15.133:0/818634581 conn(0x7f3af4071560 msgr2=0x7f3af40fa450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:54:29.816 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:29.814+0000 7f3afb32f700 1 -- 172.21.15.133:0/818634581 shutdown_connections 2024-02-20T07:54:29.816 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:29.814+0000 7f3afb32f700 1 -- 172.21.15.133:0/818634581 wait complete. 2024-02-20T07:54:30.140 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:54:29 smithi133 ceph-mon[29117]: from='client.? 172.21.15.133:0/818634581' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-02-20T07:54:30.159 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:30.158+0000 7f5e770d0700 1 -- 172.21.15.133:0/3653303420 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f5e701027e0 msgr2=0x7f5e700fbea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:54:30.159 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:30.158+0000 7f5e770d0700 1 --2- 172.21.15.133:0/3653303420 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f5e701027e0 0x7f5e700fbea0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f5e5c004940 tx=0x7f5e5c009e30 comp rx=0 tx=0).stop 2024-02-20T07:54:30.159 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:30.158+0000 7f5e770d0700 1 -- 172.21.15.133:0/3653303420 shutdown_connections 2024-02-20T07:54:30.159 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:30.158+0000 7f5e770d0700 1 --2- 172.21.15.133:0/3653303420 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f5e701027e0 0x7f5e700fbea0 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:54:30.159 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:30.158+0000 7f5e770d0700 1 --2- 172.21.15.133:0/3653303420 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5e70101e40 0x7f5e70102210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:54:30.159 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:30.158+0000 7f5e770d0700 1 -- 172.21.15.133:0/3653303420 >> 172.21.15.133:0/3653303420 conn(0x7f5e70071d90 msgr2=0x7f5e70072190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:54:30.159 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:30.158+0000 7f5e770d0700 1 -- 172.21.15.133:0/3653303420 shutdown_connections 2024-02-20T07:54:30.159 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:30.158+0000 7f5e770d0700 1 -- 172.21.15.133:0/3653303420 wait complete. 2024-02-20T07:54:30.160 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:30.158+0000 7f5e770d0700 1 Processor -- start 2024-02-20T07:54:30.160 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:30.158+0000 7f5e770d0700 1 -- start start 2024-02-20T07:54:30.161 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:30.159+0000 7f5e770d0700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f5e70101e40 0x7f5e70113a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:54:30.161 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:30.159+0000 7f5e770d0700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5e701027e0 0x7f5e70113fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:54:30.161 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:30.159+0000 7f5e770d0700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e701145f0 con 0x7f5e70101e40 2024-02-20T07:54:30.161 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:30.159+0000 7f5e770d0700 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e7010d920 con 0x7f5e701027e0 2024-02-20T07:54:30.161 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:30.159+0000 7f5e74e6c700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f5e70101e40 0x7f5e70113a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:54:30.162 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:30.159+0000 7f5e74e6c700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f5e70101e40 0x7f5e70113a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.133:53808/0 (socket says 172.21.15.133:53808) 2024-02-20T07:54:30.162 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:30.159+0000 7f5e74e6c700 1 -- 172.21.15.133:0/434289863 learned_addr learned my addr 172.21.15.133:0/434289863 (peer_addr_for_me v2:172.21.15.133:0/0) 2024-02-20T07:54:30.162 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:30.159+0000 7f5e6ffff700 1 --2- 172.21.15.133:0/434289863 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5e701027e0 0x7f5e70113fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:54:30.162 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:30.159+0000 7f5e74e6c700 1 -- 172.21.15.133:0/434289863 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5e701027e0 msgr2=0x7f5e70113fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:54:30.162 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:30.159+0000 7f5e74e6c700 1 --2- 172.21.15.133:0/434289863 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5e701027e0 0x7f5e70113fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:54:30.162 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:30.159+0000 7f5e74e6c700 1 -- 172.21.15.133:0/434289863 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e5c005040 con 0x7f5e70101e40 2024-02-20T07:54:30.162 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:30.159+0000 7f5e74e6c700 1 --2- 172.21.15.133:0/434289863 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f5e70101e40 0x7f5e70113a60 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f5e64002a50 tx=0x7f5e640046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:54:30.162 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:30.160+0000 7f5e6d7fa700 1 -- 172.21.15.133:0/434289863 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5e64007cc0 con 0x7f5e70101e40 2024-02-20T07:54:30.162 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:30.160+0000 7f5e6d7fa700 1 -- 172.21.15.133:0/434289863 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5e6400f040 con 0x7f5e70101e40 2024-02-20T07:54:30.162 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:30.160+0000 7f5e6d7fa700 1 -- 172.21.15.133:0/434289863 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5e6400da30 con 0x7f5e70101e40 2024-02-20T07:54:30.162 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:30.160+0000 7f5e770d0700 1 -- 172.21.15.133:0/434289863 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e7010dc00 con 0x7f5e70101e40 2024-02-20T07:54:30.162 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:30.160+0000 7f5e770d0700 1 -- 172.21.15.133:0/434289863 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e7010e110 con 0x7f5e70101e40 2024-02-20T07:54:30.163 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:30.161+0000 7f5e770d0700 1 -- 172.21.15.133:0/434289863 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5e700738b0 con 0x7f5e70101e40 2024-02-20T07:54:30.163 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:30.161+0000 7f5e6d7fa700 1 -- 172.21.15.133:0/434289863 <== mon.0 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7f5e6400db90 con 0x7f5e70101e40 2024-02-20T07:54:30.163 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:30.162+0000 7f5e6d7fa700 1 --2- 172.21.15.133:0/434289863 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7f5e6006d6c0 0x7f5e6006fb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:54:30.163 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:30.162+0000 7f5e6d7fa700 1 -- 172.21.15.133:0/434289863 <== mon.0 v2:172.21.15.133:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f5e64086c00 con 0x7f5e70101e40 2024-02-20T07:54:30.163 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:30.162+0000 7f5e6ffff700 1 --2- 172.21.15.133:0/434289863 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7f5e6006d6c0 0x7f5e6006fb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:54:30.163 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:30.162+0000 7f5e6ffff700 1 --2- 172.21.15.133:0/434289863 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7f5e6006d6c0 0x7f5e6006fb70 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f5e7010f390 tx=0x7f5e5c009e30 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:54:30.166 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:30.166+0000 7f5e6d7fa700 1 -- 172.21.15.133:0/434289863 <== mon.0 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f5e6408c020 con 0x7f5e70101e40 2024-02-20T07:54:30.255 INFO:teuthology.orchestra.run.smithi133.stderr:Inferring config /var/lib/ceph/4537327c-cfc4-11ee-95bb-87774f69a715/mon.smithi133/config 2024-02-20T07:54:30.297 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:54:29 smithi183 ceph-mon[35832]: from='client.? 172.21.15.133:0/818634581' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-02-20T07:54:30.363 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:30.362+0000 7f5e770d0700 1 -- 172.21.15.133:0/434289863 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f5e7010f040 con 0x7f5e70101e40 2024-02-20T07:54:30.363 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:30.362+0000 7f5e6d7fa700 1 -- 172.21.15.133:0/434289863 <== mon.0 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f5e640569f0 con 0x7f5e70101e40 2024-02-20T07:54:30.363 INFO:teuthology.orchestra.run.smithi133.stdout:60129542175 2024-02-20T07:54:30.365 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:30.364+0000 7f5e770d0700 1 -- 172.21.15.133:0/434289863 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7f5e6006d6c0 msgr2=0x7f5e6006fb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:54:30.365 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:30.364+0000 7f5e770d0700 1 --2- 172.21.15.133:0/434289863 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7f5e6006d6c0 0x7f5e6006fb70 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f5e7010f390 tx=0x7f5e5c009e30 comp rx=0 tx=0).stop 2024-02-20T07:54:30.365 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:30.364+0000 7f5e770d0700 1 -- 172.21.15.133:0/434289863 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f5e70101e40 msgr2=0x7f5e70113a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:54:30.365 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:30.364+0000 7f5e770d0700 1 --2- 172.21.15.133:0/434289863 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f5e70101e40 0x7f5e70113a60 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f5e64002a50 tx=0x7f5e640046b0 comp rx=0 tx=0).stop 2024-02-20T07:54:30.365 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:30.364+0000 7f5e770d0700 1 -- 172.21.15.133:0/434289863 shutdown_connections 2024-02-20T07:54:30.366 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:30.364+0000 7f5e770d0700 1 --2- 172.21.15.133:0/434289863 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f5e70101e40 0x7f5e70113a60 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:54:30.366 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:30.364+0000 7f5e770d0700 1 --2- 172.21.15.133:0/434289863 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7f5e6006d6c0 0x7f5e6006fb70 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:54:30.366 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:30.364+0000 7f5e770d0700 1 --2- 172.21.15.133:0/434289863 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5e701027e0 0x7f5e70113fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:54:30.366 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:30.364+0000 7f5e770d0700 1 -- 172.21.15.133:0/434289863 >> 172.21.15.133:0/434289863 conn(0x7f5e70071d90 msgr2=0x7f5e70100110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:54:30.366 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:30.364+0000 7f5e770d0700 1 -- 172.21.15.133:0/434289863 shutdown_connections 2024-02-20T07:54:30.366 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:30.364+0000 7f5e770d0700 1 -- 172.21.15.133:0/434289863 wait complete. 2024-02-20T07:54:30.853 INFO:tasks.cephadm.ceph_manager.ceph:need seq 146028888078 got 146028888080 for osd.4 2024-02-20T07:54:30.853 DEBUG:teuthology.parallel:result is None 2024-02-20T07:54:31.241 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:54:30 smithi133 ceph-mon[29117]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-20T07:54:31.242 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:54:30 smithi133 ceph-mon[29117]: from='client.? 172.21.15.133:0/434289863' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-02-20T07:54:31.298 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:54:30 smithi183 ceph-mon[35832]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-20T07:54:31.298 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:54:30 smithi183 ceph-mon[35832]: from='client.? 172.21.15.133:0/434289863' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-02-20T07:54:33.041 INFO:tasks.cephadm.ceph_manager.ceph:need seq 60129542173 got 60129542175 for osd.1 2024-02-20T07:54:33.042 DEBUG:teuthology.parallel:result is None 2024-02-20T07:54:33.241 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:54:32 smithi133 ceph-mon[29117]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-20T07:54:33.297 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:54:32 smithi183 ceph-mon[35832]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-20T07:54:34.504 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:34.501+0000 7f17afbf5700 1 -- 172.21.15.133:0/3170744888 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f17a8102f50 msgr2=0x7f17a8103320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:54:34.504 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:34.501+0000 7f17afbf5700 1 --2- 172.21.15.133:0/3170744888 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f17a8102f50 0x7f17a8103320 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f17a0004940 tx=0x7f17a0009e30 comp rx=0 tx=0).stop 2024-02-20T07:54:34.504 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:34.501+0000 7f17afbf5700 1 -- 172.21.15.133:0/3170744888 shutdown_connections 2024-02-20T07:54:34.504 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:34.501+0000 7f17afbf5700 1 --2- 172.21.15.133:0/3170744888 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f17a81038f0 0x7f17a80fcf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:54:34.505 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:34.501+0000 7f17afbf5700 1 --2- 172.21.15.133:0/3170744888 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f17a8102f50 0x7f17a8103320 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:54:34.505 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:34.501+0000 7f17afbf5700 1 -- 172.21.15.133:0/3170744888 >> 172.21.15.133:0/3170744888 conn(0x7f17a80f8ae0 msgr2=0x7f17a80faef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:54:34.505 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:34.501+0000 7f17afbf5700 1 -- 172.21.15.133:0/3170744888 shutdown_connections 2024-02-20T07:54:34.505 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:34.501+0000 7f17afbf5700 1 -- 172.21.15.133:0/3170744888 wait complete. 2024-02-20T07:54:34.505 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:34.502+0000 7f17afbf5700 1 Processor -- start 2024-02-20T07:54:34.505 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:34.502+0000 7f17afbf5700 1 -- start start 2024-02-20T07:54:34.505 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:34.502+0000 7f17afbf5700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f17a8102f50 0x7f17a8113a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:54:34.505 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:34.502+0000 7f17afbf5700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f17a81038f0 0x7f17a8113fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:54:34.505 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:34.502+0000 7f17afbf5700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17a8114600 con 0x7f17a8102f50 2024-02-20T07:54:34.505 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:34.502+0000 7f17afbf5700 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17a810d930 con 0x7f17a81038f0 2024-02-20T07:54:34.505 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:34.502+0000 7f17ad991700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f17a8102f50 0x7f17a8113a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:54:34.505 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:34.502+0000 7f17ad991700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f17a8102f50 0x7f17a8113a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.133:53826/0 (socket says 172.21.15.133:53826) 2024-02-20T07:54:34.505 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:34.502+0000 7f17ad991700 1 -- 172.21.15.133:0/2971765184 learned_addr learned my addr 172.21.15.133:0/2971765184 (peer_addr_for_me v2:172.21.15.133:0/0) 2024-02-20T07:54:34.505 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:34.503+0000 7f17ad991700 1 -- 172.21.15.133:0/2971765184 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f17a81038f0 msgr2=0x7f17a8113fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:54:34.505 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:34.503+0000 7f17ad991700 1 --2- 172.21.15.133:0/2971765184 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f17a81038f0 0x7f17a8113fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:54:34.506 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:34.503+0000 7f17ad991700 1 -- 172.21.15.133:0/2971765184 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17a0005040 con 0x7f17a8102f50 2024-02-20T07:54:34.506 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:34.503+0000 7f17ad991700 1 --2- 172.21.15.133:0/2971765184 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f17a8102f50 0x7f17a8113a70 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f17a000b3b0 tx=0x7f17a0009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:54:34.506 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:34.503+0000 7f179e7fc700 1 -- 172.21.15.133:0/2971765184 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f17a000e040 con 0x7f17a8102f50 2024-02-20T07:54:34.506 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:34.503+0000 7f179e7fc700 1 -- 172.21.15.133:0/2971765184 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f17a000bbf0 con 0x7f17a8102f50 2024-02-20T07:54:34.506 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:34.503+0000 7f17afbf5700 1 -- 172.21.15.133:0/2971765184 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f17a810dbb0 con 0x7f17a8102f50 2024-02-20T07:54:34.506 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:34.503+0000 7f179e7fc700 1 -- 172.21.15.133:0/2971765184 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f17a001c630 con 0x7f17a8102f50 2024-02-20T07:54:34.506 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:34.503+0000 7f17afbf5700 1 -- 172.21.15.133:0/2971765184 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f17a810e0f0 con 0x7f17a8102f50 2024-02-20T07:54:34.506 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:34.504+0000 7f17afbf5700 1 -- 172.21.15.133:0/2971765184 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f17a80fc6e0 con 0x7f17a8102f50 2024-02-20T07:54:34.507 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:34.504+0000 7f179e7fc700 1 -- 172.21.15.133:0/2971765184 <== mon.0 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7f17a000bd60 con 0x7f17a8102f50 2024-02-20T07:54:34.507 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:34.505+0000 7f179e7fc700 1 --2- 172.21.15.133:0/2971765184 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7f17940692a0 0x7f179406b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:54:34.507 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:34.505+0000 7f179e7fc700 1 -- 172.21.15.133:0/2971765184 <== mon.0 v2:172.21.15.133:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f17a00882b0 con 0x7f17a8102f50 2024-02-20T07:54:34.507 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:34.505+0000 7f17ad190700 1 --2- 172.21.15.133:0/2971765184 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7f17940692a0 0x7f179406b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:54:34.507 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:34.505+0000 7f17ad190700 1 --2- 172.21.15.133:0/2971765184 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7f17940692a0 0x7f179406b750 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f17a810f2a0 tx=0x7f1798007d40 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:54:34.509 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:34.509+0000 7f179e7fc700 1 -- 172.21.15.133:0/2971765184 <== mon.0 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f17a0058350 con 0x7f17a8102f50 2024-02-20T07:54:34.698 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:34.698+0000 7f17afbf5700 1 -- 172.21.15.133:0/2971765184 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7f17a804bed0 con 0x7f17a8102f50 2024-02-20T07:54:34.699 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:34.698+0000 7f179e7fc700 1 -- 172.21.15.133:0/2971765184 <== mon.0 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f17a0057ee0 con 0x7f17a8102f50 2024-02-20T07:54:34.699 INFO:teuthology.orchestra.run.smithi133.stdout:111669149718 2024-02-20T07:54:34.701 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:34.700+0000 7f17afbf5700 1 -- 172.21.15.133:0/2971765184 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7f17940692a0 msgr2=0x7f179406b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:54:34.701 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:34.700+0000 7f17afbf5700 1 --2- 172.21.15.133:0/2971765184 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7f17940692a0 0x7f179406b750 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f17a810f2a0 tx=0x7f1798007d40 comp rx=0 tx=0).stop 2024-02-20T07:54:34.701 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:34.700+0000 7f17afbf5700 1 -- 172.21.15.133:0/2971765184 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f17a8102f50 msgr2=0x7f17a8113a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:54:34.701 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:34.700+0000 7f17afbf5700 1 --2- 172.21.15.133:0/2971765184 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f17a8102f50 0x7f17a8113a70 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f17a000b3b0 tx=0x7f17a0009e30 comp rx=0 tx=0).stop 2024-02-20T07:54:34.701 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:34.700+0000 7f17afbf5700 1 -- 172.21.15.133:0/2971765184 shutdown_connections 2024-02-20T07:54:34.701 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:34.700+0000 7f17afbf5700 1 --2- 172.21.15.133:0/2971765184 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f17a8102f50 0x7f17a8113a70 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:54:34.701 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:34.700+0000 7f17afbf5700 1 --2- 172.21.15.133:0/2971765184 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7f17940692a0 0x7f179406b750 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:54:34.701 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:34.700+0000 7f17afbf5700 1 --2- 172.21.15.133:0/2971765184 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f17a81038f0 0x7f17a8113fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:54:34.701 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:34.700+0000 7f17afbf5700 1 -- 172.21.15.133:0/2971765184 >> 172.21.15.133:0/2971765184 conn(0x7f17a80f8ae0 msgr2=0x7f17a80fa500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:54:34.702 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:34.700+0000 7f17afbf5700 1 -- 172.21.15.133:0/2971765184 shutdown_connections 2024-02-20T07:54:34.702 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:34.700+0000 7f17afbf5700 1 -- 172.21.15.133:0/2971765184 wait complete. 2024-02-20T07:54:35.234 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:54:34 smithi133 ceph-mon[29117]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-20T07:54:35.234 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:54:34 smithi133 ceph-mon[29117]: from='client.? 172.21.15.133:0/2971765184' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-02-20T07:54:35.297 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:54:34 smithi183 ceph-mon[35832]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-20T07:54:35.297 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:54:34 smithi183 ceph-mon[35832]: from='client.? 172.21.15.133:0/2971765184' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-02-20T07:54:35.985 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:35.981+0000 7f0e4fd40700 1 -- 172.21.15.133:0/603554140 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f0e48102f50 msgr2=0x7f0e48103320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:54:35.985 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:35.981+0000 7f0e4fd40700 1 --2- 172.21.15.133:0/603554140 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f0e48102f50 0x7f0e48103320 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f0e44004940 tx=0x7f0e44009e30 comp rx=0 tx=0).stop 2024-02-20T07:54:35.986 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:35.981+0000 7f0e4fd40700 1 -- 172.21.15.133:0/603554140 shutdown_connections 2024-02-20T07:54:35.986 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:35.981+0000 7f0e4fd40700 1 --2- 172.21.15.133:0/603554140 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0e481038f0 0x7f0e480fcf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:54:35.986 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:35.981+0000 7f0e4fd40700 1 --2- 172.21.15.133:0/603554140 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f0e48102f50 0x7f0e48103320 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:54:35.986 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:35.981+0000 7f0e4fd40700 1 -- 172.21.15.133:0/603554140 >> 172.21.15.133:0/603554140 conn(0x7f0e480f8ae0 msgr2=0x7f0e480faef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:54:35.986 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:35.981+0000 7f0e4fd40700 1 -- 172.21.15.133:0/603554140 shutdown_connections 2024-02-20T07:54:35.986 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:35.981+0000 7f0e4fd40700 1 -- 172.21.15.133:0/603554140 wait complete. 2024-02-20T07:54:35.986 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:35.981+0000 7f0e4fd40700 1 Processor -- start 2024-02-20T07:54:35.986 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:35.981+0000 7f0e4fd40700 1 -- start start 2024-02-20T07:54:35.986 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:35.982+0000 7f0e4fd40700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0e48102f50 0x7f0e48113a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:54:35.986 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:35.982+0000 7f0e4fd40700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f0e481038f0 0x7f0e48113f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:54:35.986 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:35.982+0000 7f0e4fd40700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e481145e0 con 0x7f0e481038f0 2024-02-20T07:54:35.986 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:35.982+0000 7f0e4fd40700 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e4810d910 con 0x7f0e48102f50 2024-02-20T07:54:35.986 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:35.982+0000 7f0e4d2db700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f0e481038f0 0x7f0e48113f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:54:35.986 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:35.982+0000 7f0e4d2db700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f0e481038f0 0x7f0e48113f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.133:53840/0 (socket says 172.21.15.133:53840) 2024-02-20T07:54:35.986 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:35.982+0000 7f0e4d2db700 1 -- 172.21.15.133:0/2291594752 learned_addr learned my addr 172.21.15.133:0/2291594752 (peer_addr_for_me v2:172.21.15.133:0/0) 2024-02-20T07:54:35.987 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:35.982+0000 7f0e4dadc700 1 --2- 172.21.15.133:0/2291594752 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0e48102f50 0x7f0e48113a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:54:35.987 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:35.982+0000 7f0e4d2db700 1 -- 172.21.15.133:0/2291594752 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0e48102f50 msgr2=0x7f0e48113a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:54:35.987 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:35.982+0000 7f0e4d2db700 1 --2- 172.21.15.133:0/2291594752 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0e48102f50 0x7f0e48113a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:54:35.987 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:35.982+0000 7f0e4d2db700 1 -- 172.21.15.133:0/2291594752 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e44005040 con 0x7f0e481038f0 2024-02-20T07:54:35.987 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:35.982+0000 7f0e4dadc700 1 --2- 172.21.15.133:0/2291594752 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0e48102f50 0x7f0e48113a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-20T07:54:35.987 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:35.982+0000 7f0e4d2db700 1 --2- 172.21.15.133:0/2291594752 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f0e481038f0 0x7f0e48113f90 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f0e3800c800 tx=0x7f0e38004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:54:35.987 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:35.982+0000 7f0e3e7fc700 1 -- 172.21.15.133:0/2291594752 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0e3800a760 con 0x7f0e481038f0 2024-02-20T07:54:35.987 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:35.982+0000 7f0e3e7fc700 1 -- 172.21.15.133:0/2291594752 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0e3800ad90 con 0x7f0e481038f0 2024-02-20T07:54:35.987 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:35.982+0000 7f0e4fd40700 1 -- 172.21.15.133:0/2291594752 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0e4810dbf0 con 0x7f0e481038f0 2024-02-20T07:54:35.987 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:35.982+0000 7f0e3e7fc700 1 -- 172.21.15.133:0/2291594752 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0e380113a0 con 0x7f0e481038f0 2024-02-20T07:54:35.987 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:35.982+0000 7f0e4fd40700 1 -- 172.21.15.133:0/2291594752 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0e4810e130 con 0x7f0e481038f0 2024-02-20T07:54:35.987 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:35.983+0000 7f0e4fd40700 1 -- 172.21.15.133:0/2291594752 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0e480fc6e0 con 0x7f0e481038f0 2024-02-20T07:54:35.987 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:35.984+0000 7f0e3e7fc700 1 -- 172.21.15.133:0/2291594752 <== mon.0 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7f0e38012400 con 0x7f0e481038f0 2024-02-20T07:54:35.987 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:35.984+0000 7f0e3e7fc700 1 --2- 172.21.15.133:0/2291594752 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7f0e340692a0 0x7f0e3406b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:54:35.987 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:35.984+0000 7f0e3e7fc700 1 -- 172.21.15.133:0/2291594752 <== mon.0 v2:172.21.15.133:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f0e38086db0 con 0x7f0e481038f0 2024-02-20T07:54:35.988 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:35.984+0000 7f0e4dadc700 1 --2- 172.21.15.133:0/2291594752 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7f0e340692a0 0x7f0e3406b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:54:35.988 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:35.984+0000 7f0e4dadc700 1 --2- 172.21.15.133:0/2291594752 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7f0e340692a0 0x7f0e3406b750 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f0e44004560 tx=0x7f0e44009e30 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:54:35.991 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:35.990+0000 7f0e3e7fc700 1 -- 172.21.15.133:0/2291594752 <== mon.0 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f0e38056f50 con 0x7f0e481038f0 2024-02-20T07:54:36.180 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:36.179+0000 7f0e4fd40700 1 -- 172.21.15.133:0/2291594752 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7f0e4804bed0 con 0x7f0e481038f0 2024-02-20T07:54:36.180 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:36.179+0000 7f0e3e7fc700 1 -- 172.21.15.133:0/2291594752 <== mon.0 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f0e38056ae0 con 0x7f0e481038f0 2024-02-20T07:54:36.181 INFO:teuthology.orchestra.run.smithi133.stdout:38654705702 2024-02-20T07:54:36.182 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:36.181+0000 7f0e4fd40700 1 -- 172.21.15.133:0/2291594752 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7f0e340692a0 msgr2=0x7f0e3406b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:54:36.182 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:36.181+0000 7f0e4fd40700 1 --2- 172.21.15.133:0/2291594752 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7f0e340692a0 0x7f0e3406b750 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f0e44004560 tx=0x7f0e44009e30 comp rx=0 tx=0).stop 2024-02-20T07:54:36.182 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:36.181+0000 7f0e4fd40700 1 -- 172.21.15.133:0/2291594752 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f0e481038f0 msgr2=0x7f0e48113f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:54:36.182 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:36.181+0000 7f0e4fd40700 1 --2- 172.21.15.133:0/2291594752 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f0e481038f0 0x7f0e48113f90 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f0e3800c800 tx=0x7f0e38004470 comp rx=0 tx=0).stop 2024-02-20T07:54:36.182 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:36.181+0000 7f0e4fd40700 1 -- 172.21.15.133:0/2291594752 shutdown_connections 2024-02-20T07:54:36.182 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:36.181+0000 7f0e4fd40700 1 --2- 172.21.15.133:0/2291594752 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7f0e340692a0 0x7f0e3406b750 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:54:36.182 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:36.181+0000 7f0e4fd40700 1 --2- 172.21.15.133:0/2291594752 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0e48102f50 0x7f0e48113a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:54:36.182 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:36.181+0000 7f0e4fd40700 1 --2- 172.21.15.133:0/2291594752 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f0e481038f0 0x7f0e48113f90 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:54:36.182 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:36.181+0000 7f0e4fd40700 1 -- 172.21.15.133:0/2291594752 >> 172.21.15.133:0/2291594752 conn(0x7f0e480f8ae0 msgr2=0x7f0e480fa500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:54:36.182 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:36.181+0000 7f0e4fd40700 1 -- 172.21.15.133:0/2291594752 shutdown_connections 2024-02-20T07:54:36.182 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:36.181+0000 7f0e4fd40700 1 -- 172.21.15.133:0/2291594752 wait complete. 2024-02-20T07:54:36.694 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:36.691+0000 7f0d6ce59700 1 -- 172.21.15.133:0/3201448081 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f0d681027f0 msgr2=0x7f0d680fbe60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:54:36.694 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:36.691+0000 7f0d6ce59700 1 --2- 172.21.15.133:0/3201448081 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f0d681027f0 0x7f0d680fbe60 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f0d5c004940 tx=0x7f0d5c009e30 comp rx=0 tx=0).stop 2024-02-20T07:54:36.694 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:36.691+0000 7f0d6ce59700 1 -- 172.21.15.133:0/3201448081 shutdown_connections 2024-02-20T07:54:36.694 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:36.691+0000 7f0d6ce59700 1 --2- 172.21.15.133:0/3201448081 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f0d681027f0 0x7f0d680fbe60 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:54:36.695 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:36.691+0000 7f0d6ce59700 1 --2- 172.21.15.133:0/3201448081 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0d68101e50 0x7f0d68102220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:54:36.695 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:36.691+0000 7f0d6ce59700 1 -- 172.21.15.133:0/3201448081 >> 172.21.15.133:0/3201448081 conn(0x7f0d680f79e0 msgr2=0x7f0d680f9df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:54:36.695 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:36.691+0000 7f0d6ce59700 1 -- 172.21.15.133:0/3201448081 shutdown_connections 2024-02-20T07:54:36.695 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:36.691+0000 7f0d6ce59700 1 -- 172.21.15.133:0/3201448081 wait complete. 2024-02-20T07:54:36.695 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:36.692+0000 7f0d6ce59700 1 Processor -- start 2024-02-20T07:54:36.695 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:36.692+0000 7f0d6ce59700 1 -- start start 2024-02-20T07:54:36.695 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:36.692+0000 7f0d6ce59700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f0d68101e50 0x7f0d6810ec30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:54:36.695 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:36.692+0000 7f0d6ce59700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0d681027f0 0x7f0d6810f170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:54:36.695 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:36.692+0000 7f0d6ce59700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d6810f7c0 con 0x7f0d68101e50 2024-02-20T07:54:36.695 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:36.692+0000 7f0d6ce59700 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d6810f930 con 0x7f0d681027f0 2024-02-20T07:54:36.695 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:36.692+0000 7f0d6659c700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f0d68101e50 0x7f0d6810ec30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:54:36.695 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:36.692+0000 7f0d6659c700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f0d68101e50 0x7f0d6810ec30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.133:47556/0 (socket says 172.21.15.133:47556) 2024-02-20T07:54:36.696 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:36.692+0000 7f0d6659c700 1 -- 172.21.15.133:0/2579222140 learned_addr learned my addr 172.21.15.133:0/2579222140 (peer_addr_for_me v2:172.21.15.133:0/0) 2024-02-20T07:54:36.696 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:36.692+0000 7f0d65d9b700 1 --2- 172.21.15.133:0/2579222140 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0d681027f0 0x7f0d6810f170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:54:36.696 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:36.692+0000 7f0d6659c700 1 -- 172.21.15.133:0/2579222140 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0d681027f0 msgr2=0x7f0d6810f170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:54:36.696 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:36.692+0000 7f0d6659c700 1 --2- 172.21.15.133:0/2579222140 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0d681027f0 0x7f0d6810f170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:54:36.696 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:36.692+0000 7f0d6659c700 1 -- 172.21.15.133:0/2579222140 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d5c005040 con 0x7f0d68101e50 2024-02-20T07:54:36.696 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:36.692+0000 7f0d6659c700 1 --2- 172.21.15.133:0/2579222140 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f0d68101e50 0x7f0d6810ec30 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f0d5800b790 tx=0x7f0d58009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:54:36.696 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:36.692+0000 7f0d56ffd700 1 -- 172.21.15.133:0/2579222140 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0d5800f6f0 con 0x7f0d68101e50 2024-02-20T07:54:36.696 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:36.693+0000 7f0d6ce59700 1 -- 172.21.15.133:0/2579222140 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0d68117c80 con 0x7f0d68101e50 2024-02-20T07:54:36.696 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:36.693+0000 7f0d6ce59700 1 -- 172.21.15.133:0/2579222140 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0d681181c0 con 0x7f0d68101e50 2024-02-20T07:54:36.696 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:36.693+0000 7f0d56ffd700 1 -- 172.21.15.133:0/2579222140 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0d5800fd20 con 0x7f0d68101e50 2024-02-20T07:54:36.696 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:36.693+0000 7f0d56ffd700 1 -- 172.21.15.133:0/2579222140 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0d580103a0 con 0x7f0d68101e50 2024-02-20T07:54:36.696 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:36.694+0000 7f0d6ce59700 1 -- 172.21.15.133:0/2579222140 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0d480052e0 con 0x7f0d68101e50 2024-02-20T07:54:36.697 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:36.694+0000 7f0d56ffd700 1 -- 172.21.15.133:0/2579222140 <== mon.0 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7f0d5800f850 con 0x7f0d68101e50 2024-02-20T07:54:36.697 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:36.695+0000 7f0d56ffd700 1 --2- 172.21.15.133:0/2579222140 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7f0d500692b0 0x7f0d5006b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:54:36.697 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:36.695+0000 7f0d56ffd700 1 -- 172.21.15.133:0/2579222140 <== mon.0 v2:172.21.15.133:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f0d58086b40 con 0x7f0d68101e50 2024-02-20T07:54:36.697 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:36.695+0000 7f0d65d9b700 1 --2- 172.21.15.133:0/2579222140 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7f0d500692b0 0x7f0d5006b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:54:36.697 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:36.695+0000 7f0d65d9b700 1 --2- 172.21.15.133:0/2579222140 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7f0d500692b0 0x7f0d5006b760 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f0d68066a30 tx=0x7f0d5c009e30 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:54:36.700 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:36.699+0000 7f0d56ffd700 1 -- 172.21.15.133:0/2579222140 <== mon.0 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f0d58053250 con 0x7f0d68101e50 2024-02-20T07:54:36.892 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:36.891+0000 7f0d6ce59700 1 -- 172.21.15.133:0/2579222140 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7f0d48005590 con 0x7f0d68101e50 2024-02-20T07:54:36.892 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:36.891+0000 7f0d56ffd700 1 -- 172.21.15.133:0/2579222140 <== mon.0 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f0d58056870 con 0x7f0d68101e50 2024-02-20T07:54:36.892 INFO:teuthology.orchestra.run.smithi133.stdout:167503724556 2024-02-20T07:54:36.894 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:36.893+0000 7f0d6ce59700 1 -- 172.21.15.133:0/2579222140 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7f0d500692b0 msgr2=0x7f0d5006b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:54:36.894 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:36.893+0000 7f0d6ce59700 1 --2- 172.21.15.133:0/2579222140 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7f0d500692b0 0x7f0d5006b760 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f0d68066a30 tx=0x7f0d5c009e30 comp rx=0 tx=0).stop 2024-02-20T07:54:36.894 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:36.893+0000 7f0d6ce59700 1 -- 172.21.15.133:0/2579222140 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f0d68101e50 msgr2=0x7f0d6810ec30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:54:36.894 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:36.893+0000 7f0d6ce59700 1 --2- 172.21.15.133:0/2579222140 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f0d68101e50 0x7f0d6810ec30 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f0d5800b790 tx=0x7f0d58009250 comp rx=0 tx=0).stop 2024-02-20T07:54:36.894 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:36.893+0000 7f0d6ce59700 1 -- 172.21.15.133:0/2579222140 shutdown_connections 2024-02-20T07:54:36.894 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:36.893+0000 7f0d6ce59700 1 --2- 172.21.15.133:0/2579222140 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f0d68101e50 0x7f0d6810ec30 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:54:36.894 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:36.893+0000 7f0d6ce59700 1 --2- 172.21.15.133:0/2579222140 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7f0d500692b0 0x7f0d5006b760 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:54:36.895 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:36.893+0000 7f0d6ce59700 1 --2- 172.21.15.133:0/2579222140 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0d681027f0 0x7f0d6810f170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:54:36.895 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:36.893+0000 7f0d6ce59700 1 -- 172.21.15.133:0/2579222140 >> 172.21.15.133:0/2579222140 conn(0x7f0d680f79e0 msgr2=0x7f0d680f94b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:54:36.895 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:36.893+0000 7f0d6ce59700 1 -- 172.21.15.133:0/2579222140 shutdown_connections 2024-02-20T07:54:36.895 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:36.893+0000 7f0d6ce59700 1 -- 172.21.15.133:0/2579222140 wait complete. 2024-02-20T07:54:37.202 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:54:36 smithi133 ceph-mon[29117]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-20T07:54:37.202 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:54:36 smithi133 ceph-mon[29117]: from='client.? 172.21.15.133:0/2291594752' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-02-20T07:54:37.202 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:54:36 smithi133 ceph-mon[29117]: from='client.? 172.21.15.133:0/2579222140' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-02-20T07:54:37.297 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:54:36 smithi183 ceph-mon[35832]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-20T07:54:37.297 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:54:36 smithi183 ceph-mon[35832]: from='client.? 172.21.15.133:0/2291594752' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-02-20T07:54:37.297 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:54:36 smithi183 ceph-mon[35832]: from='client.? 172.21.15.133:0/2579222140' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-02-20T07:54:37.747 INFO:tasks.cephadm.ceph_manager.ceph:need seq 111669149716 got 111669149718 for osd.3 2024-02-20T07:54:37.747 DEBUG:teuthology.parallel:result is None 2024-02-20T07:54:38.112 INFO:tasks.cephadm.ceph_manager.ceph:need seq 38654705700 got 38654705702 for osd.0 2024-02-20T07:54:38.113 DEBUG:teuthology.parallel:result is None 2024-02-20T07:54:38.479 INFO:tasks.cephadm.ceph_manager.ceph:need seq 167503724554 got 167503724556 for osd.5 2024-02-20T07:54:38.479 DEBUG:teuthology.parallel:result is None 2024-02-20T07:54:39.218 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:54:38 smithi133 ceph-mon[29117]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-20T07:54:39.218 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:54:38 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-20T07:54:39.297 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:54:38 smithi183 ceph-mon[35832]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-20T07:54:39.297 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:54:38 smithi183 ceph-mon[35832]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-20T07:54:39.450 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:39.447+0000 7fab763ee700 1 -- 172.21.15.133:0/4242984132 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fab700fbd20 msgr2=0x7fab700fc0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:54:39.450 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:39.447+0000 7fab763ee700 1 --2- 172.21.15.133:0/4242984132 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fab700fbd20 0x7fab700fc0f0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fab60004940 tx=0x7fab60009e30 comp rx=0 tx=0).stop 2024-02-20T07:54:39.450 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:39.447+0000 7fab763ee700 1 -- 172.21.15.133:0/4242984132 shutdown_connections 2024-02-20T07:54:39.450 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:39.447+0000 7fab763ee700 1 --2- 172.21.15.133:0/4242984132 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fab700fc630 0x7fab70109390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:54:39.450 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:39.447+0000 7fab763ee700 1 --2- 172.21.15.133:0/4242984132 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fab700fbd20 0x7fab700fc0f0 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:54:39.450 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:39.447+0000 7fab763ee700 1 -- 172.21.15.133:0/4242984132 >> 172.21.15.133:0/4242984132 conn(0x7fab700f7a10 msgr2=0x7fab700f9e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:54:39.450 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:39.447+0000 7fab763ee700 1 -- 172.21.15.133:0/4242984132 shutdown_connections 2024-02-20T07:54:39.451 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:39.447+0000 7fab763ee700 1 -- 172.21.15.133:0/4242984132 wait complete. 2024-02-20T07:54:39.451 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:39.448+0000 7fab763ee700 1 Processor -- start 2024-02-20T07:54:39.451 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:39.448+0000 7fab763ee700 1 -- start start 2024-02-20T07:54:39.451 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:39.448+0000 7fab763ee700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fab700fbd20 0x7fab70115bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:54:39.451 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:39.448+0000 7fab763ee700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fab700fc630 0x7fab70116110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:54:39.451 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:39.448+0000 7fab763ee700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab70116760 con 0x7fab700fbd20 2024-02-20T07:54:39.451 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:39.448+0000 7fab763ee700 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab7010fa90 con 0x7fab700fc630 2024-02-20T07:54:39.451 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:39.448+0000 7fab6ffff700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fab700fbd20 0x7fab70115bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:54:39.451 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:39.448+0000 7fab6ffff700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fab700fbd20 0x7fab70115bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.133:47578/0 (socket says 172.21.15.133:47578) 2024-02-20T07:54:39.451 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:39.448+0000 7fab6ffff700 1 -- 172.21.15.133:0/1705517695 learned_addr learned my addr 172.21.15.133:0/1705517695 (peer_addr_for_me v2:172.21.15.133:0/0) 2024-02-20T07:54:39.451 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:39.448+0000 7fab6f7fe700 1 --2- 172.21.15.133:0/1705517695 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fab700fc630 0x7fab70116110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:54:39.451 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:39.449+0000 7fab6ffff700 1 -- 172.21.15.133:0/1705517695 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fab700fc630 msgr2=0x7fab70116110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:54:39.452 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:39.449+0000 7fab6ffff700 1 --2- 172.21.15.133:0/1705517695 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fab700fc630 0x7fab70116110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:54:39.452 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:39.449+0000 7fab6ffff700 1 -- 172.21.15.133:0/1705517695 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab60005040 con 0x7fab700fbd20 2024-02-20T07:54:39.452 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:39.449+0000 7fab6f7fe700 1 --2- 172.21.15.133:0/1705517695 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fab700fc630 0x7fab70116110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-20T07:54:39.452 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:39.449+0000 7fab6ffff700 1 --2- 172.21.15.133:0/1705517695 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fab700fbd20 0x7fab70115bd0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fab60004890 tx=0x7fab60009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:54:39.452 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:39.449+0000 7fab6cff9700 1 -- 172.21.15.133:0/1705517695 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fab6000e040 con 0x7fab700fbd20 2024-02-20T07:54:39.452 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:39.449+0000 7fab6cff9700 1 -- 172.21.15.133:0/1705517695 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fab60004290 con 0x7fab700fbd20 2024-02-20T07:54:39.452 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:39.449+0000 7fab6cff9700 1 -- 172.21.15.133:0/1705517695 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fab6001c630 con 0x7fab700fbd20 2024-02-20T07:54:39.452 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:39.449+0000 7fab763ee700 1 -- 172.21.15.133:0/1705517695 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fab7010fd70 con 0x7fab700fbd20 2024-02-20T07:54:39.452 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:39.449+0000 7fab763ee700 1 -- 172.21.15.133:0/1705517695 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fab70110260 con 0x7fab700fbd20 2024-02-20T07:54:39.453 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:39.450+0000 7fab763ee700 1 -- 172.21.15.133:0/1705517695 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fab70106ac0 con 0x7fab700fbd20 2024-02-20T07:54:39.453 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:39.450+0000 7fab6cff9700 1 -- 172.21.15.133:0/1705517695 <== mon.0 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7fab6000bbf0 con 0x7fab700fbd20 2024-02-20T07:54:39.453 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:39.451+0000 7fab6cff9700 1 --2- 172.21.15.133:0/1705517695 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7fab580692a0 0x7fab5806b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:54:39.453 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:39.451+0000 7fab6cff9700 1 -- 172.21.15.133:0/1705517695 <== mon.0 v2:172.21.15.133:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fab60088300 con 0x7fab700fbd20 2024-02-20T07:54:39.453 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:39.451+0000 7fab6f7fe700 1 --2- 172.21.15.133:0/1705517695 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7fab580692a0 0x7fab5806b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:54:39.453 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:39.451+0000 7fab6f7fe700 1 --2- 172.21.15.133:0/1705517695 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7fab580692a0 0x7fab5806b750 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fab70111400 tx=0x7fab64009040 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:54:39.455 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:39.455+0000 7fab6cff9700 1 -- 172.21.15.133:0/1705517695 <== mon.0 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7fab600583a0 con 0x7fab700fbd20 2024-02-20T07:54:39.641 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:39.640+0000 7fab763ee700 1 -- 172.21.15.133:0/1705517695 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7fab7004bed0 con 0x7fab700fbd20 2024-02-20T07:54:39.641 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:39.641+0000 7fab6cff9700 1 -- 172.21.15.133:0/1705517695 <== mon.0 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fab60057f30 con 0x7fab700fbd20 2024-02-20T07:54:39.642 INFO:teuthology.orchestra.run.smithi133.stdout:81604378652 2024-02-20T07:54:39.643 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:39.643+0000 7fab763ee700 1 -- 172.21.15.133:0/1705517695 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7fab580692a0 msgr2=0x7fab5806b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:54:39.643 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:39.643+0000 7fab763ee700 1 --2- 172.21.15.133:0/1705517695 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7fab580692a0 0x7fab5806b750 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fab70111400 tx=0x7fab64009040 comp rx=0 tx=0).stop 2024-02-20T07:54:39.644 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:39.643+0000 7fab763ee700 1 -- 172.21.15.133:0/1705517695 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fab700fbd20 msgr2=0x7fab70115bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:54:39.644 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:39.643+0000 7fab763ee700 1 --2- 172.21.15.133:0/1705517695 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fab700fbd20 0x7fab70115bd0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fab60004890 tx=0x7fab60009e30 comp rx=0 tx=0).stop 2024-02-20T07:54:39.644 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:39.643+0000 7fab763ee700 1 -- 172.21.15.133:0/1705517695 shutdown_connections 2024-02-20T07:54:39.644 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:39.643+0000 7fab763ee700 1 --2- 172.21.15.133:0/1705517695 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fab700fbd20 0x7fab70115bd0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:54:39.644 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:39.643+0000 7fab763ee700 1 --2- 172.21.15.133:0/1705517695 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7fab580692a0 0x7fab5806b750 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:54:39.644 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:39.643+0000 7fab763ee700 1 --2- 172.21.15.133:0/1705517695 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fab700fc630 0x7fab70116110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:54:39.644 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:39.643+0000 7fab763ee700 1 -- 172.21.15.133:0/1705517695 >> 172.21.15.133:0/1705517695 conn(0x7fab700f7a10 msgr2=0x7fab700f9100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:54:39.644 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:39.643+0000 7fab763ee700 1 -- 172.21.15.133:0/1705517695 shutdown_connections 2024-02-20T07:54:39.644 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:39.643+0000 7fab763ee700 1 -- 172.21.15.133:0/1705517695 wait complete. 2024-02-20T07:54:40.242 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:54:39 smithi133 ceph-mon[29117]: from='client.? 172.21.15.133:0/1705517695' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-02-20T07:54:40.297 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:54:39 smithi183 ceph-mon[35832]: from='client.? 172.21.15.133:0/1705517695' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-02-20T07:54:40.301 INFO:tasks.cephadm.ceph_manager.ceph:need seq 81604378649 got 81604378652 for osd.2 2024-02-20T07:54:40.301 DEBUG:teuthology.parallel:result is None 2024-02-20T07:54:40.301 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-02-20T07:54:40.301 DEBUG:teuthology.orchestra.run.smithi133:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 4537327c-cfc4-11ee-95bb-87774f69a715 -- ceph pg dump --format=json 2024-02-20T07:54:40.555 INFO:teuthology.orchestra.run.smithi133.stderr:Inferring config /var/lib/ceph/4537327c-cfc4-11ee-95bb-87774f69a715/mon.smithi133/config 2024-02-20T07:54:41.241 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:54:40 smithi133 ceph-mon[29117]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-20T07:54:41.297 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:54:40 smithi183 ceph-mon[35832]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-20T07:54:42.194 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:42.192+0000 7f6410505700 1 -- 172.21.15.133:0/2586686648 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f64080fb570 msgr2=0x7f64080fb9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:54:42.195 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:42.192+0000 7f6410505700 1 --2- 172.21.15.133:0/2586686648 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f64080fb570 0x7f64080fb9c0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f63f8004940 tx=0x7f63f8009e30 comp rx=0 tx=0).stop 2024-02-20T07:54:42.195 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:42.192+0000 7f6410505700 1 -- 172.21.15.133:0/2586686648 shutdown_connections 2024-02-20T07:54:42.195 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:42.192+0000 7f6410505700 1 --2- 172.21.15.133:0/2586686648 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f64080fb570 0x7f64080fb9c0 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:54:42.195 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:42.192+0000 7f6410505700 1 --2- 172.21.15.133:0/2586686648 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f6408103140 0x7f6408103510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:54:42.195 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:42.192+0000 7f6410505700 1 -- 172.21.15.133:0/2586686648 >> 172.21.15.133:0/2586686648 conn(0x7f6408071560 msgr2=0x7f6408071960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:54:42.195 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:42.192+0000 7f6410505700 1 -- 172.21.15.133:0/2586686648 shutdown_connections 2024-02-20T07:54:42.195 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:42.192+0000 7f6410505700 1 -- 172.21.15.133:0/2586686648 wait complete. 2024-02-20T07:54:42.195 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:42.193+0000 7f6410505700 1 Processor -- start 2024-02-20T07:54:42.195 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:42.193+0000 7f6410505700 1 -- start start 2024-02-20T07:54:42.195 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:42.193+0000 7f6410505700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f64080fb570 0x7f64080fc750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:54:42.195 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:42.193+0000 7f6410505700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f6408103140 0x7f64080fcc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:54:42.195 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:42.193+0000 7f6410505700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64080fd320 con 0x7f6408103140 2024-02-20T07:54:42.195 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:42.193+0000 7f6410505700 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64080fd490 con 0x7f64080fb570 2024-02-20T07:54:42.196 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:42.193+0000 7f640daa0700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f6408103140 0x7f64080fcc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:54:42.196 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:42.193+0000 7f640daa0700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f6408103140 0x7f64080fcc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.133:47598/0 (socket says 172.21.15.133:47598) 2024-02-20T07:54:42.196 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:42.193+0000 7f640daa0700 1 -- 172.21.15.133:0/2174040806 learned_addr learned my addr 172.21.15.133:0/2174040806 (peer_addr_for_me v2:172.21.15.133:0/0) 2024-02-20T07:54:42.196 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:42.193+0000 7f640e2a1700 1 --2- 172.21.15.133:0/2174040806 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f64080fb570 0x7f64080fc750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:54:42.196 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:42.193+0000 7f640daa0700 1 -- 172.21.15.133:0/2174040806 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f64080fb570 msgr2=0x7f64080fc750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:54:42.196 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:42.193+0000 7f640daa0700 1 --2- 172.21.15.133:0/2174040806 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f64080fb570 0x7f64080fc750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:54:42.196 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:42.193+0000 7f640daa0700 1 -- 172.21.15.133:0/2174040806 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63f8005040 con 0x7f6408103140 2024-02-20T07:54:42.196 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:42.193+0000 7f640daa0700 1 --2- 172.21.15.133:0/2174040806 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f6408103140 0x7f64080fcc90 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f63f8009fd0 tx=0x7f63f8009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:54:42.196 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:42.193+0000 7f63feffd700 1 -- 172.21.15.133:0/2174040806 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f63f800e040 con 0x7f6408103140 2024-02-20T07:54:42.196 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:42.193+0000 7f6410505700 1 -- 172.21.15.133:0/2174040806 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6408119e80 con 0x7f6408103140 2024-02-20T07:54:42.196 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:42.194+0000 7f63feffd700 1 -- 172.21.15.133:0/2174040806 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f63f800bbf0 con 0x7f6408103140 2024-02-20T07:54:42.196 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:42.194+0000 7f63feffd700 1 -- 172.21.15.133:0/2174040806 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f63f801c3f0 con 0x7f6408103140 2024-02-20T07:54:42.196 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:42.194+0000 7f6410505700 1 -- 172.21.15.133:0/2174040806 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f640811a340 con 0x7f6408103140 2024-02-20T07:54:42.197 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:42.194+0000 7f6410505700 1 -- 172.21.15.133:0/2174040806 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6408105570 con 0x7f6408103140 2024-02-20T07:54:42.197 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:42.195+0000 7f63feffd700 1 -- 172.21.15.133:0/2174040806 <== mon.0 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7f63f800bd60 con 0x7f6408103140 2024-02-20T07:54:42.197 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:42.195+0000 7f63feffd700 1 --2- 172.21.15.133:0/2174040806 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7f63f40692b0 0x7f63f406b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:54:42.197 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:42.195+0000 7f63feffd700 1 -- 172.21.15.133:0/2174040806 <== mon.0 v2:172.21.15.133:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f63f8088130 con 0x7f6408103140 2024-02-20T07:54:42.198 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:42.195+0000 7f640e2a1700 1 --2- 172.21.15.133:0/2174040806 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7f63f40692b0 0x7f63f406b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:54:42.198 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:42.196+0000 7f640e2a1700 1 --2- 172.21.15.133:0/2174040806 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7f63f40692b0 0x7f63f406b760 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f6404006fd0 tx=0x7f6404008040 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:54:42.200 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:42.200+0000 7f63feffd700 1 -- 172.21.15.133:0/2174040806 <== mon.0 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f63f80547a0 con 0x7f6408103140 2024-02-20T07:54:42.379 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:42.378+0000 7f6410505700 1 -- 172.21.15.133:0/2174040806 --> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f640804bed0 con 0x7f63f40692b0 2024-02-20T07:54:42.380 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:42.379+0000 7f63feffd700 1 -- 172.21.15.133:0/2174040806 <== mgr.14208 v2:172.21.15.133:6800/2703586938 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+28041 (secure 0 0 0) 0x7f640804bed0 con 0x7f63f40692b0 2024-02-20T07:54:42.381 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-02-20T07:54:42.382 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:42.382+0000 7f6410505700 1 -- 172.21.15.133:0/2174040806 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7f63f40692b0 msgr2=0x7f63f406b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:54:42.382 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:42.382+0000 7f6410505700 1 --2- 172.21.15.133:0/2174040806 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7f63f40692b0 0x7f63f406b760 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f6404006fd0 tx=0x7f6404008040 comp rx=0 tx=0).stop 2024-02-20T07:54:42.383 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:42.382+0000 7f6410505700 1 -- 172.21.15.133:0/2174040806 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f6408103140 msgr2=0x7f64080fcc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:54:42.383 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:42.382+0000 7f6410505700 1 --2- 172.21.15.133:0/2174040806 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f6408103140 0x7f64080fcc90 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f63f8009fd0 tx=0x7f63f8009e30 comp rx=0 tx=0).stop 2024-02-20T07:54:42.383 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:42.382+0000 7f6410505700 1 -- 172.21.15.133:0/2174040806 shutdown_connections 2024-02-20T07:54:42.383 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:42.382+0000 7f6410505700 1 --2- 172.21.15.133:0/2174040806 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7f63f40692b0 0x7f63f406b760 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:54:42.383 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:42.382+0000 7f6410505700 1 --2- 172.21.15.133:0/2174040806 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f64080fb570 0x7f64080fc750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:54:42.383 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:42.382+0000 7f6410505700 1 --2- 172.21.15.133:0/2174040806 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f6408103140 0x7f64080fcc90 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:54:42.383 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:42.382+0000 7f6410505700 1 -- 172.21.15.133:0/2174040806 >> 172.21.15.133:0/2174040806 conn(0x7f6408071560 msgr2=0x7f64080f9740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:54:42.383 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:42.382+0000 7f6410505700 1 -- 172.21.15.133:0/2174040806 shutdown_connections 2024-02-20T07:54:42.383 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:42.382+0000 7f6410505700 1 -- 172.21.15.133:0/2174040806 wait complete. 2024-02-20T07:54:42.383 INFO:teuthology.orchestra.run.smithi133.stderr:dumped all 2024-02-20T07:54:43.117 INFO:teuthology.orchestra.run.smithi133.stdout:{"pg_ready":true,"pg_map":{"version":157,"stamp":"2024-02-20T07:54:41.571014+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":3,"kb":562470912,"kb_used":1787604,"kb_used_data":3876,"kb_used_omap":0,"kb_used_meta":1783680,"kb_avail":560683308,"statfs":{"total":575970213888,"available":574139707392,"internally_reserved":0,"allocated":3969024,"data_stored":2606262,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1826488320},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002442"},"pg_stats":[{"pgid":"1.0","version":"23'117","reported_seq":190,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-02-20T07:53:49.690123+0000","last_change":"2024-02-20T07:53:24.386260+0000","last_active":"2024-02-20T07:53:49.690123+0000","last_peered":"2024-02-20T07:53:49.690123+0000","last_clean":"2024-02-20T07:53:49.690123+0000","last_became_active":"2024-02-20T07:53:24.385972+0000","last_became_peered":"2024-02-20T07:53:24.385972+0000","last_unstale":"2024-02-20T07:53:49.690123+0000","last_undegraded":"2024-02-20T07:53:49.690123+0000","last_fullsized":"2024-02-20T07:53:49.690123+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-20T07:52:35.900545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-20T07:52:35.900545+0000","last_clean_scrub_stamp":"2024-02-20T07:52:35.900545+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T17:19:10.262261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2375680,"data_stored":2361472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":3,"acting":3,"num_store_stats":4}],"osd_stats":[{"osd":5,"up_from":39,"seq":167503724558,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":297644,"kb_used_data":356,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":93447508,"statfs":{"total":95995035648,"available":95690248192,"internally_reserved":0,"allocated":364544,"data_stored":139193,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.439}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47299999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58399999999999996}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.32800000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39200000000000002}]}]},{"osd":4,"up_from":34,"seq":146028888083,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":297644,"kb_used_data":356,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":93447508,"statfs":{"total":95995035648,"available":95690248192,"internally_reserved":0,"allocated":364544,"data_stored":139193,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Feb 20 07:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.47999999999999998},"min":{"1min":0.26600000000000001,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.64700000000000002,"5min":0.64700000000000002,"15min":0.64700000000000002},"last":0.53800000000000003},{"interface":"front","average":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.63100000000000001,"5min":0.63100000000000001,"15min":0.63100000000000001},"last":0.66000000000000003}]},{"osd":1,"last update":"Tue Feb 20 07:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"min":{"1min":0.29099999999999998,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"last":0.51000000000000001},{"interface":"front","average":{"1min":0.46600000000000003,"5min":0.46600000000000003,"15min":0.46600000000000003},"min":{"1min":0.312,"5min":0.312,"15min":0.312},"max":{"1min":0.61699999999999999,"5min":0.61699999999999999,"15min":0.61699999999999999},"last":0.58499999999999996}]},{"osd":2,"last update":"Tue Feb 20 07:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":0.58299999999999996,"5min":0.58299999999999996,"15min":0.58299999999999996},"last":0.64700000000000002},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.48499999999999999,"15min":0.48499999999999999},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.64400000000000002,"5min":0.64400000000000002,"15min":0.64400000000000002},"last":0.60199999999999998}]},{"osd":3,"last update":"Tue Feb 20 07:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.378,"5min":0.378,"15min":0.378},"min":{"1min":0.26400000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"last":0.55400000000000005},{"interface":"front","average":{"1min":0.371,"5min":0.371,"15min":0.371},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"last":0.56799999999999995}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.62}]}]},{"osd":3,"up_from":26,"seq":111669149719,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298224,"kb_used_data":936,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":93446928,"statfs":{"total":95995035648,"available":95689654272,"internally_reserved":0,"allocated":958464,"data_stored":729561,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Feb 20 07:54:00 2024","interfaces":[{"interface":"back","average":{"1min":0.50700000000000001,"5min":0.50700000000000001,"15min":0.50700000000000001},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.754,"5min":0.754,"15min":0.754},"last":0.47899999999999998},{"interface":"front","average":{"1min":0.505,"5min":0.505,"15min":0.505},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.70599999999999996,"5min":0.70599999999999996,"15min":0.70599999999999996},"last":0.44600000000000001}]},{"osd":1,"last update":"Tue Feb 20 07:54:00 2024","interfaces":[{"interface":"back","average":{"1min":0.51500000000000001,"5min":0.51500000000000001,"15min":0.51500000000000001},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.69099999999999995,"5min":0.69099999999999995,"15min":0.69099999999999995},"last":0.51200000000000001},{"interface":"front","average":{"1min":0.501,"5min":0.501,"15min":0.501},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.72599999999999998,"5min":0.72599999999999998,"15min":0.72599999999999998},"last":0.56799999999999995}]},{"osd":2,"last update":"Tue Feb 20 07:54:00 2024","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.501,"15min":0.501},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":0.752,"5min":0.752,"15min":0.752},"last":0.498},{"interface":"front","average":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.68999999999999995,"5min":0.68999999999999995,"15min":0.68999999999999995},"last":0.54400000000000004}]},{"osd":4,"last update":"Tue Feb 20 07:54:24 2024","interfaces":[{"interface":"back","average":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.61599999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"last":0.42699999999999999},{"interface":"front","average":{"1min":0.48699999999999999,"5min":0.48699999999999999,"15min":0.48699999999999999},"min":{"1min":0.27600000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.68899999999999995,"5min":0.68899999999999995,"15min":0.68899999999999995},"last":0.59299999999999997}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52600000000000002}]}]},{"osd":2,"up_from":19,"seq":81604378653,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":297644,"kb_used_data":356,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":93447508,"statfs":{"total":95995035648,"available":95690248192,"internally_reserved":0,"allocated":364544,"data_stored":139193,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Feb 20 07:54:39 2024","interfaces":[{"interface":"back","average":{"1min":0.32100000000000001,"5min":0.33900000000000002,"15min":0.34200000000000003},"min":{"1min":0.224,"5min":0.224,"15min":0.224},"max":{"1min":0.40999999999999998,"5min":0.58899999999999997,"15min":0.58899999999999997},"last":0.23699999999999999},{"interface":"front","average":{"1min":0.33500000000000002,"5min":0.35599999999999998,"15min":0.35999999999999999},"min":{"1min":0.24399999999999999,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":0.442,"5min":0.57399999999999995,"15min":0.57399999999999995},"last":0.28000000000000003}]},{"osd":1,"last update":"Tue Feb 20 07:54:39 2024","interfaces":[{"interface":"back","average":{"1min":0.34200000000000003,"5min":0.36599999999999999,"15min":0.37},"min":{"1min":0.249,"5min":0.249,"15min":0.249},"max":{"1min":0.45600000000000002,"5min":0.51500000000000001,"15min":0.51500000000000001},"last":0.26300000000000001},{"interface":"front","average":{"1min":0.34200000000000003,"5min":0.36299999999999999,"15min":0.36699999999999999},"min":{"1min":0.249,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.497,"5min":0.55400000000000005,"15min":0.55400000000000005},"last":0.249}]},{"osd":3,"last update":"Tue Feb 20 07:54:01 2024","interfaces":[{"interface":"back","average":{"1min":0.47899999999999998,"5min":0.47899999999999998,"15min":0.47899999999999998},"min":{"1min":0.30399999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.61199999999999999,"5min":0.61199999999999999,"15min":0.61199999999999999},"last":0.54700000000000004},{"interface":"front","average":{"1min":0.46800000000000003,"5min":0.46800000000000003,"15min":0.46800000000000003},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.59799999999999998,"5min":0.59799999999999998,"15min":0.59799999999999998},"last":0.56399999999999995}]},{"osd":4,"last update":"Tue Feb 20 07:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.51200000000000001,"5min":0.51200000000000001,"15min":0.51200000000000001},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.67600000000000005},"last":0.59799999999999998},{"interface":"front","average":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.52600000000000002},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.77700000000000002,"5min":0.77700000000000002,"15min":0.77700000000000002},"last":0.51900000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57599999999999996}]}]},{"osd":0,"up_from":9,"seq":38654705704,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298224,"kb_used_data":936,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":93446928,"statfs":{"total":95995035648,"available":95689654272,"internally_reserved":0,"allocated":958464,"data_stored":729561,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Feb 20 07:54:13 2024","interfaces":[{"interface":"back","average":{"1min":0.35499999999999998,"5min":0.33100000000000002,"15min":0.32700000000000001},"min":{"1min":0.26400000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":0.45400000000000001,"5min":0.47299999999999998,"15min":0.47299999999999998},"last":0.245},{"interface":"front","average":{"1min":0.36099999999999999,"5min":0.34499999999999997,"15min":0.34200000000000003},"min":{"1min":0.252,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":0.48799999999999999,"5min":0.54400000000000004,"15min":0.54400000000000004},"last":0.30099999999999999}]},{"osd":2,"last update":"Tue Feb 20 07:53:39 2024","interfaces":[{"interface":"back","average":{"1min":0.376,"5min":0.376,"15min":0.376},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"last":0.26200000000000001},{"interface":"front","average":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.505,"5min":0.505,"15min":0.505},"last":0.28899999999999998}]},{"osd":3,"last update":"Tue Feb 20 07:53:58 2024","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.73999999999999999,"5min":0.73999999999999999,"15min":0.73999999999999999},"last":0.495},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.79000000000000004,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.45100000000000001}]},{"osd":4,"last update":"Tue Feb 20 07:54:27 2024","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.56299999999999994,"15min":0.56299999999999994},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.78600000000000003,"5min":0.78600000000000003,"15min":0.78600000000000003},"last":0.54100000000000004},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.57899999999999996,"15min":0.57899999999999996},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"last":0.51000000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52300000000000002}]}]},{"osd":1,"up_from":14,"seq":60129542177,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298224,"kb_used_data":936,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":93446928,"statfs":{"total":95995035648,"available":95689654272,"internally_reserved":0,"allocated":958464,"data_stored":729561,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Feb 20 07:54:12 2024","interfaces":[{"interface":"back","average":{"1min":0.32900000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"min":{"1min":0.24199999999999999,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.45700000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"last":0.33400000000000002},{"interface":"front","average":{"1min":0.34100000000000003,"5min":0.34899999999999998,"15min":0.35099999999999998},"min":{"1min":0.20899999999999999,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"last":0.30099999999999999}]},{"osd":2,"last update":"Tue Feb 20 07:54:38 2024","interfaces":[{"interface":"back","average":{"1min":0.33900000000000002,"5min":0.38300000000000001,"15min":0.39100000000000001},"min":{"1min":0.245,"5min":0.245,"15min":0.245},"max":{"1min":0.47599999999999998,"5min":0.51200000000000001,"15min":0.51200000000000001},"last":0.28299999999999997},{"interface":"front","average":{"1min":0.34699999999999998,"5min":0.377,"15min":0.38200000000000001},"min":{"1min":0.255,"5min":0.255,"15min":0.255},"max":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"last":0.314}]},{"osd":3,"last update":"Tue Feb 20 07:54:01 2024","interfaces":[{"interface":"back","average":{"1min":0.49199999999999999,"5min":0.49199999999999999,"15min":0.49199999999999999},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.61599999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"last":0.52400000000000002},{"interface":"front","average":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.61599999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"last":0.46600000000000003}]},{"osd":4,"last update":"Tue Feb 20 07:54:25 2024","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.68000000000000005,"5min":0.68000000000000005,"15min":0.68000000000000005},"last":0.49399999999999999},{"interface":"front","average":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.66600000000000004,"5min":0.66600000000000004,"15min":0.66600000000000004},"last":0.48299999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.44500000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-20T07:54:43.119 DEBUG:teuthology.orchestra.run.smithi133:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 4537327c-cfc4-11ee-95bb-87774f69a715 -- ceph pg dump --format=json 2024-02-20T07:54:43.297 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:54:43 smithi183 ceph-mon[35832]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-20T07:54:43.371 INFO:teuthology.orchestra.run.smithi133.stderr:Inferring config /var/lib/ceph/4537327c-cfc4-11ee-95bb-87774f69a715/mon.smithi133/config 2024-02-20T07:54:43.491 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:54:43 smithi133 ceph-mon[29117]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-20T07:54:44.242 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:54:44 smithi133 ceph-mon[29117]: from='client.14426 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-20T07:54:44.297 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:54:44 smithi183 ceph-mon[35832]: from='client.14426 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-20T07:54:44.960 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:44.956+0000 7fb6ba5a2700 1 -- 172.21.15.133:0/4224881191 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb6b41038f0 msgr2=0x7fb6b40fcf60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:54:44.960 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:44.956+0000 7fb6ba5a2700 1 --2- 172.21.15.133:0/4224881191 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb6b41038f0 0x7fb6b40fcf60 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fb6a8004940 tx=0x7fb6a8009e30 comp rx=0 tx=0).stop 2024-02-20T07:54:44.960 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:44.956+0000 7fb6ba5a2700 1 -- 172.21.15.133:0/4224881191 shutdown_connections 2024-02-20T07:54:44.960 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:44.956+0000 7fb6ba5a2700 1 --2- 172.21.15.133:0/4224881191 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb6b41038f0 0x7fb6b40fcf60 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:54:44.960 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:44.956+0000 7fb6ba5a2700 1 --2- 172.21.15.133:0/4224881191 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb6b4102f50 0x7fb6b4103320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:54:44.960 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:44.956+0000 7fb6ba5a2700 1 -- 172.21.15.133:0/4224881191 >> 172.21.15.133:0/4224881191 conn(0x7fb6b40f8ae0 msgr2=0x7fb6b40faef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:54:44.960 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:44.956+0000 7fb6ba5a2700 1 -- 172.21.15.133:0/4224881191 shutdown_connections 2024-02-20T07:54:44.960 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:44.956+0000 7fb6ba5a2700 1 -- 172.21.15.133:0/4224881191 wait complete. 2024-02-20T07:54:44.961 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:44.957+0000 7fb6ba5a2700 1 Processor -- start 2024-02-20T07:54:44.961 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:44.957+0000 7fb6ba5a2700 1 -- start start 2024-02-20T07:54:44.961 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:44.957+0000 7fb6ba5a2700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb6b4102f50 0x7fb6b4113ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:54:44.961 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:44.957+0000 7fb6ba5a2700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb6b41038f0 0x7fb6b4114000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:54:44.961 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:44.957+0000 7fb6ba5a2700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6b41146e0 con 0x7fb6b41038f0 2024-02-20T07:54:44.961 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:44.957+0000 7fb6ba5a2700 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6b410d980 con 0x7fb6b4102f50 2024-02-20T07:54:44.961 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:44.958+0000 7fb6b37fe700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb6b41038f0 0x7fb6b4114000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:54:44.961 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:44.958+0000 7fb6b37fe700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb6b41038f0 0x7fb6b4114000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.133:47608/0 (socket says 172.21.15.133:47608) 2024-02-20T07:54:44.961 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:44.958+0000 7fb6b37fe700 1 -- 172.21.15.133:0/584230362 learned_addr learned my addr 172.21.15.133:0/584230362 (peer_addr_for_me v2:172.21.15.133:0/0) 2024-02-20T07:54:44.961 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:44.958+0000 7fb6b37fe700 1 -- 172.21.15.133:0/584230362 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb6b4102f50 msgr2=0x7fb6b4113ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:54:44.961 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:44.958+0000 7fb6b37fe700 1 --2- 172.21.15.133:0/584230362 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb6b4102f50 0x7fb6b4113ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:54:44.961 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:44.958+0000 7fb6b37fe700 1 -- 172.21.15.133:0/584230362 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6a8005040 con 0x7fb6b41038f0 2024-02-20T07:54:44.961 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:44.958+0000 7fb6b37fe700 1 --2- 172.21.15.133:0/584230362 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb6b41038f0 0x7fb6b4114000 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fb6a80090d0 tx=0x7fb6a8009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:54:44.961 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:44.958+0000 7fb6b0ff9700 1 -- 172.21.15.133:0/584230362 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb6a800e040 con 0x7fb6b41038f0 2024-02-20T07:54:44.961 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:44.958+0000 7fb6b0ff9700 1 -- 172.21.15.133:0/584230362 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb6a800bbf0 con 0x7fb6b41038f0 2024-02-20T07:54:44.962 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:44.958+0000 7fb6b0ff9700 1 -- 172.21.15.133:0/584230362 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb6a801c630 con 0x7fb6b41038f0 2024-02-20T07:54:44.962 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:44.958+0000 7fb6ba5a2700 1 -- 172.21.15.133:0/584230362 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb6b410dc00 con 0x7fb6b41038f0 2024-02-20T07:54:44.962 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:44.958+0000 7fb6ba5a2700 1 -- 172.21.15.133:0/584230362 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb6b410e0f0 con 0x7fb6b41038f0 2024-02-20T07:54:44.963 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:44.959+0000 7fb6ba5a2700 1 -- 172.21.15.133:0/584230362 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb6b40fc6e0 con 0x7fb6b41038f0 2024-02-20T07:54:44.963 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:44.960+0000 7fb6b0ff9700 1 -- 172.21.15.133:0/584230362 <== mon.0 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7fb6a800bd60 con 0x7fb6b41038f0 2024-02-20T07:54:44.963 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:44.960+0000 7fb6b0ff9700 1 --2- 172.21.15.133:0/584230362 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7fb69c0692a0 0x7fb69c06b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:54:44.963 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:44.960+0000 7fb6b3fff700 1 --2- 172.21.15.133:0/584230362 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7fb69c0692a0 0x7fb69c06b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:54:44.963 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:44.960+0000 7fb6b0ff9700 1 -- 172.21.15.133:0/584230362 <== mon.0 v2:172.21.15.133:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fb6a80881b0 con 0x7fb6b41038f0 2024-02-20T07:54:44.963 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:44.961+0000 7fb6b3fff700 1 --2- 172.21.15.133:0/584230362 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7fb69c0692a0 0x7fb69c06b750 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fb6a40098f0 tx=0x7fb6a4006da0 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:54:44.966 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:44.964+0000 7fb6b0ff9700 1 -- 172.21.15.133:0/584230362 <== mon.0 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7fb6a80582b0 con 0x7fb6b41038f0 2024-02-20T07:54:45.145 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:45.143+0000 7fb6ba5a2700 1 -- 172.21.15.133:0/584230362 --> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fb6b404bf70 con 0x7fb69c0692a0 2024-02-20T07:54:45.146 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:45.145+0000 7fb6b0ff9700 1 -- 172.21.15.133:0/584230362 <== mgr.14208 v2:172.21.15.133:6800/2703586938 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+28041 (secure 0 0 0) 0x7fb6a4009dc0 con 0x7fb69c0692a0 2024-02-20T07:54:45.147 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-02-20T07:54:45.149 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:45.147+0000 7fb6ba5a2700 1 -- 172.21.15.133:0/584230362 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7fb69c0692a0 msgr2=0x7fb69c06b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:54:45.149 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:45.147+0000 7fb6ba5a2700 1 --2- 172.21.15.133:0/584230362 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7fb69c0692a0 0x7fb69c06b750 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fb6a40098f0 tx=0x7fb6a4006da0 comp rx=0 tx=0).stop 2024-02-20T07:54:45.149 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:45.147+0000 7fb6ba5a2700 1 -- 172.21.15.133:0/584230362 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb6b41038f0 msgr2=0x7fb6b4114000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:54:45.149 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:45.147+0000 7fb6ba5a2700 1 --2- 172.21.15.133:0/584230362 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb6b41038f0 0x7fb6b4114000 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fb6a80090d0 tx=0x7fb6a8009e30 comp rx=0 tx=0).stop 2024-02-20T07:54:45.149 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:45.147+0000 7fb6ba5a2700 1 -- 172.21.15.133:0/584230362 shutdown_connections 2024-02-20T07:54:45.149 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:45.147+0000 7fb6ba5a2700 1 --2- 172.21.15.133:0/584230362 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7fb69c0692a0 0x7fb69c06b750 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:54:45.149 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:45.147+0000 7fb6ba5a2700 1 --2- 172.21.15.133:0/584230362 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb6b4102f50 0x7fb6b4113ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:54:45.149 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:45.147+0000 7fb6ba5a2700 1 --2- 172.21.15.133:0/584230362 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb6b41038f0 0x7fb6b4114000 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:54:45.149 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:45.147+0000 7fb6ba5a2700 1 -- 172.21.15.133:0/584230362 >> 172.21.15.133:0/584230362 conn(0x7fb6b40f8ae0 msgr2=0x7fb6b40fa510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:54:45.149 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:45.147+0000 7fb6ba5a2700 1 -- 172.21.15.133:0/584230362 shutdown_connections 2024-02-20T07:54:45.149 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:45.147+0000 7fb6ba5a2700 1 -- 172.21.15.133:0/584230362 wait complete. 2024-02-20T07:54:45.150 INFO:teuthology.orchestra.run.smithi133.stderr:dumped all 2024-02-20T07:54:45.297 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:54:45 smithi183 ceph-mon[35832]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-20T07:54:45.491 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:54:45 smithi133 ceph-mon[29117]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-20T07:54:45.814 INFO:teuthology.orchestra.run.smithi133.stdout:{"pg_ready":true,"pg_map":{"version":158,"stamp":"2024-02-20T07:54:43.571376+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":3,"kb":562470912,"kb_used":1787604,"kb_used_data":3876,"kb_used_omap":0,"kb_used_meta":1783680,"kb_avail":560683308,"statfs":{"total":575970213888,"available":574139707392,"internally_reserved":0,"allocated":3969024,"data_stored":2606262,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1826488320},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002361"},"pg_stats":[{"pgid":"1.0","version":"23'117","reported_seq":190,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-02-20T07:53:49.690123+0000","last_change":"2024-02-20T07:53:24.386260+0000","last_active":"2024-02-20T07:53:49.690123+0000","last_peered":"2024-02-20T07:53:49.690123+0000","last_clean":"2024-02-20T07:53:49.690123+0000","last_became_active":"2024-02-20T07:53:24.385972+0000","last_became_peered":"2024-02-20T07:53:24.385972+0000","last_unstale":"2024-02-20T07:53:49.690123+0000","last_undegraded":"2024-02-20T07:53:49.690123+0000","last_fullsized":"2024-02-20T07:53:49.690123+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-20T07:52:35.900545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-20T07:52:35.900545+0000","last_clean_scrub_stamp":"2024-02-20T07:52:35.900545+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T17:19:10.262261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2375680,"data_stored":2361472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":3,"acting":3,"num_store_stats":4}],"osd_stats":[{"osd":5,"up_from":39,"seq":167503724558,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":297644,"kb_used_data":356,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":93447508,"statfs":{"total":95995035648,"available":95690248192,"internally_reserved":0,"allocated":364544,"data_stored":139193,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.439}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47299999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58399999999999996}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.32800000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39200000000000002}]}]},{"osd":4,"up_from":34,"seq":146028888083,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":297644,"kb_used_data":356,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":93447508,"statfs":{"total":95995035648,"available":95690248192,"internally_reserved":0,"allocated":364544,"data_stored":139193,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Feb 20 07:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.47999999999999998},"min":{"1min":0.26600000000000001,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.64700000000000002,"5min":0.64700000000000002,"15min":0.64700000000000002},"last":0.53800000000000003},{"interface":"front","average":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.63100000000000001,"5min":0.63100000000000001,"15min":0.63100000000000001},"last":0.66000000000000003}]},{"osd":1,"last update":"Tue Feb 20 07:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"min":{"1min":0.29099999999999998,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"last":0.51000000000000001},{"interface":"front","average":{"1min":0.46600000000000003,"5min":0.46600000000000003,"15min":0.46600000000000003},"min":{"1min":0.312,"5min":0.312,"15min":0.312},"max":{"1min":0.61699999999999999,"5min":0.61699999999999999,"15min":0.61699999999999999},"last":0.58499999999999996}]},{"osd":2,"last update":"Tue Feb 20 07:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":0.58299999999999996,"5min":0.58299999999999996,"15min":0.58299999999999996},"last":0.64700000000000002},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.48499999999999999,"15min":0.48499999999999999},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.64400000000000002,"5min":0.64400000000000002,"15min":0.64400000000000002},"last":0.60199999999999998}]},{"osd":3,"last update":"Tue Feb 20 07:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.378,"5min":0.378,"15min":0.378},"min":{"1min":0.26400000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"last":0.55400000000000005},{"interface":"front","average":{"1min":0.371,"5min":0.371,"15min":0.371},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"last":0.56799999999999995}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.62}]}]},{"osd":3,"up_from":26,"seq":111669149720,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298224,"kb_used_data":936,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":93446928,"statfs":{"total":95995035648,"available":95689654272,"internally_reserved":0,"allocated":958464,"data_stored":729561,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Feb 20 07:54:00 2024","interfaces":[{"interface":"back","average":{"1min":0.50700000000000001,"5min":0.50700000000000001,"15min":0.50700000000000001},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.754,"5min":0.754,"15min":0.754},"last":0.51800000000000002},{"interface":"front","average":{"1min":0.505,"5min":0.505,"15min":0.505},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.70599999999999996,"5min":0.70599999999999996,"15min":0.70599999999999996},"last":0.59399999999999997}]},{"osd":1,"last update":"Tue Feb 20 07:54:00 2024","interfaces":[{"interface":"back","average":{"1min":0.51500000000000001,"5min":0.51500000000000001,"15min":0.51500000000000001},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.69099999999999995,"5min":0.69099999999999995,"15min":0.69099999999999995},"last":0.60699999999999998},{"interface":"front","average":{"1min":0.501,"5min":0.501,"15min":0.501},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.72599999999999998,"5min":0.72599999999999998,"15min":0.72599999999999998},"last":0.57999999999999996}]},{"osd":2,"last update":"Tue Feb 20 07:54:00 2024","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.501,"15min":0.501},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":0.752,"5min":0.752,"15min":0.752},"last":0.54000000000000004},{"interface":"front","average":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.68999999999999995,"5min":0.68999999999999995,"15min":0.68999999999999995},"last":0.496}]},{"osd":4,"last update":"Tue Feb 20 07:54:24 2024","interfaces":[{"interface":"back","average":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.61599999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"last":0.61599999999999999},{"interface":"front","average":{"1min":0.48699999999999999,"5min":0.48699999999999999,"15min":0.48699999999999999},"min":{"1min":0.27600000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.68899999999999995,"5min":0.68899999999999995,"15min":0.68899999999999995},"last":0.63600000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47799999999999998}]}]},{"osd":2,"up_from":19,"seq":81604378653,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":297644,"kb_used_data":356,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":93447508,"statfs":{"total":95995035648,"available":95690248192,"internally_reserved":0,"allocated":364544,"data_stored":139193,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Feb 20 07:54:39 2024","interfaces":[{"interface":"back","average":{"1min":0.32100000000000001,"5min":0.33900000000000002,"15min":0.34200000000000003},"min":{"1min":0.224,"5min":0.224,"15min":0.224},"max":{"1min":0.40999999999999998,"5min":0.58899999999999997,"15min":0.58899999999999997},"last":0.23699999999999999},{"interface":"front","average":{"1min":0.33500000000000002,"5min":0.35599999999999998,"15min":0.35999999999999999},"min":{"1min":0.24399999999999999,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":0.442,"5min":0.57399999999999995,"15min":0.57399999999999995},"last":0.28000000000000003}]},{"osd":1,"last update":"Tue Feb 20 07:54:39 2024","interfaces":[{"interface":"back","average":{"1min":0.34200000000000003,"5min":0.36599999999999999,"15min":0.37},"min":{"1min":0.249,"5min":0.249,"15min":0.249},"max":{"1min":0.45600000000000002,"5min":0.51500000000000001,"15min":0.51500000000000001},"last":0.26300000000000001},{"interface":"front","average":{"1min":0.34200000000000003,"5min":0.36299999999999999,"15min":0.36699999999999999},"min":{"1min":0.249,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.497,"5min":0.55400000000000005,"15min":0.55400000000000005},"last":0.249}]},{"osd":3,"last update":"Tue Feb 20 07:54:01 2024","interfaces":[{"interface":"back","average":{"1min":0.47899999999999998,"5min":0.47899999999999998,"15min":0.47899999999999998},"min":{"1min":0.30399999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.61199999999999999,"5min":0.61199999999999999,"15min":0.61199999999999999},"last":0.54700000000000004},{"interface":"front","average":{"1min":0.46800000000000003,"5min":0.46800000000000003,"15min":0.46800000000000003},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.59799999999999998,"5min":0.59799999999999998,"15min":0.59799999999999998},"last":0.56399999999999995}]},{"osd":4,"last update":"Tue Feb 20 07:54:26 2024","interfaces":[{"interface":"back","average":{"1min":0.51200000000000001,"5min":0.51200000000000001,"15min":0.51200000000000001},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.67600000000000005},"last":0.59799999999999998},{"interface":"front","average":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.52600000000000002},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.77700000000000002,"5min":0.77700000000000002,"15min":0.77700000000000002},"last":0.51900000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57599999999999996}]}]},{"osd":0,"up_from":9,"seq":38654705704,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298224,"kb_used_data":936,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":93446928,"statfs":{"total":95995035648,"available":95689654272,"internally_reserved":0,"allocated":958464,"data_stored":729561,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Feb 20 07:54:13 2024","interfaces":[{"interface":"back","average":{"1min":0.35499999999999998,"5min":0.33100000000000002,"15min":0.32700000000000001},"min":{"1min":0.26400000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":0.45400000000000001,"5min":0.47299999999999998,"15min":0.47299999999999998},"last":0.245},{"interface":"front","average":{"1min":0.36099999999999999,"5min":0.34499999999999997,"15min":0.34200000000000003},"min":{"1min":0.252,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":0.48799999999999999,"5min":0.54400000000000004,"15min":0.54400000000000004},"last":0.30099999999999999}]},{"osd":2,"last update":"Tue Feb 20 07:53:39 2024","interfaces":[{"interface":"back","average":{"1min":0.376,"5min":0.376,"15min":0.376},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"last":0.26200000000000001},{"interface":"front","average":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.505,"5min":0.505,"15min":0.505},"last":0.28899999999999998}]},{"osd":3,"last update":"Tue Feb 20 07:53:58 2024","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.73999999999999999,"5min":0.73999999999999999,"15min":0.73999999999999999},"last":0.495},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.79000000000000004,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.45100000000000001}]},{"osd":4,"last update":"Tue Feb 20 07:54:27 2024","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.56299999999999994,"15min":0.56299999999999994},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.78600000000000003,"5min":0.78600000000000003,"15min":0.78600000000000003},"last":0.54100000000000004},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.57899999999999996,"15min":0.57899999999999996},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"last":0.51000000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52300000000000002}]}]},{"osd":1,"up_from":14,"seq":60129542178,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298224,"kb_used_data":936,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":93446928,"statfs":{"total":95995035648,"available":95689654272,"internally_reserved":0,"allocated":958464,"data_stored":729561,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Feb 20 07:54:12 2024","interfaces":[{"interface":"back","average":{"1min":0.32900000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"min":{"1min":0.24199999999999999,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.45700000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"last":0.33400000000000002},{"interface":"front","average":{"1min":0.34100000000000003,"5min":0.34899999999999998,"15min":0.35099999999999998},"min":{"1min":0.20899999999999999,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"last":0.30099999999999999}]},{"osd":2,"last update":"Tue Feb 20 07:54:38 2024","interfaces":[{"interface":"back","average":{"1min":0.33900000000000002,"5min":0.38300000000000001,"15min":0.39100000000000001},"min":{"1min":0.245,"5min":0.245,"15min":0.245},"max":{"1min":0.47599999999999998,"5min":0.51200000000000001,"15min":0.51200000000000001},"last":0.28299999999999997},{"interface":"front","average":{"1min":0.34699999999999998,"5min":0.377,"15min":0.38200000000000001},"min":{"1min":0.255,"5min":0.255,"15min":0.255},"max":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"last":0.314}]},{"osd":3,"last update":"Tue Feb 20 07:54:01 2024","interfaces":[{"interface":"back","average":{"1min":0.49199999999999999,"5min":0.49199999999999999,"15min":0.49199999999999999},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.61599999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"last":0.52400000000000002},{"interface":"front","average":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.61599999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"last":0.46600000000000003}]},{"osd":4,"last update":"Tue Feb 20 07:54:25 2024","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.68000000000000005,"5min":0.68000000000000005,"15min":0.68000000000000005},"last":0.49399999999999999},{"interface":"front","average":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.66600000000000004,"5min":0.66600000000000004,"15min":0.66600000000000004},"last":0.48299999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.44500000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-20T07:54:45.815 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-02-20T07:54:45.815 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-02-20T07:54:45.815 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-02-20T07:54:45.815 DEBUG:teuthology.orchestra.run.smithi133:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 4537327c-cfc4-11ee-95bb-87774f69a715 -- ceph health --format=json 2024-02-20T07:54:46.067 INFO:teuthology.orchestra.run.smithi133.stderr:Inferring config /var/lib/ceph/4537327c-cfc4-11ee-95bb-87774f69a715/mon.smithi133/config 2024-02-20T07:54:47.179 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:54:47 smithi133 ceph-mon[29117]: from='client.14430 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-20T07:54:47.180 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:54:47 smithi133 ceph-mon[29117]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-20T07:54:47.297 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:54:47 smithi183 ceph-mon[35832]: from='client.14430 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-20T07:54:47.298 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:54:47 smithi183 ceph-mon[35832]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-20T07:54:47.700 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:47.694+0000 7f6fee027700 1 -- 172.21.15.133:0/421294251 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f6fe80fe8f0 msgr2=0x7f6fe8106fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:54:47.701 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:47.694+0000 7f6fee027700 1 --2- 172.21.15.133:0/421294251 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f6fe80fe8f0 0x7f6fe8106fc0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f6fdc004940 tx=0x7f6fdc009e30 comp rx=0 tx=0).stop 2024-02-20T07:54:47.701 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:47.694+0000 7f6fee027700 1 -- 172.21.15.133:0/421294251 shutdown_connections 2024-02-20T07:54:47.701 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:47.694+0000 7f6fee027700 1 --2- 172.21.15.133:0/421294251 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f6fe80fe8f0 0x7f6fe8106fc0 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:54:47.701 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:47.694+0000 7f6fee027700 1 --2- 172.21.15.133:0/421294251 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f6fe80fdf50 0x7f6fe80fe320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:54:47.701 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:47.694+0000 7f6fee027700 1 -- 172.21.15.133:0/421294251 >> 172.21.15.133:0/421294251 conn(0x7f6fe80f7a10 msgr2=0x7f6fe80f9e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:54:47.701 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:47.694+0000 7f6fee027700 1 -- 172.21.15.133:0/421294251 shutdown_connections 2024-02-20T07:54:47.702 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:47.694+0000 7f6fee027700 1 -- 172.21.15.133:0/421294251 wait complete. 2024-02-20T07:54:47.702 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:47.695+0000 7f6fee027700 1 Processor -- start 2024-02-20T07:54:47.702 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:47.695+0000 7f6fee027700 1 -- start start 2024-02-20T07:54:47.702 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:47.695+0000 7f6fee027700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f6fe80fdf50 0x7f6fe8115c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:54:47.702 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:47.695+0000 7f6fee027700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f6fe80fe8f0 0x7f6fe8116160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:54:47.702 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:47.695+0000 7f6fee027700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fe81167b0 con 0x7f6fe80fe8f0 2024-02-20T07:54:47.702 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:47.695+0000 7f6fee027700 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fe810fb00 con 0x7f6fe80fdf50 2024-02-20T07:54:47.703 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:47.695+0000 7f6fe6ffd700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f6fe80fe8f0 0x7f6fe8116160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:54:47.703 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:47.695+0000 7f6fe6ffd700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f6fe80fe8f0 0x7f6fe8116160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.133:60398/0 (socket says 172.21.15.133:60398) 2024-02-20T07:54:47.703 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:47.695+0000 7f6fe6ffd700 1 -- 172.21.15.133:0/829777664 learned_addr learned my addr 172.21.15.133:0/829777664 (peer_addr_for_me v2:172.21.15.133:0/0) 2024-02-20T07:54:47.703 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:47.695+0000 7f6fe77fe700 1 --2- 172.21.15.133:0/829777664 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f6fe80fdf50 0x7f6fe8115c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:54:47.703 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:47.695+0000 7f6fe6ffd700 1 -- 172.21.15.133:0/829777664 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f6fe80fdf50 msgr2=0x7f6fe8115c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:54:47.703 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:47.695+0000 7f6fe6ffd700 1 --2- 172.21.15.133:0/829777664 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f6fe80fdf50 0x7f6fe8115c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:54:47.703 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:47.695+0000 7f6fe6ffd700 1 -- 172.21.15.133:0/829777664 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fdc005040 con 0x7f6fe80fe8f0 2024-02-20T07:54:47.703 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:47.695+0000 7f6fe77fe700 1 --2- 172.21.15.133:0/829777664 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f6fe80fdf50 0x7f6fe8115c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-20T07:54:47.703 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:47.696+0000 7f6fe6ffd700 1 --2- 172.21.15.133:0/829777664 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f6fe80fe8f0 0x7f6fe8116160 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f6fdc000c00 tx=0x7f6fdc009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:54:47.703 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:47.696+0000 7f6fcffff700 1 -- 172.21.15.133:0/829777664 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6fdc00e040 con 0x7f6fe80fe8f0 2024-02-20T07:54:47.703 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:47.696+0000 7f6fcffff700 1 -- 172.21.15.133:0/829777664 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6fdc009af0 con 0x7f6fe80fe8f0 2024-02-20T07:54:47.704 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:47.696+0000 7f6fee027700 1 -- 172.21.15.133:0/829777664 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6fe810fd80 con 0x7f6fe80fe8f0 2024-02-20T07:54:47.704 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:47.696+0000 7f6fcffff700 1 -- 172.21.15.133:0/829777664 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6fdc01c630 con 0x7f6fe80fe8f0 2024-02-20T07:54:47.704 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:47.696+0000 7f6fee027700 1 -- 172.21.15.133:0/829777664 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6fe81102c0 con 0x7f6fe80fe8f0 2024-02-20T07:54:47.704 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:47.697+0000 7f6fee027700 1 -- 172.21.15.133:0/829777664 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6fe8106a80 con 0x7f6fe80fe8f0 2024-02-20T07:54:47.704 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:47.697+0000 7f6fcffff700 1 -- 172.21.15.133:0/829777664 <== mon.0 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7f6fdc00bbf0 con 0x7f6fe80fe8f0 2024-02-20T07:54:47.704 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:47.698+0000 7f6fcffff700 1 --2- 172.21.15.133:0/829777664 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7f6fd00692a0 0x7f6fd006b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:54:47.704 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:47.698+0000 7f6fcffff700 1 -- 172.21.15.133:0/829777664 <== mon.0 v2:172.21.15.133:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f6fdc088240 con 0x7f6fe80fe8f0 2024-02-20T07:54:47.704 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:47.698+0000 7f6fe77fe700 1 --2- 172.21.15.133:0/829777664 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7f6fd00692a0 0x7f6fd006b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:54:47.705 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:47.698+0000 7f6fe77fe700 1 --2- 172.21.15.133:0/829777664 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7f6fd00692a0 0x7f6fd006b750 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f6fd8009fd0 tx=0x7f6fd8009ee0 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:54:47.705 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:47.702+0000 7f6fcffff700 1 -- 172.21.15.133:0/829777664 <== mon.0 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f6fdc0548b0 con 0x7f6fe80fe8f0 2024-02-20T07:54:47.928 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:47.926+0000 7f6fee027700 1 -- 172.21.15.133:0/829777664 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f6fe804bed0 con 0x7f6fe80fe8f0 2024-02-20T07:54:47.928 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:47.926+0000 7f6fcffff700 1 -- 172.21.15.133:0/829777664 <== mon.0 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+46 (secure 0 0 0) 0x7f6fdc057ed0 con 0x7f6fe80fe8f0 2024-02-20T07:54:47.928 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-02-20T07:54:47.928 INFO:teuthology.orchestra.run.smithi133.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-02-20T07:54:47.930 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:47.928+0000 7f6fee027700 1 -- 172.21.15.133:0/829777664 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7f6fd00692a0 msgr2=0x7f6fd006b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:54:47.930 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:47.928+0000 7f6fee027700 1 --2- 172.21.15.133:0/829777664 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7f6fd00692a0 0x7f6fd006b750 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f6fd8009fd0 tx=0x7f6fd8009ee0 comp rx=0 tx=0).stop 2024-02-20T07:54:47.930 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:47.928+0000 7f6fee027700 1 -- 172.21.15.133:0/829777664 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f6fe80fe8f0 msgr2=0x7f6fe8116160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:54:47.930 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:47.928+0000 7f6fee027700 1 --2- 172.21.15.133:0/829777664 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f6fe80fe8f0 0x7f6fe8116160 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f6fdc000c00 tx=0x7f6fdc009e30 comp rx=0 tx=0).stop 2024-02-20T07:54:47.930 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:47.928+0000 7f6fee027700 1 -- 172.21.15.133:0/829777664 shutdown_connections 2024-02-20T07:54:47.930 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:47.928+0000 7f6fee027700 1 --2- 172.21.15.133:0/829777664 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7f6fd00692a0 0x7f6fd006b750 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:54:47.931 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:47.928+0000 7f6fee027700 1 --2- 172.21.15.133:0/829777664 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f6fe80fdf50 0x7f6fe8115c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:54:47.931 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:47.928+0000 7f6fee027700 1 --2- 172.21.15.133:0/829777664 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f6fe80fe8f0 0x7f6fe8116160 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:54:47.931 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:47.928+0000 7f6fee027700 1 -- 172.21.15.133:0/829777664 >> 172.21.15.133:0/829777664 conn(0x7f6fe80f7a10 msgr2=0x7f6fe80fc580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:54:47.931 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:47.928+0000 7f6fee027700 1 -- 172.21.15.133:0/829777664 shutdown_connections 2024-02-20T07:54:47.931 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:47.928+0000 7f6fee027700 1 -- 172.21.15.133:0/829777664 wait complete. 2024-02-20T07:54:48.241 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:54:48 smithi133 ceph-mon[29117]: from='client.? 172.21.15.133:0/829777664' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-02-20T07:54:48.297 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:54:48 smithi183 ceph-mon[35832]: from='client.? 172.21.15.133:0/829777664' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-02-20T07:54:48.673 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-02-20T07:54:48.673 INFO:tasks.cephadm:Setup complete, yielding 2024-02-20T07:54:48.674 INFO:teuthology.run_tasks:Running task print... 2024-02-20T07:54:48.683 INFO:teuthology.task.print:**** done end installing quincy cephadm ... 2024-02-20T07:54:48.683 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-02-20T07:54:48.715 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi133.front.sepia.ceph.com 2024-02-20T07:54:48.716 DEBUG:teuthology.orchestra.run.smithi133:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4537327c-cfc4-11ee-95bb-87774f69a715 -- bash -c 'ceph config set mgr mgr/cephadm/use_repo_digest true --force' 2024-02-20T07:54:48.973 INFO:teuthology.orchestra.run.smithi133.stderr:Inferring config /var/lib/ceph/4537327c-cfc4-11ee-95bb-87774f69a715/mon.smithi133/config 2024-02-20T07:54:49.297 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:54:49 smithi183 ceph-mon[35832]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-20T07:54:49.365 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:54:49 smithi133 ceph-mon[29117]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-20T07:54:50.627 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:50.624+0000 7f2b73713700 1 -- 172.21.15.133:0/3236273418 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f2b6c1000e0 msgr2=0x7f2b6c102520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:54:50.627 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:50.624+0000 7f2b73713700 1 --2- 172.21.15.133:0/3236273418 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f2b6c1000e0 0x7f2b6c102520 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f2b5c004940 tx=0x7f2b5c009e30 comp rx=0 tx=0).stop 2024-02-20T07:54:50.627 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:50.624+0000 7f2b73713700 1 -- 172.21.15.133:0/3236273418 shutdown_connections 2024-02-20T07:54:50.627 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:50.624+0000 7f2b73713700 1 --2- 172.21.15.133:0/3236273418 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f2b6c1000e0 0x7f2b6c102520 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:54:50.627 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:50.624+0000 7f2b73713700 1 --2- 172.21.15.133:0/3236273418 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2b6c0663c0 0x7f2b6c0ffb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:54:50.627 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:50.624+0000 7f2b73713700 1 -- 172.21.15.133:0/3236273418 >> 172.21.15.133:0/3236273418 conn(0x7f2b6c0f74e0 msgr2=0x7f2b6c0f9930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:54:50.627 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:50.624+0000 7f2b73713700 1 -- 172.21.15.133:0/3236273418 shutdown_connections 2024-02-20T07:54:50.628 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:50.624+0000 7f2b73713700 1 -- 172.21.15.133:0/3236273418 wait complete. 2024-02-20T07:54:50.628 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:50.624+0000 7f2b73713700 1 Processor -- start 2024-02-20T07:54:50.628 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:50.624+0000 7f2b73713700 1 -- start start 2024-02-20T07:54:50.628 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:50.625+0000 7f2b73713700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f2b6c0663c0 0x7f2b6c10d9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:54:50.628 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:50.625+0000 7f2b73713700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2b6c1000e0 0x7f2b6c10dee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:54:50.628 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:50.625+0000 7f2b73713700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b6c10e4e0 con 0x7f2b6c0663c0 2024-02-20T07:54:50.628 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:50.625+0000 7f2b73713700 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b6c10ed80 con 0x7f2b6c1000e0 2024-02-20T07:54:50.628 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:50.625+0000 7f2b714af700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f2b6c0663c0 0x7f2b6c10d9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:54:50.628 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:50.625+0000 7f2b714af700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f2b6c0663c0 0x7f2b6c10d9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.133:60410/0 (socket says 172.21.15.133:60410) 2024-02-20T07:54:50.628 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:50.625+0000 7f2b714af700 1 -- 172.21.15.133:0/2497648516 learned_addr learned my addr 172.21.15.133:0/2497648516 (peer_addr_for_me v2:172.21.15.133:0/0) 2024-02-20T07:54:50.628 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:50.625+0000 7f2b70cae700 1 --2- 172.21.15.133:0/2497648516 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2b6c1000e0 0x7f2b6c10dee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:54:50.628 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:50.625+0000 7f2b714af700 1 -- 172.21.15.133:0/2497648516 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2b6c1000e0 msgr2=0x7f2b6c10dee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:54:50.628 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:50.625+0000 7f2b714af700 1 --2- 172.21.15.133:0/2497648516 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2b6c1000e0 0x7f2b6c10dee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:54:50.628 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:50.625+0000 7f2b714af700 1 -- 172.21.15.133:0/2497648516 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b5c005040 con 0x7f2b6c0663c0 2024-02-20T07:54:50.628 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:50.625+0000 7f2b714af700 1 --2- 172.21.15.133:0/2497648516 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f2b6c0663c0 0x7f2b6c10d9a0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f2b6800cb10 tx=0x7f2b68007f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:54:50.629 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:50.625+0000 7f2b61ffb700 1 -- 172.21.15.133:0/2497648516 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2b68004200 con 0x7f2b6c0663c0 2024-02-20T07:54:50.629 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:50.625+0000 7f2b61ffb700 1 -- 172.21.15.133:0/2497648516 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2b6800acc0 con 0x7f2b6c0663c0 2024-02-20T07:54:50.629 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:50.625+0000 7f2b73713700 1 -- 172.21.15.133:0/2497648516 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2b6c10f060 con 0x7f2b6c0663c0 2024-02-20T07:54:50.629 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:50.625+0000 7f2b61ffb700 1 -- 172.21.15.133:0/2497648516 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2b68003d90 con 0x7f2b6c0663c0 2024-02-20T07:54:50.629 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:50.626+0000 7f2b73713700 1 -- 172.21.15.133:0/2497648516 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2b6c10f5a0 con 0x7f2b6c0663c0 2024-02-20T07:54:50.630 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:50.626+0000 7f2b73713700 1 -- 172.21.15.133:0/2497648516 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2b6c107a50 con 0x7f2b6c0663c0 2024-02-20T07:54:50.630 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:50.627+0000 7f2b61ffb700 1 -- 172.21.15.133:0/2497648516 <== mon.0 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7f2b68003430 con 0x7f2b6c0663c0 2024-02-20T07:54:50.630 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:50.627+0000 7f2b61ffb700 1 --2- 172.21.15.133:0/2497648516 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7f2b580692a0 0x7f2b5806b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:54:50.630 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:50.627+0000 7f2b61ffb700 1 -- 172.21.15.133:0/2497648516 <== mon.0 v2:172.21.15.133:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f2b680873e0 con 0x7f2b6c0663c0 2024-02-20T07:54:50.630 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:50.627+0000 7f2b70cae700 1 --2- 172.21.15.133:0/2497648516 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7f2b580692a0 0x7f2b5806b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:54:50.630 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:50.628+0000 7f2b70cae700 1 --2- 172.21.15.133:0/2497648516 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7f2b580692a0 0x7f2b5806b750 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f2b5c009fd0 tx=0x7f2b5c009e30 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:54:50.633 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:50.631+0000 7f2b61ffb700 1 -- 172.21.15.133:0/2497648516 <== mon.0 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f2b680574e0 con 0x7f2b6c0663c0 2024-02-20T07:54:50.815 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:50.813+0000 7f2b73713700 1 -- 172.21.15.133:0/2497648516 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/use_repo_digest}] v 0) v1 -- 0x7f2b6c04bed0 con 0x7f2b6c0663c0 2024-02-20T07:54:50.819 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:50.817+0000 7f2b61ffb700 1 -- 172.21.15.133:0/2497648516 <== mon.0 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/use_repo_digest}]=0 v20)=0 v20) v1 ==== 143+0+0 (secure 0 0 0) 0x7f2b68057070 con 0x7f2b6c0663c0 2024-02-20T07:54:50.821 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:50.820+0000 7f2b73713700 1 -- 172.21.15.133:0/2497648516 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7f2b580692a0 msgr2=0x7f2b5806b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:54:50.821 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:50.820+0000 7f2b73713700 1 --2- 172.21.15.133:0/2497648516 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7f2b580692a0 0x7f2b5806b750 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f2b5c009fd0 tx=0x7f2b5c009e30 comp rx=0 tx=0).stop 2024-02-20T07:54:50.821 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:50.820+0000 7f2b73713700 1 -- 172.21.15.133:0/2497648516 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f2b6c0663c0 msgr2=0x7f2b6c10d9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:54:50.821 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:50.820+0000 7f2b73713700 1 --2- 172.21.15.133:0/2497648516 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f2b6c0663c0 0x7f2b6c10d9a0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f2b6800cb10 tx=0x7f2b68007f90 comp rx=0 tx=0).stop 2024-02-20T07:54:50.822 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:50.820+0000 7f2b73713700 1 -- 172.21.15.133:0/2497648516 shutdown_connections 2024-02-20T07:54:50.822 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:50.820+0000 7f2b73713700 1 --2- 172.21.15.133:0/2497648516 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f2b6c0663c0 0x7f2b6c10d9a0 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:54:50.822 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:50.820+0000 7f2b73713700 1 --2- 172.21.15.133:0/2497648516 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7f2b580692a0 0x7f2b5806b750 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:54:50.822 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:50.820+0000 7f2b73713700 1 --2- 172.21.15.133:0/2497648516 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2b6c1000e0 0x7f2b6c10dee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:54:50.822 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:50.820+0000 7f2b73713700 1 -- 172.21.15.133:0/2497648516 >> 172.21.15.133:0/2497648516 conn(0x7f2b6c0f74e0 msgr2=0x7f2b6c0f9930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:54:50.822 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:50.820+0000 7f2b73713700 1 -- 172.21.15.133:0/2497648516 shutdown_connections 2024-02-20T07:54:50.822 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:50.820+0000 7f2b73713700 1 -- 172.21.15.133:0/2497648516 wait complete. 2024-02-20T07:54:51.241 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:54:51 smithi133 ceph-mon[29117]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-20T07:54:51.241 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:54:51 smithi133 ceph-mon[29117]: from='client.? 172.21.15.133:0/2497648516' entity='client.admin' 2024-02-20T07:54:51.241 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:54:51 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-20T07:54:51.297 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:54:51 smithi183 ceph-mon[35832]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-20T07:54:51.297 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:54:51 smithi183 ceph-mon[35832]: from='client.? 172.21.15.133:0/2497648516' entity='client.admin' 2024-02-20T07:54:51.297 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:54:51 smithi183 ceph-mon[35832]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-20T07:54:51.460 INFO:teuthology.run_tasks:Running task print... 2024-02-20T07:54:51.469 INFO:teuthology.task.print:**** done cephadm.shell ceph config set mgr... 2024-02-20T07:54:51.469 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-02-20T07:54:51.483 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi133.front.sepia.ceph.com 2024-02-20T07:54:51.484 DEBUG:teuthology.orchestra.run.smithi133:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4537327c-cfc4-11ee-95bb-87774f69a715 -- bash -c 'ceph orch status' 2024-02-20T07:54:51.752 INFO:teuthology.orchestra.run.smithi133.stderr:Inferring config /var/lib/ceph/4537327c-cfc4-11ee-95bb-87774f69a715/mon.smithi133/config 2024-02-20T07:54:52.179 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:54:52 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-20T07:54:52.180 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:54:52 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-20T07:54:52.180 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:54:52 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:54:52.547 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:54:52 smithi183 ceph-mon[35832]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-20T07:54:52.547 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:54:52 smithi183 ceph-mon[35832]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-20T07:54:52.547 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:54:52 smithi183 ceph-mon[35832]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:54:53.061 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:54:53 smithi133 ceph-mon[29117]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-20T07:54:53.261 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:53.257+0000 7fc0a0bd3700 1 -- 172.21.15.133:0/2583269499 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fc09c0fe2b0 msgr2=0x7fc09c0fe700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:54:53.262 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:53.257+0000 7fc0a0bd3700 1 --2- 172.21.15.133:0/2583269499 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fc09c0fe2b0 0x7fc09c0fe700 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fc090004940 tx=0x7fc090009e30 comp rx=0 tx=0).stop 2024-02-20T07:54:53.262 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:53.257+0000 7fc0a0bd3700 1 -- 172.21.15.133:0/2583269499 shutdown_connections 2024-02-20T07:54:53.262 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:53.257+0000 7fc0a0bd3700 1 --2- 172.21.15.133:0/2583269499 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fc09c0fe2b0 0x7fc09c0fe700 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:54:53.262 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:53.257+0000 7fc0a0bd3700 1 --2- 172.21.15.133:0/2583269499 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc09c0fb220 0x7fc09c0fb630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:54:53.262 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:53.257+0000 7fc0a0bd3700 1 -- 172.21.15.133:0/2583269499 >> 172.21.15.133:0/2583269499 conn(0x7fc09c0f6ca0 msgr2=0x7fc09c0f90d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:54:53.262 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:53.257+0000 7fc0a0bd3700 1 -- 172.21.15.133:0/2583269499 shutdown_connections 2024-02-20T07:54:53.262 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:53.257+0000 7fc0a0bd3700 1 -- 172.21.15.133:0/2583269499 wait complete. 2024-02-20T07:54:53.262 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:53.258+0000 7fc0a0bd3700 1 Processor -- start 2024-02-20T07:54:53.262 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:53.258+0000 7fc0a0bd3700 1 -- start start 2024-02-20T07:54:53.262 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:53.258+0000 7fc0a0bd3700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fc09c0fb220 0x7fc09c10da10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:54:53.262 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:53.258+0000 7fc0a0bd3700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc09c0fe2b0 0x7fc09c10df50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:54:53.263 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:53.258+0000 7fc0a0bd3700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc09c10e550 con 0x7fc09c0fb220 2024-02-20T07:54:53.263 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:53.258+0000 7fc0a0bd3700 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc09c10edf0 con 0x7fc09c0fe2b0 2024-02-20T07:54:53.263 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:53.258+0000 7fc09a59c700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fc09c0fb220 0x7fc09c10da10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:54:53.263 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:53.258+0000 7fc09a59c700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fc09c0fb220 0x7fc09c10da10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.133:60430/0 (socket says 172.21.15.133:60430) 2024-02-20T07:54:53.263 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:53.258+0000 7fc09a59c700 1 -- 172.21.15.133:0/1138214775 learned_addr learned my addr 172.21.15.133:0/1138214775 (peer_addr_for_me v2:172.21.15.133:0/0) 2024-02-20T07:54:53.263 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:53.258+0000 7fc09a59c700 1 -- 172.21.15.133:0/1138214775 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc09c0fe2b0 msgr2=0x7fc09c10df50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:54:53.263 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:53.258+0000 7fc099d9b700 1 --2- 172.21.15.133:0/1138214775 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc09c0fe2b0 0x7fc09c10df50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:54:53.263 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:53.258+0000 7fc09a59c700 1 --2- 172.21.15.133:0/1138214775 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc09c0fe2b0 0x7fc09c10df50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:54:53.263 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:53.258+0000 7fc09a59c700 1 -- 172.21.15.133:0/1138214775 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc090005040 con 0x7fc09c0fb220 2024-02-20T07:54:53.263 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:53.259+0000 7fc09a59c700 1 --2- 172.21.15.133:0/1138214775 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fc09c0fb220 0x7fc09c10da10 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fc08c00c8e0 tx=0x7fc08c0044c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:54:53.263 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:53.259+0000 7fc08affd700 1 -- 172.21.15.133:0/1138214775 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc08c00a940 con 0x7fc09c0fb220 2024-02-20T07:54:53.263 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:53.259+0000 7fc0a0bd3700 1 -- 172.21.15.133:0/1138214775 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc09c10f0d0 con 0x7fc09c0fb220 2024-02-20T07:54:53.263 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:53.259+0000 7fc08affd700 1 -- 172.21.15.133:0/1138214775 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc08c00ce80 con 0x7fc09c0fb220 2024-02-20T07:54:53.263 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:53.259+0000 7fc08affd700 1 -- 172.21.15.133:0/1138214775 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc08c0113a0 con 0x7fc09c0fb220 2024-02-20T07:54:53.264 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:53.259+0000 7fc0a0bd3700 1 -- 172.21.15.133:0/1138214775 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc09c10f610 con 0x7fc09c0fb220 2024-02-20T07:54:53.264 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:53.260+0000 7fc0a0bd3700 1 -- 172.21.15.133:0/1138214775 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc09c107a80 con 0x7fc09c0fb220 2024-02-20T07:54:53.265 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:53.261+0000 7fc08affd700 1 -- 172.21.15.133:0/1138214775 <== mon.0 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7fc08c012400 con 0x7fc09c0fb220 2024-02-20T07:54:53.265 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:53.261+0000 7fc08affd700 1 --2- 172.21.15.133:0/1138214775 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7fc0840692a0 0x7fc08406b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:54:53.265 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:53.261+0000 7fc08affd700 1 -- 172.21.15.133:0/1138214775 <== mon.0 v2:172.21.15.133:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fc08c086f50 con 0x7fc09c0fb220 2024-02-20T07:54:53.265 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:53.261+0000 7fc099d9b700 1 --2- 172.21.15.133:0/1138214775 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7fc0840692a0 0x7fc08406b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:54:53.265 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:53.262+0000 7fc099d9b700 1 --2- 172.21.15.133:0/1138214775 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7fc0840692a0 0x7fc08406b750 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fc090004560 tx=0x7fc090009e30 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:54:53.267 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:53.266+0000 7fc08affd700 1 -- 172.21.15.133:0/1138214775 <== mon.0 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7fc08c057050 con 0x7fc09c0fb220 2024-02-20T07:54:53.463 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:53.461+0000 7fc0a0bd3700 1 -- 172.21.15.133:0/1138214775 --> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] -- mgr_command(tid 0: {"prefix": "orch status", "target": ["mon-mgr", ""]}) v1 -- 0x7fc09c10f8f0 con 0x7fc0840692a0 2024-02-20T07:54:53.464 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:53.463+0000 7fc08affd700 1 -- 172.21.15.133:0/1138214775 <== mgr.14208 v2:172.21.15.133:6800/2703586938 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+43 (secure 0 0 0) 0x7fc09c10f8f0 con 0x7fc0840692a0 2024-02-20T07:54:53.464 INFO:teuthology.orchestra.run.smithi133.stdout:Backend: cephadm 2024-02-20T07:54:53.464 INFO:teuthology.orchestra.run.smithi133.stdout:Available: Yes 2024-02-20T07:54:53.464 INFO:teuthology.orchestra.run.smithi133.stdout:Paused: No 2024-02-20T07:54:53.466 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:53.464+0000 7fc0a0bd3700 1 -- 172.21.15.133:0/1138214775 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7fc0840692a0 msgr2=0x7fc08406b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:54:53.466 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:53.464+0000 7fc0a0bd3700 1 --2- 172.21.15.133:0/1138214775 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7fc0840692a0 0x7fc08406b750 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fc090004560 tx=0x7fc090009e30 comp rx=0 tx=0).stop 2024-02-20T07:54:53.466 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:53.464+0000 7fc0a0bd3700 1 -- 172.21.15.133:0/1138214775 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fc09c0fb220 msgr2=0x7fc09c10da10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:54:53.466 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:53.464+0000 7fc0a0bd3700 1 --2- 172.21.15.133:0/1138214775 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fc09c0fb220 0x7fc09c10da10 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fc08c00c8e0 tx=0x7fc08c0044c0 comp rx=0 tx=0).stop 2024-02-20T07:54:53.467 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:53.464+0000 7fc0a0bd3700 1 -- 172.21.15.133:0/1138214775 shutdown_connections 2024-02-20T07:54:53.467 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:53.464+0000 7fc0a0bd3700 1 --2- 172.21.15.133:0/1138214775 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fc09c0fb220 0x7fc09c10da10 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:54:53.467 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:53.465+0000 7fc0a0bd3700 1 --2- 172.21.15.133:0/1138214775 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7fc0840692a0 0x7fc08406b750 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:54:53.467 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:53.465+0000 7fc0a0bd3700 1 --2- 172.21.15.133:0/1138214775 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc09c0fe2b0 0x7fc09c10df50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:54:53.467 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:53.465+0000 7fc0a0bd3700 1 -- 172.21.15.133:0/1138214775 >> 172.21.15.133:0/1138214775 conn(0x7fc09c0f6ca0 msgr2=0x7fc09c0f8f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:54:53.467 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:53.465+0000 7fc0a0bd3700 1 -- 172.21.15.133:0/1138214775 shutdown_connections 2024-02-20T07:54:53.467 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:53.465+0000 7fc0a0bd3700 1 -- 172.21.15.133:0/1138214775 wait complete. 2024-02-20T07:54:53.547 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:54:53 smithi183 ceph-mon[35832]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-20T07:54:54.166 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:54:54 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-20T07:54:54.190 DEBUG:teuthology.orchestra.run.smithi133:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4537327c-cfc4-11ee-95bb-87774f69a715 -- bash -c 'ceph orch ps' 2024-02-20T07:54:54.445 INFO:teuthology.orchestra.run.smithi133.stderr:Inferring config /var/lib/ceph/4537327c-cfc4-11ee-95bb-87774f69a715/mon.smithi133/config 2024-02-20T07:54:54.547 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:54:54 smithi183 ceph-mon[35832]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-20T07:54:55.241 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:54:55 smithi133 ceph-mon[29117]: from='client.14442 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-20T07:54:55.242 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:54:55 smithi133 ceph-mon[29117]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-20T07:54:55.547 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:54:55 smithi183 ceph-mon[35832]: from='client.14442 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-20T07:54:55.547 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:54:55 smithi183 ceph-mon[35832]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-20T07:54:55.943 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:55.939+0000 7f99b69f6700 1 -- 172.21.15.133:0/1072415069 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f99b00fcd80 msgr2=0x7f99b00fd1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:54:55.943 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:55.939+0000 7f99b69f6700 1 --2- 172.21.15.133:0/1072415069 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f99b00fcd80 0x7f99b00fd1f0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f99a4004940 tx=0x7f99a4009e30 comp rx=0 tx=0).stop 2024-02-20T07:54:55.943 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:55.939+0000 7f99b69f6700 1 -- 172.21.15.133:0/1072415069 shutdown_connections 2024-02-20T07:54:55.943 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:55.939+0000 7f99b69f6700 1 --2- 172.21.15.133:0/1072415069 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f99b00fcd80 0x7f99b00fd1f0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:54:55.943 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:55.939+0000 7f99b69f6700 1 --2- 172.21.15.133:0/1072415069 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f99b00fc430 0x7f99b00fc840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:54:55.943 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:55.939+0000 7f99b69f6700 1 -- 172.21.15.133:0/1072415069 >> 172.21.15.133:0/1072415069 conn(0x7f99b00f6d20 msgr2=0x7f99b00f9150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:54:55.943 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:55.939+0000 7f99b69f6700 1 -- 172.21.15.133:0/1072415069 shutdown_connections 2024-02-20T07:54:55.943 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:55.939+0000 7f99b69f6700 1 -- 172.21.15.133:0/1072415069 wait complete. 2024-02-20T07:54:55.943 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:55.940+0000 7f99b69f6700 1 Processor -- start 2024-02-20T07:54:55.943 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:55.940+0000 7f99b69f6700 1 -- start start 2024-02-20T07:54:55.943 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:55.940+0000 7f99b69f6700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f99b00fc430 0x7f99b010cba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:54:55.943 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:55.940+0000 7f99b69f6700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f99b00fcd80 0x7f99b010d0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:54:55.943 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:55.940+0000 7f99b69f6700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99b010d770 con 0x7f99b00fcd80 2024-02-20T07:54:55.944 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:55.940+0000 7f99b69f6700 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99b010dfb0 con 0x7f99b00fc430 2024-02-20T07:54:55.944 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:55.940+0000 7f99af7fe700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f99b00fcd80 0x7f99b010d0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:54:55.944 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:55.940+0000 7f99af7fe700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f99b00fcd80 0x7f99b010d0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.133:60444/0 (socket says 172.21.15.133:60444) 2024-02-20T07:54:55.944 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:55.940+0000 7f99af7fe700 1 -- 172.21.15.133:0/2635699344 learned_addr learned my addr 172.21.15.133:0/2635699344 (peer_addr_for_me v2:172.21.15.133:0/0) 2024-02-20T07:54:55.944 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:55.940+0000 7f99affff700 1 --2- 172.21.15.133:0/2635699344 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f99b00fc430 0x7f99b010cba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:54:55.944 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:55.940+0000 7f99af7fe700 1 -- 172.21.15.133:0/2635699344 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f99b00fc430 msgr2=0x7f99b010cba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:54:55.944 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:55.940+0000 7f99af7fe700 1 --2- 172.21.15.133:0/2635699344 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f99b00fc430 0x7f99b010cba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:54:55.944 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:55.941+0000 7f99af7fe700 1 -- 172.21.15.133:0/2635699344 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99a4005040 con 0x7f99b00fcd80 2024-02-20T07:54:55.944 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:55.941+0000 7f99af7fe700 1 --2- 172.21.15.133:0/2635699344 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f99b00fcd80 0x7f99b010d0e0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f99a40093b0 tx=0x7f99a4009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:54:55.944 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:55.941+0000 7f99acff9700 1 -- 172.21.15.133:0/2635699344 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f99a400e040 con 0x7f99b00fcd80 2024-02-20T07:54:55.944 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:55.941+0000 7f99b69f6700 1 -- 172.21.15.133:0/2635699344 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f99b010e230 con 0x7f99b00fcd80 2024-02-20T07:54:55.944 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:55.941+0000 7f99acff9700 1 -- 172.21.15.133:0/2635699344 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f99a400bcc0 con 0x7f99b00fcd80 2024-02-20T07:54:55.944 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:55.941+0000 7f99acff9700 1 -- 172.21.15.133:0/2635699344 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f99a401c630 con 0x7f99b00fcd80 2024-02-20T07:54:55.944 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:55.941+0000 7f99b69f6700 1 -- 172.21.15.133:0/2635699344 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f99b010e770 con 0x7f99b00fcd80 2024-02-20T07:54:55.945 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:55.942+0000 7f99b69f6700 1 -- 172.21.15.133:0/2635699344 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f99b004bed0 con 0x7f99b00fcd80 2024-02-20T07:54:55.945 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:55.942+0000 7f99acff9700 1 -- 172.21.15.133:0/2635699344 <== mon.0 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7f99a40139f0 con 0x7f99b00fcd80 2024-02-20T07:54:55.945 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:55.943+0000 7f99acff9700 1 --2- 172.21.15.133:0/2635699344 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7f999806b4b0 0x7f999806d960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:54:55.946 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:55.943+0000 7f99acff9700 1 -- 172.21.15.133:0/2635699344 <== mon.0 v2:172.21.15.133:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f99a4088250 con 0x7f99b00fcd80 2024-02-20T07:54:55.946 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:55.943+0000 7f99affff700 1 --2- 172.21.15.133:0/2635699344 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7f999806b4b0 0x7f999806d960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:54:55.946 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:55.943+0000 7f99affff700 1 --2- 172.21.15.133:0/2635699344 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7f999806b4b0 0x7f999806d960 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f99a0006fd0 tx=0x7f99a0008040 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:54:55.948 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:55.947+0000 7f99acff9700 1 -- 172.21.15.133:0/2635699344 <== mon.0 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f99a4058350 con 0x7f99b00fcd80 2024-02-20T07:54:56.142 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:56.140+0000 7f99b69f6700 1 -- 172.21.15.133:0/2635699344 --> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f99b010e990 con 0x7f999806b4b0 2024-02-20T07:54:56.150 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:56.148+0000 7f99acff9700 1 -- 172.21.15.133:0/2635699344 <== mgr.14208 v2:172.21.15.133:6800/2703586938 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+2682 (secure 0 0 0) 0x7f99b010e990 con 0x7f999806b4b0 2024-02-20T07:54:56.150 INFO:teuthology.orchestra.run.smithi133.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-20T07:54:56.150 INFO:teuthology.orchestra.run.smithi133.stdout:alertmanager.smithi133 smithi133 *:9093,9094 running (3m) 2m ago 5m 26.1M - 0.25.0 c8568f914cd2 26689e669607 2024-02-20T07:54:56.150 INFO:teuthology.orchestra.run.smithi133.stdout:crash.smithi133 smithi133 running (5m) 2m ago 5m 6660k - 17.2.7-521-gc80e7168 10b0c032c913 ba7610707152 2024-02-20T07:54:56.150 INFO:teuthology.orchestra.run.smithi133.stdout:crash.smithi183 smithi183 running (4m) 68s ago 4m 6669k - 17.2.7-521-gc80e7168 10b0c032c913 55659076790f 2024-02-20T07:54:56.150 INFO:teuthology.orchestra.run.smithi133.stdout:grafana.smithi133 smithi133 *:3000 running (3m) 2m ago 4m 74.9M - 9.4.7 954c08fa6188 d35d6b4393e9 2024-02-20T07:54:56.150 INFO:teuthology.orchestra.run.smithi133.stdout:mgr.smithi133.mrvcbp smithi133 *:9283 running (6m) 2m ago 6m 469M - 17.2.7-521-gc80e7168 10b0c032c913 58ed54a355b1 2024-02-20T07:54:56.150 INFO:teuthology.orchestra.run.smithi133.stdout:mgr.smithi183.hebfng smithi183 *:8443,9283 running (4m) 68s ago 4m 433M - 17.2.7-521-gc80e7168 10b0c032c913 a0dc319ab2bd 2024-02-20T07:54:56.150 INFO:teuthology.orchestra.run.smithi133.stdout:mon.smithi133 smithi133 running (6m) 2m ago 6m 39.8M 2048M 17.2.7-521-gc80e7168 10b0c032c913 fe7046a20c87 2024-02-20T07:54:56.150 INFO:teuthology.orchestra.run.smithi133.stdout:mon.smithi183 smithi183 running (4m) 68s ago 4m 38.7M 2048M 17.2.7-521-gc80e7168 10b0c032c913 971a48e1d959 2024-02-20T07:54:56.151 INFO:teuthology.orchestra.run.smithi133.stdout:node-exporter.smithi133 smithi133 *:9100 running (5m) 2m ago 5m 16.4M - 1.5.0 0da6a335fe13 71e70059b12f 2024-02-20T07:54:56.151 INFO:teuthology.orchestra.run.smithi133.stdout:node-exporter.smithi183 smithi183 *:9100 running (4m) 68s ago 4m 16.4M - 1.5.0 0da6a335fe13 c567d872acb6 2024-02-20T07:54:56.151 INFO:teuthology.orchestra.run.smithi133.stdout:osd.0 smithi133 running (3m) 2m ago 3m 38.1M 4288M 17.2.7-521-gc80e7168 10b0c032c913 eee5e0d5322f 2024-02-20T07:54:56.151 INFO:teuthology.orchestra.run.smithi133.stdout:osd.1 smithi133 running (2m) 2m ago 2m 36.3M 4288M 17.2.7-521-gc80e7168 10b0c032c913 aa50c25fdf11 2024-02-20T07:54:56.151 INFO:teuthology.orchestra.run.smithi133.stdout:osd.2 smithi133 running (2m) 2m ago 2m 10.2M 4288M 17.2.7-521-gc80e7168 10b0c032c913 91e34ab8741f 2024-02-20T07:54:56.151 INFO:teuthology.orchestra.run.smithi133.stdout:osd.3 smithi183 running (2m) 68s ago 2m 41.3M 5312M 17.2.7-521-gc80e7168 10b0c032c913 4e2ae9566662 2024-02-20T07:54:56.151 INFO:teuthology.orchestra.run.smithi133.stdout:osd.4 smithi183 running (99s) 68s ago 98s 40.5M 5312M 17.2.7-521-gc80e7168 10b0c032c913 79015466a807 2024-02-20T07:54:56.151 INFO:teuthology.orchestra.run.smithi133.stdout:osd.5 smithi183 running (73s) 68s ago 73s 10.9M 5312M 17.2.7-521-gc80e7168 10b0c032c913 f85f43119a0b 2024-02-20T07:54:56.151 INFO:teuthology.orchestra.run.smithi133.stdout:prometheus.smithi133 smithi133 *:9095 running (3m) 2m ago 4m 35.8M - 2.43.0 a07b618ecd1d 568c27a3eecc 2024-02-20T07:54:56.152 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:56.150+0000 7f99b69f6700 1 -- 172.21.15.133:0/2635699344 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7f999806b4b0 msgr2=0x7f999806d960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:54:56.152 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:56.150+0000 7f99b69f6700 1 --2- 172.21.15.133:0/2635699344 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7f999806b4b0 0x7f999806d960 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f99a0006fd0 tx=0x7f99a0008040 comp rx=0 tx=0).stop 2024-02-20T07:54:56.152 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:56.150+0000 7f99b69f6700 1 -- 172.21.15.133:0/2635699344 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f99b00fcd80 msgr2=0x7f99b010d0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:54:56.152 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:56.150+0000 7f99b69f6700 1 --2- 172.21.15.133:0/2635699344 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f99b00fcd80 0x7f99b010d0e0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f99a40093b0 tx=0x7f99a4009e30 comp rx=0 tx=0).stop 2024-02-20T07:54:56.152 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:56.150+0000 7f99b69f6700 1 -- 172.21.15.133:0/2635699344 shutdown_connections 2024-02-20T07:54:56.152 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:56.150+0000 7f99b69f6700 1 --2- 172.21.15.133:0/2635699344 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7f999806b4b0 0x7f999806d960 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:54:56.153 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:56.150+0000 7f99b69f6700 1 --2- 172.21.15.133:0/2635699344 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f99b00fc430 0x7f99b010cba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:54:56.153 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:56.150+0000 7f99b69f6700 1 --2- 172.21.15.133:0/2635699344 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f99b00fcd80 0x7f99b010d0e0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:54:56.153 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:56.150+0000 7f99b69f6700 1 -- 172.21.15.133:0/2635699344 >> 172.21.15.133:0/2635699344 conn(0x7f99b00f6d20 msgr2=0x7f99b01005c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:54:56.153 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:56.150+0000 7f99b69f6700 1 -- 172.21.15.133:0/2635699344 shutdown_connections 2024-02-20T07:54:56.153 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:56.150+0000 7f99b69f6700 1 -- 172.21.15.133:0/2635699344 wait complete. 2024-02-20T07:54:56.770 DEBUG:teuthology.orchestra.run.smithi133:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4537327c-cfc4-11ee-95bb-87774f69a715 -- bash -c 'ceph orch ls' 2024-02-20T07:54:57.021 INFO:teuthology.orchestra.run.smithi133.stderr:Inferring config /var/lib/ceph/4537327c-cfc4-11ee-95bb-87774f69a715/mon.smithi133/config 2024-02-20T07:54:57.356 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:54:57 smithi133 ceph-mon[29117]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-20T07:54:57.547 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:54:57 smithi183 ceph-mon[35832]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-20T07:54:58.380 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:54:58 smithi133 ceph-mon[29117]: from='client.14446 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-20T07:54:58.482 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:58.477+0000 7fc7936af700 1 -- 172.21.15.133:0/2081576215 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fc78c0fc1a0 msgr2=0x7fc78c0fc5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:54:58.482 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:58.477+0000 7fc7936af700 1 --2- 172.21.15.133:0/2081576215 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fc78c0fc1a0 0x7fc78c0fc5f0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fc77c004940 tx=0x7fc77c009e30 comp rx=0 tx=0).stop 2024-02-20T07:54:58.482 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:58.477+0000 7fc7936af700 1 -- 172.21.15.133:0/2081576215 shutdown_connections 2024-02-20T07:54:58.482 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:58.477+0000 7fc7936af700 1 --2- 172.21.15.133:0/2081576215 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fc78c0fc1a0 0x7fc78c0fc5f0 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:54:58.482 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:58.477+0000 7fc7936af700 1 --2- 172.21.15.133:0/2081576215 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc78c0fd440 0x7fc78c0fd850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:54:58.482 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:58.477+0000 7fc7936af700 1 -- 172.21.15.133:0/2081576215 >> 172.21.15.133:0/2081576215 conn(0x7fc78c0f6ca0 msgr2=0x7fc78c0f90d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:54:58.483 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:58.477+0000 7fc7936af700 1 -- 172.21.15.133:0/2081576215 shutdown_connections 2024-02-20T07:54:58.483 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:58.477+0000 7fc7936af700 1 -- 172.21.15.133:0/2081576215 wait complete. 2024-02-20T07:54:58.483 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:58.477+0000 7fc7936af700 1 Processor -- start 2024-02-20T07:54:58.483 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:58.478+0000 7fc7936af700 1 -- start start 2024-02-20T07:54:58.483 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:58.478+0000 7fc7936af700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fc78c0fc1a0 0x7fc78c1106f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:54:58.483 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:58.478+0000 7fc7936af700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc78c0fd440 0x7fc78c112c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:54:58.483 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:58.478+0000 7fc7936af700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc78c0fbe50 con 0x7fc78c0fc1a0 2024-02-20T07:54:58.483 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:58.478+0000 7fc7936af700 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc78c0fbf90 con 0x7fc78c0fd440 2024-02-20T07:54:58.483 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:58.478+0000 7fc79144b700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fc78c0fc1a0 0x7fc78c1106f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:54:58.483 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:58.478+0000 7fc79144b700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fc78c0fc1a0 0x7fc78c1106f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.133:47454/0 (socket says 172.21.15.133:47454) 2024-02-20T07:54:58.483 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:58.478+0000 7fc79144b700 1 -- 172.21.15.133:0/2356610147 learned_addr learned my addr 172.21.15.133:0/2356610147 (peer_addr_for_me v2:172.21.15.133:0/0) 2024-02-20T07:54:58.483 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:58.478+0000 7fc790c4a700 1 --2- 172.21.15.133:0/2356610147 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc78c0fd440 0x7fc78c112c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:54:58.483 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:58.478+0000 7fc79144b700 1 -- 172.21.15.133:0/2356610147 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc78c0fd440 msgr2=0x7fc78c112c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:54:58.484 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:58.478+0000 7fc79144b700 1 --2- 172.21.15.133:0/2356610147 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc78c0fd440 0x7fc78c112c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:54:58.484 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:58.478+0000 7fc79144b700 1 -- 172.21.15.133:0/2356610147 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc77c005040 con 0x7fc78c0fc1a0 2024-02-20T07:54:58.484 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:58.478+0000 7fc790c4a700 1 --2- 172.21.15.133:0/2356610147 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc78c0fd440 0x7fc78c112c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-20T07:54:58.484 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:58.478+0000 7fc79144b700 1 --2- 172.21.15.133:0/2356610147 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fc78c0fc1a0 0x7fc78c1106f0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fc78800ca90 tx=0x7fc788007f00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:54:58.484 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:58.478+0000 7fc781ffb700 1 -- 172.21.15.133:0/2356610147 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc78800aa50 con 0x7fc78c0fc1a0 2024-02-20T07:54:58.484 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:58.478+0000 7fc781ffb700 1 -- 172.21.15.133:0/2356610147 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc78800abb0 con 0x7fc78c0fc1a0 2024-02-20T07:54:58.484 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:58.478+0000 7fc7936af700 1 -- 172.21.15.133:0/2356610147 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc78c1132a0 con 0x7fc78c0fc1a0 2024-02-20T07:54:58.484 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:58.478+0000 7fc781ffb700 1 -- 172.21.15.133:0/2356610147 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc7880115e0 con 0x7fc78c0fc1a0 2024-02-20T07:54:58.484 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:58.479+0000 7fc7936af700 1 -- 172.21.15.133:0/2356610147 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc78c113790 con 0x7fc78c0fc1a0 2024-02-20T07:54:58.484 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:58.480+0000 7fc7936af700 1 -- 172.21.15.133:0/2356610147 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc78c106980 con 0x7fc78c0fc1a0 2024-02-20T07:54:58.484 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:58.480+0000 7fc781ffb700 1 -- 172.21.15.133:0/2356610147 <== mon.0 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7fc78800ad20 con 0x7fc78c0fc1a0 2024-02-20T07:54:58.484 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:58.480+0000 7fc781ffb700 1 --2- 172.21.15.133:0/2356610147 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7fc7780692a0 0x7fc77806b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:54:58.484 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:58.480+0000 7fc781ffb700 1 -- 172.21.15.133:0/2356610147 <== mon.0 v2:172.21.15.133:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fc7880874f0 con 0x7fc78c0fc1a0 2024-02-20T07:54:58.485 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:58.480+0000 7fc790c4a700 1 --2- 172.21.15.133:0/2356610147 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7fc7780692a0 0x7fc77806b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:54:58.485 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:58.481+0000 7fc790c4a700 1 --2- 172.21.15.133:0/2356610147 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7fc7780692a0 0x7fc77806b750 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fc77c004f90 tx=0x7fc77c009e30 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:54:58.486 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:58.485+0000 7fc781ffb700 1 -- 172.21.15.133:0/2356610147 <== mon.0 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7fc788053b60 con 0x7fc78c0fc1a0 2024-02-20T07:54:58.547 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:54:58 smithi183 ceph-mon[35832]: from='client.14446 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-20T07:54:58.679 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:58.677+0000 7fc7936af700 1 -- 172.21.15.133:0/2356610147 --> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] -- mgr_command(tid 0: {"prefix": "orch ls", "target": ["mon-mgr", ""]}) v1 -- 0x7fc78c02b340 con 0x7fc7780692a0 2024-02-20T07:54:58.684 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:58.682+0000 7fc781ffb700 1 -- 172.21.15.133:0/2356610147 <== mgr.14208 v2:172.21.15.133:6800/2703586938 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+1179 (secure 0 0 0) 0x7fc78c02b340 con 0x7fc7780692a0 2024-02-20T07:54:58.684 INFO:teuthology.orchestra.run.smithi133.stdout:NAME PORTS RUNNING REFRESHED AGE PLACEMENT 2024-02-20T07:54:58.684 INFO:teuthology.orchestra.run.smithi133.stdout:alertmanager ?:9093,9094 1/1 2m ago 5m count:1 2024-02-20T07:54:58.684 INFO:teuthology.orchestra.run.smithi133.stdout:crash 2/2 2m ago 5m * 2024-02-20T07:54:58.684 INFO:teuthology.orchestra.run.smithi133.stdout:grafana ?:3000 1/1 2m ago 5m count:1 2024-02-20T07:54:58.684 INFO:teuthology.orchestra.run.smithi133.stdout:mgr 2/2 2m ago 5m count:2 2024-02-20T07:54:58.684 INFO:teuthology.orchestra.run.smithi133.stdout:mon 2/2 2m ago 5m smithi133:172.21.15.133=smithi133;smithi183:172.21.15.183=smithi183;count:2 2024-02-20T07:54:58.684 INFO:teuthology.orchestra.run.smithi133.stdout:node-exporter ?:9100 2/2 2m ago 5m * 2024-02-20T07:54:58.684 INFO:teuthology.orchestra.run.smithi133.stdout:osd 6 2m ago - 2024-02-20T07:54:58.685 INFO:teuthology.orchestra.run.smithi133.stdout:prometheus ?:9095 1/1 2m ago 5m count:1 2024-02-20T07:54:58.688 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:58.684+0000 7fc7936af700 1 -- 172.21.15.133:0/2356610147 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7fc7780692a0 msgr2=0x7fc77806b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:54:58.689 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:58.684+0000 7fc7936af700 1 --2- 172.21.15.133:0/2356610147 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7fc7780692a0 0x7fc77806b750 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fc77c004f90 tx=0x7fc77c009e30 comp rx=0 tx=0).stop 2024-02-20T07:54:58.689 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:58.684+0000 7fc7936af700 1 -- 172.21.15.133:0/2356610147 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fc78c0fc1a0 msgr2=0x7fc78c1106f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:54:58.689 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:58.684+0000 7fc7936af700 1 --2- 172.21.15.133:0/2356610147 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fc78c0fc1a0 0x7fc78c1106f0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fc78800ca90 tx=0x7fc788007f00 comp rx=0 tx=0).stop 2024-02-20T07:54:58.690 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:58.684+0000 7fc7936af700 1 -- 172.21.15.133:0/2356610147 shutdown_connections 2024-02-20T07:54:58.690 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:58.684+0000 7fc7936af700 1 --2- 172.21.15.133:0/2356610147 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fc78c0fc1a0 0x7fc78c1106f0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:54:58.690 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:58.684+0000 7fc7936af700 1 --2- 172.21.15.133:0/2356610147 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7fc7780692a0 0x7fc77806b750 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:54:58.690 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:58.684+0000 7fc7936af700 1 --2- 172.21.15.133:0/2356610147 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc78c0fd440 0x7fc78c112c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:54:58.691 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:58.684+0000 7fc7936af700 1 -- 172.21.15.133:0/2356610147 >> 172.21.15.133:0/2356610147 conn(0x7fc78c0f6ca0 msgr2=0x7fc78c0f8f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:54:58.691 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:58.684+0000 7fc7936af700 1 -- 172.21.15.133:0/2356610147 shutdown_connections 2024-02-20T07:54:58.691 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:54:58.684+0000 7fc7936af700 1 -- 172.21.15.133:0/2356610147 wait complete. 2024-02-20T07:54:59.301 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:54:59 smithi133 ceph-mon[29117]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-20T07:54:59.326 DEBUG:teuthology.orchestra.run.smithi133:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4537327c-cfc4-11ee-95bb-87774f69a715 -- bash -c 'ceph orch host ls' 2024-02-20T07:54:59.547 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:54:59 smithi183 ceph-mon[35832]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-20T07:54:59.581 INFO:teuthology.orchestra.run.smithi133.stderr:Inferring config /var/lib/ceph/4537327c-cfc4-11ee-95bb-87774f69a715/mon.smithi133/config 2024-02-20T07:55:00.491 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:55:00 smithi133 ceph-mon[29117]: from='client.14450 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-02-20T07:55:00.547 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:55:00 smithi183 ceph-mon[35832]: from='client.14450 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-02-20T07:55:01.104 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:01.101+0000 7f2d2272d700 1 -- 172.21.15.133:0/1747375036 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f2d1c0663e0 msgr2=0x7f2d1c0ffaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:55:01.105 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:01.101+0000 7f2d2272d700 1 --2- 172.21.15.133:0/1747375036 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f2d1c0663e0 0x7f2d1c0ffaf0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f2d10004940 tx=0x7f2d10009e30 comp rx=0 tx=0).stop 2024-02-20T07:55:01.105 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:01.101+0000 7f2d2272d700 1 -- 172.21.15.133:0/1747375036 shutdown_connections 2024-02-20T07:55:01.105 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:01.101+0000 7f2d2272d700 1 --2- 172.21.15.133:0/1747375036 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2d1c1000c0 0x7f2d1c102520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:55:01.105 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:01.101+0000 7f2d2272d700 1 --2- 172.21.15.133:0/1747375036 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f2d1c0663e0 0x7f2d1c0ffaf0 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:55:01.105 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:01.101+0000 7f2d2272d700 1 -- 172.21.15.133:0/1747375036 >> 172.21.15.133:0/1747375036 conn(0x7f2d1c0f74e0 msgr2=0x7f2d1c0f9930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:55:01.105 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:01.101+0000 7f2d2272d700 1 -- 172.21.15.133:0/1747375036 shutdown_connections 2024-02-20T07:55:01.105 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:01.101+0000 7f2d2272d700 1 -- 172.21.15.133:0/1747375036 wait complete. 2024-02-20T07:55:01.105 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:01.101+0000 7f2d2272d700 1 Processor -- start 2024-02-20T07:55:01.105 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:01.102+0000 7f2d2272d700 1 -- start start 2024-02-20T07:55:01.105 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:01.102+0000 7f2d2272d700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f2d1c0663e0 0x7f2d1c10d980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:55:01.105 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:01.102+0000 7f2d2272d700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2d1c1000c0 0x7f2d1c10dec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:55:01.105 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:01.102+0000 7f2d2272d700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d1c10e550 con 0x7f2d1c0663e0 2024-02-20T07:55:01.105 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:01.102+0000 7f2d2272d700 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d1c10ed90 con 0x7f2d1c1000c0 2024-02-20T07:55:01.105 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:01.102+0000 7f2d1bfff700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f2d1c0663e0 0x7f2d1c10d980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:55:01.106 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:01.102+0000 7f2d1bfff700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f2d1c0663e0 0x7f2d1c10d980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.133:47472/0 (socket says 172.21.15.133:47472) 2024-02-20T07:55:01.106 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:01.102+0000 7f2d1bfff700 1 -- 172.21.15.133:0/3304345661 learned_addr learned my addr 172.21.15.133:0/3304345661 (peer_addr_for_me v2:172.21.15.133:0/0) 2024-02-20T07:55:01.106 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:01.102+0000 7f2d1b7fe700 1 --2- 172.21.15.133:0/3304345661 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2d1c1000c0 0x7f2d1c10dec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:55:01.106 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:01.102+0000 7f2d1bfff700 1 -- 172.21.15.133:0/3304345661 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2d1c1000c0 msgr2=0x7f2d1c10dec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:55:01.106 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:01.102+0000 7f2d1bfff700 1 --2- 172.21.15.133:0/3304345661 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2d1c1000c0 0x7f2d1c10dec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:55:01.106 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:01.102+0000 7f2d1bfff700 1 -- 172.21.15.133:0/3304345661 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d10005040 con 0x7f2d1c0663e0 2024-02-20T07:55:01.106 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:01.102+0000 7f2d1b7fe700 1 --2- 172.21.15.133:0/3304345661 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2d1c1000c0 0x7f2d1c10dec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-20T07:55:01.106 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:01.102+0000 7f2d1bfff700 1 --2- 172.21.15.133:0/3304345661 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f2d1c0663e0 0x7f2d1c10d980 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f2d100093b0 tx=0x7f2d10009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:55:01.106 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:01.102+0000 7f2d18ff9700 1 -- 172.21.15.133:0/3304345661 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2d1000e040 con 0x7f2d1c0663e0 2024-02-20T07:55:01.106 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:01.103+0000 7f2d18ff9700 1 -- 172.21.15.133:0/3304345661 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2d1000bcc0 con 0x7f2d1c0663e0 2024-02-20T07:55:01.106 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:01.103+0000 7f2d2272d700 1 -- 172.21.15.133:0/3304345661 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2d1c10f010 con 0x7f2d1c0663e0 2024-02-20T07:55:01.106 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:01.103+0000 7f2d18ff9700 1 -- 172.21.15.133:0/3304345661 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2d1001c630 con 0x7f2d1c0663e0 2024-02-20T07:55:01.106 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:01.103+0000 7f2d2272d700 1 -- 172.21.15.133:0/3304345661 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2d1c10f550 con 0x7f2d1c0663e0 2024-02-20T07:55:01.107 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:01.104+0000 7f2d2272d700 1 -- 172.21.15.133:0/3304345661 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2d1c107af0 con 0x7f2d1c0663e0 2024-02-20T07:55:01.107 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:01.104+0000 7f2d18ff9700 1 -- 172.21.15.133:0/3304345661 <== mon.0 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7f2d100139f0 con 0x7f2d1c0663e0 2024-02-20T07:55:01.108 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:01.104+0000 7f2d18ff9700 1 --2- 172.21.15.133:0/3304345661 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7f2d040692a0 0x7f2d0406b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:55:01.108 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:01.105+0000 7f2d18ff9700 1 -- 172.21.15.133:0/3304345661 <== mon.0 v2:172.21.15.133:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f2d10088250 con 0x7f2d1c0663e0 2024-02-20T07:55:01.108 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:01.105+0000 7f2d1b7fe700 1 --2- 172.21.15.133:0/3304345661 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7f2d040692a0 0x7f2d0406b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:55:01.108 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:01.105+0000 7f2d1b7fe700 1 --2- 172.21.15.133:0/3304345661 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7f2d040692a0 0x7f2d0406b750 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f2d0c006fd0 tx=0x7f2d0c008040 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:55:01.110 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:01.109+0000 7f2d18ff9700 1 -- 172.21.15.133:0/3304345661 <== mon.0 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f2d10058350 con 0x7f2d1c0663e0 2024-02-20T07:55:01.241 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:55:01 smithi133 ceph-mon[29117]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-20T07:55:01.299 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:01.298+0000 7f2d2272d700 1 -- 172.21.15.133:0/3304345661 --> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f2d1c02b340 con 0x7f2d040692a0 2024-02-20T07:55:01.301 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:01.299+0000 7f2d18ff9700 1 -- 172.21.15.133:0/3304345661 <== mgr.14208 v2:172.21.15.133:6800/2703586938 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+148 (secure 0 0 0) 0x7f2d1c02b340 con 0x7f2d040692a0 2024-02-20T07:55:01.301 INFO:teuthology.orchestra.run.smithi133.stdout:HOST ADDR LABELS STATUS 2024-02-20T07:55:01.301 INFO:teuthology.orchestra.run.smithi133.stdout:smithi133 172.21.15.133 2024-02-20T07:55:01.301 INFO:teuthology.orchestra.run.smithi133.stdout:smithi183 172.21.15.183 2024-02-20T07:55:01.301 INFO:teuthology.orchestra.run.smithi133.stdout:2 hosts in cluster 2024-02-20T07:55:01.303 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:01.301+0000 7f2d2272d700 1 -- 172.21.15.133:0/3304345661 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7f2d040692a0 msgr2=0x7f2d0406b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:55:01.303 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:01.301+0000 7f2d2272d700 1 --2- 172.21.15.133:0/3304345661 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7f2d040692a0 0x7f2d0406b750 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f2d0c006fd0 tx=0x7f2d0c008040 comp rx=0 tx=0).stop 2024-02-20T07:55:01.303 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:01.301+0000 7f2d2272d700 1 -- 172.21.15.133:0/3304345661 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f2d1c0663e0 msgr2=0x7f2d1c10d980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:55:01.303 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:01.301+0000 7f2d2272d700 1 --2- 172.21.15.133:0/3304345661 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f2d1c0663e0 0x7f2d1c10d980 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f2d100093b0 tx=0x7f2d10009e30 comp rx=0 tx=0).stop 2024-02-20T07:55:01.303 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:01.301+0000 7f2d2272d700 1 -- 172.21.15.133:0/3304345661 shutdown_connections 2024-02-20T07:55:01.304 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:01.301+0000 7f2d2272d700 1 --2- 172.21.15.133:0/3304345661 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f2d1c0663e0 0x7f2d1c10d980 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:55:01.304 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:01.301+0000 7f2d2272d700 1 --2- 172.21.15.133:0/3304345661 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7f2d040692a0 0x7f2d0406b750 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:55:01.304 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:01.301+0000 7f2d2272d700 1 --2- 172.21.15.133:0/3304345661 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2d1c1000c0 0x7f2d1c10dec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:55:01.304 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:01.301+0000 7f2d2272d700 1 -- 172.21.15.133:0/3304345661 >> 172.21.15.133:0/3304345661 conn(0x7f2d1c0f74e0 msgr2=0x7f2d1c0f9930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:55:01.304 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:01.301+0000 7f2d2272d700 1 -- 172.21.15.133:0/3304345661 shutdown_connections 2024-02-20T07:55:01.304 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:01.301+0000 7f2d2272d700 1 -- 172.21.15.133:0/3304345661 wait complete. 2024-02-20T07:55:01.449 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:55:01 smithi183 ceph-mon[35832]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-20T07:55:01.948 DEBUG:teuthology.orchestra.run.smithi133:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4537327c-cfc4-11ee-95bb-87774f69a715 -- bash -c 'ceph orch device ls' 2024-02-20T07:55:02.205 INFO:teuthology.orchestra.run.smithi133.stderr:Inferring config /var/lib/ceph/4537327c-cfc4-11ee-95bb-87774f69a715/mon.smithi133/config 2024-02-20T07:55:02.992 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:55:02 smithi133 ceph-mon[29117]: from='client.14454 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-02-20T07:55:02.992 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:55:02 smithi133 ceph-mon[29117]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-20T07:55:03.047 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:55:02 smithi183 ceph-mon[35832]: from='client.14454 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-02-20T07:55:03.048 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:55:02 smithi183 ceph-mon[35832]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-20T07:55:03.987 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:03.982+0000 7f5493ce4700 1 -- 172.21.15.133:0/3926029885 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f548c0fe0e0 msgr2=0x7f548c0fe550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:55:03.987 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:03.982+0000 7f5493ce4700 1 --2- 172.21.15.133:0/3926029885 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f548c0fe0e0 0x7f548c0fe550 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f547c004940 tx=0x7f547c009e30 comp rx=0 tx=0).stop 2024-02-20T07:55:03.987 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:03.982+0000 7f5493ce4700 1 -- 172.21.15.133:0/3926029885 shutdown_connections 2024-02-20T07:55:03.987 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:03.982+0000 7f5493ce4700 1 --2- 172.21.15.133:0/3926029885 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f548c0fe0e0 0x7f548c0fe550 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:55:03.987 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:03.982+0000 7f5493ce4700 1 --2- 172.21.15.133:0/3926029885 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f548c0fcef0 0x7f548c0fd320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:55:03.987 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:03.982+0000 7f5493ce4700 1 -- 172.21.15.133:0/3926029885 >> 172.21.15.133:0/3926029885 conn(0x7f548c0f8620 msgr2=0x7f548c0faa30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:55:03.988 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:03.982+0000 7f5493ce4700 1 -- 172.21.15.133:0/3926029885 shutdown_connections 2024-02-20T07:55:03.988 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:03.982+0000 7f5493ce4700 1 -- 172.21.15.133:0/3926029885 wait complete. 2024-02-20T07:55:03.988 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:03.982+0000 7f5493ce4700 1 Processor -- start 2024-02-20T07:55:03.988 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:03.982+0000 7f5493ce4700 1 -- start start 2024-02-20T07:55:03.988 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:03.982+0000 7f5493ce4700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f548c0fcef0 0x7f548c10d960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:55:03.988 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:03.982+0000 7f5493ce4700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f548c0fe0e0 0x7f548c10dea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:55:03.988 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:03.982+0000 7f5493ce4700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f548c10e4a0 con 0x7f548c0fcef0 2024-02-20T07:55:03.988 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:03.982+0000 7f5493ce4700 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f548c10ed40 con 0x7f548c0fe0e0 2024-02-20T07:55:03.988 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:03.982+0000 7f5491a80700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f548c0fcef0 0x7f548c10d960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:55:03.988 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:03.982+0000 7f5491a80700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f548c0fcef0 0x7f548c10d960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.133:47482/0 (socket says 172.21.15.133:47482) 2024-02-20T07:55:03.989 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:03.982+0000 7f5491a80700 1 -- 172.21.15.133:0/4061097112 learned_addr learned my addr 172.21.15.133:0/4061097112 (peer_addr_for_me v2:172.21.15.133:0/0) 2024-02-20T07:55:03.989 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:03.983+0000 7f549127f700 1 --2- 172.21.15.133:0/4061097112 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f548c0fe0e0 0x7f548c10dea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:55:03.989 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:03.983+0000 7f5491a80700 1 -- 172.21.15.133:0/4061097112 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f548c0fe0e0 msgr2=0x7f548c10dea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:55:03.989 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:03.983+0000 7f5491a80700 1 --2- 172.21.15.133:0/4061097112 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f548c0fe0e0 0x7f548c10dea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:55:03.989 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:03.983+0000 7f5491a80700 1 -- 172.21.15.133:0/4061097112 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f547c005040 con 0x7f548c0fcef0 2024-02-20T07:55:03.989 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:03.983+0000 7f5491a80700 1 --2- 172.21.15.133:0/4061097112 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f548c0fcef0 0x7f548c10d960 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f548800c8b0 tx=0x7f5488004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:55:03.989 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:03.983+0000 7f54827fc700 1 -- 172.21.15.133:0/4061097112 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f54880106f0 con 0x7f548c0fcef0 2024-02-20T07:55:03.989 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:03.983+0000 7f5493ce4700 1 -- 172.21.15.133:0/4061097112 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f548c10f020 con 0x7f548c0fcef0 2024-02-20T07:55:03.989 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:03.983+0000 7f54827fc700 1 -- 172.21.15.133:0/4061097112 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5488010d20 con 0x7f548c0fcef0 2024-02-20T07:55:03.989 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:03.983+0000 7f54827fc700 1 -- 172.21.15.133:0/4061097112 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5488012e90 con 0x7f548c0fcef0 2024-02-20T07:55:03.989 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:03.983+0000 7f5493ce4700 1 -- 172.21.15.133:0/4061097112 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f548c10f560 con 0x7f548c0fcef0 2024-02-20T07:55:03.989 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:03.984+0000 7f5493ce4700 1 -- 172.21.15.133:0/4061097112 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f548c04bed0 con 0x7f548c0fcef0 2024-02-20T07:55:03.990 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:03.984+0000 7f54827fc700 1 -- 172.21.15.133:0/4061097112 <== mon.0 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7f5488010850 con 0x7f548c0fcef0 2024-02-20T07:55:03.990 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:03.985+0000 7f54827fc700 1 --2- 172.21.15.133:0/4061097112 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7f54780692a0 0x7f547806b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:55:03.990 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:03.985+0000 7f54827fc700 1 -- 172.21.15.133:0/4061097112 <== mon.0 v2:172.21.15.133:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f5488086df0 con 0x7f548c0fcef0 2024-02-20T07:55:03.990 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:03.985+0000 7f549127f700 1 --2- 172.21.15.133:0/4061097112 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7f54780692a0 0x7f547806b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:55:03.990 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:03.985+0000 7f549127f700 1 --2- 172.21.15.133:0/4061097112 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7f54780692a0 0x7f547806b750 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f547c00b3b0 tx=0x7f547c009e30 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:55:03.991 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:03.989+0000 7f54827fc700 1 -- 172.21.15.133:0/4061097112 <== mon.0 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f5488056e70 con 0x7f548c0fcef0 2024-02-20T07:55:04.178 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:04.176+0000 7f5493ce4700 1 -- 172.21.15.133:0/4061097112 --> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] -- mgr_command(tid 0: {"prefix": "orch device ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f548c10fa10 con 0x7f54780692a0 2024-02-20T07:55:04.180 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:04.178+0000 7f54827fc700 1 -- 172.21.15.133:0/4061097112 <== mgr.14208 v2:172.21.15.133:6800/2703586938 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+399 (secure 0 0 0) 0x7f548c10fa10 con 0x7f54780692a0 2024-02-20T07:55:04.180 INFO:teuthology.orchestra.run.smithi133.stdout:HOST PATH TYPE DEVICE ID SIZE AVAILABLE REFRESHED REJECT REASONS 2024-02-20T07:55:04.180 INFO:teuthology.orchestra.run.smithi133.stdout:smithi133 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_PHFT620401AQ400BGN 372G No 2m ago Has a FileSystem, LVM detected 2024-02-20T07:55:04.180 INFO:teuthology.orchestra.run.smithi133.stdout:smithi183 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_PHFT620500QJ400BGN 372G No 73s ago Has a FileSystem, LVM detected 2024-02-20T07:55:04.182 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:04.180+0000 7f5493ce4700 1 -- 172.21.15.133:0/4061097112 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7f54780692a0 msgr2=0x7f547806b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:55:04.182 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:04.180+0000 7f5493ce4700 1 --2- 172.21.15.133:0/4061097112 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7f54780692a0 0x7f547806b750 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f547c00b3b0 tx=0x7f547c009e30 comp rx=0 tx=0).stop 2024-02-20T07:55:04.182 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:04.180+0000 7f5493ce4700 1 -- 172.21.15.133:0/4061097112 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f548c0fcef0 msgr2=0x7f548c10d960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:55:04.182 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:04.180+0000 7f5493ce4700 1 --2- 172.21.15.133:0/4061097112 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f548c0fcef0 0x7f548c10d960 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f548800c8b0 tx=0x7f5488004470 comp rx=0 tx=0).stop 2024-02-20T07:55:04.183 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:04.180+0000 7f5493ce4700 1 -- 172.21.15.133:0/4061097112 shutdown_connections 2024-02-20T07:55:04.183 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:04.180+0000 7f5493ce4700 1 --2- 172.21.15.133:0/4061097112 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f548c0fcef0 0x7f548c10d960 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:55:04.183 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:04.180+0000 7f5493ce4700 1 --2- 172.21.15.133:0/4061097112 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7f54780692a0 0x7f547806b750 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:55:04.183 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:04.180+0000 7f5493ce4700 1 --2- 172.21.15.133:0/4061097112 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f548c0fe0e0 0x7f548c10dea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:55:04.183 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:04.180+0000 7f5493ce4700 1 -- 172.21.15.133:0/4061097112 >> 172.21.15.133:0/4061097112 conn(0x7f548c0f8620 msgr2=0x7f548c1013a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:55:04.183 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:04.180+0000 7f5493ce4700 1 -- 172.21.15.133:0/4061097112 shutdown_connections 2024-02-20T07:55:04.183 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:04.180+0000 7f5493ce4700 1 -- 172.21.15.133:0/4061097112 wait complete. 2024-02-20T07:55:04.837 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:55:04 smithi133 ceph-mon[29117]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-20T07:55:04.863 INFO:teuthology.run_tasks:Running task full_sequential_finally... 2024-02-20T07:55:04.874 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-02-20T07:55:04.953 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi133.front.sepia.ceph.com 2024-02-20T07:55:04.953 DEBUG:teuthology.orchestra.run.smithi133:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4537327c-cfc4-11ee-95bb-87774f69a715 -- bash -c 'ceph fs volume create cephfs --placement=4' 2024-02-20T07:55:05.047 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:55:04 smithi183 ceph-mon[35832]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-20T07:55:05.212 INFO:teuthology.orchestra.run.smithi133.stderr:Inferring config /var/lib/ceph/4537327c-cfc4-11ee-95bb-87774f69a715/mon.smithi133/config 2024-02-20T07:55:05.991 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:55:05 smithi133 ceph-mon[29117]: from='client.14458 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-02-20T07:55:06.047 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:55:05 smithi183 ceph-mon[35832]: from='client.14458 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-02-20T07:55:06.861 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:06.859+0000 7f3a69758700 1 -- 172.21.15.133:0/1998872980 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f3a640fe2b0 msgr2=0x7f3a640fe700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:55:06.861 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:06.859+0000 7f3a69758700 1 --2- 172.21.15.133:0/1998872980 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f3a640fe2b0 0x7f3a640fe700 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f3a58004940 tx=0x7f3a58009e30 comp rx=0 tx=0).stop 2024-02-20T07:55:06.861 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:06.859+0000 7f3a69758700 1 -- 172.21.15.133:0/1998872980 shutdown_connections 2024-02-20T07:55:06.861 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:06.859+0000 7f3a69758700 1 --2- 172.21.15.133:0/1998872980 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f3a640fe2b0 0x7f3a640fe700 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:55:06.862 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:06.859+0000 7f3a69758700 1 --2- 172.21.15.133:0/1998872980 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f3a640fb220 0x7f3a640fb630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:55:06.862 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:06.859+0000 7f3a69758700 1 -- 172.21.15.133:0/1998872980 >> 172.21.15.133:0/1998872980 conn(0x7f3a640f6ca0 msgr2=0x7f3a640f90d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:55:06.862 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:06.859+0000 7f3a69758700 1 -- 172.21.15.133:0/1998872980 shutdown_connections 2024-02-20T07:55:06.862 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:06.859+0000 7f3a69758700 1 -- 172.21.15.133:0/1998872980 wait complete. 2024-02-20T07:55:06.862 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:55:06 smithi133 ceph-mon[29117]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-20T07:55:06.864 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:06.860+0000 7f3a69758700 1 Processor -- start 2024-02-20T07:55:06.864 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:06.860+0000 7f3a69758700 1 -- start start 2024-02-20T07:55:06.864 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:06.860+0000 7f3a69758700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f3a640fb220 0x7f3a6410da20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:55:06.864 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:06.860+0000 7f3a69758700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f3a640fe2b0 0x7f3a6410df60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:55:06.864 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:06.860+0000 7f3a69758700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a6410e560 con 0x7f3a640fb220 2024-02-20T07:55:06.864 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:06.860+0000 7f3a69758700 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a6410ee00 con 0x7f3a640fe2b0 2024-02-20T07:55:06.864 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:06.860+0000 7f3a62ffd700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f3a640fb220 0x7f3a6410da20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:55:06.864 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:06.860+0000 7f3a62ffd700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f3a640fb220 0x7f3a6410da20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.133:34022/0 (socket says 172.21.15.133:34022) 2024-02-20T07:55:06.864 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:06.860+0000 7f3a62ffd700 1 -- 172.21.15.133:0/1076687526 learned_addr learned my addr 172.21.15.133:0/1076687526 (peer_addr_for_me v2:172.21.15.133:0/0) 2024-02-20T07:55:06.864 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:06.860+0000 7f3a627fc700 1 --2- 172.21.15.133:0/1076687526 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f3a640fe2b0 0x7f3a6410df60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:55:06.864 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:06.860+0000 7f3a62ffd700 1 -- 172.21.15.133:0/1076687526 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f3a640fe2b0 msgr2=0x7f3a6410df60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:55:06.864 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:06.860+0000 7f3a62ffd700 1 --2- 172.21.15.133:0/1076687526 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f3a640fe2b0 0x7f3a6410df60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:55:06.865 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:06.860+0000 7f3a62ffd700 1 -- 172.21.15.133:0/1076687526 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a58005040 con 0x7f3a640fb220 2024-02-20T07:55:06.865 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:06.861+0000 7f3a62ffd700 1 --2- 172.21.15.133:0/1076687526 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f3a640fb220 0x7f3a6410da20 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f3a54002a80 tx=0x7f3a54004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:55:06.865 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:06.861+0000 7f3a4b7fe700 1 -- 172.21.15.133:0/1076687526 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3a54007cc0 con 0x7f3a640fb220 2024-02-20T07:55:06.865 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:06.861+0000 7f3a69758700 1 -- 172.21.15.133:0/1076687526 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3a6410f0e0 con 0x7f3a640fb220 2024-02-20T07:55:06.865 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:06.861+0000 7f3a4b7fe700 1 -- 172.21.15.133:0/1076687526 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3a5400f040 con 0x7f3a640fb220 2024-02-20T07:55:06.865 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:06.861+0000 7f3a4b7fe700 1 -- 172.21.15.133:0/1076687526 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3a5400da30 con 0x7f3a640fb220 2024-02-20T07:55:06.865 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:06.861+0000 7f3a69758700 1 -- 172.21.15.133:0/1076687526 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3a6410f620 con 0x7f3a640fb220 2024-02-20T07:55:06.865 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:06.862+0000 7f3a69758700 1 -- 172.21.15.133:0/1076687526 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3a6404bed0 con 0x7f3a640fb220 2024-02-20T07:55:06.866 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:06.863+0000 7f3a4b7fe700 1 -- 172.21.15.133:0/1076687526 <== mon.0 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7f3a5400db90 con 0x7f3a640fb220 2024-02-20T07:55:06.866 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:06.863+0000 7f3a4b7fe700 1 --2- 172.21.15.133:0/1076687526 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7f3a4c0692a0 0x7f3a4c06b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:55:06.866 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:06.864+0000 7f3a4b7fe700 1 -- 172.21.15.133:0/1076687526 <== mon.0 v2:172.21.15.133:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f3a54086b30 con 0x7f3a640fb220 2024-02-20T07:55:06.866 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:06.864+0000 7f3a627fc700 1 --2- 172.21.15.133:0/1076687526 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7f3a4c0692a0 0x7f3a4c06b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:55:06.866 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:06.864+0000 7f3a627fc700 1 --2- 172.21.15.133:0/1076687526 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7f3a4c0692a0 0x7f3a4c06b750 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f3a58004560 tx=0x7f3a58009e30 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:55:06.869 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:06.868+0000 7f3a4b7fe700 1 -- 172.21.15.133:0/1076687526 <== mon.0 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f3a54056c30 con 0x7f3a640fb220 2024-02-20T07:55:07.047 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:55:06 smithi183 ceph-mon[35832]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-20T07:55:07.087 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:07.084+0000 7f3a69758700 1 -- 172.21.15.133:0/1076687526 --> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] -- mgr_command(tid 0: {"prefix": "fs volume create", "name": "cephfs", "placement": "4", "target": ["mon-mgr", ""]}) v1 -- 0x7f3a6410fa90 con 0x7f3a4c0692a0 2024-02-20T07:55:07.991 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:55:07 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "osd pool create", "pool": "cephfs.cephfs.meta"}]: dispatch 2024-02-20T07:55:08.047 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:55:07 smithi183 ceph-mon[35832]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "osd pool create", "pool": "cephfs.cephfs.meta"}]: dispatch 2024-02-20T07:55:08.704 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:08.701+0000 7f3a4b7fe700 1 -- 172.21.15.133:0/1076687526 <== mgr.14208 v2:172.21.15.133:6800/2703586938 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f3a6410fa90 con 0x7f3a4c0692a0 2024-02-20T07:55:08.706 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:08.704+0000 7f3a69758700 1 -- 172.21.15.133:0/1076687526 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7f3a4c0692a0 msgr2=0x7f3a4c06b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:55:08.706 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:08.704+0000 7f3a69758700 1 --2- 172.21.15.133:0/1076687526 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7f3a4c0692a0 0x7f3a4c06b750 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f3a58004560 tx=0x7f3a58009e30 comp rx=0 tx=0).stop 2024-02-20T07:55:08.707 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:08.704+0000 7f3a69758700 1 -- 172.21.15.133:0/1076687526 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f3a640fb220 msgr2=0x7f3a6410da20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:55:08.707 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:08.704+0000 7f3a69758700 1 --2- 172.21.15.133:0/1076687526 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f3a640fb220 0x7f3a6410da20 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f3a54002a80 tx=0x7f3a54004600 comp rx=0 tx=0).stop 2024-02-20T07:55:08.707 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:08.704+0000 7f3a69758700 1 -- 172.21.15.133:0/1076687526 shutdown_connections 2024-02-20T07:55:08.707 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:08.704+0000 7f3a69758700 1 --2- 172.21.15.133:0/1076687526 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f3a640fb220 0x7f3a6410da20 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:55:08.707 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:08.704+0000 7f3a69758700 1 --2- 172.21.15.133:0/1076687526 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7f3a4c0692a0 0x7f3a4c06b750 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:55:08.707 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:08.704+0000 7f3a69758700 1 --2- 172.21.15.133:0/1076687526 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f3a640fe2b0 0x7f3a6410df60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:55:08.708 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:08.704+0000 7f3a69758700 1 -- 172.21.15.133:0/1076687526 >> 172.21.15.133:0/1076687526 conn(0x7f3a640f6ca0 msgr2=0x7f3a641014e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:55:08.708 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:08.704+0000 7f3a69758700 1 -- 172.21.15.133:0/1076687526 shutdown_connections 2024-02-20T07:55:08.708 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:08.704+0000 7f3a69758700 1 -- 172.21.15.133:0/1076687526 wait complete. 2024-02-20T07:55:08.996 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:55:08 smithi133 ceph-mon[29117]: from='client.14462 -' entity='client.admin' cmd=[{"prefix": "fs volume create", "name": "cephfs", "placement": "4", "target": ["mon-mgr", ""]}]: dispatch 2024-02-20T07:55:08.996 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:55:08 smithi133 ceph-mon[29117]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-20T07:55:08.996 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:55:08 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd='[{"prefix": "osd pool create", "pool": "cephfs.cephfs.meta"}]': finished 2024-02-20T07:55:08.996 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:55:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T07:55:08.676+0000 7fa82d9f3700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-02-20T07:55:08.997 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:55:08 smithi133 ceph-mon[29117]: osdmap e41: 6 total, 6 up, 6 in 2024-02-20T07:55:08.997 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:55:08 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"bulk": true, "prefix": "osd pool create", "pool": "cephfs.cephfs.data"}]: dispatch 2024-02-20T07:55:08.997 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:55:08 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-20T07:55:09.047 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:55:08 smithi183 ceph-mon[35832]: from='client.14462 -' entity='client.admin' cmd=[{"prefix": "fs volume create", "name": "cephfs", "placement": "4", "target": ["mon-mgr", ""]}]: dispatch 2024-02-20T07:55:09.048 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:55:08 smithi183 ceph-mon[35832]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-20T07:55:09.048 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:55:08 smithi183 ceph-mon[35832]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd='[{"prefix": "osd pool create", "pool": "cephfs.cephfs.meta"}]': finished 2024-02-20T07:55:09.048 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:55:08 smithi183 ceph-mon[35832]: osdmap e41: 6 total, 6 up, 6 in 2024-02-20T07:55:09.048 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:55:08 smithi183 ceph-mon[35832]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"bulk": true, "prefix": "osd pool create", "pool": "cephfs.cephfs.data"}]: dispatch 2024-02-20T07:55:09.048 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:55:08 smithi183 ceph-mon[35832]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-20T07:55:09.353 DEBUG:teuthology.orchestra.run.smithi133:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4537327c-cfc4-11ee-95bb-87774f69a715 -- bash -c 'ceph fs dump' 2024-02-20T07:55:09.992 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:55:09 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd='[{"bulk": true, "prefix": "osd pool create", "pool": "cephfs.cephfs.data"}]': finished 2024-02-20T07:55:09.992 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:55:09 smithi133 ceph-mon[29117]: osdmap e42: 6 total, 6 up, 6 in 2024-02-20T07:55:09.992 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:55:09 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs.cephfs.meta", "data": "cephfs.cephfs.data"}]: dispatch 2024-02-20T07:55:09.992 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:55:09 smithi133 ceph-mon[29117]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-02-20T07:55:09.992 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:55:09 smithi133 ceph-mon[29117]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-02-20T07:55:09.992 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:55:09 smithi133 ceph-mon[29117]: osdmap e43: 6 total, 6 up, 6 in 2024-02-20T07:55:09.993 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:55:09 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs.cephfs.meta", "data": "cephfs.cephfs.data"}]': finished 2024-02-20T07:55:09.993 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:55:09 smithi133 ceph-mon[29117]: fsmap cephfs:0 2024-02-20T07:55:09.993 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:55:09 smithi133 ceph-mon[29117]: Saving service mds.cephfs spec with placement count:4 2024-02-20T07:55:09.993 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:55:09 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:55:09.993 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:55:09 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-20T07:55:09.993 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:55:09 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-20T07:55:09.993 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:55:09 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-20T07:55:09.993 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:55:09 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:55:09.993 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:55:09 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi133.itlmbm", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-02-20T07:55:09.993 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:55:09 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi133.itlmbm", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-02-20T07:55:09.993 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:55:09 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-20T07:55:09.993 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:55:09 smithi133 ceph-mon[29117]: Deploying daemon mds.cephfs.smithi133.itlmbm on smithi133 2024-02-20T07:55:10.047 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:55:09 smithi183 ceph-mon[35832]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd='[{"bulk": true, "prefix": "osd pool create", "pool": "cephfs.cephfs.data"}]': finished 2024-02-20T07:55:10.047 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:55:09 smithi183 ceph-mon[35832]: osdmap e42: 6 total, 6 up, 6 in 2024-02-20T07:55:10.048 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:55:09 smithi183 ceph-mon[35832]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs.cephfs.meta", "data": "cephfs.cephfs.data"}]: dispatch 2024-02-20T07:55:10.048 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:55:09 smithi183 ceph-mon[35832]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-02-20T07:55:10.048 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:55:09 smithi183 ceph-mon[35832]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-02-20T07:55:10.048 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:55:09 smithi183 ceph-mon[35832]: osdmap e43: 6 total, 6 up, 6 in 2024-02-20T07:55:10.048 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:55:09 smithi183 ceph-mon[35832]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs.cephfs.meta", "data": "cephfs.cephfs.data"}]': finished 2024-02-20T07:55:10.048 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:55:09 smithi183 ceph-mon[35832]: fsmap cephfs:0 2024-02-20T07:55:10.048 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:55:09 smithi183 ceph-mon[35832]: Saving service mds.cephfs spec with placement count:4 2024-02-20T07:55:10.048 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:55:09 smithi183 ceph-mon[35832]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:55:10.048 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:55:09 smithi183 ceph-mon[35832]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-20T07:55:10.048 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:55:09 smithi183 ceph-mon[35832]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-20T07:55:10.048 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:55:09 smithi183 ceph-mon[35832]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-20T07:55:10.048 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:55:09 smithi183 ceph-mon[35832]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:55:10.048 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:55:09 smithi183 ceph-mon[35832]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi133.itlmbm", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-02-20T07:55:10.048 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:55:09 smithi183 ceph-mon[35832]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi133.itlmbm", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-02-20T07:55:10.048 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:55:09 smithi183 ceph-mon[35832]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-20T07:55:10.049 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:55:09 smithi183 ceph-mon[35832]: Deploying daemon mds.cephfs.smithi133.itlmbm on smithi133 2024-02-20T07:55:10.125 INFO:teuthology.orchestra.run.smithi133.stderr:Inferring config /var/lib/ceph/4537327c-cfc4-11ee-95bb-87774f69a715/mon.smithi133/config 2024-02-20T07:55:10.913 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:55:10 smithi133 ceph-mon[29117]: pgmap v174: 65 pgs: 16 creating+peering, 48 unknown, 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-20T07:55:10.913 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:55:10 smithi133 ceph-mon[29117]: osdmap e44: 6 total, 6 up, 6 in 2024-02-20T07:55:11.047 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:55:10 smithi183 ceph-mon[35832]: pgmap v174: 65 pgs: 16 creating+peering, 48 unknown, 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-20T07:55:11.047 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:55:10 smithi183 ceph-mon[35832]: osdmap e44: 6 total, 6 up, 6 in 2024-02-20T07:55:11.741 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:55:11 smithi133 ceph-mon[29117]: osdmap e45: 6 total, 6 up, 6 in 2024-02-20T07:55:12.047 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:55:11 smithi183 ceph-mon[35832]: osdmap e45: 6 total, 6 up, 6 in 2024-02-20T07:55:12.259 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:12.255+0000 7f26e2fca700 1 -- 172.21.15.133:0/2401857785 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f26dc0fbe40 msgr2=0x7f26dc0fc250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:55:12.259 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:12.255+0000 7f26e2fca700 1 --2- 172.21.15.133:0/2401857785 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f26dc0fbe40 0x7f26dc0fc250 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f26cc004940 tx=0x7f26cc009e30 comp rx=0 tx=0).stop 2024-02-20T07:55:12.259 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:12.255+0000 7f26e2fca700 1 -- 172.21.15.133:0/2401857785 shutdown_connections 2024-02-20T07:55:12.259 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:12.255+0000 7f26e2fca700 1 --2- 172.21.15.133:0/2401857785 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f26dc0fcfb0 0x7f26dc0fd420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:55:12.259 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:12.255+0000 7f26e2fca700 1 --2- 172.21.15.133:0/2401857785 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f26dc0fbe40 0x7f26dc0fc250 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:55:12.259 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:12.255+0000 7f26e2fca700 1 -- 172.21.15.133:0/2401857785 >> 172.21.15.133:0/2401857785 conn(0x7f26dc0f7520 msgr2=0x7f26dc0f9930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:55:12.259 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:12.255+0000 7f26e2fca700 1 -- 172.21.15.133:0/2401857785 shutdown_connections 2024-02-20T07:55:12.259 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:12.256+0000 7f26e2fca700 1 -- 172.21.15.133:0/2401857785 wait complete. 2024-02-20T07:55:12.259 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:12.256+0000 7f26e2fca700 1 Processor -- start 2024-02-20T07:55:12.260 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:12.256+0000 7f26e2fca700 1 -- start start 2024-02-20T07:55:12.260 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:12.256+0000 7f26e2fca700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f26dc0fbe40 0x7f26dc10c830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:55:12.260 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:12.256+0000 7f26e2fca700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f26dc0fcfb0 0x7f26dc10cd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:55:12.260 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:12.256+0000 7f26e2fca700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26dc10d370 con 0x7f26dc0fbe40 2024-02-20T07:55:12.260 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:12.256+0000 7f26e2fca700 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26dc10dc10 con 0x7f26dc0fcfb0 2024-02-20T07:55:12.260 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:12.256+0000 7f26e0d66700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f26dc0fbe40 0x7f26dc10c830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:55:12.260 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:12.256+0000 7f26e0d66700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f26dc0fbe40 0x7f26dc10c830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.133:34038/0 (socket says 172.21.15.133:34038) 2024-02-20T07:55:12.260 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:12.256+0000 7f26e0d66700 1 -- 172.21.15.133:0/4103674360 learned_addr learned my addr 172.21.15.133:0/4103674360 (peer_addr_for_me v2:172.21.15.133:0/0) 2024-02-20T07:55:12.260 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:12.257+0000 7f26dbfff700 1 --2- 172.21.15.133:0/4103674360 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f26dc0fcfb0 0x7f26dc10cd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:55:12.260 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:12.257+0000 7f26e0d66700 1 -- 172.21.15.133:0/4103674360 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f26dc0fcfb0 msgr2=0x7f26dc10cd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:55:12.260 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:12.257+0000 7f26e0d66700 1 --2- 172.21.15.133:0/4103674360 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f26dc0fcfb0 0x7f26dc10cd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:55:12.260 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:12.257+0000 7f26e0d66700 1 -- 172.21.15.133:0/4103674360 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26cc005040 con 0x7f26dc0fbe40 2024-02-20T07:55:12.260 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:12.257+0000 7f26e0d66700 1 --2- 172.21.15.133:0/4103674360 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f26dc0fbe40 0x7f26dc10c830 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f26cc0093b0 tx=0x7f26cc009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:55:12.261 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:12.257+0000 7f26d97fa700 1 -- 172.21.15.133:0/4103674360 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f26cc00e040 con 0x7f26dc0fbe40 2024-02-20T07:55:12.261 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:12.257+0000 7f26e2fca700 1 -- 172.21.15.133:0/4103674360 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f26dc10de90 con 0x7f26dc0fbe40 2024-02-20T07:55:12.261 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:12.257+0000 7f26d97fa700 1 -- 172.21.15.133:0/4103674360 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f26cc00bcc0 con 0x7f26dc0fbe40 2024-02-20T07:55:12.261 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:12.257+0000 7f26d97fa700 1 -- 172.21.15.133:0/4103674360 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f26cc01c630 con 0x7f26dc0fbe40 2024-02-20T07:55:12.261 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:12.257+0000 7f26e2fca700 1 -- 172.21.15.133:0/4103674360 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f26dc10e3d0 con 0x7f26dc0fbe40 2024-02-20T07:55:12.262 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:12.259+0000 7f26e2fca700 1 -- 172.21.15.133:0/4103674360 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f26dc04bed0 con 0x7f26dc0fbe40 2024-02-20T07:55:12.262 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:12.259+0000 7f26d97fa700 1 -- 172.21.15.133:0/4103674360 <== mon.0 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7f26cc013980 con 0x7f26dc0fbe40 2024-02-20T07:55:12.262 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:12.259+0000 7f26d97fa700 1 --2- 172.21.15.133:0/4103674360 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7f26c80692a0 0x7f26c806b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:55:12.262 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:12.259+0000 7f26d97fa700 1 -- 172.21.15.133:0/4103674360 <== mon.0 v2:172.21.15.133:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5464+0+0 (secure 0 0 0) 0x7f26cc088450 con 0x7f26dc0fbe40 2024-02-20T07:55:12.262 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:12.259+0000 7f26dbfff700 1 --2- 172.21.15.133:0/4103674360 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7f26c80692a0 0x7f26c806b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:55:12.262 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:12.260+0000 7f26dbfff700 1 --2- 172.21.15.133:0/4103674360 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7f26c80692a0 0x7f26c806b750 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f26c4004f40 tx=0x7f26c400b040 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:55:12.268 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:12.266+0000 7f26d97fa700 1 -- 172.21.15.133:0/4103674360 <== mon.0 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f26cc0582b0 con 0x7f26dc0fbe40 2024-02-20T07:55:12.508 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:12.507+0000 7f26e2fca700 1 -- 172.21.15.133:0/4103674360 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f26dc112e50 con 0x7f26dc0fbe40 2024-02-20T07:55:12.509 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:12.507+0000 7f26d97fa700 1 -- 172.21.15.133:0/4103674360 <== mon.0 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 2 v2) v1 ==== 75+0+1076 (secure 0 0 0) 0x7f26cc057e40 con 0x7f26dc0fbe40 2024-02-20T07:55:12.509 INFO:teuthology.orchestra.run.smithi133.stdout:e2 2024-02-20T07:55:12.509 INFO:teuthology.orchestra.run.smithi133.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-02-20T07:55:12.509 INFO:teuthology.orchestra.run.smithi133.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-20T07:55:12.509 INFO:teuthology.orchestra.run.smithi133.stdout:legacy client fscid: 1 2024-02-20T07:55:12.509 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-02-20T07:55:12.509 INFO:teuthology.orchestra.run.smithi133.stdout:Filesystem 'cephfs' (1) 2024-02-20T07:55:12.509 INFO:teuthology.orchestra.run.smithi133.stdout:fs_name cephfs 2024-02-20T07:55:12.509 INFO:teuthology.orchestra.run.smithi133.stdout:epoch 2 2024-02-20T07:55:12.510 INFO:teuthology.orchestra.run.smithi133.stdout:flags 12 joinable allow_snaps allow_multimds_snaps 2024-02-20T07:55:12.510 INFO:teuthology.orchestra.run.smithi133.stdout:created 2024-02-20T07:55:08.678040+0000 2024-02-20T07:55:12.510 INFO:teuthology.orchestra.run.smithi133.stdout:modified 2024-02-20T07:55:08.678102+0000 2024-02-20T07:55:12.510 INFO:teuthology.orchestra.run.smithi133.stdout:tableserver 0 2024-02-20T07:55:12.510 INFO:teuthology.orchestra.run.smithi133.stdout:root 0 2024-02-20T07:55:12.510 INFO:teuthology.orchestra.run.smithi133.stdout:session_timeout 60 2024-02-20T07:55:12.510 INFO:teuthology.orchestra.run.smithi133.stdout:session_autoclose 300 2024-02-20T07:55:12.510 INFO:teuthology.orchestra.run.smithi133.stdout:max_file_size 1099511627776 2024-02-20T07:55:12.510 INFO:teuthology.orchestra.run.smithi133.stdout:required_client_features {} 2024-02-20T07:55:12.510 INFO:teuthology.orchestra.run.smithi133.stdout:last_failure 0 2024-02-20T07:55:12.510 INFO:teuthology.orchestra.run.smithi133.stdout:last_failure_osd_epoch 0 2024-02-20T07:55:12.510 INFO:teuthology.orchestra.run.smithi133.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-20T07:55:12.510 INFO:teuthology.orchestra.run.smithi133.stdout:max_mds 1 2024-02-20T07:55:12.511 INFO:teuthology.orchestra.run.smithi133.stdout:in 2024-02-20T07:55:12.511 INFO:teuthology.orchestra.run.smithi133.stdout:up {} 2024-02-20T07:55:12.511 INFO:teuthology.orchestra.run.smithi133.stdout:failed 2024-02-20T07:55:12.511 INFO:teuthology.orchestra.run.smithi133.stdout:damaged 2024-02-20T07:55:12.511 INFO:teuthology.orchestra.run.smithi133.stdout:stopped 2024-02-20T07:55:12.511 INFO:teuthology.orchestra.run.smithi133.stdout:data_pools [3] 2024-02-20T07:55:12.511 INFO:teuthology.orchestra.run.smithi133.stdout:metadata_pool 2 2024-02-20T07:55:12.511 INFO:teuthology.orchestra.run.smithi133.stdout:inline_data disabled 2024-02-20T07:55:12.511 INFO:teuthology.orchestra.run.smithi133.stdout:balancer 2024-02-20T07:55:12.511 INFO:teuthology.orchestra.run.smithi133.stdout:standby_count_wanted 0 2024-02-20T07:55:12.511 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-02-20T07:55:12.511 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-02-20T07:55:12.512 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:12.509+0000 7f26e2fca700 1 -- 172.21.15.133:0/4103674360 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7f26c80692a0 msgr2=0x7f26c806b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:55:12.512 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:12.509+0000 7f26e2fca700 1 --2- 172.21.15.133:0/4103674360 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7f26c80692a0 0x7f26c806b750 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f26c4004f40 tx=0x7f26c400b040 comp rx=0 tx=0).stop 2024-02-20T07:55:12.512 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:12.509+0000 7f26e2fca700 1 -- 172.21.15.133:0/4103674360 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f26dc0fbe40 msgr2=0x7f26dc10c830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:55:12.512 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:12.509+0000 7f26e2fca700 1 --2- 172.21.15.133:0/4103674360 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f26dc0fbe40 0x7f26dc10c830 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f26cc0093b0 tx=0x7f26cc009e30 comp rx=0 tx=0).stop 2024-02-20T07:55:12.513 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:12.509+0000 7f26e2fca700 1 -- 172.21.15.133:0/4103674360 shutdown_connections 2024-02-20T07:55:12.513 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:12.509+0000 7f26e2fca700 1 --2- 172.21.15.133:0/4103674360 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f26dc0fbe40 0x7f26dc10c830 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:55:12.513 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:12.509+0000 7f26e2fca700 1 --2- 172.21.15.133:0/4103674360 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7f26c80692a0 0x7f26c806b750 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:55:12.513 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:12.509+0000 7f26e2fca700 1 --2- 172.21.15.133:0/4103674360 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f26dc0fcfb0 0x7f26dc10cd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:55:12.513 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:12.509+0000 7f26e2fca700 1 -- 172.21.15.133:0/4103674360 >> 172.21.15.133:0/4103674360 conn(0x7f26dc0f7520 msgr2=0x7f26dc100270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:55:12.513 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:12.509+0000 7f26e2fca700 1 -- 172.21.15.133:0/4103674360 shutdown_connections 2024-02-20T07:55:12.513 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:12.509+0000 7f26e2fca700 1 -- 172.21.15.133:0/4103674360 wait complete. 2024-02-20T07:55:12.513 INFO:teuthology.orchestra.run.smithi133.stderr:dumped fsmap epoch 2 2024-02-20T07:55:12.863 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:55:12 smithi133 ceph-mon[29117]: pgmap v177: 65 pgs: 16 creating+peering, 11 unknown, 38 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-20T07:55:12.863 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:55:12 smithi133 ceph-mon[29117]: from='client.? 172.21.15.133:0/4103674360' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-02-20T07:55:13.047 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:55:12 smithi183 ceph-mon[35832]: pgmap v177: 65 pgs: 16 creating+peering, 11 unknown, 38 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-20T07:55:13.047 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:55:12 smithi183 ceph-mon[35832]: from='client.? 172.21.15.133:0/4103674360' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-02-20T07:55:13.441 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-02-20T07:55:13.451 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi133.front.sepia.ceph.com 2024-02-20T07:55:13.452 DEBUG:teuthology.orchestra.run.smithi133:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4537327c-cfc4-11ee-95bb-87774f69a715 -- bash -c 'ceph fs set cephfs max_mds 2' 2024-02-20T07:55:13.710 INFO:teuthology.orchestra.run.smithi133.stderr:Inferring config /var/lib/ceph/4537327c-cfc4-11ee-95bb-87774f69a715/mon.smithi133/config 2024-02-20T07:55:14.047 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:55:14 smithi183 ceph-mon[35832]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:55:14.048 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:55:14 smithi183 ceph-mon[35832]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:55:14.048 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:55:14 smithi183 ceph-mon[35832]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:55:14.048 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:55:14 smithi183 ceph-mon[35832]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi183.eipfta", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-02-20T07:55:14.048 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:55:14 smithi183 ceph-mon[35832]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi183.eipfta", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-02-20T07:55:14.048 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:55:14 smithi183 ceph-mon[35832]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-20T07:55:14.048 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:55:14 smithi183 ceph-mon[35832]: Deploying daemon mds.cephfs.smithi183.eipfta on smithi183 2024-02-20T07:55:14.048 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:55:14 smithi183 ceph-mon[35832]: mds.? [v2:172.21.15.133:6826/461346469,v1:172.21.15.133:6827/461346469] up:boot 2024-02-20T07:55:14.048 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:55:14 smithi183 ceph-mon[35832]: daemon mds.cephfs.smithi133.itlmbm assigned to filesystem cephfs as rank 0 (now has 1 ranks) 2024-02-20T07:55:14.048 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:55:14 smithi183 ceph-mon[35832]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-02-20T07:55:14.048 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:55:14 smithi183 ceph-mon[35832]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-02-20T07:55:14.048 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:55:14 smithi183 ceph-mon[35832]: Cluster is now healthy 2024-02-20T07:55:14.048 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:55:14 smithi183 ceph-mon[35832]: fsmap cephfs:0 1 up:standby 2024-02-20T07:55:14.049 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:55:14 smithi183 ceph-mon[35832]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi133.itlmbm"}]: dispatch 2024-02-20T07:55:14.049 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:55:14 smithi183 ceph-mon[35832]: fsmap cephfs:1 {0=cephfs.smithi133.itlmbm=up:creating} 2024-02-20T07:55:14.049 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:55:14 smithi183 ceph-mon[35832]: daemon mds.cephfs.smithi133.itlmbm is now active in filesystem cephfs as rank 0 2024-02-20T07:55:14.229 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:55:14 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:55:14.230 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:55:14 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:55:14.230 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:55:14 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:55:14.230 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:55:14 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi183.eipfta", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-02-20T07:55:14.230 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:55:14 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi183.eipfta", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-02-20T07:55:14.230 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:55:14 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-20T07:55:14.230 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:55:14 smithi133 ceph-mon[29117]: Deploying daemon mds.cephfs.smithi183.eipfta on smithi183 2024-02-20T07:55:14.230 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:55:14 smithi133 ceph-mon[29117]: mds.? [v2:172.21.15.133:6826/461346469,v1:172.21.15.133:6827/461346469] up:boot 2024-02-20T07:55:14.230 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:55:14 smithi133 ceph-mon[29117]: daemon mds.cephfs.smithi133.itlmbm assigned to filesystem cephfs as rank 0 (now has 1 ranks) 2024-02-20T07:55:14.231 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:55:14 smithi133 ceph-mon[29117]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-02-20T07:55:14.231 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:55:14 smithi133 ceph-mon[29117]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-02-20T07:55:14.231 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:55:14 smithi133 ceph-mon[29117]: Cluster is now healthy 2024-02-20T07:55:14.231 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:55:14 smithi133 ceph-mon[29117]: fsmap cephfs:0 1 up:standby 2024-02-20T07:55:14.231 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:55:14 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi133.itlmbm"}]: dispatch 2024-02-20T07:55:14.231 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:55:14 smithi133 ceph-mon[29117]: fsmap cephfs:1 {0=cephfs.smithi133.itlmbm=up:creating} 2024-02-20T07:55:14.231 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:55:14 smithi133 ceph-mon[29117]: daemon mds.cephfs.smithi133.itlmbm is now active in filesystem cephfs as rank 0 2024-02-20T07:55:15.172 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:55:15 smithi183 ceph-mon[35832]: pgmap v178: 65 pgs: 65 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-20T07:55:15.172 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:55:15 smithi183 ceph-mon[35832]: mds.? [v2:172.21.15.133:6826/461346469,v1:172.21.15.133:6827/461346469] up:active 2024-02-20T07:55:15.172 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:55:15 smithi183 ceph-mon[35832]: fsmap cephfs:1 {0=cephfs.smithi133.itlmbm=up:active} 2024-02-20T07:55:15.241 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:55:15 smithi133 ceph-mon[29117]: pgmap v178: 65 pgs: 65 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-20T07:55:15.241 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:55:15 smithi133 ceph-mon[29117]: mds.? [v2:172.21.15.133:6826/461346469,v1:172.21.15.133:6827/461346469] up:active 2024-02-20T07:55:15.241 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:55:15 smithi133 ceph-mon[29117]: fsmap cephfs:1 {0=cephfs.smithi133.itlmbm=up:active} 2024-02-20T07:55:15.487 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:15.484+0000 7fbe66307700 1 -- 172.21.15.133:0/4267752189 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fbe601003b0 msgr2=0x7fbe60102790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:55:15.487 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:15.484+0000 7fbe66307700 1 --2- 172.21.15.133:0/4267752189 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fbe601003b0 0x7fbe60102790 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fbe54004940 tx=0x7fbe54009e30 comp rx=0 tx=0).stop 2024-02-20T07:55:15.487 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:15.484+0000 7fbe66307700 1 -- 172.21.15.133:0/4267752189 shutdown_connections 2024-02-20T07:55:15.487 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:15.484+0000 7fbe66307700 1 --2- 172.21.15.133:0/4267752189 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fbe601003b0 0x7fbe60102790 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:55:15.487 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:15.484+0000 7fbe66307700 1 --2- 172.21.15.133:0/4267752189 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fbe600fda90 0x7fbe600ffe70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:55:15.488 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:15.484+0000 7fbe66307700 1 -- 172.21.15.133:0/4267752189 >> 172.21.15.133:0/4267752189 conn(0x7fbe600f7510 msgr2=0x7fbe600f9940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:55:15.488 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:15.484+0000 7fbe66307700 1 -- 172.21.15.133:0/4267752189 shutdown_connections 2024-02-20T07:55:15.488 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:15.484+0000 7fbe66307700 1 -- 172.21.15.133:0/4267752189 wait complete. 2024-02-20T07:55:15.488 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:15.484+0000 7fbe66307700 1 Processor -- start 2024-02-20T07:55:15.488 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:15.485+0000 7fbe66307700 1 -- start start 2024-02-20T07:55:15.488 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:15.485+0000 7fbe66307700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fbe600fda90 0x7fbe600fd520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:55:15.488 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:15.485+0000 7fbe66307700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fbe601003b0 0x7fbe600fb9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:55:15.488 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:15.485+0000 7fbe66307700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe600fbfa0 con 0x7fbe600fda90 2024-02-20T07:55:15.488 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:15.485+0000 7fbe66307700 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe600fc110 con 0x7fbe601003b0 2024-02-20T07:55:15.488 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:15.485+0000 7fbe5ffff700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fbe600fda90 0x7fbe600fd520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:55:15.488 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:15.485+0000 7fbe5ffff700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fbe600fda90 0x7fbe600fd520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.133:34090/0 (socket says 172.21.15.133:34090) 2024-02-20T07:55:15.488 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:15.485+0000 7fbe5ffff700 1 -- 172.21.15.133:0/3711320302 learned_addr learned my addr 172.21.15.133:0/3711320302 (peer_addr_for_me v2:172.21.15.133:0/0) 2024-02-20T07:55:15.488 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:15.485+0000 7fbe5f7fe700 1 --2- 172.21.15.133:0/3711320302 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fbe601003b0 0x7fbe600fb9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:55:15.489 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:15.485+0000 7fbe5ffff700 1 -- 172.21.15.133:0/3711320302 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fbe601003b0 msgr2=0x7fbe600fb9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:55:15.489 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:15.485+0000 7fbe5ffff700 1 --2- 172.21.15.133:0/3711320302 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fbe601003b0 0x7fbe600fb9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:55:15.489 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:15.485+0000 7fbe5ffff700 1 -- 172.21.15.133:0/3711320302 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe54005040 con 0x7fbe600fda90 2024-02-20T07:55:15.489 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:15.485+0000 7fbe5ffff700 1 --2- 172.21.15.133:0/3711320302 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fbe600fda90 0x7fbe600fd520 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fbe50002a50 tx=0x7fbe500046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:55:15.489 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:15.485+0000 7fbe5cff9700 1 -- 172.21.15.133:0/3711320302 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbe50007cc0 con 0x7fbe600fda90 2024-02-20T07:55:15.489 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:15.485+0000 7fbe5cff9700 1 -- 172.21.15.133:0/3711320302 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fbe5000f040 con 0x7fbe600fda90 2024-02-20T07:55:15.489 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:15.485+0000 7fbe66307700 1 -- 172.21.15.133:0/3711320302 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbe600fc3f0 con 0x7fbe600fda90 2024-02-20T07:55:15.489 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:15.486+0000 7fbe5cff9700 1 -- 172.21.15.133:0/3711320302 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbe5000da30 con 0x7fbe600fda90 2024-02-20T07:55:15.489 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:15.486+0000 7fbe66307700 1 -- 172.21.15.133:0/3711320302 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbe60115970 con 0x7fbe600fda90 2024-02-20T07:55:15.490 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:15.487+0000 7fbe66307700 1 -- 172.21.15.133:0/3711320302 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbe6004bed0 con 0x7fbe600fda90 2024-02-20T07:55:15.490 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:15.487+0000 7fbe5cff9700 1 -- 172.21.15.133:0/3711320302 <== mon.0 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7fbe5000db90 con 0x7fbe600fda90 2024-02-20T07:55:15.490 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:15.487+0000 7fbe5cff9700 1 --2- 172.21.15.133:0/3711320302 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7fbe480692a0 0x7fbe4806b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:55:15.490 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:15.487+0000 7fbe5cff9700 1 -- 172.21.15.133:0/3711320302 <== mon.0 v2:172.21.15.133:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5464+0+0 (secure 0 0 0) 0x7fbe50086e60 con 0x7fbe600fda90 2024-02-20T07:55:15.490 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:15.487+0000 7fbe5f7fe700 1 --2- 172.21.15.133:0/3711320302 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7fbe480692a0 0x7fbe4806b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:55:15.490 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:15.488+0000 7fbe5f7fe700 1 --2- 172.21.15.133:0/3711320302 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7fbe480692a0 0x7fbe4806b750 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fbe5400b3b0 tx=0x7fbe54009e30 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:55:15.493 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:15.492+0000 7fbe5cff9700 1 -- 172.21.15.133:0/3711320302 <== mon.0 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7fbe50056c20 con 0x7fbe600fda90 2024-02-20T07:55:15.731 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:15.729+0000 7fbe66307700 1 -- 172.21.15.133:0/3711320302 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "2"} v 0) v1 -- 0x7fbe60115b60 con 0x7fbe600fda90 2024-02-20T07:55:16.050 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:16.049+0000 7fbe5cff9700 1 -- 172.21.15.133:0/3711320302 <== mon.0 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "2"}]=0 v6) v1 ==== 105+0+0 (secure 0 0 0) 0x7fbe60115b60 con 0x7fbe600fda90 2024-02-20T07:55:16.052 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:16.051+0000 7fbe66307700 1 -- 172.21.15.133:0/3711320302 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7fbe480692a0 msgr2=0x7fbe4806b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:55:16.052 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:16.051+0000 7fbe66307700 1 --2- 172.21.15.133:0/3711320302 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7fbe480692a0 0x7fbe4806b750 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fbe5400b3b0 tx=0x7fbe54009e30 comp rx=0 tx=0).stop 2024-02-20T07:55:16.052 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:16.051+0000 7fbe66307700 1 -- 172.21.15.133:0/3711320302 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fbe600fda90 msgr2=0x7fbe600fd520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:55:16.052 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:16.051+0000 7fbe66307700 1 --2- 172.21.15.133:0/3711320302 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fbe600fda90 0x7fbe600fd520 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fbe50002a50 tx=0x7fbe500046b0 comp rx=0 tx=0).stop 2024-02-20T07:55:16.052 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:16.051+0000 7fbe66307700 1 -- 172.21.15.133:0/3711320302 shutdown_connections 2024-02-20T07:55:16.052 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:16.051+0000 7fbe66307700 1 --2- 172.21.15.133:0/3711320302 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fbe600fda90 0x7fbe600fd520 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:55:16.052 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:16.051+0000 7fbe66307700 1 --2- 172.21.15.133:0/3711320302 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7fbe480692a0 0x7fbe4806b750 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:55:16.052 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:16.051+0000 7fbe66307700 1 --2- 172.21.15.133:0/3711320302 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fbe601003b0 0x7fbe600fb9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:55:16.052 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:16.051+0000 7fbe66307700 1 -- 172.21.15.133:0/3711320302 >> 172.21.15.133:0/3711320302 conn(0x7fbe600f7510 msgr2=0x7fbe600f9940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:55:16.053 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:16.051+0000 7fbe66307700 1 -- 172.21.15.133:0/3711320302 shutdown_connections 2024-02-20T07:55:16.053 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:16.052+0000 7fbe66307700 1 -- 172.21.15.133:0/3711320302 wait complete. 2024-02-20T07:55:16.742 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:55:16 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:55:16.742 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:55:16 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:55:16.742 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:55:16 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:55:16.742 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:55:16 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi133.cmofvy", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-02-20T07:55:16.742 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:55:16 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi133.cmofvy", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-02-20T07:55:16.742 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:55:16 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-20T07:55:16.742 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:55:16 smithi133 ceph-mon[29117]: Deploying daemon mds.cephfs.smithi133.cmofvy on smithi133 2024-02-20T07:55:16.742 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:55:16 smithi133 ceph-mon[29117]: pgmap v179: 65 pgs: 65 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 148 B/s wr, 0 op/s 2024-02-20T07:55:16.742 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:55:16 smithi133 ceph-mon[29117]: from='client.? 172.21.15.133:0/3711320302' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "2"}]: dispatch 2024-02-20T07:55:16.742 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:55:16 smithi133 ceph-mon[29117]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-02-20T07:55:16.743 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:55:16 smithi133 ceph-mon[29117]: mds.? [v2:172.21.15.183:6824/3202657453,v1:172.21.15.183:6825/3202657453] up:boot 2024-02-20T07:55:16.743 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:55:16 smithi133 ceph-mon[29117]: from='client.? 172.21.15.133:0/3711320302' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "2"}]': finished 2024-02-20T07:55:16.743 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:55:16 smithi133 ceph-mon[29117]: daemon mds.cephfs.smithi183.eipfta assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-02-20T07:55:16.743 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:55:16 smithi133 ceph-mon[29117]: Health check failed: insufficient standby MDS daemons available (MDS_INSUFFICIENT_STANDBY) 2024-02-20T07:55:16.743 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:55:16 smithi133 ceph-mon[29117]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-02-20T07:55:16.743 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:55:16 smithi133 ceph-mon[29117]: fsmap cephfs:1 {0=cephfs.smithi133.itlmbm=up:active} 1 up:standby 2024-02-20T07:55:16.743 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:55:16 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi183.eipfta"}]: dispatch 2024-02-20T07:55:16.743 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:55:16 smithi133 ceph-mon[29117]: fsmap cephfs:2 {0=cephfs.smithi133.itlmbm=up:active,1=cephfs.smithi183.eipfta=up:creating} 2024-02-20T07:55:16.743 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:55:16 smithi133 ceph-mon[29117]: daemon mds.cephfs.smithi183.eipfta is now active in filesystem cephfs as rank 1 2024-02-20T07:55:16.783 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:55:16 smithi183 ceph-mon[35832]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:55:16.783 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:55:16 smithi183 ceph-mon[35832]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:55:16.783 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:55:16 smithi183 ceph-mon[35832]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:55:16.783 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:55:16 smithi183 ceph-mon[35832]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi133.cmofvy", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-02-20T07:55:16.783 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:55:16 smithi183 ceph-mon[35832]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi133.cmofvy", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-02-20T07:55:16.783 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:55:16 smithi183 ceph-mon[35832]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-20T07:55:16.783 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:55:16 smithi183 ceph-mon[35832]: Deploying daemon mds.cephfs.smithi133.cmofvy on smithi133 2024-02-20T07:55:16.783 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:55:16 smithi183 ceph-mon[35832]: pgmap v179: 65 pgs: 65 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 148 B/s wr, 0 op/s 2024-02-20T07:55:16.783 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:55:16 smithi183 ceph-mon[35832]: from='client.? 172.21.15.133:0/3711320302' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "2"}]: dispatch 2024-02-20T07:55:16.783 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:55:16 smithi183 ceph-mon[35832]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-02-20T07:55:16.783 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:55:16 smithi183 ceph-mon[35832]: mds.? [v2:172.21.15.183:6824/3202657453,v1:172.21.15.183:6825/3202657453] up:boot 2024-02-20T07:55:16.783 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:55:16 smithi183 ceph-mon[35832]: from='client.? 172.21.15.133:0/3711320302' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "2"}]': finished 2024-02-20T07:55:16.784 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:55:16 smithi183 ceph-mon[35832]: daemon mds.cephfs.smithi183.eipfta assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-02-20T07:55:16.784 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:55:16 smithi183 ceph-mon[35832]: Health check failed: insufficient standby MDS daemons available (MDS_INSUFFICIENT_STANDBY) 2024-02-20T07:55:16.784 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:55:16 smithi183 ceph-mon[35832]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-02-20T07:55:16.784 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:55:16 smithi183 ceph-mon[35832]: fsmap cephfs:1 {0=cephfs.smithi133.itlmbm=up:active} 1 up:standby 2024-02-20T07:55:16.784 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:55:16 smithi183 ceph-mon[35832]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi183.eipfta"}]: dispatch 2024-02-20T07:55:16.784 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:55:16 smithi183 ceph-mon[35832]: fsmap cephfs:2 {0=cephfs.smithi133.itlmbm=up:active,1=cephfs.smithi183.eipfta=up:creating} 2024-02-20T07:55:16.784 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:55:16 smithi183 ceph-mon[35832]: daemon mds.cephfs.smithi183.eipfta is now active in filesystem cephfs as rank 1 2024-02-20T07:55:17.070 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-02-20T07:55:17.081 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi133.front.sepia.ceph.com 2024-02-20T07:55:17.081 DEBUG:teuthology.orchestra.run.smithi133:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4537327c-cfc4-11ee-95bb-87774f69a715 -- bash -c 'ceph fs set cephfs allow_standby_replay true' 2024-02-20T07:55:17.683 INFO:teuthology.orchestra.run.smithi133.stderr:Inferring config /var/lib/ceph/4537327c-cfc4-11ee-95bb-87774f69a715/mon.smithi133/config 2024-02-20T07:55:18.487 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:55:18 smithi133 ceph-mon[29117]: mds.? [v2:172.21.15.183:6824/3202657453,v1:172.21.15.183:6825/3202657453] up:active 2024-02-20T07:55:18.487 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:55:18 smithi133 ceph-mon[29117]: fsmap cephfs:2 {0=cephfs.smithi133.itlmbm=up:active,1=cephfs.smithi183.eipfta=up:active} 2024-02-20T07:55:18.547 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:55:18 smithi183 ceph-mon[35832]: mds.? [v2:172.21.15.183:6824/3202657453,v1:172.21.15.183:6825/3202657453] up:active 2024-02-20T07:55:18.547 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:55:18 smithi183 ceph-mon[35832]: fsmap cephfs:2 {0=cephfs.smithi133.itlmbm=up:active,1=cephfs.smithi183.eipfta=up:active} 2024-02-20T07:55:19.283 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:19.261+0000 7f96ce1eb700 1 -- 172.21.15.133:0/3747299965 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f96c80fbe40 msgr2=0x7f96c80fc250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:55:19.283 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:19.261+0000 7f96ce1eb700 1 --2- 172.21.15.133:0/3747299965 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f96c80fbe40 0x7f96c80fc250 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f96b8004940 tx=0x7f96b8009e30 comp rx=0 tx=0).stop 2024-02-20T07:55:19.283 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:19.261+0000 7f96ce1eb700 1 -- 172.21.15.133:0/3747299965 shutdown_connections 2024-02-20T07:55:19.283 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:19.261+0000 7f96ce1eb700 1 --2- 172.21.15.133:0/3747299965 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f96c80fcfb0 0x7f96c80fd420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:55:19.283 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:19.261+0000 7f96ce1eb700 1 --2- 172.21.15.133:0/3747299965 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f96c80fbe40 0x7f96c80fc250 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:55:19.283 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:19.261+0000 7f96ce1eb700 1 -- 172.21.15.133:0/3747299965 >> 172.21.15.133:0/3747299965 conn(0x7f96c80f7500 msgr2=0x7f96c80f9930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:55:19.283 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:19.261+0000 7f96ce1eb700 1 -- 172.21.15.133:0/3747299965 shutdown_connections 2024-02-20T07:55:19.283 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:19.261+0000 7f96ce1eb700 1 -- 172.21.15.133:0/3747299965 wait complete. 2024-02-20T07:55:19.284 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:19.261+0000 7f96ce1eb700 1 Processor -- start 2024-02-20T07:55:19.284 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:19.262+0000 7f96ce1eb700 1 -- start start 2024-02-20T07:55:19.284 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:19.262+0000 7f96ce1eb700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f96c80fbe40 0x7f96c8112a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:55:19.284 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:19.262+0000 7f96ce1eb700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f96c80fcfb0 0x7f96c8114fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:55:19.284 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:19.262+0000 7f96ce1eb700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96c8066be0 con 0x7f96c80fbe40 2024-02-20T07:55:19.284 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:19.262+0000 7f96ce1eb700 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96c8066d20 con 0x7f96c80fcfb0 2024-02-20T07:55:19.284 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:19.262+0000 7f96c77fe700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f96c80fbe40 0x7f96c8112a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:55:19.284 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:19.262+0000 7f96c77fe700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f96c80fbe40 0x7f96c8112a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.133:55950/0 (socket says 172.21.15.133:55950) 2024-02-20T07:55:19.284 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:19.262+0000 7f96c77fe700 1 -- 172.21.15.133:0/843618449 learned_addr learned my addr 172.21.15.133:0/843618449 (peer_addr_for_me v2:172.21.15.133:0/0) 2024-02-20T07:55:19.284 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:19.262+0000 7f96c6ffd700 1 --2- 172.21.15.133:0/843618449 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f96c80fcfb0 0x7f96c8114fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:55:19.284 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:19.262+0000 7f96c77fe700 1 -- 172.21.15.133:0/843618449 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f96c80fcfb0 msgr2=0x7f96c8114fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:55:19.285 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:19.262+0000 7f96c77fe700 1 --2- 172.21.15.133:0/843618449 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f96c80fcfb0 0x7f96c8114fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:55:19.285 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:19.262+0000 7f96c77fe700 1 -- 172.21.15.133:0/843618449 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96b8005040 con 0x7f96c80fbe40 2024-02-20T07:55:19.285 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:19.262+0000 7f96c6ffd700 1 --2- 172.21.15.133:0/843618449 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f96c80fcfb0 0x7f96c8114fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-20T07:55:19.285 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:19.262+0000 7f96c77fe700 1 --2- 172.21.15.133:0/843618449 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f96c80fbe40 0x7f96c8112a70 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f96b80093b0 tx=0x7f96b8009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:55:19.285 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:19.262+0000 7f96affff700 1 -- 172.21.15.133:0/843618449 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f96b800e040 con 0x7f96c80fbe40 2024-02-20T07:55:19.285 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:19.263+0000 7f96ce1eb700 1 -- 172.21.15.133:0/843618449 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f96c8115500 con 0x7f96c80fbe40 2024-02-20T07:55:19.285 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:19.263+0000 7f96affff700 1 -- 172.21.15.133:0/843618449 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f96b800bcc0 con 0x7f96c80fbe40 2024-02-20T07:55:19.285 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:19.263+0000 7f96affff700 1 -- 172.21.15.133:0/843618449 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f96b801c630 con 0x7f96c80fbe40 2024-02-20T07:55:19.285 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:19.263+0000 7f96ce1eb700 1 -- 172.21.15.133:0/843618449 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f96c8115960 con 0x7f96c80fbe40 2024-02-20T07:55:19.285 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:19.264+0000 7f96ce1eb700 1 -- 172.21.15.133:0/843618449 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f96c804bed0 con 0x7f96c80fbe40 2024-02-20T07:55:19.285 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:19.264+0000 7f96affff700 1 -- 172.21.15.133:0/843618449 <== mon.0 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7f96b80139f0 con 0x7f96c80fbe40 2024-02-20T07:55:19.285 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:19.264+0000 7f96affff700 1 --2- 172.21.15.133:0/843618449 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7f96b00692a0 0x7f96b006b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:55:19.285 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:19.264+0000 7f96affff700 1 -- 172.21.15.133:0/843618449 <== mon.0 v2:172.21.15.133:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5464+0+0 (secure 0 0 0) 0x7f96b80885e0 con 0x7f96c80fbe40 2024-02-20T07:55:19.285 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:19.264+0000 7f96c6ffd700 1 --2- 172.21.15.133:0/843618449 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7f96b00692a0 0x7f96b006b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:55:19.285 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:19.265+0000 7f96c6ffd700 1 --2- 172.21.15.133:0/843618449 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7f96b00692a0 0x7f96b006b750 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f96bc006fd0 tx=0x7f96bc008040 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:55:19.286 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:19.269+0000 7f96affff700 1 -- 172.21.15.133:0/843618449 <== mon.0 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f96b8054910 con 0x7f96c80fbe40 2024-02-20T07:55:19.388 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:55:19 smithi133 ceph-mon[29117]: pgmap v180: 65 pgs: 65 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1023 B/s wr, 3 op/s 2024-02-20T07:55:19.502 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:19.501+0000 7f96ce1eb700 1 -- 172.21.15.133:0/843618449 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"} v 0) v1 -- 0x7f96c81049b0 con 0x7f96c80fbe40 2024-02-20T07:55:19.547 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:55:19 smithi183 ceph-mon[35832]: pgmap v180: 65 pgs: 65 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1023 B/s wr, 3 op/s 2024-02-20T07:55:20.067 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:20.067+0000 7f96affff700 1 -- 172.21.15.133:0/843618449 <== mon.0 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]=0 v9) v1 ==== 121+0+0 (secure 0 0 0) 0x7f96b8057f30 con 0x7f96c80fbe40 2024-02-20T07:55:20.070 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:20.069+0000 7f96ce1eb700 1 -- 172.21.15.133:0/843618449 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7f96b00692a0 msgr2=0x7f96b006b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:55:20.070 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:20.069+0000 7f96ce1eb700 1 --2- 172.21.15.133:0/843618449 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7f96b00692a0 0x7f96b006b750 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f96bc006fd0 tx=0x7f96bc008040 comp rx=0 tx=0).stop 2024-02-20T07:55:20.070 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:20.069+0000 7f96ce1eb700 1 -- 172.21.15.133:0/843618449 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f96c80fbe40 msgr2=0x7f96c8112a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:55:20.070 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:20.069+0000 7f96ce1eb700 1 --2- 172.21.15.133:0/843618449 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f96c80fbe40 0x7f96c8112a70 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f96b80093b0 tx=0x7f96b8009e30 comp rx=0 tx=0).stop 2024-02-20T07:55:20.070 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:20.069+0000 7f96ce1eb700 1 -- 172.21.15.133:0/843618449 shutdown_connections 2024-02-20T07:55:20.070 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:20.069+0000 7f96ce1eb700 1 --2- 172.21.15.133:0/843618449 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f96c80fbe40 0x7f96c8112a70 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:55:20.070 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:20.069+0000 7f96ce1eb700 1 --2- 172.21.15.133:0/843618449 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7f96b00692a0 0x7f96b006b750 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:55:20.070 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:20.069+0000 7f96ce1eb700 1 --2- 172.21.15.133:0/843618449 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f96c80fcfb0 0x7f96c8114fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:55:20.070 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:20.069+0000 7f96ce1eb700 1 -- 172.21.15.133:0/843618449 >> 172.21.15.133:0/843618449 conn(0x7f96c80f7500 msgr2=0x7f96c8100270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:55:20.070 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:20.069+0000 7f96ce1eb700 1 -- 172.21.15.133:0/843618449 shutdown_connections 2024-02-20T07:55:20.070 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:20.069+0000 7f96ce1eb700 1 -- 172.21.15.133:0/843618449 wait complete. 2024-02-20T07:55:20.400 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:55:20 smithi133 ceph-mon[29117]: from='client.? 172.21.15.133:0/843618449' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-02-20T07:55:20.400 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:55:20 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:55:20.400 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:55:20 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:55:20.400 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:55:20 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:55:20.400 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:55:20 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi183.cwfunk", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-02-20T07:55:20.400 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:55:20 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi183.cwfunk", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-02-20T07:55:20.400 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:55:20 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-20T07:55:20.434 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:55:20 smithi183 ceph-mon[35832]: from='client.? 172.21.15.133:0/843618449' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-02-20T07:55:20.434 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:55:20 smithi183 ceph-mon[35832]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:55:20.434 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:55:20 smithi183 ceph-mon[35832]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:55:20.434 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:55:20 smithi183 ceph-mon[35832]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:55:20.434 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:55:20 smithi183 ceph-mon[35832]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi183.cwfunk", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-02-20T07:55:20.435 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:55:20 smithi183 ceph-mon[35832]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi183.cwfunk", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-02-20T07:55:20.435 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:55:20 smithi183 ceph-mon[35832]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-20T07:55:20.666 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-02-20T07:55:20.677 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi133.front.sepia.ceph.com 2024-02-20T07:55:20.677 DEBUG:teuthology.orchestra.run.smithi133:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4537327c-cfc4-11ee-95bb-87774f69a715 -- bash -c 'ceph fs set cephfs inline_data true --yes-i-really-really-mean-it' 2024-02-20T07:55:20.937 INFO:teuthology.orchestra.run.smithi133.stderr:Inferring config /var/lib/ceph/4537327c-cfc4-11ee-95bb-87774f69a715/mon.smithi133/config 2024-02-20T07:55:21.318 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:55:21 smithi183 ceph-mon[35832]: pgmap v181: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.8 KiB/s wr, 6 op/s 2024-02-20T07:55:21.318 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:55:21 smithi183 ceph-mon[35832]: Deploying daemon mds.cephfs.smithi183.cwfunk on smithi183 2024-02-20T07:55:21.318 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:55:21 smithi183 ceph-mon[35832]: Health check cleared: MDS_INSUFFICIENT_STANDBY (was: insufficient standby MDS daemons available) 2024-02-20T07:55:21.318 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:55:21 smithi183 ceph-mon[35832]: Cluster is now healthy 2024-02-20T07:55:21.318 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:55:21 smithi183 ceph-mon[35832]: from='client.? 172.21.15.133:0/843618449' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-02-20T07:55:21.318 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:55:21 smithi183 ceph-mon[35832]: mds.? [v2:172.21.15.133:6828/3316004709,v1:172.21.15.133:6829/3316004709] up:boot 2024-02-20T07:55:21.318 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:55:21 smithi183 ceph-mon[35832]: fsmap cephfs:2 {0=cephfs.smithi133.itlmbm=up:active,1=cephfs.smithi183.eipfta=up:active} 1 up:standby 2024-02-20T07:55:21.318 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:55:21 smithi183 ceph-mon[35832]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi133.cmofvy"}]: dispatch 2024-02-20T07:55:21.318 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:55:21 smithi183 ceph-mon[35832]: fsmap cephfs:2 {0=cephfs.smithi133.itlmbm=up:active,1=cephfs.smithi183.eipfta=up:active} 1 up:standby-replay 2024-02-20T07:55:21.427 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:55:21 smithi133 ceph-mon[29117]: pgmap v181: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.8 KiB/s wr, 6 op/s 2024-02-20T07:55:21.427 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:55:21 smithi133 ceph-mon[29117]: Deploying daemon mds.cephfs.smithi183.cwfunk on smithi183 2024-02-20T07:55:21.427 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:55:21 smithi133 ceph-mon[29117]: Health check cleared: MDS_INSUFFICIENT_STANDBY (was: insufficient standby MDS daemons available) 2024-02-20T07:55:21.427 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:55:21 smithi133 ceph-mon[29117]: Cluster is now healthy 2024-02-20T07:55:21.427 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:55:21 smithi133 ceph-mon[29117]: from='client.? 172.21.15.133:0/843618449' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-02-20T07:55:21.427 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:55:21 smithi133 ceph-mon[29117]: mds.? [v2:172.21.15.133:6828/3316004709,v1:172.21.15.133:6829/3316004709] up:boot 2024-02-20T07:55:21.427 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:55:21 smithi133 ceph-mon[29117]: fsmap cephfs:2 {0=cephfs.smithi133.itlmbm=up:active,1=cephfs.smithi183.eipfta=up:active} 1 up:standby 2024-02-20T07:55:21.427 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:55:21 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi133.cmofvy"}]: dispatch 2024-02-20T07:55:21.427 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:55:21 smithi133 ceph-mon[29117]: fsmap cephfs:2 {0=cephfs.smithi133.itlmbm=up:active,1=cephfs.smithi183.eipfta=up:active} 1 up:standby-replay 2024-02-20T07:55:22.581 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:22.580+0000 7f764e371700 1 -- 172.21.15.133:0/1876228989 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f7648066380 msgr2=0x7f76480ffb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:55:22.581 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:22.580+0000 7f764e371700 1 --2- 172.21.15.133:0/1876228989 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f7648066380 0x7f76480ffb50 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f763c004940 tx=0x7f763c009e30 comp rx=0 tx=0).stop 2024-02-20T07:55:22.582 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:22.580+0000 7f764e371700 1 -- 172.21.15.133:0/1876228989 shutdown_connections 2024-02-20T07:55:22.582 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:22.580+0000 7f764e371700 1 --2- 172.21.15.133:0/1876228989 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f7648100120 0x7f7648102520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:55:22.582 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:22.580+0000 7f764e371700 1 --2- 172.21.15.133:0/1876228989 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f7648066380 0x7f76480ffb50 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:55:22.582 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:22.580+0000 7f764e371700 1 -- 172.21.15.133:0/1876228989 >> 172.21.15.133:0/1876228989 conn(0x7f76480f74e0 msgr2=0x7f76480f9930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:55:22.582 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:22.580+0000 7f764e371700 1 -- 172.21.15.133:0/1876228989 shutdown_connections 2024-02-20T07:55:22.582 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:22.580+0000 7f764e371700 1 -- 172.21.15.133:0/1876228989 wait complete. 2024-02-20T07:55:22.582 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:22.581+0000 7f764e371700 1 Processor -- start 2024-02-20T07:55:22.582 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:22.581+0000 7f764e371700 1 -- start start 2024-02-20T07:55:22.583 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:22.581+0000 7f764e371700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f7648066380 0x7f764810d8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:55:22.584 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:22.581+0000 7f764e371700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f7648100120 0x7f764810de30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:55:22.584 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:22.581+0000 7f764e371700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f764810e430 con 0x7f7648066380 2024-02-20T07:55:22.584 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:22.581+0000 7f764e371700 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f764810ecd0 con 0x7f7648100120 2024-02-20T07:55:22.584 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:22.581+0000 7f7647fff700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f7648066380 0x7f764810d8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:55:22.584 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:22.581+0000 7f7647fff700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f7648066380 0x7f764810d8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.133:55996/0 (socket says 172.21.15.133:55996) 2024-02-20T07:55:22.584 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:22.581+0000 7f7647fff700 1 -- 172.21.15.133:0/1980274698 learned_addr learned my addr 172.21.15.133:0/1980274698 (peer_addr_for_me v2:172.21.15.133:0/0) 2024-02-20T07:55:22.584 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:22.581+0000 7f76477fe700 1 --2- 172.21.15.133:0/1980274698 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f7648100120 0x7f764810de30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:55:22.584 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:22.582+0000 7f7647fff700 1 -- 172.21.15.133:0/1980274698 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f7648100120 msgr2=0x7f764810de30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:55:22.584 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:22.582+0000 7f7647fff700 1 --2- 172.21.15.133:0/1980274698 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f7648100120 0x7f764810de30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:55:22.584 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:22.582+0000 7f7647fff700 1 -- 172.21.15.133:0/1980274698 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f763c005040 con 0x7f7648066380 2024-02-20T07:55:22.584 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:22.582+0000 7f76477fe700 1 --2- 172.21.15.133:0/1980274698 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f7648100120 0x7f764810de30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-20T07:55:22.584 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:22.582+0000 7f7647fff700 1 --2- 172.21.15.133:0/1980274698 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f7648066380 0x7f764810d8f0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f763c000c00 tx=0x7f763c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:55:22.585 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:22.582+0000 7f7644ff9700 1 -- 172.21.15.133:0/1980274698 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f763c021070 con 0x7f7648066380 2024-02-20T07:55:22.585 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:22.582+0000 7f764e371700 1 -- 172.21.15.133:0/1980274698 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f764810ef50 con 0x7f7648066380 2024-02-20T07:55:22.585 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:22.582+0000 7f7644ff9700 1 -- 172.21.15.133:0/1980274698 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f763c01c400 con 0x7f7648066380 2024-02-20T07:55:22.585 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:22.582+0000 7f7644ff9700 1 -- 172.21.15.133:0/1980274698 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f763c00e070 con 0x7f7648066380 2024-02-20T07:55:22.585 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:22.582+0000 7f764e371700 1 -- 172.21.15.133:0/1980274698 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f764810f430 con 0x7f7648066380 2024-02-20T07:55:22.585 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:22.583+0000 7f764e371700 1 -- 172.21.15.133:0/1980274698 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f764804bed0 con 0x7f7648066380 2024-02-20T07:55:22.585 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:22.584+0000 7f7644ff9700 1 -- 172.21.15.133:0/1980274698 <== mon.0 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7f763c01ca50 con 0x7f7648066380 2024-02-20T07:55:22.586 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:22.584+0000 7f7644ff9700 1 --2- 172.21.15.133:0/1980274698 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7f76300692a0 0x7f763006b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:55:22.586 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:22.584+0000 7f7644ff9700 1 -- 172.21.15.133:0/1980274698 <== mon.0 v2:172.21.15.133:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5464+0+0 (secure 0 0 0) 0x7f763c088390 con 0x7f7648066380 2024-02-20T07:55:22.586 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:22.584+0000 7f76477fe700 1 --2- 172.21.15.133:0/1980274698 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7f76300692a0 0x7f763006b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:55:22.586 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:22.584+0000 7f76477fe700 1 --2- 172.21.15.133:0/1980274698 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7f76300692a0 0x7f763006b750 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f7638009b80 tx=0x7f763800b040 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:55:22.589 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:22.588+0000 7f7644ff9700 1 -- 172.21.15.133:0/1980274698 <== mon.0 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f763c058150 con 0x7f7648066380 2024-02-20T07:55:22.830 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:22.829+0000 7f764e371700 1 -- 172.21.15.133:0/1980274698 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "inline_data", "val": "true", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f764810f890 con 0x7f7648066380 2024-02-20T07:55:23.074 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:23.073+0000 7f7644ff9700 1 -- 172.21.15.133:0/1980274698 <== mon.0 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "inline_data", "val": "true", "yes_i_really_really_mean_it": true}]=0 inline data enabled v11) v1 ==== 168+0+0 (secure 0 0 0) 0x7f763c057ce0 con 0x7f7648066380 2024-02-20T07:55:23.077 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:23.076+0000 7f764e371700 1 -- 172.21.15.133:0/1980274698 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7f76300692a0 msgr2=0x7f763006b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:55:23.077 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:23.076+0000 7f764e371700 1 --2- 172.21.15.133:0/1980274698 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7f76300692a0 0x7f763006b750 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f7638009b80 tx=0x7f763800b040 comp rx=0 tx=0).stop 2024-02-20T07:55:23.077 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:23.076+0000 7f764e371700 1 -- 172.21.15.133:0/1980274698 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f7648066380 msgr2=0x7f764810d8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:55:23.077 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:23.076+0000 7f764e371700 1 --2- 172.21.15.133:0/1980274698 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f7648066380 0x7f764810d8f0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f763c000c00 tx=0x7f763c009e30 comp rx=0 tx=0).stop 2024-02-20T07:55:23.077 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:23.076+0000 7f764e371700 1 -- 172.21.15.133:0/1980274698 shutdown_connections 2024-02-20T07:55:23.077 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:23.076+0000 7f764e371700 1 --2- 172.21.15.133:0/1980274698 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f7648066380 0x7f764810d8f0 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:55:23.077 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:23.076+0000 7f764e371700 1 --2- 172.21.15.133:0/1980274698 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7f76300692a0 0x7f763006b750 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:55:23.077 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:23.076+0000 7f764e371700 1 --2- 172.21.15.133:0/1980274698 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f7648100120 0x7f764810de30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:55:23.077 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:23.076+0000 7f764e371700 1 -- 172.21.15.133:0/1980274698 >> 172.21.15.133:0/1980274698 conn(0x7f76480f74e0 msgr2=0x7f76480f9930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:55:23.077 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:23.076+0000 7f764e371700 1 -- 172.21.15.133:0/1980274698 shutdown_connections 2024-02-20T07:55:23.078 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:23.076+0000 7f764e371700 1 -- 172.21.15.133:0/1980274698 wait complete. 2024-02-20T07:55:23.078 INFO:teuthology.orchestra.run.smithi133.stderr:inline data enabled 2024-02-20T07:55:23.297 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:55:23 smithi183 ceph-mon[35832]: pgmap v182: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 282 B/s rd, 2.2 KiB/s wr, 7 op/s 2024-02-20T07:55:23.297 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:55:23 smithi183 ceph-mon[35832]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:55:23.297 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:55:23 smithi183 ceph-mon[35832]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:55:23.297 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:55:23 smithi183 ceph-mon[35832]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:55:23.298 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:55:23 smithi183 ceph-mon[35832]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:55:23.298 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:55:23 smithi183 ceph-mon[35832]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:55:23.298 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:55:23 smithi183 ceph-mon[35832]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-20T07:55:23.298 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:55:23 smithi183 ceph-mon[35832]: from='client.? 172.21.15.133:0/1980274698' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "inline_data", "val": "true", "yes_i_really_really_mean_it": true}]: dispatch 2024-02-20T07:55:23.358 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:55:23 smithi133 ceph-mon[29117]: pgmap v182: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 282 B/s rd, 2.2 KiB/s wr, 7 op/s 2024-02-20T07:55:23.358 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:55:23 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:55:23.358 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:55:23 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:55:23.358 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:55:23 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:55:23.358 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:55:23 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:55:23.359 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:55:23 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:55:23.359 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:55:23 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-20T07:55:23.359 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:55:23 smithi133 ceph-mon[29117]: from='client.? 172.21.15.133:0/1980274698' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "inline_data", "val": "true", "yes_i_really_really_mean_it": true}]: dispatch 2024-02-20T07:55:23.756 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-02-20T07:55:23.766 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi133.front.sepia.ceph.com 2024-02-20T07:55:23.766 DEBUG:teuthology.orchestra.run.smithi133:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4537327c-cfc4-11ee-95bb-87774f69a715 -- bash -c 'ceph fs dump' 2024-02-20T07:55:24.172 INFO:teuthology.orchestra.run.smithi133.stderr:Inferring config /var/lib/ceph/4537327c-cfc4-11ee-95bb-87774f69a715/mon.smithi133/config 2024-02-20T07:55:24.349 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:55:24 smithi133 ceph-mon[29117]: Health check failed: 1 filesystem with deprecated feature inline_data (FS_INLINE_DATA_DEPRECATED) 2024-02-20T07:55:24.349 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:55:24 smithi133 ceph-mon[29117]: mds.? [v2:172.21.15.183:6826/1729948973,v1:172.21.15.183:6827/1729948973] up:boot 2024-02-20T07:55:24.349 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:55:24 smithi133 ceph-mon[29117]: from='client.? 172.21.15.133:0/1980274698' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "inline_data", "val": "true", "yes_i_really_really_mean_it": true}]': finished 2024-02-20T07:55:24.349 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:55:24 smithi133 ceph-mon[29117]: fsmap cephfs:2 {0=cephfs.smithi133.itlmbm=up:active,1=cephfs.smithi183.eipfta=up:active} 1 up:standby-replay 1 up:standby 2024-02-20T07:55:24.349 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:55:24 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi183.cwfunk"}]: dispatch 2024-02-20T07:55:24.350 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:55:24 smithi133 ceph-mon[29117]: fsmap cephfs:2 {0=cephfs.smithi133.itlmbm=up:active,1=cephfs.smithi183.eipfta=up:active} 2 up:standby-replay 2024-02-20T07:55:24.350 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:55:24 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:55:24.350 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:55:24 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-20T07:55:24.350 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:55:24 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:55:24.547 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:55:24 smithi183 ceph-mon[35832]: Health check failed: 1 filesystem with deprecated feature inline_data (FS_INLINE_DATA_DEPRECATED) 2024-02-20T07:55:24.547 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:55:24 smithi183 ceph-mon[35832]: mds.? [v2:172.21.15.183:6826/1729948973,v1:172.21.15.183:6827/1729948973] up:boot 2024-02-20T07:55:24.548 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:55:24 smithi183 ceph-mon[35832]: from='client.? 172.21.15.133:0/1980274698' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "inline_data", "val": "true", "yes_i_really_really_mean_it": true}]': finished 2024-02-20T07:55:24.548 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:55:24 smithi183 ceph-mon[35832]: fsmap cephfs:2 {0=cephfs.smithi133.itlmbm=up:active,1=cephfs.smithi183.eipfta=up:active} 1 up:standby-replay 1 up:standby 2024-02-20T07:55:24.548 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:55:24 smithi183 ceph-mon[35832]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi183.cwfunk"}]: dispatch 2024-02-20T07:55:24.548 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:55:24 smithi183 ceph-mon[35832]: fsmap cephfs:2 {0=cephfs.smithi133.itlmbm=up:active,1=cephfs.smithi183.eipfta=up:active} 2 up:standby-replay 2024-02-20T07:55:24.548 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:55:24 smithi183 ceph-mon[35832]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:55:24.548 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:55:24 smithi183 ceph-mon[35832]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-20T07:55:24.548 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:55:24 smithi183 ceph-mon[35832]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:55:25.242 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:55:25 smithi133 ceph-mon[29117]: pgmap v183: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.3 KiB/s rd, 2.0 KiB/s wr, 8 op/s 2024-02-20T07:55:25.242 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:55:25 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:55:25.242 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:55:25 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:55:25.242 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:55:25 smithi133 ceph-mon[29117]: mds.? [v2:172.21.15.133:6828/3316004709,v1:172.21.15.133:6829/3316004709] up:standby-replay 2024-02-20T07:55:25.242 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:55:25 smithi133 ceph-mon[29117]: mds.? [v2:172.21.15.183:6824/3202657453,v1:172.21.15.183:6825/3202657453] up:active 2024-02-20T07:55:25.242 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:55:25 smithi133 ceph-mon[29117]: Dropping low affinity active daemon mds.cephfs.smithi133.itlmbm in favor of higher affinity standby. 2024-02-20T07:55:25.242 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:55:25 smithi133 ceph-mon[29117]: Replacing daemon mds.cephfs.smithi133.itlmbm as rank 0 with standby daemon mds.cephfs.smithi133.cmofvy 2024-02-20T07:55:25.242 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:55:25 smithi133 ceph-mon[29117]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-02-20T07:55:25.242 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:55:25 smithi133 ceph-mon[29117]: fsmap cephfs:2 {0=cephfs.smithi133.itlmbm=up:active,1=cephfs.smithi183.eipfta=up:active} 2 up:standby-replay 2024-02-20T07:55:25.242 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:55:25 smithi133 ceph-mon[29117]: osdmap e46: 6 total, 6 up, 6 in 2024-02-20T07:55:25.242 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:55:25 smithi133 ceph-mon[29117]: fsmap cephfs:2/2 {0=cephfs.smithi133.cmofvy=up:replay,1=cephfs.smithi183.eipfta=up:active} 1 up:standby-replay 2024-02-20T07:55:25.547 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:55:25 smithi183 ceph-mon[35832]: pgmap v183: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.3 KiB/s rd, 2.0 KiB/s wr, 8 op/s 2024-02-20T07:55:25.548 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:55:25 smithi183 ceph-mon[35832]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:55:25.548 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:55:25 smithi183 ceph-mon[35832]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:55:25.548 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:55:25 smithi183 ceph-mon[35832]: mds.? [v2:172.21.15.133:6828/3316004709,v1:172.21.15.133:6829/3316004709] up:standby-replay 2024-02-20T07:55:25.548 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:55:25 smithi183 ceph-mon[35832]: mds.? [v2:172.21.15.183:6824/3202657453,v1:172.21.15.183:6825/3202657453] up:active 2024-02-20T07:55:25.548 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:55:25 smithi183 ceph-mon[35832]: Dropping low affinity active daemon mds.cephfs.smithi133.itlmbm in favor of higher affinity standby. 2024-02-20T07:55:25.548 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:55:25 smithi183 ceph-mon[35832]: Replacing daemon mds.cephfs.smithi133.itlmbm as rank 0 with standby daemon mds.cephfs.smithi133.cmofvy 2024-02-20T07:55:25.548 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:55:25 smithi183 ceph-mon[35832]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-02-20T07:55:25.548 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:55:25 smithi183 ceph-mon[35832]: fsmap cephfs:2 {0=cephfs.smithi133.itlmbm=up:active,1=cephfs.smithi183.eipfta=up:active} 2 up:standby-replay 2024-02-20T07:55:25.548 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:55:25 smithi183 ceph-mon[35832]: osdmap e46: 6 total, 6 up, 6 in 2024-02-20T07:55:25.548 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:55:25 smithi183 ceph-mon[35832]: fsmap cephfs:2/2 {0=cephfs.smithi133.cmofvy=up:replay,1=cephfs.smithi183.eipfta=up:active} 1 up:standby-replay 2024-02-20T07:55:26.380 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:26.377+0000 7f0e347a1700 1 -- 172.21.15.133:0/661197094 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f0e2c066c50 msgr2=0x7f0e2c0ffb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:55:26.380 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:26.377+0000 7f0e347a1700 1 --2- 172.21.15.133:0/661197094 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f0e2c066c50 0x7f0e2c0ffb20 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f0e28004940 tx=0x7f0e28009e30 comp rx=0 tx=0).stop 2024-02-20T07:55:26.380 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:26.378+0000 7f0e347a1700 1 -- 172.21.15.133:0/661197094 shutdown_connections 2024-02-20T07:55:26.381 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:26.378+0000 7f0e347a1700 1 --2- 172.21.15.133:0/661197094 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0e2c100060 0x7f0e2c102480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:55:26.381 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:26.378+0000 7f0e347a1700 1 --2- 172.21.15.133:0/661197094 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f0e2c066c50 0x7f0e2c0ffb20 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:55:26.381 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:26.378+0000 7f0e347a1700 1 -- 172.21.15.133:0/661197094 >> 172.21.15.133:0/661197094 conn(0x7f0e2c0f7590 msgr2=0x7f0e2c0f99c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:55:26.381 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:26.378+0000 7f0e347a1700 1 -- 172.21.15.133:0/661197094 shutdown_connections 2024-02-20T07:55:26.381 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:26.378+0000 7f0e347a1700 1 -- 172.21.15.133:0/661197094 wait complete. 2024-02-20T07:55:26.381 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:26.378+0000 7f0e347a1700 1 Processor -- start 2024-02-20T07:55:26.381 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:26.378+0000 7f0e347a1700 1 -- start start 2024-02-20T07:55:26.381 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:26.378+0000 7f0e347a1700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f0e2c066c50 0x7f0e2c0fd5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:55:26.381 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:26.378+0000 7f0e347a1700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0e2c100060 0x7f0e2c0fba20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:55:26.381 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:26.378+0000 7f0e347a1700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e2c0fc020 con 0x7f0e2c066c50 2024-02-20T07:55:26.381 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:26.378+0000 7f0e347a1700 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e2c0fc190 con 0x7f0e2c100060 2024-02-20T07:55:26.381 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:26.378+0000 7f0e3253d700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f0e2c066c50 0x7f0e2c0fd5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:55:26.381 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:26.378+0000 7f0e3253d700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f0e2c066c50 0x7f0e2c0fd5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.133:56016/0 (socket says 172.21.15.133:56016) 2024-02-20T07:55:26.382 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:26.378+0000 7f0e3253d700 1 -- 172.21.15.133:0/1005587274 learned_addr learned my addr 172.21.15.133:0/1005587274 (peer_addr_for_me v2:172.21.15.133:0/0) 2024-02-20T07:55:26.382 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:26.378+0000 7f0e31d3c700 1 --2- 172.21.15.133:0/1005587274 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0e2c100060 0x7f0e2c0fba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:55:26.382 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:26.379+0000 7f0e3253d700 1 -- 172.21.15.133:0/1005587274 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0e2c100060 msgr2=0x7f0e2c0fba20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:55:26.382 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:26.379+0000 7f0e3253d700 1 --2- 172.21.15.133:0/1005587274 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0e2c100060 0x7f0e2c0fba20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:55:26.382 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:26.379+0000 7f0e3253d700 1 -- 172.21.15.133:0/1005587274 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e28005040 con 0x7f0e2c066c50 2024-02-20T07:55:26.382 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:26.379+0000 7f0e3253d700 1 --2- 172.21.15.133:0/1005587274 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f0e2c066c50 0x7f0e2c0fd5a0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f0e2800b3b0 tx=0x7f0e28009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:55:26.382 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:26.379+0000 7f0e22ffd700 1 -- 172.21.15.133:0/1005587274 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0e2800e040 con 0x7f0e2c066c50 2024-02-20T07:55:26.382 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:26.379+0000 7f0e22ffd700 1 -- 172.21.15.133:0/1005587274 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0e2800bcc0 con 0x7f0e2c066c50 2024-02-20T07:55:26.382 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:26.379+0000 7f0e22ffd700 1 -- 172.21.15.133:0/1005587274 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0e2801c630 con 0x7f0e2c066c50 2024-02-20T07:55:26.382 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:26.379+0000 7f0e347a1700 1 -- 172.21.15.133:0/1005587274 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0e2c0fc410 con 0x7f0e2c066c50 2024-02-20T07:55:26.382 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:26.379+0000 7f0e347a1700 1 -- 172.21.15.133:0/1005587274 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0e2c1159a0 con 0x7f0e2c066c50 2024-02-20T07:55:26.383 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:26.380+0000 7f0e347a1700 1 -- 172.21.15.133:0/1005587274 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0e2c04bed0 con 0x7f0e2c066c50 2024-02-20T07:55:26.383 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:26.380+0000 7f0e22ffd700 1 -- 172.21.15.133:0/1005587274 <== mon.0 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7f0e28013980 con 0x7f0e2c066c50 2024-02-20T07:55:26.383 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:26.381+0000 7f0e22ffd700 1 --2- 172.21.15.133:0/1005587274 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7f0e180692a0 0x7f0e1806b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:55:26.383 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:26.381+0000 7f0e22ffd700 1 -- 172.21.15.133:0/1005587274 <== mon.0 v2:172.21.15.133:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f0e28088450 con 0x7f0e2c066c50 2024-02-20T07:55:26.383 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:26.381+0000 7f0e31d3c700 1 --2- 172.21.15.133:0/1005587274 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7f0e180692a0 0x7f0e1806b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:55:26.384 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:26.381+0000 7f0e31d3c700 1 --2- 172.21.15.133:0/1005587274 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7f0e180692a0 0x7f0e1806b750 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f0e1c0041d0 tx=0x7f0e1c0043c0 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:55:26.386 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:26.385+0000 7f0e22ffd700 1 -- 172.21.15.133:0/1005587274 <== mon.0 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f0e28058270 con 0x7f0e2c066c50 2024-02-20T07:55:26.620 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:26.619+0000 7f0e347a1700 1 -- 172.21.15.133:0/1005587274 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f0e2c115d10 con 0x7f0e2c066c50 2024-02-20T07:55:26.620 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:26.620+0000 7f0e22ffd700 1 -- 172.21.15.133:0/1005587274 <== mon.0 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 15 v15) v1 ==== 76+0+1838 (secure 0 0 0) 0x7f0e28057e00 con 0x7f0e2c066c50 2024-02-20T07:55:26.620 INFO:teuthology.orchestra.run.smithi133.stdout:e15 2024-02-20T07:55:26.621 INFO:teuthology.orchestra.run.smithi133.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-02-20T07:55:26.621 INFO:teuthology.orchestra.run.smithi133.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-20T07:55:26.621 INFO:teuthology.orchestra.run.smithi133.stdout:legacy client fscid: 1 2024-02-20T07:55:26.621 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-02-20T07:55:26.621 INFO:teuthology.orchestra.run.smithi133.stdout:Filesystem 'cephfs' (1) 2024-02-20T07:55:26.621 INFO:teuthology.orchestra.run.smithi133.stdout:fs_name cephfs 2024-02-20T07:55:26.621 INFO:teuthology.orchestra.run.smithi133.stdout:epoch 15 2024-02-20T07:55:26.621 INFO:teuthology.orchestra.run.smithi133.stdout:flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2024-02-20T07:55:26.621 INFO:teuthology.orchestra.run.smithi133.stdout:created 2024-02-20T07:55:08.678040+0000 2024-02-20T07:55:26.621 INFO:teuthology.orchestra.run.smithi133.stdout:modified 2024-02-20T07:55:25.627831+0000 2024-02-20T07:55:26.621 INFO:teuthology.orchestra.run.smithi133.stdout:tableserver 0 2024-02-20T07:55:26.621 INFO:teuthology.orchestra.run.smithi133.stdout:root 0 2024-02-20T07:55:26.621 INFO:teuthology.orchestra.run.smithi133.stdout:session_timeout 60 2024-02-20T07:55:26.622 INFO:teuthology.orchestra.run.smithi133.stdout:session_autoclose 300 2024-02-20T07:55:26.622 INFO:teuthology.orchestra.run.smithi133.stdout:max_file_size 1099511627776 2024-02-20T07:55:26.622 INFO:teuthology.orchestra.run.smithi133.stdout:required_client_features {} 2024-02-20T07:55:26.622 INFO:teuthology.orchestra.run.smithi133.stdout:last_failure 0 2024-02-20T07:55:26.622 INFO:teuthology.orchestra.run.smithi133.stdout:last_failure_osd_epoch 46 2024-02-20T07:55:26.622 INFO:teuthology.orchestra.run.smithi133.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-20T07:55:26.622 INFO:teuthology.orchestra.run.smithi133.stdout:max_mds 2 2024-02-20T07:55:26.622 INFO:teuthology.orchestra.run.smithi133.stdout:in 0,1 2024-02-20T07:55:26.622 INFO:teuthology.orchestra.run.smithi133.stdout:up {0=14486,1=24257} 2024-02-20T07:55:26.622 INFO:teuthology.orchestra.run.smithi133.stdout:failed 2024-02-20T07:55:26.622 INFO:teuthology.orchestra.run.smithi133.stdout:damaged 2024-02-20T07:55:26.622 INFO:teuthology.orchestra.run.smithi133.stdout:stopped 2024-02-20T07:55:26.622 INFO:teuthology.orchestra.run.smithi133.stdout:data_pools [3] 2024-02-20T07:55:26.623 INFO:teuthology.orchestra.run.smithi133.stdout:metadata_pool 2 2024-02-20T07:55:26.623 INFO:teuthology.orchestra.run.smithi133.stdout:inline_data enabled 2024-02-20T07:55:26.623 INFO:teuthology.orchestra.run.smithi133.stdout:balancer 2024-02-20T07:55:26.623 INFO:teuthology.orchestra.run.smithi133.stdout:standby_count_wanted 1 2024-02-20T07:55:26.623 INFO:teuthology.orchestra.run.smithi133.stdout:[mds.cephfs.smithi133.cmofvy{0:14486} state up:resolve seq 3 join_fscid=1 addr [v2:172.21.15.133:6828/3316004709,v1:172.21.15.133:6829/3316004709] compat {c=[1],r=[1],i=[7ff]}] 2024-02-20T07:55:26.623 INFO:teuthology.orchestra.run.smithi133.stdout:[mds.cephfs.smithi183.eipfta{1:24257} state up:active seq 4 join_fscid=1 addr [v2:172.21.15.183:6824/3202657453,v1:172.21.15.183:6825/3202657453] compat {c=[1],r=[1],i=[7ff]}] 2024-02-20T07:55:26.623 INFO:teuthology.orchestra.run.smithi133.stdout:[mds.cephfs.smithi183.cwfunk{1:24265} state up:standby-replay seq 1 addr [v2:172.21.15.183:6826/1729948973,v1:172.21.15.183:6827/1729948973] compat {c=[1],r=[1],i=[7ff]}] 2024-02-20T07:55:26.623 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-02-20T07:55:26.623 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-02-20T07:55:26.623 INFO:teuthology.orchestra.run.smithi133.stdout:Standby daemons: 2024-02-20T07:55:26.623 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-02-20T07:55:26.623 INFO:teuthology.orchestra.run.smithi133.stdout:[mds.cephfs.smithi133.itlmbm{-1:14496} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.133:6826/1557353480,v1:172.21.15.133:6827/1557353480] compat {c=[1],r=[1],i=[7ff]}] 2024-02-20T07:55:26.624 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:26.622+0000 7f0e347a1700 1 -- 172.21.15.133:0/1005587274 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7f0e180692a0 msgr2=0x7f0e1806b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:55:26.624 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:26.622+0000 7f0e347a1700 1 --2- 172.21.15.133:0/1005587274 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7f0e180692a0 0x7f0e1806b750 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f0e1c0041d0 tx=0x7f0e1c0043c0 comp rx=0 tx=0).stop 2024-02-20T07:55:26.624 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:26.622+0000 7f0e347a1700 1 -- 172.21.15.133:0/1005587274 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f0e2c066c50 msgr2=0x7f0e2c0fd5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:55:26.624 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:26.622+0000 7f0e347a1700 1 --2- 172.21.15.133:0/1005587274 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f0e2c066c50 0x7f0e2c0fd5a0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f0e2800b3b0 tx=0x7f0e28009e30 comp rx=0 tx=0).stop 2024-02-20T07:55:26.624 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:26.622+0000 7f0e347a1700 1 -- 172.21.15.133:0/1005587274 shutdown_connections 2024-02-20T07:55:26.624 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:26.622+0000 7f0e347a1700 1 --2- 172.21.15.133:0/1005587274 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f0e2c066c50 0x7f0e2c0fd5a0 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:55:26.624 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:26.622+0000 7f0e347a1700 1 --2- 172.21.15.133:0/1005587274 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7f0e180692a0 0x7f0e1806b750 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:55:26.624 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:26.622+0000 7f0e347a1700 1 --2- 172.21.15.133:0/1005587274 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0e2c100060 0x7f0e2c0fba20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:55:26.625 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:26.622+0000 7f0e347a1700 1 -- 172.21.15.133:0/1005587274 >> 172.21.15.133:0/1005587274 conn(0x7f0e2c0f7590 msgr2=0x7f0e2c0f99c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:55:26.625 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:26.622+0000 7f0e347a1700 1 -- 172.21.15.133:0/1005587274 shutdown_connections 2024-02-20T07:55:26.625 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:26.622+0000 7f0e347a1700 1 -- 172.21.15.133:0/1005587274 wait complete. 2024-02-20T07:55:26.625 INFO:teuthology.orchestra.run.smithi133.stderr:dumped fsmap epoch 15 2024-02-20T07:55:26.991 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:55:26 smithi133 ceph-mon[29117]: pgmap v185: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.6 KiB/s rd, 2.3 KiB/s wr, 10 op/s 2024-02-20T07:55:26.991 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:55:26 smithi133 ceph-mon[29117]: mds.? [v2:172.21.15.133:6826/1557353480,v1:172.21.15.133:6827/1557353480] up:boot 2024-02-20T07:55:26.992 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:55:26 smithi133 ceph-mon[29117]: mds.? [v2:172.21.15.133:6828/3316004709,v1:172.21.15.133:6829/3316004709] up:resolve 2024-02-20T07:55:26.992 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:55:26 smithi133 ceph-mon[29117]: fsmap cephfs:2/2 {0=cephfs.smithi133.cmofvy=up:resolve,1=cephfs.smithi183.eipfta=up:active} 1 up:standby-replay 1 up:standby 2024-02-20T07:55:26.992 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:55:26 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi133.itlmbm"}]: dispatch 2024-02-20T07:55:26.992 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:55:26 smithi133 ceph-mon[29117]: from='client.? 172.21.15.133:0/1005587274' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-02-20T07:55:27.047 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:55:26 smithi183 ceph-mon[35832]: pgmap v185: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.6 KiB/s rd, 2.3 KiB/s wr, 10 op/s 2024-02-20T07:55:27.047 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:55:26 smithi183 ceph-mon[35832]: mds.? [v2:172.21.15.133:6826/1557353480,v1:172.21.15.133:6827/1557353480] up:boot 2024-02-20T07:55:27.047 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:55:26 smithi183 ceph-mon[35832]: mds.? [v2:172.21.15.133:6828/3316004709,v1:172.21.15.133:6829/3316004709] up:resolve 2024-02-20T07:55:27.047 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:55:26 smithi183 ceph-mon[35832]: fsmap cephfs:2/2 {0=cephfs.smithi133.cmofvy=up:resolve,1=cephfs.smithi183.eipfta=up:active} 1 up:standby-replay 1 up:standby 2024-02-20T07:55:27.048 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:55:26 smithi183 ceph-mon[35832]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi133.itlmbm"}]: dispatch 2024-02-20T07:55:27.048 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:55:26 smithi183 ceph-mon[35832]: from='client.? 172.21.15.133:0/1005587274' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-02-20T07:55:27.361 DEBUG:teuthology.orchestra.run.smithi133:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4537327c-cfc4-11ee-95bb-87774f69a715 -- bash -c 'ceph --format=json fs dump | jq -e ".filesystems | length == 1"' 2024-02-20T07:55:27.939 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:55:27 smithi133 ceph-mon[29117]: mds.? [v2:172.21.15.133:6828/3316004709,v1:172.21.15.133:6829/3316004709] up:reconnect 2024-02-20T07:55:27.940 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:55:27 smithi133 ceph-mon[29117]: mds.? [v2:172.21.15.183:6826/1729948973,v1:172.21.15.183:6827/1729948973] up:standby-replay 2024-02-20T07:55:27.940 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:55:27 smithi133 ceph-mon[29117]: fsmap cephfs:2/2 {0=cephfs.smithi133.cmofvy=up:reconnect,1=cephfs.smithi183.eipfta=up:active} 1 up:standby-replay 1 up:standby 2024-02-20T07:55:27.940 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:55:27 smithi133 ceph-mon[29117]: mds.? [v2:172.21.15.133:6828/3316004709,v1:172.21.15.133:6829/3316004709] up:rejoin 2024-02-20T07:55:27.940 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:55:27 smithi133 ceph-mon[29117]: fsmap cephfs:2/2 {0=cephfs.smithi133.cmofvy=up:rejoin,1=cephfs.smithi183.eipfta=up:active} 1 up:standby-replay 1 up:standby 2024-02-20T07:55:27.940 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:55:27 smithi133 ceph-mon[29117]: daemon mds.cephfs.smithi133.cmofvy is now active in filesystem cephfs as rank 0 2024-02-20T07:55:28.047 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:55:27 smithi183 ceph-mon[35832]: mds.? [v2:172.21.15.133:6828/3316004709,v1:172.21.15.133:6829/3316004709] up:reconnect 2024-02-20T07:55:28.047 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:55:27 smithi183 ceph-mon[35832]: mds.? [v2:172.21.15.183:6826/1729948973,v1:172.21.15.183:6827/1729948973] up:standby-replay 2024-02-20T07:55:28.047 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:55:27 smithi183 ceph-mon[35832]: fsmap cephfs:2/2 {0=cephfs.smithi133.cmofvy=up:reconnect,1=cephfs.smithi183.eipfta=up:active} 1 up:standby-replay 1 up:standby 2024-02-20T07:55:28.047 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:55:27 smithi183 ceph-mon[35832]: mds.? [v2:172.21.15.133:6828/3316004709,v1:172.21.15.133:6829/3316004709] up:rejoin 2024-02-20T07:55:28.047 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:55:27 smithi183 ceph-mon[35832]: fsmap cephfs:2/2 {0=cephfs.smithi133.cmofvy=up:rejoin,1=cephfs.smithi183.eipfta=up:active} 1 up:standby-replay 1 up:standby 2024-02-20T07:55:28.048 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:55:27 smithi183 ceph-mon[35832]: daemon mds.cephfs.smithi133.cmofvy is now active in filesystem cephfs as rank 0 2024-02-20T07:55:28.336 INFO:teuthology.orchestra.run.smithi133.stderr:Inferring config /var/lib/ceph/4537327c-cfc4-11ee-95bb-87774f69a715/mon.smithi133/config 2024-02-20T07:55:28.711 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:55:28 smithi133 ceph-mon[29117]: pgmap v186: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 1.6 KiB/s wr, 8 op/s 2024-02-20T07:55:28.711 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:55:28 smithi133 ceph-mon[29117]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-02-20T07:55:28.711 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:55:28 smithi133 ceph-mon[29117]: mds.? [v2:172.21.15.133:6828/3316004709,v1:172.21.15.133:6829/3316004709] up:active 2024-02-20T07:55:28.711 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:55:28 smithi133 ceph-mon[29117]: fsmap cephfs:2 {0=cephfs.smithi133.cmofvy=up:active,1=cephfs.smithi183.eipfta=up:active} 1 up:standby-replay 1 up:standby 2024-02-20T07:55:28.711 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:55:28 smithi133 ceph-mon[29117]: fsmap cephfs:2 {0=cephfs.smithi133.cmofvy=up:active,1=cephfs.smithi183.eipfta=up:active} 2 up:standby-replay 2024-02-20T07:55:29.047 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:55:28 smithi183 ceph-mon[35832]: pgmap v186: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 1.6 KiB/s wr, 8 op/s 2024-02-20T07:55:29.047 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:55:28 smithi183 ceph-mon[35832]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-02-20T07:55:29.047 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:55:28 smithi183 ceph-mon[35832]: mds.? [v2:172.21.15.133:6828/3316004709,v1:172.21.15.133:6829/3316004709] up:active 2024-02-20T07:55:29.047 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:55:28 smithi183 ceph-mon[35832]: fsmap cephfs:2 {0=cephfs.smithi133.cmofvy=up:active,1=cephfs.smithi183.eipfta=up:active} 1 up:standby-replay 1 up:standby 2024-02-20T07:55:29.047 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:55:28 smithi183 ceph-mon[35832]: fsmap cephfs:2 {0=cephfs.smithi133.cmofvy=up:active,1=cephfs.smithi183.eipfta=up:active} 2 up:standby-replay 2024-02-20T07:55:30.230 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:30.229+0000 7f0fef0d7700 1 -- 172.21.15.133:0/241404929 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f0fe80fe0e0 msgr2=0x7f0fe80fe550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:55:30.230 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:30.229+0000 7f0fef0d7700 1 --2- 172.21.15.133:0/241404929 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f0fe80fe0e0 0x7f0fe80fe550 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f0fd4004940 tx=0x7f0fd4009e30 comp rx=0 tx=0).stop 2024-02-20T07:55:30.230 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:30.229+0000 7f0fef0d7700 1 -- 172.21.15.133:0/241404929 shutdown_connections 2024-02-20T07:55:30.230 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:30.229+0000 7f0fef0d7700 1 --2- 172.21.15.133:0/241404929 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f0fe80fe0e0 0x7f0fe80fe550 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:55:30.231 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:30.229+0000 7f0fef0d7700 1 --2- 172.21.15.133:0/241404929 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0fe80fcef0 0x7f0fe80fd320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:55:30.231 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:30.229+0000 7f0fef0d7700 1 -- 172.21.15.133:0/241404929 >> 172.21.15.133:0/241404929 conn(0x7f0fe80f8620 msgr2=0x7f0fe80faa30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:55:30.231 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:30.229+0000 7f0fef0d7700 1 -- 172.21.15.133:0/241404929 shutdown_connections 2024-02-20T07:55:30.231 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:30.229+0000 7f0fef0d7700 1 -- 172.21.15.133:0/241404929 wait complete. 2024-02-20T07:55:30.231 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:30.230+0000 7f0fef0d7700 1 Processor -- start 2024-02-20T07:55:30.231 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:30.230+0000 7f0fef0d7700 1 -- start start 2024-02-20T07:55:30.232 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:30.230+0000 7f0fef0d7700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0fe80fcef0 0x7f0fe810d960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:55:30.232 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:30.230+0000 7f0fef0d7700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f0fe80fe0e0 0x7f0fe810dea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:55:30.233 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:30.230+0000 7f0fef0d7700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fe810e4a0 con 0x7f0fe80fe0e0 2024-02-20T07:55:30.233 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:30.230+0000 7f0fef0d7700 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fe810ed40 con 0x7f0fe80fcef0 2024-02-20T07:55:30.233 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:30.230+0000 7f0fe7fff700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f0fe80fe0e0 0x7f0fe810dea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:55:30.233 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:30.230+0000 7f0fe7fff700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f0fe80fe0e0 0x7f0fe810dea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.133:46438/0 (socket says 172.21.15.133:46438) 2024-02-20T07:55:30.233 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:30.230+0000 7f0fe7fff700 1 -- 172.21.15.133:0/1241330615 learned_addr learned my addr 172.21.15.133:0/1241330615 (peer_addr_for_me v2:172.21.15.133:0/0) 2024-02-20T07:55:30.233 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:30.230+0000 7f0fece73700 1 --2- 172.21.15.133:0/1241330615 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0fe80fcef0 0x7f0fe810d960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:55:30.233 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:30.231+0000 7f0fe7fff700 1 -- 172.21.15.133:0/1241330615 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0fe80fcef0 msgr2=0x7f0fe810d960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:55:30.233 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:30.231+0000 7f0fe7fff700 1 --2- 172.21.15.133:0/1241330615 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0fe80fcef0 0x7f0fe810d960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:55:30.233 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:30.231+0000 7f0fe7fff700 1 -- 172.21.15.133:0/1241330615 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0fd4005040 con 0x7f0fe80fe0e0 2024-02-20T07:55:30.233 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:30.231+0000 7f0fe7fff700 1 --2- 172.21.15.133:0/1241330615 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f0fe80fe0e0 0x7f0fe810dea0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f0fd40093b0 tx=0x7f0fd4009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:55:30.233 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:30.231+0000 7f0fe57fa700 1 -- 172.21.15.133:0/1241330615 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0fd400e040 con 0x7f0fe80fe0e0 2024-02-20T07:55:30.233 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:30.231+0000 7f0fe57fa700 1 -- 172.21.15.133:0/1241330615 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0fd400bcc0 con 0x7f0fe80fe0e0 2024-02-20T07:55:30.233 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:30.231+0000 7f0fef0d7700 1 -- 172.21.15.133:0/1241330615 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0fe810efc0 con 0x7f0fe80fe0e0 2024-02-20T07:55:30.233 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:30.231+0000 7f0fe57fa700 1 -- 172.21.15.133:0/1241330615 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0fd401c630 con 0x7f0fe80fe0e0 2024-02-20T07:55:30.234 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:30.231+0000 7f0fef0d7700 1 -- 172.21.15.133:0/1241330615 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0fe810f500 con 0x7f0fe80fe0e0 2024-02-20T07:55:30.234 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:30.232+0000 7f0fef0d7700 1 -- 172.21.15.133:0/1241330615 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0fe804bed0 con 0x7f0fe80fe0e0 2024-02-20T07:55:30.234 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:30.232+0000 7f0fe57fa700 1 -- 172.21.15.133:0/1241330615 <== mon.0 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7f0fd4013980 con 0x7f0fe80fe0e0 2024-02-20T07:55:30.234 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:30.233+0000 7f0fe57fa700 1 --2- 172.21.15.133:0/1241330615 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7f0fd80692a0 0x7f0fd806b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:55:30.234 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:30.233+0000 7f0fece73700 1 --2- 172.21.15.133:0/1241330615 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7f0fd80692a0 0x7f0fd806b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:55:30.235 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:30.233+0000 7f0fe57fa700 1 -- 172.21.15.133:0/1241330615 <== mon.0 v2:172.21.15.133:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f0fd4088570 con 0x7f0fe80fe0e0 2024-02-20T07:55:30.235 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:30.233+0000 7f0fece73700 1 --2- 172.21.15.133:0/1241330615 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7f0fd80692a0 0x7f0fd806b750 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f0fe80fcc30 tx=0x7f0fdc00b040 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:55:30.238 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:30.237+0000 7f0fe57fa700 1 -- 172.21.15.133:0/1241330615 <== mon.0 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f0fd40582e0 con 0x7f0fe80fe0e0 2024-02-20T07:55:30.467 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:30.466+0000 7f0fef0d7700 1 -- 172.21.15.133:0/1241330615 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0fe8105a30 con 0x7f0fe80fe0e0 2024-02-20T07:55:30.467 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:30.467+0000 7f0fe57fa700 1 -- 172.21.15.133:0/1241330615 <== mon.0 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 19 v19) v1 ==== 94+0+4731 (secure 0 0 0) 0x7f0fd4057e70 con 0x7f0fe80fe0e0 2024-02-20T07:55:30.470 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:30.469+0000 7f0fef0d7700 1 -- 172.21.15.133:0/1241330615 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7f0fd80692a0 msgr2=0x7f0fd806b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:55:30.470 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:30.469+0000 7f0fef0d7700 1 --2- 172.21.15.133:0/1241330615 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7f0fd80692a0 0x7f0fd806b750 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f0fe80fcc30 tx=0x7f0fdc00b040 comp rx=0 tx=0).stop 2024-02-20T07:55:30.470 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:30.469+0000 7f0fef0d7700 1 -- 172.21.15.133:0/1241330615 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f0fe80fe0e0 msgr2=0x7f0fe810dea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:55:30.470 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:30.469+0000 7f0fef0d7700 1 --2- 172.21.15.133:0/1241330615 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f0fe80fe0e0 0x7f0fe810dea0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f0fd40093b0 tx=0x7f0fd4009e30 comp rx=0 tx=0).stop 2024-02-20T07:55:30.470 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:30.469+0000 7f0fef0d7700 1 -- 172.21.15.133:0/1241330615 shutdown_connections 2024-02-20T07:55:30.471 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:30.469+0000 7f0fef0d7700 1 --2- 172.21.15.133:0/1241330615 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7f0fd80692a0 0x7f0fd806b750 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:55:30.471 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:30.469+0000 7f0fef0d7700 1 --2- 172.21.15.133:0/1241330615 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0fe80fcef0 0x7f0fe810d960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:55:30.471 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:30.469+0000 7f0fef0d7700 1 --2- 172.21.15.133:0/1241330615 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f0fe80fe0e0 0x7f0fe810dea0 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:55:30.471 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:30.470+0000 7f0fef0d7700 1 -- 172.21.15.133:0/1241330615 >> 172.21.15.133:0/1241330615 conn(0x7f0fe80f8620 msgr2=0x7f0fe81013a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:55:30.471 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:30.470+0000 7f0fef0d7700 1 -- 172.21.15.133:0/1241330615 shutdown_connections 2024-02-20T07:55:30.471 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:30.470+0000 7f0fef0d7700 1 -- 172.21.15.133:0/1241330615 wait complete. 2024-02-20T07:55:30.471 INFO:teuthology.orchestra.run.smithi133.stderr:dumped fsmap epoch 19 2024-02-20T07:55:30.481 INFO:teuthology.orchestra.run.smithi133.stdout:true 2024-02-20T07:55:30.741 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:55:30 smithi133 ceph-mon[29117]: pgmap v187: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.7 KiB/s rd, 614 B/s wr, 5 op/s 2024-02-20T07:55:30.741 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:55:30 smithi133 ceph-mon[29117]: from='client.? 172.21.15.133:0/1241330615' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-02-20T07:55:31.047 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:55:30 smithi183 ceph-mon[35832]: pgmap v187: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.7 KiB/s rd, 614 B/s wr, 5 op/s 2024-02-20T07:55:31.047 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:55:30 smithi183 ceph-mon[35832]: from='client.? 172.21.15.133:0/1241330615' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-02-20T07:55:31.090 DEBUG:teuthology.orchestra.run.smithi133:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4537327c-cfc4-11ee-95bb-87774f69a715 -- bash -c 'while ! ceph --format=json mds versions | jq -e ". | add == 4"; do sleep 1; done' 2024-02-20T07:55:31.368 INFO:teuthology.orchestra.run.smithi133.stderr:Inferring config /var/lib/ceph/4537327c-cfc4-11ee-95bb-87774f69a715/mon.smithi133/config 2024-02-20T07:55:32.578 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:55:32 smithi183 ceph-mon[35832]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:55:32.578 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:55:32 smithi183 ceph-mon[35832]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:55:32.578 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:55:32 smithi183 ceph-mon[35832]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-20T07:55:32.578 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:55:32 smithi183 ceph-mon[35832]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-20T07:55:32.579 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:55:32 smithi183 ceph-mon[35832]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:55:32.579 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:55:32 smithi183 ceph-mon[35832]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-20T07:55:32.579 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:55:32 smithi183 ceph-mon[35832]: pgmap v188: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 5.9 KiB/s rd, 0 B/s wr, 6 op/s 2024-02-20T07:55:32.671 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:55:32 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:55:32.671 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:55:32 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:55:32.671 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:55:32 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-20T07:55:32.671 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:55:32 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-20T07:55:32.671 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:55:32 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:55:32.671 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:55:32 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-20T07:55:32.671 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:55:32 smithi133 ceph-mon[29117]: pgmap v188: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 5.9 KiB/s rd, 0 B/s wr, 6 op/s 2024-02-20T07:55:32.887 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:32.885+0000 7fb79809c700 1 -- 172.21.15.133:0/3739104353 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb7900fcef0 msgr2=0x7fb7900fd320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:55:32.887 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:32.885+0000 7fb79809c700 1 --2- 172.21.15.133:0/3739104353 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb7900fcef0 0x7fb7900fd320 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fb78c004940 tx=0x7fb78c009e30 comp rx=0 tx=0).stop 2024-02-20T07:55:32.887 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:32.885+0000 7fb79809c700 1 -- 172.21.15.133:0/3739104353 shutdown_connections 2024-02-20T07:55:32.887 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:32.885+0000 7fb79809c700 1 --2- 172.21.15.133:0/3739104353 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb7900fe0e0 0x7fb7900fe550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:55:32.888 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:32.885+0000 7fb79809c700 1 --2- 172.21.15.133:0/3739104353 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb7900fcef0 0x7fb7900fd320 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:55:32.888 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:32.885+0000 7fb79809c700 1 -- 172.21.15.133:0/3739104353 >> 172.21.15.133:0/3739104353 conn(0x7fb7900f8620 msgr2=0x7fb7900faa30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:55:32.888 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:32.885+0000 7fb79809c700 1 -- 172.21.15.133:0/3739104353 shutdown_connections 2024-02-20T07:55:32.888 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:32.885+0000 7fb79809c700 1 -- 172.21.15.133:0/3739104353 wait complete. 2024-02-20T07:55:32.888 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:32.885+0000 7fb79809c700 1 Processor -- start 2024-02-20T07:55:32.888 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:32.885+0000 7fb79809c700 1 -- start start 2024-02-20T07:55:32.888 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:32.886+0000 7fb79809c700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb7900fcef0 0x7fb790071e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:55:32.888 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:32.886+0000 7fb79809c700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb7900fe0e0 0x7fb7900702e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:55:32.888 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:32.886+0000 7fb79809c700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7900708e0 con 0x7fb7900fcef0 2024-02-20T07:55:32.888 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:32.886+0000 7fb79809c700 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb790070a50 con 0x7fb7900fe0e0 2024-02-20T07:55:32.888 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:32.886+0000 7fb795e38700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb7900fcef0 0x7fb790071e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:55:32.888 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:32.886+0000 7fb795e38700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb7900fcef0 0x7fb790071e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.133:46466/0 (socket says 172.21.15.133:46466) 2024-02-20T07:55:32.889 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:32.886+0000 7fb795e38700 1 -- 172.21.15.133:0/2748201619 learned_addr learned my addr 172.21.15.133:0/2748201619 (peer_addr_for_me v2:172.21.15.133:0/0) 2024-02-20T07:55:32.889 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:32.886+0000 7fb795637700 1 --2- 172.21.15.133:0/2748201619 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb7900fe0e0 0x7fb7900702e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:55:32.889 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:32.886+0000 7fb795e38700 1 -- 172.21.15.133:0/2748201619 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb7900fe0e0 msgr2=0x7fb7900702e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:55:32.889 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:32.886+0000 7fb795e38700 1 --2- 172.21.15.133:0/2748201619 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb7900fe0e0 0x7fb7900702e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:55:32.889 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:32.886+0000 7fb795e38700 1 -- 172.21.15.133:0/2748201619 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb78c005040 con 0x7fb7900fcef0 2024-02-20T07:55:32.889 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:32.886+0000 7fb795e38700 1 --2- 172.21.15.133:0/2748201619 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb7900fcef0 0x7fb790071e60 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fb78c009380 tx=0x7fb78c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:55:32.889 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:32.886+0000 7fb7867fc700 1 -- 172.21.15.133:0/2748201619 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb78c00e040 con 0x7fb7900fcef0 2024-02-20T07:55:32.889 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:32.886+0000 7fb7867fc700 1 -- 172.21.15.133:0/2748201619 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb78c00bbf0 con 0x7fb7900fcef0 2024-02-20T07:55:32.889 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:32.886+0000 7fb79809c700 1 -- 172.21.15.133:0/2748201619 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb790070cd0 con 0x7fb7900fcef0 2024-02-20T07:55:32.889 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:32.886+0000 7fb7867fc700 1 -- 172.21.15.133:0/2748201619 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb78c01c630 con 0x7fb7900fcef0 2024-02-20T07:55:32.889 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:32.886+0000 7fb79809c700 1 -- 172.21.15.133:0/2748201619 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb7901159a0 con 0x7fb7900fcef0 2024-02-20T07:55:32.890 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:32.887+0000 7fb79809c700 1 -- 172.21.15.133:0/2748201619 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb79004bed0 con 0x7fb7900fcef0 2024-02-20T07:55:32.890 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:32.888+0000 7fb7867fc700 1 -- 172.21.15.133:0/2748201619 <== mon.0 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7fb78c013980 con 0x7fb7900fcef0 2024-02-20T07:55:32.890 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:32.888+0000 7fb7867fc700 1 --2- 172.21.15.133:0/2748201619 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7fb77c0692a0 0x7fb77c06b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:55:32.890 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:32.888+0000 7fb7867fc700 1 -- 172.21.15.133:0/2748201619 <== mon.0 v2:172.21.15.133:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5550+0+0 (secure 0 0 0) 0x7fb78c088470 con 0x7fb7900fcef0 2024-02-20T07:55:32.890 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:32.888+0000 7fb795637700 1 --2- 172.21.15.133:0/2748201619 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7fb77c0692a0 0x7fb77c06b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:55:32.890 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:32.888+0000 7fb795637700 1 --2- 172.21.15.133:0/2748201619 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7fb77c0692a0 0x7fb77c06b750 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fb780004f40 tx=0x7fb78000b040 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:55:32.893 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:32.892+0000 7fb7867fc700 1 -- 172.21.15.133:0/2748201619 <== mon.0 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7fb78c058290 con 0x7fb7900fcef0 2024-02-20T07:55:33.177 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:33.176+0000 7fb79809c700 1 -- 172.21.15.133:0/2748201619 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "mds versions", "format": "json"} v 0) v1 -- 0x7fb790105a30 con 0x7fb7900fcef0 2024-02-20T07:55:33.177 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:33.177+0000 7fb7867fc700 1 -- 172.21.15.133:0/2748201619 <== mon.0 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "mds versions", "format": "json"}]=0 v19) v1 ==== 78+0+99 (secure 0 0 0) 0x7fb78c057e20 con 0x7fb7900fcef0 2024-02-20T07:55:33.179 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:33.178+0000 7fb79809c700 1 -- 172.21.15.133:0/2748201619 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7fb77c0692a0 msgr2=0x7fb77c06b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:55:33.179 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:33.179+0000 7fb79809c700 1 --2- 172.21.15.133:0/2748201619 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7fb77c0692a0 0x7fb77c06b750 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fb780004f40 tx=0x7fb78000b040 comp rx=0 tx=0).stop 2024-02-20T07:55:33.179 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:33.179+0000 7fb79809c700 1 -- 172.21.15.133:0/2748201619 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb7900fcef0 msgr2=0x7fb790071e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:55:33.179 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:33.179+0000 7fb79809c700 1 --2- 172.21.15.133:0/2748201619 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb7900fcef0 0x7fb790071e60 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fb78c009380 tx=0x7fb78c009e30 comp rx=0 tx=0).stop 2024-02-20T07:55:33.179 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:33.179+0000 7fb79809c700 1 -- 172.21.15.133:0/2748201619 shutdown_connections 2024-02-20T07:55:33.179 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:33.179+0000 7fb79809c700 1 --2- 172.21.15.133:0/2748201619 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb7900fcef0 0x7fb790071e60 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:55:33.180 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:33.179+0000 7fb79809c700 1 --2- 172.21.15.133:0/2748201619 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7fb77c0692a0 0x7fb77c06b750 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:55:33.180 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:33.179+0000 7fb79809c700 1 --2- 172.21.15.133:0/2748201619 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb7900fe0e0 0x7fb7900702e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:55:33.180 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:33.179+0000 7fb79809c700 1 -- 172.21.15.133:0/2748201619 >> 172.21.15.133:0/2748201619 conn(0x7fb7900f8620 msgr2=0x7fb7901013a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:55:33.180 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:33.179+0000 7fb79809c700 1 -- 172.21.15.133:0/2748201619 shutdown_connections 2024-02-20T07:55:33.180 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:33.179+0000 7fb79809c700 1 -- 172.21.15.133:0/2748201619 wait complete. 2024-02-20T07:55:33.189 INFO:teuthology.orchestra.run.smithi133.stdout:true 2024-02-20T07:55:33.491 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:55:33 smithi133 ceph-mon[29117]: from='client.? 172.21.15.133:0/2748201619' entity='client.admin' cmd=[{"prefix": "mds versions", "format": "json"}]: dispatch 2024-02-20T07:55:33.797 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:55:33 smithi183 ceph-mon[35832]: from='client.? 172.21.15.133:0/2748201619' entity='client.admin' cmd=[{"prefix": "mds versions", "format": "json"}]: dispatch 2024-02-20T07:55:33.840 INFO:teuthology.run_tasks:Running task fs.pre_upgrade_save... 2024-02-20T07:55:33.855 DEBUG:teuthology.orchestra.run.smithi133:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-20T07:55:33.992 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:33.989+0000 7f9b81225640 1 -- 172.21.15.133:0/3549882655 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f9b7c0fc3b0 msgr2=0x7f9b7c0fc820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:55:33.992 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:33.989+0000 7f9b81225640 1 --2- 172.21.15.133:0/3549882655 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f9b7c0fc3b0 0x7f9b7c0fc820 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f9b700048d0 tx=0x7f9b70009da0 comp rx=0 tx=0).stop 2024-02-20T07:55:33.992 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:33.989+0000 7f9b81225640 1 -- 172.21.15.133:0/3549882655 shutdown_connections 2024-02-20T07:55:33.992 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:33.989+0000 7f9b81225640 1 --2- 172.21.15.133:0/3549882655 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f9b7c0fc3b0 0x7f9b7c0fc820 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:55:33.992 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:33.989+0000 7f9b81225640 1 --2- 172.21.15.133:0/3549882655 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9b7c0fb180 0x7f9b7c0fb590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:55:33.992 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:33.989+0000 7f9b81225640 1 -- 172.21.15.133:0/3549882655 >> 172.21.15.133:0/3549882655 conn(0x7f9b7c0f6860 msgr2=0x7f9b7c0f8cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:55:33.992 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:33.990+0000 7f9b81225640 1 -- 172.21.15.133:0/3549882655 shutdown_connections 2024-02-20T07:55:33.992 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:33.990+0000 7f9b81225640 1 -- 172.21.15.133:0/3549882655 wait complete. 2024-02-20T07:55:33.993 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:33.990+0000 7f9b81225640 1 Processor -- start 2024-02-20T07:55:33.993 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:33.990+0000 7f9b81225640 1 -- start start 2024-02-20T07:55:33.993 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:33.990+0000 7f9b81225640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9b7c0fb180 0x7f9b7c0ff070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:55:33.993 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:33.990+0000 7f9b81225640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f9b7c0fc3b0 0x7f9b7c0fd6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:55:33.993 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:33.990+0000 7f9b81225640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b7c0fdc30 con 0x7f9b7c0fc3b0 2024-02-20T07:55:33.993 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:33.990+0000 7f9b81225640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b7c0fdda0 con 0x7f9b7c0fb180 2024-02-20T07:55:33.993 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:33.990+0000 7f9b7a59c640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f9b7c0fc3b0 0x7f9b7c0fd6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:55:33.993 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:33.990+0000 7f9b7a59c640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f9b7c0fc3b0 0x7f9b7c0fd6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.133:46476/0 (socket says 172.21.15.133:46476) 2024-02-20T07:55:33.993 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:33.990+0000 7f9b7a59c640 1 -- 172.21.15.133:0/3762827397 learned_addr learned my addr 172.21.15.133:0/3762827397 (peer_addr_for_me v2:172.21.15.133:0/0) 2024-02-20T07:55:33.993 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:33.990+0000 7f9b7ad9d640 1 --2- 172.21.15.133:0/3762827397 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9b7c0fb180 0x7f9b7c0ff070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:55:33.993 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:33.991+0000 7f9b7a59c640 1 -- 172.21.15.133:0/3762827397 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9b7c0fb180 msgr2=0x7f9b7c0ff070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:55:33.994 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:33.991+0000 7f9b7a59c640 1 --2- 172.21.15.133:0/3762827397 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9b7c0fb180 0x7f9b7c0ff070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:55:33.994 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:33.991+0000 7f9b7a59c640 1 -- 172.21.15.133:0/3762827397 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b70005040 con 0x7f9b7c0fc3b0 2024-02-20T07:55:33.994 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:33.991+0000 7f9b7ad9d640 1 --2- 172.21.15.133:0/3762827397 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9b7c0fb180 0x7f9b7c0ff070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-20T07:55:33.994 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:33.991+0000 7f9b7a59c640 1 --2- 172.21.15.133:0/3762827397 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f9b7c0fc3b0 0x7f9b7c0fd6f0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f9b7002bda0 tx=0x7f9b70009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:55:33.994 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:33.991+0000 7f9b637fe640 1 -- 172.21.15.133:0/3762827397 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9b700094c0 con 0x7f9b7c0fc3b0 2024-02-20T07:55:33.994 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:33.991+0000 7f9b81225640 1 -- 172.21.15.133:0/3762827397 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9b7c0fe020 con 0x7f9b7c0fc3b0 2024-02-20T07:55:33.994 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:33.991+0000 7f9b637fe640 1 -- 172.21.15.133:0/3762827397 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9b70009620 con 0x7f9b7c0fc3b0 2024-02-20T07:55:33.994 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:33.991+0000 7f9b637fe640 1 -- 172.21.15.133:0/3762827397 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9b7003b640 con 0x7f9b7c0fc3b0 2024-02-20T07:55:33.994 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:33.991+0000 7f9b81225640 1 -- 172.21.15.133:0/3762827397 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9b7c0fe540 con 0x7f9b7c0fc3b0 2024-02-20T07:55:33.995 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:33.992+0000 7f9b81225640 1 -- 172.21.15.133:0/3762827397 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9b7c103cf0 con 0x7f9b7c0fc3b0 2024-02-20T07:55:33.995 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:33.992+0000 7f9b637fe640 1 -- 172.21.15.133:0/3762827397 <== mon.0 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7f9b7002f070 con 0x7f9b7c0fc3b0 2024-02-20T07:55:33.995 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:33.993+0000 7f9b637fe640 1 --2- 172.21.15.133:0/3762827397 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7f9b64069210 0x7f9b6406b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:55:33.995 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:33.993+0000 7f9b637fe640 1 -- 172.21.15.133:0/3762827397 <== mon.0 v2:172.21.15.133:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f9b700a9780 con 0x7f9b7c0fc3b0 2024-02-20T07:55:33.995 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:33.993+0000 7f9b7ad9d640 1 --2- 172.21.15.133:0/3762827397 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7f9b64069210 0x7f9b6406b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:55:33.995 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:33.993+0000 7f9b7ad9d640 1 --2- 172.21.15.133:0/3762827397 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7f9b64069210 0x7f9b6406b6c0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f9b6c004fd0 tx=0x7f9b6c00b040 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:55:33.997 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:33.997+0000 7f9b637fe640 1 -- 172.21.15.133:0/3762827397 <== mon.0 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f9b700759e0 con 0x7f9b7c0fc3b0 2024-02-20T07:55:34.166 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:34.166+0000 7f9b81225640 1 -- 172.21.15.133:0/3762827397 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9b7c114c90 con 0x7f9b7c0fc3b0 2024-02-20T07:55:34.167 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:34.167+0000 7f9b637fe640 1 -- 172.21.15.133:0/3762827397 <== mon.0 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v46) v1 ==== 74+0+14426 (secure 0 0 0) 0x7f9b70079000 con 0x7f9b7c0fc3b0 2024-02-20T07:55:34.167 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-02-20T07:55:34.168 INFO:teuthology.orchestra.run.smithi133.stdout:{"epoch":46,"fsid":"4537327c-cfc4-11ee-95bb-87774f69a715","created":"2024-02-20T07:48:13.550549+0000","modified":"2024-02-20T07:55:24.626511+0000","last_up_change":"2024-02-20T07:53:49.684637+0000","last_in_change":"2024-02-20T07:53:30.651509+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":14,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-20T07:52:35.839244+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"cephfs.cephfs.meta","create_time":"2024-02-20T07:55:07.331887+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"44","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":3,"pool_name":"cephfs.cephfs.data","create_time":"2024-02-20T07:55:07.913654+0000","flags":131073,"flags_names":"hashpspool,bulk","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"45","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"0a20b241-99e8-4e0b-ba5e-be07c8fdff94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6802","nonce":261892236},{"type":"v1","addr":"172.21.15.133:6803","nonce":261892236}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6804","nonce":261892236},{"type":"v1","addr":"172.21.15.133:6805","nonce":261892236}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6808","nonce":261892236},{"type":"v1","addr":"172.21.15.133:6809","nonce":261892236}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6806","nonce":261892236},{"type":"v1","addr":"172.21.15.133:6807","nonce":261892236}]},"public_addr":"172.21.15.133:6803/261892236","cluster_addr":"172.21.15.133:6805/261892236","heartbeat_back_addr":"172.21.15.133:6809/261892236","heartbeat_front_addr":"172.21.15.133:6807/261892236","state":["exists","up"]},{"osd":1,"uuid":"2b1b8e8f-74b4-4d41-8753-502296eb87df","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6810","nonce":1652695599},{"type":"v1","addr":"172.21.15.133:6811","nonce":1652695599}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6812","nonce":1652695599},{"type":"v1","addr":"172.21.15.133:6813","nonce":1652695599}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6816","nonce":1652695599},{"type":"v1","addr":"172.21.15.133:6817","nonce":1652695599}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6814","nonce":1652695599},{"type":"v1","addr":"172.21.15.133:6815","nonce":1652695599}]},"public_addr":"172.21.15.133:6811/1652695599","cluster_addr":"172.21.15.133:6813/1652695599","heartbeat_back_addr":"172.21.15.133:6817/1652695599","heartbeat_front_addr":"172.21.15.133:6815/1652695599","state":["exists","up"]},{"osd":2,"uuid":"95848108-ed6d-4030-9534-d687a769350f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6818","nonce":3838005412},{"type":"v1","addr":"172.21.15.133:6819","nonce":3838005412}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6820","nonce":3838005412},{"type":"v1","addr":"172.21.15.133:6821","nonce":3838005412}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6824","nonce":3838005412},{"type":"v1","addr":"172.21.15.133:6825","nonce":3838005412}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6822","nonce":3838005412},{"type":"v1","addr":"172.21.15.133:6823","nonce":3838005412}]},"public_addr":"172.21.15.133:6819/3838005412","cluster_addr":"172.21.15.133:6821/3838005412","heartbeat_back_addr":"172.21.15.133:6825/3838005412","heartbeat_front_addr":"172.21.15.133:6823/3838005412","state":["exists","up"]},{"osd":3,"uuid":"c2015ce7-f7fb-428d-b42f-e5fb665933e0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6800","nonce":1053980618},{"type":"v1","addr":"172.21.15.183:6801","nonce":1053980618}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6802","nonce":1053980618},{"type":"v1","addr":"172.21.15.183:6803","nonce":1053980618}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6806","nonce":1053980618},{"type":"v1","addr":"172.21.15.183:6807","nonce":1053980618}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6804","nonce":1053980618},{"type":"v1","addr":"172.21.15.183:6805","nonce":1053980618}]},"public_addr":"172.21.15.183:6801/1053980618","cluster_addr":"172.21.15.183:6803/1053980618","heartbeat_back_addr":"172.21.15.183:6807/1053980618","heartbeat_front_addr":"172.21.15.183:6805/1053980618","state":["exists","up"]},{"osd":4,"uuid":"55148881-b92f-40e6-bfe8-b8320406c96a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6808","nonce":3697465174},{"type":"v1","addr":"172.21.15.183:6809","nonce":3697465174}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6810","nonce":3697465174},{"type":"v1","addr":"172.21.15.183:6811","nonce":3697465174}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6814","nonce":3697465174},{"type":"v1","addr":"172.21.15.183:6815","nonce":3697465174}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6812","nonce":3697465174},{"type":"v1","addr":"172.21.15.183:6813","nonce":3697465174}]},"public_addr":"172.21.15.183:6809/3697465174","cluster_addr":"172.21.15.183:6811/3697465174","heartbeat_back_addr":"172.21.15.183:6815/3697465174","heartbeat_front_addr":"172.21.15.183:6813/3697465174","state":["exists","up"]},{"osd":5,"uuid":"ee5510f9-fab7-4aa3-8372-3856a0787e92","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6816","nonce":81997900},{"type":"v1","addr":"172.21.15.183:6817","nonce":81997900}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6818","nonce":81997900},{"type":"v1","addr":"172.21.15.183:6819","nonce":81997900}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6822","nonce":81997900},{"type":"v1","addr":"172.21.15.183:6823","nonce":81997900}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6820","nonce":81997900},{"type":"v1","addr":"172.21.15.183:6821","nonce":81997900}]},"public_addr":"172.21.15.183:6817/81997900","cluster_addr":"172.21.15.183:6819/81997900","heartbeat_back_addr":"172.21.15.183:6823/81997900","heartbeat_front_addr":"172.21.15.183:6821/81997900","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-02-20T07:51:38.297291+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-02-20T07:52:05.983942+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-02-20T07:52:33.478124+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-02-20T07:52:56.569795+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-02-20T07:53:22.416562+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-02-20T07:53:47.770454+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.133:0/1481070311":"2024-02-21T07:48:49.130220+0000","172.21.15.133:0/2831283357":"2024-02-21T07:49:19.684526+0000","172.21.15.133:0/2763668929":"2024-02-21T07:48:49.130220+0000","172.21.15.133:0/3354686539":"2024-02-21T07:49:19.684526+0000","172.21.15.133:6801/3944459195":"2024-02-21T07:50:23.523666+0000","172.21.15.133:0/129232921":"2024-02-21T07:48:49.130220+0000","172.21.15.133:0/2942023244":"2024-02-21T07:50:23.523666+0000","172.21.15.133:0/3813170035":"2024-02-21T07:49:19.684526+0000","172.21.15.133:6827/461346469":"2024-02-21T07:55:24.626490+0000","172.21.15.133:0/3014285121":"2024-02-21T07:50:23.523666+0000","172.21.15.133:6800/1670376163":"2024-02-21T07:49:19.684526+0000","172.21.15.133:6801/3480413474":"2024-02-21T07:48:49.130220+0000","172.21.15.133:6826/461346469":"2024-02-21T07:55:24.626490+0000","172.21.15.133:6800/3480413474":"2024-02-21T07:48:49.130220+0000","172.21.15.133:6801/1670376163":"2024-02-21T07:49:19.684526+0000","172.21.15.133:0/1714017955":"2024-02-21T07:50:23.523666+0000","172.21.15.133:6800/3944459195":"2024-02-21T07:50:23.523666+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-20T07:55:34.169 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:34.169+0000 7f9b81225640 1 -- 172.21.15.133:0/3762827397 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7f9b64069210 msgr2=0x7f9b6406b6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:55:34.170 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:34.169+0000 7f9b81225640 1 --2- 172.21.15.133:0/3762827397 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7f9b64069210 0x7f9b6406b6c0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f9b6c004fd0 tx=0x7f9b6c00b040 comp rx=0 tx=0).stop 2024-02-20T07:55:34.170 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:34.169+0000 7f9b81225640 1 -- 172.21.15.133:0/3762827397 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f9b7c0fc3b0 msgr2=0x7f9b7c0fd6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:55:34.170 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:34.169+0000 7f9b81225640 1 --2- 172.21.15.133:0/3762827397 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f9b7c0fc3b0 0x7f9b7c0fd6f0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f9b7002bda0 tx=0x7f9b70009da0 comp rx=0 tx=0).stop 2024-02-20T07:55:34.170 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:34.169+0000 7f9b81225640 1 -- 172.21.15.133:0/3762827397 shutdown_connections 2024-02-20T07:55:34.170 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:34.169+0000 7f9b81225640 1 --2- 172.21.15.133:0/3762827397 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7f9b64069210 0x7f9b6406b6c0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:55:34.170 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:34.169+0000 7f9b81225640 1 --2- 172.21.15.133:0/3762827397 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f9b7c0fc3b0 0x7f9b7c0fd6f0 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:55:34.170 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:34.169+0000 7f9b81225640 1 --2- 172.21.15.133:0/3762827397 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9b7c0fb180 0x7f9b7c0ff070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:55:34.170 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:34.169+0000 7f9b81225640 1 -- 172.21.15.133:0/3762827397 >> 172.21.15.133:0/3762827397 conn(0x7f9b7c0f6860 msgr2=0x7f9b7c064480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:55:34.170 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:34.169+0000 7f9b81225640 1 -- 172.21.15.133:0/3762827397 shutdown_connections 2024-02-20T07:55:34.170 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:34.169+0000 7f9b81225640 1 -- 172.21.15.133:0/3762827397 wait complete. 2024-02-20T07:55:34.181 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-02-20T07:52:35.839244+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}, {'pool': 2, 'pool_name': 'cephfs.cephfs.meta', 'create_time': '2024-02-20T07:55:07.331887+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '44', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 3, 'pool_name': 'cephfs.cephfs.data', 'create_time': '2024-02-20T07:55:07.913654+0000', 'flags': 131073, 'flags_names': 'hashpspool,bulk', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '45', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-02-20T07:55:34.181 DEBUG:teuthology.orchestra.run.smithi133:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-02-20T07:55:34.284 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:34.282+0000 7fb6b759d640 1 -- 172.21.15.133:0/1825944894 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb6b00fce30 msgr2=0x7fb6b00ff210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:55:34.285 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:34.282+0000 7fb6b759d640 1 --2- 172.21.15.133:0/1825944894 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb6b00fce30 0x7fb6b00ff210 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fb6ac0048d0 tx=0x7fb6ac009da0 comp rx=0 tx=0).stop 2024-02-20T07:55:34.285 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:34.282+0000 7fb6b759d640 1 -- 172.21.15.133:0/1825944894 shutdown_connections 2024-02-20T07:55:34.285 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:34.282+0000 7fb6b759d640 1 --2- 172.21.15.133:0/1825944894 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb6b00ff750 0x7fb6b0101b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:55:34.285 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:34.282+0000 7fb6b759d640 1 --2- 172.21.15.133:0/1825944894 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb6b00fce30 0x7fb6b00ff210 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:55:34.285 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:34.282+0000 7fb6b759d640 1 -- 172.21.15.133:0/1825944894 >> 172.21.15.133:0/1825944894 conn(0x7fb6b00f68d0 msgr2=0x7fb6b00f8d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:55:34.285 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:34.282+0000 7fb6b759d640 1 -- 172.21.15.133:0/1825944894 shutdown_connections 2024-02-20T07:55:34.285 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:34.282+0000 7fb6b759d640 1 -- 172.21.15.133:0/1825944894 wait complete. 2024-02-20T07:55:34.285 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:34.283+0000 7fb6b759d640 1 Processor -- start 2024-02-20T07:55:34.285 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:34.283+0000 7fb6b759d640 1 -- start start 2024-02-20T07:55:34.285 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:34.283+0000 7fb6b759d640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb6b00fce30 0x7fb6b010cbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:55:34.285 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:34.283+0000 7fb6b759d640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb6b00ff750 0x7fb6b010d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:55:34.285 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:34.283+0000 7fb6b759d640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6b010d740 con 0x7fb6b00fce30 2024-02-20T07:55:34.285 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:34.283+0000 7fb6b759d640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6b0112190 con 0x7fb6b00ff750 2024-02-20T07:55:34.286 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:34.283+0000 7fb6b5339640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb6b00fce30 0x7fb6b010cbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:55:34.286 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:34.283+0000 7fb6b5339640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb6b00fce30 0x7fb6b010cbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.133:46500/0 (socket says 172.21.15.133:46500) 2024-02-20T07:55:34.286 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:34.283+0000 7fb6b5339640 1 -- 172.21.15.133:0/2673476386 learned_addr learned my addr 172.21.15.133:0/2673476386 (peer_addr_for_me v2:172.21.15.133:0/0) 2024-02-20T07:55:34.286 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:34.283+0000 7fb6b4b38640 1 --2- 172.21.15.133:0/2673476386 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb6b00ff750 0x7fb6b010d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:55:34.286 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:34.283+0000 7fb6b5339640 1 -- 172.21.15.133:0/2673476386 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb6b00ff750 msgr2=0x7fb6b010d0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:55:34.286 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:34.283+0000 7fb6b5339640 1 --2- 172.21.15.133:0/2673476386 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb6b00ff750 0x7fb6b010d0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:55:34.286 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:34.283+0000 7fb6b5339640 1 -- 172.21.15.133:0/2673476386 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6ac005040 con 0x7fb6b00fce30 2024-02-20T07:55:34.286 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:34.283+0000 7fb6b5339640 1 --2- 172.21.15.133:0/2673476386 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb6b00fce30 0x7fb6b010cbb0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fb6ac007da0 tx=0x7fb6ac009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:55:34.286 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:34.283+0000 7fb6a5ffb640 1 -- 172.21.15.133:0/2673476386 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb6ac0094c0 con 0x7fb6b00fce30 2024-02-20T07:55:34.286 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:34.283+0000 7fb6a5ffb640 1 -- 172.21.15.133:0/2673476386 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb6ac009620 con 0x7fb6b00fce30 2024-02-20T07:55:34.286 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:34.283+0000 7fb6b759d640 1 -- 172.21.15.133:0/2673476386 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb6b01122d0 con 0x7fb6b00fce30 2024-02-20T07:55:34.286 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:34.284+0000 7fb6a5ffb640 1 -- 172.21.15.133:0/2673476386 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb6ac03b620 con 0x7fb6b00fce30 2024-02-20T07:55:34.287 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:34.284+0000 7fb6b759d640 1 -- 172.21.15.133:0/2673476386 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb6b01127c0 con 0x7fb6b00fce30 2024-02-20T07:55:34.287 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:34.285+0000 7fb6b759d640 1 -- 172.21.15.133:0/2673476386 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb6b0106e00 con 0x7fb6b00fce30 2024-02-20T07:55:34.287 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:34.285+0000 7fb6a5ffb640 1 -- 172.21.15.133:0/2673476386 <== mon.0 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7fb6ac02f070 con 0x7fb6b00fce30 2024-02-20T07:55:34.288 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:34.286+0000 7fb6a5ffb640 1 --2- 172.21.15.133:0/2673476386 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7fb69c069210 0x7fb69c06b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:55:34.288 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:34.286+0000 7fb6a5ffb640 1 -- 172.21.15.133:0/2673476386 <== mon.0 v2:172.21.15.133:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5550+0+0 (secure 0 0 0) 0x7fb6ac0a97b0 con 0x7fb6b00fce30 2024-02-20T07:55:34.288 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:34.286+0000 7fb6b4b38640 1 --2- 172.21.15.133:0/2673476386 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7fb69c069210 0x7fb69c06b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:55:34.288 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:34.286+0000 7fb6b4b38640 1 --2- 172.21.15.133:0/2673476386 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7fb69c069210 0x7fb69c06b6c0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fb6b010e100 tx=0x7fb6a000b040 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:55:34.293 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:34.291+0000 7fb6a5ffb640 1 -- 172.21.15.133:0/2673476386 <== mon.0 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7fb6ac075a10 con 0x7fb6b00fce30 2024-02-20T07:55:34.501 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:34.501+0000 7fb6b759d640 1 -- 172.21.15.133:0/2673476386 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fb6b005e0a0 con 0x7fb6b00fce30 2024-02-20T07:55:34.502 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:34.501+0000 7fb6a5ffb640 1 -- 172.21.15.133:0/2673476386 <== mon.0 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v46) v1 ==== 93+0+10 (secure 0 0 0) 0x7fb6ac079030 con 0x7fb6b00fce30 2024-02-20T07:55:34.502 INFO:teuthology.orchestra.run.smithi133.stdout:pg_num: 1 2024-02-20T07:55:34.504 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:34.503+0000 7fb6b759d640 1 -- 172.21.15.133:0/2673476386 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7fb69c069210 msgr2=0x7fb69c06b6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:55:34.504 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:34.503+0000 7fb6b759d640 1 --2- 172.21.15.133:0/2673476386 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7fb69c069210 0x7fb69c06b6c0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fb6b010e100 tx=0x7fb6a000b040 comp rx=0 tx=0).stop 2024-02-20T07:55:34.504 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:34.503+0000 7fb6b759d640 1 -- 172.21.15.133:0/2673476386 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb6b00fce30 msgr2=0x7fb6b010cbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:55:34.504 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:34.503+0000 7fb6b759d640 1 --2- 172.21.15.133:0/2673476386 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb6b00fce30 0x7fb6b010cbb0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fb6ac007da0 tx=0x7fb6ac009da0 comp rx=0 tx=0).stop 2024-02-20T07:55:34.504 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:34.503+0000 7fb6b759d640 1 -- 172.21.15.133:0/2673476386 shutdown_connections 2024-02-20T07:55:34.504 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:34.503+0000 7fb6b759d640 1 --2- 172.21.15.133:0/2673476386 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7fb69c069210 0x7fb69c06b6c0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:55:34.504 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:34.503+0000 7fb6b759d640 1 --2- 172.21.15.133:0/2673476386 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb6b00ff750 0x7fb6b010d0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:55:34.504 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:34.503+0000 7fb6b759d640 1 --2- 172.21.15.133:0/2673476386 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb6b00fce30 0x7fb6b010cbb0 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:55:34.505 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:34.503+0000 7fb6b759d640 1 -- 172.21.15.133:0/2673476386 >> 172.21.15.133:0/2673476386 conn(0x7fb6b00f68d0 msgr2=0x7fb6b00f8d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:55:34.505 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:34.503+0000 7fb6b759d640 1 -- 172.21.15.133:0/2673476386 shutdown_connections 2024-02-20T07:55:34.505 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:34.503+0000 7fb6b759d640 1 -- 172.21.15.133:0/2673476386 wait complete. 2024-02-20T07:55:34.514 DEBUG:teuthology.orchestra.run.smithi133:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs.cephfs.meta pg_num 2024-02-20T07:55:34.620 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:34.617+0000 7f9bd4eae640 1 -- 172.21.15.133:0/662791294 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f9bd00fc400 msgr2=0x7f9bd00fc870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:55:34.620 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:34.617+0000 7f9bd4eae640 1 --2- 172.21.15.133:0/662791294 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f9bd00fc400 0x7f9bd00fc870 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f9bb80048d0 tx=0x7f9bb8009da0 comp rx=0 tx=0).stop 2024-02-20T07:55:34.620 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:34.617+0000 7f9bd4eae640 1 -- 172.21.15.133:0/662791294 shutdown_connections 2024-02-20T07:55:34.620 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:34.617+0000 7f9bd4eae640 1 --2- 172.21.15.133:0/662791294 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f9bd00fc400 0x7f9bd00fc870 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:55:34.620 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:34.617+0000 7f9bd4eae640 1 --2- 172.21.15.133:0/662791294 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9bd00fb1d0 0x7f9bd00fb5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:55:34.620 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:34.617+0000 7f9bd4eae640 1 -- 172.21.15.133:0/662791294 >> 172.21.15.133:0/662791294 conn(0x7f9bd00f68d0 msgr2=0x7f9bd00f8d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:55:34.620 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:34.617+0000 7f9bd4eae640 1 -- 172.21.15.133:0/662791294 shutdown_connections 2024-02-20T07:55:34.620 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:34.617+0000 7f9bd4eae640 1 -- 172.21.15.133:0/662791294 wait complete. 2024-02-20T07:55:34.620 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:34.618+0000 7f9bd4eae640 1 Processor -- start 2024-02-20T07:55:34.620 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:34.618+0000 7f9bd4eae640 1 -- start start 2024-02-20T07:55:34.620 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:34.618+0000 7f9bd4eae640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9bd00fb1d0 0x7f9bd0064f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:55:34.621 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:34.618+0000 7f9bd4eae640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f9bd00fc400 0x7f9bd0111950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:55:34.621 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:34.618+0000 7f9bd4eae640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bd0111e90 con 0x7f9bd00fc400 2024-02-20T07:55:34.621 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:34.618+0000 7f9bd4eae640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bd0111fd0 con 0x7f9bd00fb1d0 2024-02-20T07:55:34.621 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:34.618+0000 7f9bc5d9b640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f9bd00fc400 0x7f9bd0111950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:55:34.621 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:34.618+0000 7f9bc5d9b640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f9bd00fc400 0x7f9bd0111950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.133:46524/0 (socket says 172.21.15.133:46524) 2024-02-20T07:55:34.621 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:34.618+0000 7f9bc5d9b640 1 -- 172.21.15.133:0/1444093595 learned_addr learned my addr 172.21.15.133:0/1444093595 (peer_addr_for_me v2:172.21.15.133:0/0) 2024-02-20T07:55:34.621 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:34.618+0000 7f9bc5d9b640 1 -- 172.21.15.133:0/1444093595 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9bd00fb1d0 msgr2=0x7f9bd0064f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:55:34.621 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:34.618+0000 7f9bce59c640 1 --2- 172.21.15.133:0/1444093595 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9bd00fb1d0 0x7f9bd0064f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:55:34.621 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:34.619+0000 7f9bc5d9b640 1 --2- 172.21.15.133:0/1444093595 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9bd00fb1d0 0x7f9bd0064f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:55:34.621 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:34.619+0000 7f9bc5d9b640 1 -- 172.21.15.133:0/1444093595 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9bb8005040 con 0x7f9bd00fc400 2024-02-20T07:55:34.621 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:34.619+0000 7f9bc5d9b640 1 --2- 172.21.15.133:0/1444093595 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f9bd00fc400 0x7f9bd0111950 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f9bb8003030 tx=0x7f9bb8009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:55:34.621 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:34.619+0000 7f9bc77fe640 1 -- 172.21.15.133:0/1444093595 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9bb80094c0 con 0x7f9bd00fc400 2024-02-20T07:55:34.621 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:34.619+0000 7f9bd4eae640 1 -- 172.21.15.133:0/1444093595 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9bd0112220 con 0x7f9bd00fc400 2024-02-20T07:55:34.621 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:34.619+0000 7f9bc77fe640 1 -- 172.21.15.133:0/1444093595 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9bb8009620 con 0x7f9bd00fc400 2024-02-20T07:55:34.621 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:34.619+0000 7f9bc77fe640 1 -- 172.21.15.133:0/1444093595 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9bb803b640 con 0x7f9bd00fc400 2024-02-20T07:55:34.622 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:34.619+0000 7f9bd4eae640 1 -- 172.21.15.133:0/1444093595 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9bd0112740 con 0x7f9bd00fc400 2024-02-20T07:55:34.622 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:34.620+0000 7f9bd4eae640 1 -- 172.21.15.133:0/1444093595 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9bd0103d40 con 0x7f9bd00fc400 2024-02-20T07:55:34.622 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:34.620+0000 7f9bc77fe640 1 -- 172.21.15.133:0/1444093595 <== mon.0 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7f9bb802f070 con 0x7f9bd00fc400 2024-02-20T07:55:34.622 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:34.621+0000 7f9bc77fe640 1 --2- 172.21.15.133:0/1444093595 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7f9bbc069210 0x7f9bbc06b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:55:34.623 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:34.621+0000 7f9bce59c640 1 --2- 172.21.15.133:0/1444093595 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7f9bbc069210 0x7f9bbc06b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:55:34.623 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:34.621+0000 7f9bc77fe640 1 -- 172.21.15.133:0/1444093595 <== mon.0 v2:172.21.15.133:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f9bb80a97e0 con 0x7f9bd00fc400 2024-02-20T07:55:34.623 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:34.621+0000 7f9bce59c640 1 --2- 172.21.15.133:0/1444093595 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7f9bbc069210 0x7f9bbc06b6c0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f9bd00fc260 tx=0x7f9bc000b040 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:55:34.625 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:34.625+0000 7f9bc77fe640 1 -- 172.21.15.133:0/1444093595 <== mon.0 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f9bb8075a40 con 0x7f9bd00fc400 2024-02-20T07:55:34.741 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:55:34 smithi133 ceph-mon[29117]: pgmap v189: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 6.8 KiB/s rd, 0 B/s wr, 8 op/s 2024-02-20T07:55:34.741 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:55:34 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:55:34.742 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:55:34 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:55:34.742 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:55:34 smithi133 ceph-mon[29117]: from='client.? 172.21.15.133:0/3762827397' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-20T07:55:34.742 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:55:34 smithi133 ceph-mon[29117]: from='client.? 172.21.15.133:0/2673476386' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-02-20T07:55:34.797 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:34.796+0000 7f9bd4eae640 1 -- 172.21.15.133:0/1444093595 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"} v 0) v1 -- 0x7f9bd01128d0 con 0x7f9bd00fc400 2024-02-20T07:55:34.797 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:34.796+0000 7f9bc77fe640 1 -- 172.21.15.133:0/1444093595 <== mon.0 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]=0 v46) v1 ==== 107+0+11 (secure 0 0 0) 0x7f9bb8079060 con 0x7f9bd00fc400 2024-02-20T07:55:34.797 INFO:teuthology.orchestra.run.smithi133.stdout:pg_num: 32 2024-02-20T07:55:34.799 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:34.798+0000 7f9bd4eae640 1 -- 172.21.15.133:0/1444093595 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7f9bbc069210 msgr2=0x7f9bbc06b6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:55:34.799 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:34.798+0000 7f9bd4eae640 1 --2- 172.21.15.133:0/1444093595 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7f9bbc069210 0x7f9bbc06b6c0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f9bd00fc260 tx=0x7f9bc000b040 comp rx=0 tx=0).stop 2024-02-20T07:55:34.799 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:34.798+0000 7f9bd4eae640 1 -- 172.21.15.133:0/1444093595 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f9bd00fc400 msgr2=0x7f9bd0111950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:55:34.799 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:34.798+0000 7f9bd4eae640 1 --2- 172.21.15.133:0/1444093595 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f9bd00fc400 0x7f9bd0111950 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f9bb8003030 tx=0x7f9bb8009da0 comp rx=0 tx=0).stop 2024-02-20T07:55:34.799 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:34.798+0000 7f9bd4eae640 1 -- 172.21.15.133:0/1444093595 shutdown_connections 2024-02-20T07:55:34.799 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:34.798+0000 7f9bd4eae640 1 --2- 172.21.15.133:0/1444093595 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7f9bbc069210 0x7f9bbc06b6c0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:55:34.800 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:34.798+0000 7f9bd4eae640 1 --2- 172.21.15.133:0/1444093595 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f9bd00fc400 0x7f9bd0111950 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:55:34.800 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:34.798+0000 7f9bd4eae640 1 --2- 172.21.15.133:0/1444093595 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9bd00fb1d0 0x7f9bd0064f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:55:34.800 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:34.798+0000 7f9bd4eae640 1 -- 172.21.15.133:0/1444093595 >> 172.21.15.133:0/1444093595 conn(0x7f9bd00f68d0 msgr2=0x7f9bd00ff630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:55:34.800 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:34.798+0000 7f9bd4eae640 1 -- 172.21.15.133:0/1444093595 shutdown_connections 2024-02-20T07:55:34.800 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:34.798+0000 7f9bd4eae640 1 -- 172.21.15.133:0/1444093595 wait complete. 2024-02-20T07:55:34.811 DEBUG:teuthology.orchestra.run.smithi133:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs.cephfs.data pg_num 2024-02-20T07:55:34.913 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:34.912+0000 7fb919f9d640 1 -- 172.21.15.133:0/3683140853 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb9140fb1d0 msgr2=0x7fb9140fb5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:55:34.934 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:34.912+0000 7fb919f9d640 1 --2- 172.21.15.133:0/3683140853 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb9140fb1d0 0x7fb9140fb5e0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fb9040048d0 tx=0x7fb904009da0 comp rx=0 tx=0).stop 2024-02-20T07:55:34.934 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:34.912+0000 7fb919f9d640 1 -- 172.21.15.133:0/3683140853 shutdown_connections 2024-02-20T07:55:34.934 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:34.912+0000 7fb919f9d640 1 --2- 172.21.15.133:0/3683140853 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb9140fc400 0x7fb9140fc870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:55:34.934 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:34.912+0000 7fb919f9d640 1 --2- 172.21.15.133:0/3683140853 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb9140fb1d0 0x7fb9140fb5e0 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:55:34.934 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:34.912+0000 7fb919f9d640 1 -- 172.21.15.133:0/3683140853 >> 172.21.15.133:0/3683140853 conn(0x7fb9140f68d0 msgr2=0x7fb9140f8d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:55:34.934 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:34.912+0000 7fb919f9d640 1 -- 172.21.15.133:0/3683140853 shutdown_connections 2024-02-20T07:55:34.934 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:34.912+0000 7fb919f9d640 1 -- 172.21.15.133:0/3683140853 wait complete. 2024-02-20T07:55:34.935 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:34.913+0000 7fb919f9d640 1 Processor -- start 2024-02-20T07:55:34.935 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:34.913+0000 7fb919f9d640 1 -- start start 2024-02-20T07:55:34.935 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:34.913+0000 7fb919f9d640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb9140fb1d0 0x7fb914064f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:55:34.935 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:34.913+0000 7fb919f9d640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb9140fc400 0x7fb9141119a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:55:34.935 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:34.913+0000 7fb919f9d640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb914111ee0 con 0x7fb9140fb1d0 2024-02-20T07:55:34.935 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:34.913+0000 7fb919f9d640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb914112020 con 0x7fb9140fc400 2024-02-20T07:55:34.937 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:34.913+0000 7fb9137fe640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb9140fb1d0 0x7fb914064f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:55:34.937 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:34.913+0000 7fb9137fe640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb9140fb1d0 0x7fb914064f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.133:46550/0 (socket says 172.21.15.133:46550) 2024-02-20T07:55:34.937 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:34.913+0000 7fb9137fe640 1 -- 172.21.15.133:0/3551805995 learned_addr learned my addr 172.21.15.133:0/3551805995 (peer_addr_for_me v2:172.21.15.133:0/0) 2024-02-20T07:55:34.937 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:34.913+0000 7fb912ffd640 1 --2- 172.21.15.133:0/3551805995 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb9140fc400 0x7fb9141119a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:55:34.937 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:34.913+0000 7fb9137fe640 1 -- 172.21.15.133:0/3551805995 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb9140fc400 msgr2=0x7fb9141119a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:55:34.937 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:34.913+0000 7fb9137fe640 1 --2- 172.21.15.133:0/3551805995 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb9140fc400 0x7fb9141119a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:55:34.937 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:34.913+0000 7fb9137fe640 1 -- 172.21.15.133:0/3551805995 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb904005040 con 0x7fb9140fb1d0 2024-02-20T07:55:34.938 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:34.913+0000 7fb9137fe640 1 --2- 172.21.15.133:0/3551805995 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb9140fb1d0 0x7fb914064f30 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fb904007da0 tx=0x7fb904009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:55:34.938 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:34.914+0000 7fb8fbfff640 1 -- 172.21.15.133:0/3551805995 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb9040094c0 con 0x7fb9140fb1d0 2024-02-20T07:55:34.938 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:34.914+0000 7fb919f9d640 1 -- 172.21.15.133:0/3551805995 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb914112270 con 0x7fb9140fb1d0 2024-02-20T07:55:34.938 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:34.914+0000 7fb8fbfff640 1 -- 172.21.15.133:0/3551805995 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb904009620 con 0x7fb9140fb1d0 2024-02-20T07:55:34.938 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:34.914+0000 7fb8fbfff640 1 -- 172.21.15.133:0/3551805995 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb90403b620 con 0x7fb9140fb1d0 2024-02-20T07:55:34.938 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:34.914+0000 7fb919f9d640 1 -- 172.21.15.133:0/3551805995 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb914112790 con 0x7fb9140fb1d0 2024-02-20T07:55:34.938 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:34.915+0000 7fb919f9d640 1 -- 172.21.15.133:0/3551805995 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb914103d40 con 0x7fb9140fb1d0 2024-02-20T07:55:34.938 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:34.915+0000 7fb8fbfff640 1 -- 172.21.15.133:0/3551805995 <== mon.0 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7fb90402f070 con 0x7fb9140fb1d0 2024-02-20T07:55:34.938 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:34.915+0000 7fb8fbfff640 1 --2- 172.21.15.133:0/3551805995 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7fb8fc069210 0x7fb8fc06b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:55:34.938 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:34.916+0000 7fb8fbfff640 1 -- 172.21.15.133:0/3551805995 <== mon.0 v2:172.21.15.133:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5550+0+0 (secure 0 0 0) 0x7fb9040a9840 con 0x7fb9140fb1d0 2024-02-20T07:55:34.938 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:34.916+0000 7fb912ffd640 1 --2- 172.21.15.133:0/3551805995 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7fb8fc069210 0x7fb8fc06b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:55:34.938 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:34.916+0000 7fb912ffd640 1 --2- 172.21.15.133:0/3551805995 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7fb8fc069210 0x7fb8fc06b6c0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fb908007fd0 tx=0x7fb908009e90 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:55:34.938 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:34.920+0000 7fb8fbfff640 1 -- 172.21.15.133:0/3551805995 <== mon.0 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7fb904075aa0 con 0x7fb9140fb1d0 2024-02-20T07:55:35.047 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:55:34 smithi183 ceph-mon[35832]: pgmap v189: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 6.8 KiB/s rd, 0 B/s wr, 8 op/s 2024-02-20T07:55:35.047 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:55:34 smithi183 ceph-mon[35832]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:55:35.047 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:55:34 smithi183 ceph-mon[35832]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:55:35.047 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:55:34 smithi183 ceph-mon[35832]: from='client.? 172.21.15.133:0/3762827397' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-20T07:55:35.047 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:55:34 smithi183 ceph-mon[35832]: from='client.? 172.21.15.133:0/2673476386' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-02-20T07:55:35.089 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:35.088+0000 7fb919f9d640 1 -- 172.21.15.133:0/3551805995 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"} v 0) v1 -- 0x7fb914112c50 con 0x7fb9140fb1d0 2024-02-20T07:55:35.090 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:35.089+0000 7fb8fbfff640 1 -- 172.21.15.133:0/3551805995 <== mon.0 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]=0 v46) v1 ==== 107+0+11 (secure 0 0 0) 0x7fb9040790c0 con 0x7fb9140fb1d0 2024-02-20T07:55:35.090 INFO:teuthology.orchestra.run.smithi133.stdout:pg_num: 32 2024-02-20T07:55:35.092 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:35.091+0000 7fb919f9d640 1 -- 172.21.15.133:0/3551805995 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7fb8fc069210 msgr2=0x7fb8fc06b6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:55:35.092 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:35.091+0000 7fb919f9d640 1 --2- 172.21.15.133:0/3551805995 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7fb8fc069210 0x7fb8fc06b6c0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fb908007fd0 tx=0x7fb908009e90 comp rx=0 tx=0).stop 2024-02-20T07:55:35.092 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:35.091+0000 7fb919f9d640 1 -- 172.21.15.133:0/3551805995 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb9140fb1d0 msgr2=0x7fb914064f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:55:35.092 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:35.091+0000 7fb919f9d640 1 --2- 172.21.15.133:0/3551805995 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb9140fb1d0 0x7fb914064f30 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fb904007da0 tx=0x7fb904009da0 comp rx=0 tx=0).stop 2024-02-20T07:55:35.092 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:35.091+0000 7fb919f9d640 1 -- 172.21.15.133:0/3551805995 shutdown_connections 2024-02-20T07:55:35.092 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:35.091+0000 7fb919f9d640 1 --2- 172.21.15.133:0/3551805995 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7fb8fc069210 0x7fb8fc06b6c0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:55:35.092 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:35.091+0000 7fb919f9d640 1 --2- 172.21.15.133:0/3551805995 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb9140fc400 0x7fb9141119a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:55:35.092 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:35.091+0000 7fb919f9d640 1 --2- 172.21.15.133:0/3551805995 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb9140fb1d0 0x7fb914064f30 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:55:35.092 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:35.091+0000 7fb919f9d640 1 -- 172.21.15.133:0/3551805995 >> 172.21.15.133:0/3551805995 conn(0x7fb9140f68d0 msgr2=0x7fb9140ff630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:55:35.092 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:35.091+0000 7fb919f9d640 1 -- 172.21.15.133:0/3551805995 shutdown_connections 2024-02-20T07:55:35.092 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:35.091+0000 7fb919f9d640 1 -- 172.21.15.133:0/3551805995 wait complete. 2024-02-20T07:55:35.104 DEBUG:teuthology.orchestra.run.smithi133:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-20T07:55:35.211 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:35.209+0000 7f17dc39d640 1 -- 172.21.15.133:0/3661748458 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f17d40fd4d0 msgr2=0x7f17d40fd940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:55:35.211 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:35.209+0000 7f17dc39d640 1 --2- 172.21.15.133:0/3661748458 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f17d40fd4d0 0x7f17d40fd940 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f17c40048d0 tx=0x7f17c4009da0 comp rx=0 tx=0).stop 2024-02-20T07:55:35.211 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:35.209+0000 7f17dc39d640 1 -- 172.21.15.133:0/3661748458 shutdown_connections 2024-02-20T07:55:35.211 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:35.209+0000 7f17dc39d640 1 --2- 172.21.15.133:0/3661748458 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f17d40fd4d0 0x7f17d40fd940 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:55:35.211 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:35.209+0000 7f17dc39d640 1 --2- 172.21.15.133:0/3661748458 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f17d40fc2a0 0x7f17d40fc6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:55:35.211 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:35.209+0000 7f17dc39d640 1 -- 172.21.15.133:0/3661748458 >> 172.21.15.133:0/3661748458 conn(0x7f17d40f7980 msgr2=0x7f17d40f9df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:55:35.211 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:35.209+0000 7f17dc39d640 1 -- 172.21.15.133:0/3661748458 shutdown_connections 2024-02-20T07:55:35.212 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:35.209+0000 7f17dc39d640 1 -- 172.21.15.133:0/3661748458 wait complete. 2024-02-20T07:55:35.212 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:35.209+0000 7f17dc39d640 1 Processor -- start 2024-02-20T07:55:35.212 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:35.209+0000 7f17dc39d640 1 -- start start 2024-02-20T07:55:35.212 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:35.209+0000 7f17dc39d640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f17d40fc2a0 0x7f17d410cb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:55:35.212 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:35.209+0000 7f17dc39d640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f17d40fd4d0 0x7f17d410d090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:55:35.212 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:35.209+0000 7f17dc39d640 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17d410d6e0 con 0x7f17d40fc2a0 2024-02-20T07:55:35.212 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:35.209+0000 7f17dc39d640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17d4112130 con 0x7f17d40fd4d0 2024-02-20T07:55:35.212 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:35.209+0000 7f17da139640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f17d40fc2a0 0x7f17d410cb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:55:35.212 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:35.210+0000 7f17da139640 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f17d40fc2a0 0x7f17d410cb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.133:46574/0 (socket says 172.21.15.133:46574) 2024-02-20T07:55:35.212 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:35.210+0000 7f17da139640 1 -- 172.21.15.133:0/675535954 learned_addr learned my addr 172.21.15.133:0/675535954 (peer_addr_for_me v2:172.21.15.133:0/0) 2024-02-20T07:55:35.212 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:35.210+0000 7f17da139640 1 -- 172.21.15.133:0/675535954 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f17d40fd4d0 msgr2=0x7f17d410d090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:55:35.212 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:35.210+0000 7f17da139640 1 --2- 172.21.15.133:0/675535954 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f17d40fd4d0 0x7f17d410d090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:55:35.212 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:35.210+0000 7f17da139640 1 -- 172.21.15.133:0/675535954 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17c4005040 con 0x7f17d40fc2a0 2024-02-20T07:55:35.213 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:35.210+0000 7f17da139640 1 --2- 172.21.15.133:0/675535954 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f17d40fc2a0 0x7f17d410cb50 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f17d0002a80 tx=0x7f17d0004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:55:35.213 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:35.210+0000 7f17caffd640 1 -- 172.21.15.133:0/675535954 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f17d0007c30 con 0x7f17d40fc2a0 2024-02-20T07:55:35.213 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:35.210+0000 7f17dc39d640 1 -- 172.21.15.133:0/675535954 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f17d41122d0 con 0x7f17d40fc2a0 2024-02-20T07:55:35.213 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:35.210+0000 7f17caffd640 1 -- 172.21.15.133:0/675535954 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f17d000f040 con 0x7f17d40fc2a0 2024-02-20T07:55:35.213 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:35.210+0000 7f17caffd640 1 -- 172.21.15.133:0/675535954 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f17d0009670 con 0x7f17d40fc2a0 2024-02-20T07:55:35.213 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:35.210+0000 7f17dc39d640 1 -- 172.21.15.133:0/675535954 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f17d41127c0 con 0x7f17d40fc2a0 2024-02-20T07:55:35.214 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:35.211+0000 7f17dc39d640 1 -- 172.21.15.133:0/675535954 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f17d4104df0 con 0x7f17d40fc2a0 2024-02-20T07:55:35.214 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:35.212+0000 7f17caffd640 1 -- 172.21.15.133:0/675535954 <== mon.0 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7f17d000d6b0 con 0x7f17d40fc2a0 2024-02-20T07:55:35.214 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:35.212+0000 7f17caffd640 1 --2- 172.21.15.133:0/675535954 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7f17c0069210 0x7f17c006b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:55:35.214 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:35.212+0000 7f17caffd640 1 -- 172.21.15.133:0/675535954 <== mon.0 v2:172.21.15.133:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f17d0087040 con 0x7f17d40fc2a0 2024-02-20T07:55:35.214 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:35.212+0000 7f17d9938640 1 --2- 172.21.15.133:0/675535954 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7f17c0069210 0x7f17c006b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:55:35.214 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:35.212+0000 7f17d9938640 1 --2- 172.21.15.133:0/675535954 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7f17c0069210 0x7f17c006b6c0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f17c4034040 tx=0x7f17c4009da0 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:55:35.216 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:35.216+0000 7f17caffd640 1 -- 172.21.15.133:0/675535954 <== mon.0 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f17d00532a0 con 0x7f17d40fc2a0 2024-02-20T07:55:35.431 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:35.430+0000 7f17dc39d640 1 -- 172.21.15.133:0/675535954 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f17d41129e0 con 0x7f17d40fc2a0 2024-02-20T07:55:35.432 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:35.431+0000 7f17caffd640 1 -- 172.21.15.133:0/675535954 <== mon.0 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 19 v19) v1 ==== 94+0+4731 (secure 0 0 0) 0x7f17d00568c0 con 0x7f17d40fc2a0 2024-02-20T07:55:35.432 INFO:teuthology.orchestra.run.smithi133.stdout: 2024-02-20T07:55:35.433 INFO:teuthology.orchestra.run.smithi133.stdout:{"epoch":19,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":19,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-02-20T07:55:08.678040+0000","modified":"2024-02-20T07:55:28.636430+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":46,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":14486,"mds_1":24257},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14486":{"gid":14486,"name":"cephfs.smithi133.cmofvy","rank":0,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.133:6829/3316004709","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6828","nonce":3316004709},{"type":"v1","addr":"172.21.15.133:6829","nonce":3316004709}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_14496":{"gid":14496,"name":"cephfs.smithi133.itlmbm","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.133:6827/1557353480","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.133:6826","nonce":1557353480},{"type":"v1","addr":"172.21.15.133:6827","nonce":1557353480}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24257":{"gid":24257,"name":"cephfs.smithi183.eipfta","rank":1,"incarnation":7,"state":"up:active","state_seq":4,"addr":"172.21.15.183:6825/3202657453","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6824","nonce":3202657453},{"type":"v1","addr":"172.21.15.183:6825","nonce":3202657453}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24265":{"gid":24265,"name":"cephfs.smithi183.cwfunk","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":2,"addr":"172.21.15.183:6827/1729948973","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6826","nonce":1729948973},{"type":"v1","addr":"172.21.15.183:6827","nonce":1729948973}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2024-02-20T07:55:35.434 INFO:teuthology.orchestra.run.smithi133.stderr:dumped fsmap epoch 19 2024-02-20T07:55:35.434 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:35.433+0000 7f17dc39d640 1 -- 172.21.15.133:0/675535954 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7f17c0069210 msgr2=0x7f17c006b6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:55:35.434 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:35.433+0000 7f17dc39d640 1 --2- 172.21.15.133:0/675535954 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7f17c0069210 0x7f17c006b6c0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f17c4034040 tx=0x7f17c4009da0 comp rx=0 tx=0).stop 2024-02-20T07:55:35.434 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:35.433+0000 7f17dc39d640 1 -- 172.21.15.133:0/675535954 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f17d40fc2a0 msgr2=0x7f17d410cb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:55:35.434 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:35.433+0000 7f17dc39d640 1 --2- 172.21.15.133:0/675535954 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f17d40fc2a0 0x7f17d410cb50 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f17d0002a80 tx=0x7f17d0004600 comp rx=0 tx=0).stop 2024-02-20T07:55:35.434 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:35.433+0000 7f17dc39d640 1 -- 172.21.15.133:0/675535954 shutdown_connections 2024-02-20T07:55:35.434 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:35.433+0000 7f17dc39d640 1 --2- 172.21.15.133:0/675535954 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7f17c0069210 0x7f17c006b6c0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:55:35.434 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:35.433+0000 7f17dc39d640 1 --2- 172.21.15.133:0/675535954 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f17d40fd4d0 0x7f17d410d090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:55:35.434 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:35.433+0000 7f17dc39d640 1 --2- 172.21.15.133:0/675535954 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f17d40fc2a0 0x7f17d410cb50 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:55:35.434 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:35.433+0000 7f17dc39d640 1 -- 172.21.15.133:0/675535954 >> 172.21.15.133:0/675535954 conn(0x7f17d40f7980 msgr2=0x7f17d4100700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:55:35.435 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:35.433+0000 7f17dc39d640 1 -- 172.21.15.133:0/675535954 shutdown_connections 2024-02-20T07:55:35.435 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:35.433+0000 7f17dc39d640 1 -- 172.21.15.133:0/675535954 wait complete. 2024-02-20T07:55:35.445 DEBUG:tasks.fs:fs fscid=1,name=cephfs state = {'epoch': 19, 'max_mds': 2, 'flags': 50} 2024-02-20T07:55:35.445 INFO:teuthology.run_tasks:Running task kclient... 2024-02-20T07:55:35.477 INFO:tasks.kclient:Mounting kernel clients... 2024-02-20T07:55:35.477 INFO:tasks.kclient:config is {'client.0': {}, 'client.1': {}} 2024-02-20T07:55:35.477 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-20T07:55:35.477 DEBUG:teuthology.orchestra.run.smithi133:> ip netns list 2024-02-20T07:55:35.493 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-20T07:55:35.493 DEBUG:teuthology.orchestra.run.smithi133:> sudo ip link delete ceph-brx 2024-02-20T07:55:35.563 INFO:teuthology.orchestra.run.smithi133.stderr:Cannot find device "ceph-brx" 2024-02-20T07:55:35.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-20T07:55:35.565 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-20T07:55:35.565 DEBUG:teuthology.orchestra.run.smithi183:> ip netns list 2024-02-20T07:55:35.584 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-20T07:55:35.585 DEBUG:teuthology.orchestra.run.smithi183:> sudo ip link delete ceph-brx 2024-02-20T07:55:35.661 INFO:teuthology.orchestra.run.smithi183.stderr:Cannot find device "ceph-brx" 2024-02-20T07:55:35.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-20T07:55:35.662 INFO:tasks.kclient:client.0 config is {'syntax': 'v1'} 2024-02-20T07:55:35.662 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-02-20T07:55:35.662 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-02-20T07:55:35.663 DEBUG:teuthology.orchestra.run.smithi133:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 4537327c-cfc4-11ee-95bb-87774f69a715 -- ceph fs ls 2024-02-20T07:55:35.686 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:55:35 smithi133 ceph-mon[29117]: from='client.? 172.21.15.133:0/1444093595' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]: dispatch 2024-02-20T07:55:35.686 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:55:35 smithi133 ceph-mon[29117]: from='client.? 172.21.15.133:0/3551805995' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]: dispatch 2024-02-20T07:55:35.686 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:55:35 smithi133 ceph-mon[29117]: from='client.? 172.21.15.133:0/675535954' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-02-20T07:55:36.047 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:55:35 smithi183 ceph-mon[35832]: from='client.? 172.21.15.133:0/1444093595' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]: dispatch 2024-02-20T07:55:36.047 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:55:35 smithi183 ceph-mon[35832]: from='client.? 172.21.15.133:0/3551805995' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]: dispatch 2024-02-20T07:55:36.047 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:55:35 smithi183 ceph-mon[35832]: from='client.? 172.21.15.133:0/675535954' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-02-20T07:55:36.497 INFO:teuthology.orchestra.run.smithi133.stderr:Inferring config /var/lib/ceph/4537327c-cfc4-11ee-95bb-87774f69a715/mon.smithi133/config 2024-02-20T07:55:36.959 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:55:36 smithi133 ceph-mon[29117]: pgmap v190: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 6.2 KiB/s rd, 0 B/s wr, 7 op/s 2024-02-20T07:55:37.047 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:55:36 smithi183 ceph-mon[35832]: pgmap v190: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 6.2 KiB/s rd, 0 B/s wr, 7 op/s 2024-02-20T07:55:38.535 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:38.507+0000 7f0b814d9700 1 -- 172.21.15.133:0/1385499942 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f0b7c101e40 msgr2=0x7f0b7c102210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:55:38.535 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:38.507+0000 7f0b814d9700 1 --2- 172.21.15.133:0/1385499942 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f0b7c101e40 0x7f0b7c102210 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f0b6c004940 tx=0x7f0b6c009e30 comp rx=0 tx=0).stop 2024-02-20T07:55:38.536 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:38.507+0000 7f0b814d9700 1 -- 172.21.15.133:0/1385499942 shutdown_connections 2024-02-20T07:55:38.536 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:38.507+0000 7f0b814d9700 1 --2- 172.21.15.133:0/1385499942 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0b7c1027e0 0x7f0b7c0fbe50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:55:38.536 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:38.507+0000 7f0b814d9700 1 --2- 172.21.15.133:0/1385499942 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f0b7c101e40 0x7f0b7c102210 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:55:38.536 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:38.507+0000 7f0b814d9700 1 -- 172.21.15.133:0/1385499942 >> 172.21.15.133:0/1385499942 conn(0x7f0b7c0f79d0 msgr2=0x7f0b7c0f9de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:55:38.536 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:38.507+0000 7f0b814d9700 1 -- 172.21.15.133:0/1385499942 shutdown_connections 2024-02-20T07:55:38.536 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:38.507+0000 7f0b814d9700 1 -- 172.21.15.133:0/1385499942 wait complete. 2024-02-20T07:55:38.536 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:38.508+0000 7f0b814d9700 1 Processor -- start 2024-02-20T07:55:38.536 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:38.508+0000 7f0b814d9700 1 -- start start 2024-02-20T07:55:38.536 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:38.508+0000 7f0b814d9700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0b7c101e40 0x7f0b7c10eb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:55:38.536 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:38.508+0000 7f0b814d9700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f0b7c1027e0 0x7f0b7c10f040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:55:38.536 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:38.508+0000 7f0b814d9700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b7c10f5d0 con 0x7f0b7c1027e0 2024-02-20T07:55:38.536 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:38.508+0000 7f0b814d9700 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b7c10f740 con 0x7f0b7c101e40 2024-02-20T07:55:38.536 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:38.508+0000 7f0b7a7fc700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f0b7c1027e0 0x7f0b7c10f040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:55:38.537 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:38.508+0000 7f0b7a7fc700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f0b7c1027e0 0x7f0b7c10f040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.133:39866/0 (socket says 172.21.15.133:39866) 2024-02-20T07:55:38.537 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:38.508+0000 7f0b7a7fc700 1 -- 172.21.15.133:0/1689796033 learned_addr learned my addr 172.21.15.133:0/1689796033 (peer_addr_for_me v2:172.21.15.133:0/0) 2024-02-20T07:55:38.537 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:38.509+0000 7f0b7affd700 1 --2- 172.21.15.133:0/1689796033 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0b7c101e40 0x7f0b7c10eb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:55:38.537 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:38.509+0000 7f0b7a7fc700 1 -- 172.21.15.133:0/1689796033 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0b7c101e40 msgr2=0x7f0b7c10eb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:55:38.537 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:38.509+0000 7f0b7a7fc700 1 --2- 172.21.15.133:0/1689796033 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0b7c101e40 0x7f0b7c10eb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:55:38.537 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:38.509+0000 7f0b7a7fc700 1 -- 172.21.15.133:0/1689796033 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b6c005040 con 0x7f0b7c1027e0 2024-02-20T07:55:38.537 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:38.509+0000 7f0b7affd700 1 --2- 172.21.15.133:0/1689796033 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0b7c101e40 0x7f0b7c10eb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-20T07:55:38.537 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:38.509+0000 7f0b7a7fc700 1 --2- 172.21.15.133:0/1689796033 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f0b7c1027e0 0x7f0b7c10f040 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f0b7000c800 tx=0x7f0b70004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:55:38.537 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:38.509+0000 7f0b637fe700 1 -- 172.21.15.133:0/1689796033 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0b7000a760 con 0x7f0b7c1027e0 2024-02-20T07:55:38.537 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:38.509+0000 7f0b637fe700 1 -- 172.21.15.133:0/1689796033 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0b7000ad90 con 0x7f0b7c1027e0 2024-02-20T07:55:38.537 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:38.509+0000 7f0b814d9700 1 -- 172.21.15.133:0/1689796033 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0b7c117ad0 con 0x7f0b7c1027e0 2024-02-20T07:55:38.537 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:38.509+0000 7f0b637fe700 1 -- 172.21.15.133:0/1689796033 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0b700113a0 con 0x7f0b7c1027e0 2024-02-20T07:55:38.537 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:38.509+0000 7f0b814d9700 1 -- 172.21.15.133:0/1689796033 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b7c117f90 con 0x7f0b7c1027e0 2024-02-20T07:55:38.537 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:38.510+0000 7f0b814d9700 1 -- 172.21.15.133:0/1689796033 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0b7c0ff150 con 0x7f0b7c1027e0 2024-02-20T07:55:38.537 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:38.510+0000 7f0b637fe700 1 -- 172.21.15.133:0/1689796033 <== mon.0 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7f0b70012400 con 0x7f0b7c1027e0 2024-02-20T07:55:38.538 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:38.511+0000 7f0b637fe700 1 --2- 172.21.15.133:0/1689796033 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7f0b640692a0 0x7f0b6406b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:55:38.538 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:38.511+0000 7f0b637fe700 1 -- 172.21.15.133:0/1689796033 <== mon.0 v2:172.21.15.133:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f0b70087220 con 0x7f0b7c1027e0 2024-02-20T07:55:38.538 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:38.511+0000 7f0b7affd700 1 --2- 172.21.15.133:0/1689796033 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7f0b640692a0 0x7f0b6406b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:55:38.538 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:38.511+0000 7f0b7affd700 1 --2- 172.21.15.133:0/1689796033 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7f0b640692a0 0x7f0b6406b750 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f0b6c004560 tx=0x7f0b6c009e30 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:55:38.538 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:38.515+0000 7f0b637fe700 1 -- 172.21.15.133:0/1689796033 <== mon.0 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f0b70053500 con 0x7f0b7c1027e0 2024-02-20T07:55:38.768 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:38.766+0000 7f0b814d9700 1 -- 172.21.15.133:0/1689796033 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f0b7c066770 con 0x7f0b7c1027e0 2024-02-20T07:55:38.768 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:38.766+0000 7f0b637fe700 1 -- 172.21.15.133:0/1689796033 <== mon.0 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v19) v1 ==== 53+0+83 (secure 0 0 0) 0x7f0b70056b20 con 0x7f0b7c1027e0 2024-02-20T07:55:38.768 INFO:teuthology.orchestra.run.smithi133.stdout:name: cephfs, metadata pool: cephfs.cephfs.meta, data pools: [cephfs.cephfs.data ] 2024-02-20T07:55:38.770 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:38.768+0000 7f0b814d9700 1 -- 172.21.15.133:0/1689796033 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7f0b640692a0 msgr2=0x7f0b6406b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:55:38.770 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:38.768+0000 7f0b814d9700 1 --2- 172.21.15.133:0/1689796033 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7f0b640692a0 0x7f0b6406b750 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f0b6c004560 tx=0x7f0b6c009e30 comp rx=0 tx=0).stop 2024-02-20T07:55:38.770 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:38.768+0000 7f0b814d9700 1 -- 172.21.15.133:0/1689796033 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f0b7c1027e0 msgr2=0x7f0b7c10f040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:55:38.770 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:38.768+0000 7f0b814d9700 1 --2- 172.21.15.133:0/1689796033 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f0b7c1027e0 0x7f0b7c10f040 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f0b7000c800 tx=0x7f0b70004470 comp rx=0 tx=0).stop 2024-02-20T07:55:38.770 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:38.768+0000 7f0b814d9700 1 -- 172.21.15.133:0/1689796033 shutdown_connections 2024-02-20T07:55:38.770 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:38.768+0000 7f0b814d9700 1 --2- 172.21.15.133:0/1689796033 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7f0b640692a0 0x7f0b6406b750 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:55:38.771 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:38.769+0000 7f0b814d9700 1 --2- 172.21.15.133:0/1689796033 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0b7c101e40 0x7f0b7c10eb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:55:38.771 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:38.769+0000 7f0b814d9700 1 --2- 172.21.15.133:0/1689796033 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f0b7c1027e0 0x7f0b7c10f040 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:55:38.771 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:38.769+0000 7f0b814d9700 1 -- 172.21.15.133:0/1689796033 >> 172.21.15.133:0/1689796033 conn(0x7f0b7c0f79d0 msgr2=0x7f0b7c0f9460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:55:38.771 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:38.769+0000 7f0b814d9700 1 -- 172.21.15.133:0/1689796033 shutdown_connections 2024-02-20T07:55:38.771 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:38.769+0000 7f0b814d9700 1 -- 172.21.15.133:0/1689796033 wait complete. 2024-02-20T07:55:38.991 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:55:38 smithi133 ceph-mon[29117]: pgmap v191: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 6.1 KiB/s rd, 0 B/s wr, 7 op/s 2024-02-20T07:55:38.991 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:55:38 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-20T07:55:39.047 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:55:38 smithi183 ceph-mon[35832]: pgmap v191: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 6.1 KiB/s rd, 0 B/s wr, 7 op/s 2024-02-20T07:55:39.047 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:55:38 smithi183 ceph-mon[35832]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-20T07:55:39.381 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-02-20T07:55:39.382 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-02-20T07:55:39.382 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi133.front.sepia.ceph.com 2024-02-20T07:55:39.382 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-02-20T07:55:39.382 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-02-20T07:55:39.382 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-02-20T07:55:39.382 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-02-20T07:55:39.382 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-02-20T07:55:39.382 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.0' 2024-02-20T07:55:39.382 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-20T07:55:39.382 DEBUG:teuthology.orchestra.run.smithi133:> ip addr 2024-02-20T07:55:39.438 INFO:teuthology.orchestra.run.smithi133.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-02-20T07:55:39.439 INFO:teuthology.orchestra.run.smithi133.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-02-20T07:55:39.439 INFO:teuthology.orchestra.run.smithi133.stdout: inet 127.0.0.1/8 scope host lo 2024-02-20T07:55:39.439 INFO:teuthology.orchestra.run.smithi133.stdout: valid_lft forever preferred_lft forever 2024-02-20T07:55:39.439 INFO:teuthology.orchestra.run.smithi133.stdout: inet6 ::1/128 scope host 2024-02-20T07:55:39.439 INFO:teuthology.orchestra.run.smithi133.stdout: valid_lft forever preferred_lft forever 2024-02-20T07:55:39.439 INFO:teuthology.orchestra.run.smithi133.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-02-20T07:55:39.439 INFO:teuthology.orchestra.run.smithi133.stdout: link/ether 0c:c4:7a:d9:38:b2 brd ff:ff:ff:ff:ff:ff 2024-02-20T07:55:39.439 INFO:teuthology.orchestra.run.smithi133.stdout: altname enp7s0f0 2024-02-20T07:55:39.439 INFO:teuthology.orchestra.run.smithi133.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-02-20T07:55:39.440 INFO:teuthology.orchestra.run.smithi133.stdout: link/ether 0c:c4:7a:d9:38:b3 brd ff:ff:ff:ff:ff:ff 2024-02-20T07:55:39.440 INFO:teuthology.orchestra.run.smithi133.stdout: altname enp7s0f1 2024-02-20T07:55:39.440 INFO:teuthology.orchestra.run.smithi133.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-02-20T07:55:39.440 INFO:teuthology.orchestra.run.smithi133.stdout: link/ether 0c:c4:7a:88:80:86 brd ff:ff:ff:ff:ff:ff 2024-02-20T07:55:39.440 INFO:teuthology.orchestra.run.smithi133.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-02-20T07:55:39.440 INFO:teuthology.orchestra.run.smithi133.stdout: link/ether 0c:c4:7a:88:80:87 brd ff:ff:ff:ff:ff:ff 2024-02-20T07:55:39.440 INFO:teuthology.orchestra.run.smithi133.stdout: inet 172.21.15.133/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-02-20T07:55:39.440 INFO:teuthology.orchestra.run.smithi133.stdout: valid_lft 42284sec preferred_lft 42284sec 2024-02-20T07:55:39.440 INFO:teuthology.orchestra.run.smithi133.stdout: inet6 fe80::ec4:7aff:fe88:8087/64 scope link 2024-02-20T07:55:39.441 INFO:teuthology.orchestra.run.smithi133.stdout: valid_lft forever preferred_lft forever 2024-02-20T07:55:39.441 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-02-20T07:55:39.441 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-20T07:55:39.441 DEBUG:teuthology.orchestra.run.smithi133:> (cd / && exec bash -c ' 2024-02-20T07:55:39.441 DEBUG:teuthology.orchestra.run.smithi133:> set -e 2024-02-20T07:55:39.441 DEBUG:teuthology.orchestra.run.smithi133:> sudo ip link add name ceph-brx type bridge 2024-02-20T07:55:39.442 DEBUG:teuthology.orchestra.run.smithi133:> sudo ip addr flush dev ceph-brx 2024-02-20T07:55:39.442 DEBUG:teuthology.orchestra.run.smithi133:> sudo ip link set ceph-brx up 2024-02-20T07:55:39.442 DEBUG:teuthology.orchestra.run.smithi133:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-02-20T07:55:39.442 DEBUG:teuthology.orchestra.run.smithi133:> ') 2024-02-20T07:55:39.628 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-20T07:55:39.628 DEBUG:teuthology.orchestra.run.smithi133:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-02-20T07:55:39.655 INFO:teuthology.orchestra.run.smithi133.stdout:1 2024-02-20T07:55:39.657 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-20T07:55:39.657 DEBUG:teuthology.orchestra.run.smithi133:> route 2024-02-20T07:55:39.725 INFO:teuthology.orchestra.run.smithi133.stdout:Kernel IP routing table 2024-02-20T07:55:39.725 INFO:teuthology.orchestra.run.smithi133.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-02-20T07:55:39.725 INFO:teuthology.orchestra.run.smithi133.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-02-20T07:55:39.725 INFO:teuthology.orchestra.run.smithi133.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-02-20T07:55:39.725 INFO:teuthology.orchestra.run.smithi133.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-02-20T07:55:39.725 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-20T07:55:39.726 DEBUG:teuthology.orchestra.run.smithi133:> (cd / && exec bash -c ' 2024-02-20T07:55:39.726 DEBUG:teuthology.orchestra.run.smithi133:> set -e 2024-02-20T07:55:39.726 DEBUG:teuthology.orchestra.run.smithi133:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-02-20T07:55:39.726 DEBUG:teuthology.orchestra.run.smithi133:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-02-20T07:55:39.726 DEBUG:teuthology.orchestra.run.smithi133:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-02-20T07:55:39.726 DEBUG:teuthology.orchestra.run.smithi133:> ') 2024-02-20T07:55:39.791 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:55:39 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:55:39.791 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:55:39 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:55:39.791 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:55:39 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-20T07:55:39.791 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:55:39 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-20T07:55:39.791 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:55:39 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:55:39.791 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:55:39 smithi133 ceph-mon[29117]: from='client.? 172.21.15.133:0/1689796033' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-02-20T07:55:40.005 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-20T07:55:40.005 DEBUG:teuthology.orchestra.run.smithi133:> ip netns list 2024-02-20T07:55:40.047 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:55:39 smithi183 ceph-mon[35832]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:55:40.047 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:55:39 smithi183 ceph-mon[35832]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:55:40.047 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:55:39 smithi183 ceph-mon[35832]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-20T07:55:40.048 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:55:39 smithi183 ceph-mon[35832]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-20T07:55:40.048 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:55:39 smithi183 ceph-mon[35832]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:55:40.048 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:55:39 smithi183 ceph-mon[35832]: from='client.? 172.21.15.133:0/1689796033' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-02-20T07:55:40.062 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-20T07:55:40.063 DEBUG:teuthology.orchestra.run.smithi133:> ip netns list-id 2024-02-20T07:55:40.128 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-20T07:55:40.129 DEBUG:teuthology.orchestra.run.smithi133:> (cd / && exec bash -c ' 2024-02-20T07:55:40.129 DEBUG:teuthology.orchestra.run.smithi133:> set -e 2024-02-20T07:55:40.129 DEBUG:teuthology.orchestra.run.smithi133:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.0 2024-02-20T07:55:40.129 DEBUG:teuthology.orchestra.run.smithi133:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.0 0 2024-02-20T07:55:40.129 DEBUG:teuthology.orchestra.run.smithi133:> ') 2024-02-20T07:55:40.215 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0' with 192.168.0.1/16 2024-02-20T07:55:40.215 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-20T07:55:40.215 DEBUG:teuthology.orchestra.run.smithi133:> (cd / && exec bash -c ' 2024-02-20T07:55:40.215 DEBUG:teuthology.orchestra.run.smithi133:> set -e 2024-02-20T07:55:40.215 DEBUG:teuthology.orchestra.run.smithi133:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.0 type veth peer name brx.0 2024-02-20T07:55:40.215 DEBUG:teuthology.orchestra.run.smithi133:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-02-20T07:55:40.215 DEBUG:teuthology.orchestra.run.smithi133:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set veth0 up 2024-02-20T07:55:40.215 DEBUG:teuthology.orchestra.run.smithi133:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set lo up 2024-02-20T07:55:40.216 DEBUG:teuthology.orchestra.run.smithi133:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip route add default via 192.168.255.254 2024-02-20T07:55:40.216 DEBUG:teuthology.orchestra.run.smithi133:> ') 2024-02-20T07:55:40.362 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-20T07:55:40.362 DEBUG:teuthology.orchestra.run.smithi133:> (cd / && exec bash -c ' 2024-02-20T07:55:40.362 DEBUG:teuthology.orchestra.run.smithi133:> set -e 2024-02-20T07:55:40.362 DEBUG:teuthology.orchestra.run.smithi133:> sudo ip link set brx.0 up 2024-02-20T07:55:40.362 DEBUG:teuthology.orchestra.run.smithi133:> sudo ip link set dev brx.0 master ceph-brx 2024-02-20T07:55:40.362 DEBUG:teuthology.orchestra.run.smithi133:> ') 2024-02-20T07:55:40.404 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-02-20T07:55:40.405 DEBUG:teuthology.orchestra.run.smithi133:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-02-20T07:55:40.470 INFO:teuthology.orchestra.run.smithi133.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-02-20T07:55:40.470 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-02-20T07:55:40.471 DEBUG:teuthology.orchestra.run.smithi133:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-02-20T07:55:40.525 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2024-02-20T07:55:40.525 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-20T07:55:40.526 DEBUG:teuthology.orchestra.run.smithi133:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-02-20T07:55:40.627 INFO:teuthology.orchestra.run.smithi133.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-02-20T07:55:40.628 INFO:teuthology.orchestra.run.smithi133.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-02-20T07:55:40.628 INFO:teuthology.orchestra.run.smithi133.stdout:invalid new device string format 2024-02-20T07:55:40.742 INFO:teuthology.orchestra.run.smithi133.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-02-20T07:55:40.742 INFO:teuthology.orchestra.run.smithi133.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-02-20T07:55:40.742 INFO:teuthology.orchestra.run.smithi133.stdout:invalid new device string format 2024-02-20T07:55:40.742 INFO:teuthology.orchestra.run.smithi133.stdout:mount.ceph: resolved to: "172.21.15.133:3300,172.21.15.183:3300" 2024-02-20T07:55:40.742 INFO:teuthology.orchestra.run.smithi133.stdout:mount.ceph: switching to using v1 address with old syntax 2024-02-20T07:55:40.746 INFO:teuthology.orchestra.run.smithi133.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-02-20T07:55:40.746 INFO:teuthology.orchestra.run.smithi133.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-02-20T07:55:40.747 INFO:teuthology.orchestra.run.smithi133.stdout:invalid new device string format 2024-02-20T07:55:40.747 INFO:teuthology.orchestra.run.smithi133.stdout:mount.ceph: resolved to: "172.21.15.133:3300,172.21.15.183:3300" 2024-02-20T07:55:40.747 INFO:teuthology.orchestra.run.smithi133.stdout:mount.ceph: switching to using v1 address with old syntax 2024-02-20T07:55:40.747 INFO:teuthology.orchestra.run.smithi133.stdout:mount.ceph: resolved to: "172.21.15.133:6789,172.21.15.183:6789" 2024-02-20T07:55:40.747 INFO:teuthology.orchestra.run.smithi133.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.133:6789,172.21.15.183:6789:/ 2024-02-20T07:55:40.747 INFO:teuthology.orchestra.run.smithi133.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,key=0,fsid=4537327c-cfc4-11ee-95bb-87774f69a715" will pass to kernel 2024-02-20T07:55:40.748 INFO:teuthology.orchestra.run.smithi133.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-02-20T07:55:40.748 INFO:teuthology.orchestra.run.smithi133.stdout: You just mounted a file system that supports labels which does not 2024-02-20T07:55:40.748 INFO:teuthology.orchestra.run.smithi133.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-02-20T07:55:40.748 INFO:teuthology.orchestra.run.smithi133.stdout: applications will generate AVC messages and not be allowed access to 2024-02-20T07:55:40.748 INFO:teuthology.orchestra.run.smithi133.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-02-20T07:55:40.749 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-02-20T07:55:40.749 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-20T07:55:40.749 DEBUG:teuthology.orchestra.run.smithi133:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-02-20T07:55:40.785 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-20T07:55:40.785 DEBUG:teuthology.orchestra.run.smithi133:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-02-20T07:55:40.859 INFO:teuthology.orchestra.run.smithi133.stdout:client14534 2024-02-20T07:55:40.859 DEBUG:teuthology.orchestra.run.smithi133:> set -ex 2024-02-20T07:55:40.859 DEBUG:teuthology.orchestra.run.smithi133:> dd if=/proc/self/mounts of=/dev/stdout 2024-02-20T07:55:40.928 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-20T07:55:40.929 DEBUG:teuthology.orchestra.run.smithi133:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-02-20T07:55:40.986 INFO:teuthology.orchestra.run.smithi133.stdout:client14534 2024-02-20T07:55:40.986 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-20T07:55:40.986 DEBUG:teuthology.orchestra.run.smithi133:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-02-20T07:55:40.991 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:55:40 smithi133 ceph-mon[29117]: pgmap v192: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 6.1 KiB/s rd, 0 B/s wr, 8 op/s 2024-02-20T07:55:41.019 INFO:teuthology.orchestra.run.smithi133.stdout:client14534 2024-02-20T07:55:41.019 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-20T07:55:41.020 DEBUG:teuthology.orchestra.run.smithi133:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'sudo dd if=/sys/kernel/debug/ceph/4537327c-cfc4-11ee-95bb-87774f69a715.client14534/status') 2024-02-20T07:55:41.056 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:55:40 smithi183 ceph-mon[35832]: pgmap v192: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 6.1 KiB/s rd, 0 B/s wr, 8 op/s 2024-02-20T07:55:41.090 INFO:teuthology.orchestra.run.smithi133.stdout:instance: client.14534 (3)192.168.0.1:0/605736451 2024-02-20T07:55:41.090 INFO:teuthology.orchestra.run.smithi133.stdout:blocklisted: false 2024-02-20T07:55:41.090 INFO:teuthology.orchestra.run.smithi133.stderr:0+1 records in 2024-02-20T07:55:41.090 INFO:teuthology.orchestra.run.smithi133.stderr:0+1 records out 2024-02-20T07:55:41.091 INFO:teuthology.orchestra.run.smithi133.stderr:69 bytes copied, 2.2335e-05 s, 3.1 MB/s 2024-02-20T07:55:41.092 INFO:tasks.kclient:client.1 config is {'syntax': 'v1'} 2024-02-20T07:55:41.092 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-02-20T07:55:41.092 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-02-20T07:55:41.092 DEBUG:teuthology.orchestra.run.smithi133:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 4537327c-cfc4-11ee-95bb-87774f69a715 -- ceph fs ls 2024-02-20T07:55:41.387 INFO:teuthology.orchestra.run.smithi133.stderr:Inferring config /var/lib/ceph/4537327c-cfc4-11ee-95bb-87774f69a715/mon.smithi133/config 2024-02-20T07:55:42.910 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:42.906+0000 7f761c25f700 1 -- 172.21.15.133:0/4017538506 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f76141038f0 msgr2=0x7f76140fcf60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:55:42.910 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:42.906+0000 7f761c25f700 1 --2- 172.21.15.133:0/4017538506 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f76141038f0 0x7f76140fcf60 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f7610004940 tx=0x7f7610009e30 comp rx=0 tx=0).stop 2024-02-20T07:55:42.910 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:42.906+0000 7f761c25f700 1 -- 172.21.15.133:0/4017538506 shutdown_connections 2024-02-20T07:55:42.910 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:42.906+0000 7f761c25f700 1 --2- 172.21.15.133:0/4017538506 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f76141038f0 0x7f76140fcf60 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:55:42.910 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:42.906+0000 7f761c25f700 1 --2- 172.21.15.133:0/4017538506 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f7614102f50 0x7f7614103320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:55:42.910 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:42.906+0000 7f761c25f700 1 -- 172.21.15.133:0/4017538506 >> 172.21.15.133:0/4017538506 conn(0x7f76140f8ae0 msgr2=0x7f76140faef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:55:42.911 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:42.907+0000 7f761c25f700 1 -- 172.21.15.133:0/4017538506 shutdown_connections 2024-02-20T07:55:42.911 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:42.907+0000 7f761c25f700 1 -- 172.21.15.133:0/4017538506 wait complete. 2024-02-20T07:55:42.911 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:42.907+0000 7f761c25f700 1 Processor -- start 2024-02-20T07:55:42.911 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:42.907+0000 7f761c25f700 1 -- start start 2024-02-20T07:55:42.911 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:42.907+0000 7f761c25f700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f7614102f50 0x7f7614113a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:55:42.911 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:42.907+0000 7f761c25f700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f76141038f0 0x7f7614113fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:55:42.911 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:42.907+0000 7f761c25f700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7614114620 con 0x7f7614102f50 2024-02-20T07:55:42.911 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:42.907+0000 7f761c25f700 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f761410d950 con 0x7f76141038f0 2024-02-20T07:55:42.911 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:42.908+0000 7f7619ffb700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f7614102f50 0x7f7614113a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:55:42.911 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:42.908+0000 7f7619ffb700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f7614102f50 0x7f7614113a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.133:39892/0 (socket says 172.21.15.133:39892) 2024-02-20T07:55:42.911 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:42.908+0000 7f7619ffb700 1 -- 172.21.15.133:0/4129616650 learned_addr learned my addr 172.21.15.133:0/4129616650 (peer_addr_for_me v2:172.21.15.133:0/0) 2024-02-20T07:55:42.911 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:42.908+0000 7f76197fa700 1 --2- 172.21.15.133:0/4129616650 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f76141038f0 0x7f7614113fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:55:42.911 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:42.908+0000 7f7619ffb700 1 -- 172.21.15.133:0/4129616650 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f76141038f0 msgr2=0x7f7614113fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:55:42.912 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:42.908+0000 7f7619ffb700 1 --2- 172.21.15.133:0/4129616650 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f76141038f0 0x7f7614113fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:55:42.912 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:42.908+0000 7f7619ffb700 1 -- 172.21.15.133:0/4129616650 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7610005040 con 0x7f7614102f50 2024-02-20T07:55:42.912 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:42.908+0000 7f7619ffb700 1 --2- 172.21.15.133:0/4129616650 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f7614102f50 0x7f7614113a90 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f7604002a50 tx=0x7f76040046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:55:42.912 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:42.908+0000 7f760a7fc700 1 -- 172.21.15.133:0/4129616650 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7604007cc0 con 0x7f7614102f50 2024-02-20T07:55:42.912 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:42.908+0000 7f760a7fc700 1 -- 172.21.15.133:0/4129616650 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f760400f040 con 0x7f7614102f50 2024-02-20T07:55:42.912 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:42.908+0000 7f760a7fc700 1 -- 172.21.15.133:0/4129616650 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f760400da30 con 0x7f7614102f50 2024-02-20T07:55:42.912 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:42.908+0000 7f761c25f700 1 -- 172.21.15.133:0/4129616650 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f761410dc30 con 0x7f7614102f50 2024-02-20T07:55:42.912 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:42.908+0000 7f761c25f700 1 -- 172.21.15.133:0/4129616650 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f761410e140 con 0x7f7614102f50 2024-02-20T07:55:42.913 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:42.909+0000 7f761c25f700 1 -- 172.21.15.133:0/4129616650 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7614105420 con 0x7f7614102f50 2024-02-20T07:55:42.913 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:42.910+0000 7f760a7fc700 1 -- 172.21.15.133:0/4129616650 <== mon.0 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7f760400db90 con 0x7f7614102f50 2024-02-20T07:55:42.913 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:42.910+0000 7f760a7fc700 1 --2- 172.21.15.133:0/4129616650 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7f76000692a0 0x7f760006b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:55:42.913 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:42.910+0000 7f760a7fc700 1 -- 172.21.15.133:0/4129616650 <== mon.0 v2:172.21.15.133:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f7604086eb0 con 0x7f7614102f50 2024-02-20T07:55:42.913 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:42.910+0000 7f76197fa700 1 --2- 172.21.15.133:0/4129616650 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7f76000692a0 0x7f760006b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:55:42.913 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:42.911+0000 7f76197fa700 1 --2- 172.21.15.133:0/4129616650 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7f76000692a0 0x7f760006b750 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f761000b3b0 tx=0x7f7610009e30 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:55:42.916 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:42.914+0000 7f760a7fc700 1 -- 172.21.15.133:0/4129616650 <== mon.0 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f7604056c20 con 0x7f7614102f50 2024-02-20T07:55:42.991 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:55:42 smithi133 ceph-mon[29117]: pgmap v193: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 6.0 KiB/s rd, 0 B/s wr, 7 op/s 2024-02-20T07:55:43.018 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:55:42 smithi183 ceph-mon[35832]: pgmap v193: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 6.0 KiB/s rd, 0 B/s wr, 7 op/s 2024-02-20T07:55:43.147 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:43.146+0000 7f761c25f700 1 -- 172.21.15.133:0/4129616650 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f761404bed0 con 0x7f7614102f50 2024-02-20T07:55:43.148 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:43.146+0000 7f760a7fc700 1 -- 172.21.15.133:0/4129616650 <== mon.0 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v19) v1 ==== 53+0+83 (secure 0 0 0) 0x7f76040567b0 con 0x7f7614102f50 2024-02-20T07:55:43.148 INFO:teuthology.orchestra.run.smithi133.stdout:name: cephfs, metadata pool: cephfs.cephfs.meta, data pools: [cephfs.cephfs.data ] 2024-02-20T07:55:43.150 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:43.148+0000 7f761c25f700 1 -- 172.21.15.133:0/4129616650 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7f76000692a0 msgr2=0x7f760006b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:55:43.150 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:43.148+0000 7f761c25f700 1 --2- 172.21.15.133:0/4129616650 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7f76000692a0 0x7f760006b750 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f761000b3b0 tx=0x7f7610009e30 comp rx=0 tx=0).stop 2024-02-20T07:55:43.150 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:43.148+0000 7f761c25f700 1 -- 172.21.15.133:0/4129616650 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f7614102f50 msgr2=0x7f7614113a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:55:43.150 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:43.148+0000 7f761c25f700 1 --2- 172.21.15.133:0/4129616650 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f7614102f50 0x7f7614113a90 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f7604002a50 tx=0x7f76040046b0 comp rx=0 tx=0).stop 2024-02-20T07:55:43.150 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:43.148+0000 7f761c25f700 1 -- 172.21.15.133:0/4129616650 shutdown_connections 2024-02-20T07:55:43.150 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:43.148+0000 7f761c25f700 1 --2- 172.21.15.133:0/4129616650 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f7614102f50 0x7f7614113a90 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:55:43.150 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:43.148+0000 7f761c25f700 1 --2- 172.21.15.133:0/4129616650 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7f76000692a0 0x7f760006b750 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:55:43.150 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:43.148+0000 7f761c25f700 1 --2- 172.21.15.133:0/4129616650 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f76141038f0 0x7f7614113fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:55:43.150 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:43.148+0000 7f761c25f700 1 -- 172.21.15.133:0/4129616650 >> 172.21.15.133:0/4129616650 conn(0x7f76140f8ae0 msgr2=0x7f76140fa5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:55:43.150 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:43.148+0000 7f761c25f700 1 -- 172.21.15.133:0/4129616650 shutdown_connections 2024-02-20T07:55:43.151 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:43.148+0000 7f761c25f700 1 -- 172.21.15.133:0/4129616650 wait complete. 2024-02-20T07:55:43.769 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:55:43 smithi133 ceph-mon[29117]: from='client.? 172.21.15.133:0/4129616650' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-02-20T07:55:43.771 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-02-20T07:55:43.771 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-02-20T07:55:43.771 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi183.front.sepia.ceph.com 2024-02-20T07:55:43.771 INFO:tasks.cephfs.mount:self.client.name = client.1 2024-02-20T07:55:43.771 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-02-20T07:55:43.771 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-02-20T07:55:43.771 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-02-20T07:55:43.771 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-02-20T07:55:43.771 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.1' 2024-02-20T07:55:43.772 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-20T07:55:43.772 DEBUG:teuthology.orchestra.run.smithi183:> ip addr 2024-02-20T07:55:43.788 INFO:teuthology.orchestra.run.smithi183.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-02-20T07:55:43.788 INFO:teuthology.orchestra.run.smithi183.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-02-20T07:55:43.789 INFO:teuthology.orchestra.run.smithi183.stdout: inet 127.0.0.1/8 scope host lo 2024-02-20T07:55:43.789 INFO:teuthology.orchestra.run.smithi183.stdout: valid_lft forever preferred_lft forever 2024-02-20T07:55:43.789 INFO:teuthology.orchestra.run.smithi183.stdout: inet6 ::1/128 scope host 2024-02-20T07:55:43.789 INFO:teuthology.orchestra.run.smithi183.stdout: valid_lft forever preferred_lft forever 2024-02-20T07:55:43.789 INFO:teuthology.orchestra.run.smithi183.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-02-20T07:55:43.789 INFO:teuthology.orchestra.run.smithi183.stdout: link/ether 0c:c4:7a:d9:38:aa brd ff:ff:ff:ff:ff:ff 2024-02-20T07:55:43.789 INFO:teuthology.orchestra.run.smithi183.stdout: altname enp7s0f0 2024-02-20T07:55:43.789 INFO:teuthology.orchestra.run.smithi183.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-02-20T07:55:43.789 INFO:teuthology.orchestra.run.smithi183.stdout: link/ether 0c:c4:7a:d9:38:ab brd ff:ff:ff:ff:ff:ff 2024-02-20T07:55:43.789 INFO:teuthology.orchestra.run.smithi183.stdout: altname enp7s0f1 2024-02-20T07:55:43.789 INFO:teuthology.orchestra.run.smithi183.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-02-20T07:55:43.789 INFO:teuthology.orchestra.run.smithi183.stdout: link/ether 0c:c4:7a:88:80:84 brd ff:ff:ff:ff:ff:ff 2024-02-20T07:55:43.789 INFO:teuthology.orchestra.run.smithi183.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-02-20T07:55:43.789 INFO:teuthology.orchestra.run.smithi183.stdout: link/ether 0c:c4:7a:88:80:85 brd ff:ff:ff:ff:ff:ff 2024-02-20T07:55:43.790 INFO:teuthology.orchestra.run.smithi183.stdout: inet 172.21.15.183/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-02-20T07:55:43.790 INFO:teuthology.orchestra.run.smithi183.stdout: valid_lft 42292sec preferred_lft 42292sec 2024-02-20T07:55:43.790 INFO:teuthology.orchestra.run.smithi183.stdout: inet6 fe80::ec4:7aff:fe88:8085/64 scope link 2024-02-20T07:55:43.790 INFO:teuthology.orchestra.run.smithi183.stdout: valid_lft forever preferred_lft forever 2024-02-20T07:55:43.790 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-02-20T07:55:43.790 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-20T07:55:43.790 DEBUG:teuthology.orchestra.run.smithi183:> (cd / && exec bash -c ' 2024-02-20T07:55:43.790 DEBUG:teuthology.orchestra.run.smithi183:> set -e 2024-02-20T07:55:43.790 DEBUG:teuthology.orchestra.run.smithi183:> sudo ip link add name ceph-brx type bridge 2024-02-20T07:55:43.790 DEBUG:teuthology.orchestra.run.smithi183:> sudo ip addr flush dev ceph-brx 2024-02-20T07:55:43.790 DEBUG:teuthology.orchestra.run.smithi183:> sudo ip link set ceph-brx up 2024-02-20T07:55:43.791 DEBUG:teuthology.orchestra.run.smithi183:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-02-20T07:55:43.791 DEBUG:teuthology.orchestra.run.smithi183:> ') 2024-02-20T07:55:43.912 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:55:43 smithi183 ceph-mon[35832]: from='client.? 172.21.15.133:0/4129616650' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-02-20T07:55:43.954 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-20T07:55:43.954 DEBUG:teuthology.orchestra.run.smithi183:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-02-20T07:55:43.981 INFO:teuthology.orchestra.run.smithi183.stdout:1 2024-02-20T07:55:43.983 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-20T07:55:43.983 DEBUG:teuthology.orchestra.run.smithi183:> route 2024-02-20T07:55:44.054 INFO:teuthology.orchestra.run.smithi183.stdout:Kernel IP routing table 2024-02-20T07:55:44.054 INFO:teuthology.orchestra.run.smithi183.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-02-20T07:55:44.054 INFO:teuthology.orchestra.run.smithi183.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-02-20T07:55:44.054 INFO:teuthology.orchestra.run.smithi183.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-02-20T07:55:44.055 INFO:teuthology.orchestra.run.smithi183.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-02-20T07:55:44.055 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-20T07:55:44.055 DEBUG:teuthology.orchestra.run.smithi183:> (cd / && exec bash -c ' 2024-02-20T07:55:44.055 DEBUG:teuthology.orchestra.run.smithi183:> set -e 2024-02-20T07:55:44.056 DEBUG:teuthology.orchestra.run.smithi183:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-02-20T07:55:44.056 DEBUG:teuthology.orchestra.run.smithi183:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-02-20T07:55:44.056 DEBUG:teuthology.orchestra.run.smithi183:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-02-20T07:55:44.056 DEBUG:teuthology.orchestra.run.smithi183:> ') 2024-02-20T07:55:44.306 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-20T07:55:44.306 DEBUG:teuthology.orchestra.run.smithi183:> ip netns list 2024-02-20T07:55:44.321 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-20T07:55:44.321 DEBUG:teuthology.orchestra.run.smithi183:> ip netns list-id 2024-02-20T07:55:44.376 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-20T07:55:44.376 DEBUG:teuthology.orchestra.run.smithi183:> (cd / && exec bash -c ' 2024-02-20T07:55:44.376 DEBUG:teuthology.orchestra.run.smithi183:> set -e 2024-02-20T07:55:44.376 DEBUG:teuthology.orchestra.run.smithi183:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.1 2024-02-20T07:55:44.376 DEBUG:teuthology.orchestra.run.smithi183:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.1 0 2024-02-20T07:55:44.377 DEBUG:teuthology.orchestra.run.smithi183:> ') 2024-02-20T07:55:44.458 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.1' with 192.168.0.1/16 2024-02-20T07:55:44.458 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-20T07:55:44.458 DEBUG:teuthology.orchestra.run.smithi183:> (cd / && exec bash -c ' 2024-02-20T07:55:44.458 DEBUG:teuthology.orchestra.run.smithi183:> set -e 2024-02-20T07:55:44.459 DEBUG:teuthology.orchestra.run.smithi183:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.1 type veth peer name brx.0 2024-02-20T07:55:44.459 DEBUG:teuthology.orchestra.run.smithi183:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-02-20T07:55:44.459 DEBUG:teuthology.orchestra.run.smithi183:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip link set veth0 up 2024-02-20T07:55:44.459 DEBUG:teuthology.orchestra.run.smithi183:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip link set lo up 2024-02-20T07:55:44.459 DEBUG:teuthology.orchestra.run.smithi183:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip route add default via 192.168.255.254 2024-02-20T07:55:44.459 DEBUG:teuthology.orchestra.run.smithi183:> ') 2024-02-20T07:55:44.594 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-20T07:55:44.594 DEBUG:teuthology.orchestra.run.smithi183:> (cd / && exec bash -c ' 2024-02-20T07:55:44.594 DEBUG:teuthology.orchestra.run.smithi183:> set -e 2024-02-20T07:55:44.594 DEBUG:teuthology.orchestra.run.smithi183:> sudo ip link set brx.0 up 2024-02-20T07:55:44.594 DEBUG:teuthology.orchestra.run.smithi183:> sudo ip link set dev brx.0 master ceph-brx 2024-02-20T07:55:44.594 DEBUG:teuthology.orchestra.run.smithi183:> ') 2024-02-20T07:55:44.636 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-02-20T07:55:44.636 DEBUG:teuthology.orchestra.run.smithi183:> mkdir -p -v /home/ubuntu/cephtest/mnt.1 2024-02-20T07:55:44.694 INFO:teuthology.orchestra.run.smithi183.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.1' 2024-02-20T07:55:44.694 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-02-20T07:55:44.694 DEBUG:teuthology.orchestra.run.smithi183:> chmod 0000 /home/ubuntu/cephtest/mnt.1 2024-02-20T07:55:44.750 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2024-02-20T07:55:44.750 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-20T07:55:44.750 DEBUG:teuthology.orchestra.run.smithi183:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.1 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=1,mds_namespace=cephfs 2024-02-20T07:55:44.797 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:55:44 smithi183 ceph-mon[35832]: pgmap v194: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 3.7 KiB/s rd, 255 B/s wr, 6 op/s 2024-02-20T07:55:44.843 INFO:teuthology.orchestra.run.smithi183.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=1,mds_namespace=cephfs 2024-02-20T07:55:44.843 INFO:teuthology.orchestra.run.smithi183.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,mds_namespace=cephfs". 2024-02-20T07:55:44.843 INFO:teuthology.orchestra.run.smithi183.stdout:invalid new device string format 2024-02-20T07:55:44.949 INFO:teuthology.orchestra.run.smithi183.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=1,mds_namespace=cephfs 2024-02-20T07:55:44.949 INFO:teuthology.orchestra.run.smithi183.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,mds_namespace=cephfs". 2024-02-20T07:55:44.949 INFO:teuthology.orchestra.run.smithi183.stdout:invalid new device string format 2024-02-20T07:55:44.949 INFO:teuthology.orchestra.run.smithi183.stdout:mount.ceph: resolved to: "172.21.15.133:3300,172.21.15.183:3300" 2024-02-20T07:55:44.950 INFO:teuthology.orchestra.run.smithi183.stdout:mount.ceph: switching to using v1 address with old syntax 2024-02-20T07:55:44.954 INFO:teuthology.orchestra.run.smithi183.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=1,mds_namespace=cephfs 2024-02-20T07:55:44.954 INFO:teuthology.orchestra.run.smithi183.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,mds_namespace=cephfs". 2024-02-20T07:55:44.954 INFO:teuthology.orchestra.run.smithi183.stdout:invalid new device string format 2024-02-20T07:55:44.954 INFO:teuthology.orchestra.run.smithi183.stdout:mount.ceph: resolved to: "172.21.15.133:3300,172.21.15.183:3300" 2024-02-20T07:55:44.954 INFO:teuthology.orchestra.run.smithi183.stdout:mount.ceph: switching to using v1 address with old syntax 2024-02-20T07:55:44.954 INFO:teuthology.orchestra.run.smithi183.stdout:mount.ceph: resolved to: "172.21.15.133:6789,172.21.15.183:6789" 2024-02-20T07:55:44.954 INFO:teuthology.orchestra.run.smithi183.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.133:6789,172.21.15.183:6789:/ 2024-02-20T07:55:44.954 INFO:teuthology.orchestra.run.smithi183.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,mds_namespace=cephfs,key=1,fsid=4537327c-cfc4-11ee-95bb-87774f69a715" will pass to kernel 2024-02-20T07:55:44.955 INFO:teuthology.orchestra.run.smithi183.stdout:mount: /home/ubuntu/cephtest/mnt.1 does not contain SELinux labels. 2024-02-20T07:55:44.955 INFO:teuthology.orchestra.run.smithi183.stdout: You just mounted a file system that supports labels which does not 2024-02-20T07:55:44.955 INFO:teuthology.orchestra.run.smithi183.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-02-20T07:55:44.955 INFO:teuthology.orchestra.run.smithi183.stdout: applications will generate AVC messages and not be allowed access to 2024-02-20T07:55:44.955 INFO:teuthology.orchestra.run.smithi183.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-02-20T07:55:44.957 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-02-20T07:55:44.957 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-20T07:55:44.957 DEBUG:teuthology.orchestra.run.smithi183:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.1 2024-02-20T07:55:44.991 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:55:44 smithi133 ceph-mon[29117]: pgmap v194: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 3.7 KiB/s rd, 255 B/s wr, 6 op/s 2024-02-20T07:55:45.025 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-20T07:55:45.025 DEBUG:teuthology.orchestra.run.smithi183:> (cd /home/ubuntu/cephtest/mnt.1 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-02-20T07:55:45.081 INFO:teuthology.orchestra.run.smithi183.stdout:client14540 2024-02-20T07:55:45.082 DEBUG:teuthology.orchestra.run.smithi183:> set -ex 2024-02-20T07:55:45.082 DEBUG:teuthology.orchestra.run.smithi183:> dd if=/proc/self/mounts of=/dev/stdout 2024-02-20T07:55:45.139 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-20T07:55:45.139 DEBUG:teuthology.orchestra.run.smithi183:> (cd /home/ubuntu/cephtest/mnt.1 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-02-20T07:55:45.195 INFO:teuthology.orchestra.run.smithi183.stdout:client14540 2024-02-20T07:55:45.196 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-20T07:55:45.196 DEBUG:teuthology.orchestra.run.smithi183:> (cd /home/ubuntu/cephtest/mnt.1 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-02-20T07:55:45.253 INFO:teuthology.orchestra.run.smithi183.stdout:client14540 2024-02-20T07:55:45.253 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-20T07:55:45.253 DEBUG:teuthology.orchestra.run.smithi183:> (cd /home/ubuntu/cephtest/mnt.1 && exec bash -c 'sudo dd if=/sys/kernel/debug/ceph/4537327c-cfc4-11ee-95bb-87774f69a715.client14540/status') 2024-02-20T07:55:45.324 INFO:teuthology.orchestra.run.smithi183.stdout:instance: client.14540 (3)172.21.15.183:0/1755472417 2024-02-20T07:55:45.324 INFO:teuthology.orchestra.run.smithi183.stdout:blocklisted: false 2024-02-20T07:55:45.324 INFO:teuthology.orchestra.run.smithi183.stderr:0+1 records in 2024-02-20T07:55:45.324 INFO:teuthology.orchestra.run.smithi183.stderr:0+1 records out 2024-02-20T07:55:45.324 INFO:teuthology.orchestra.run.smithi183.stderr:72 bytes copied, 2.1065e-05 s, 3.4 MB/s 2024-02-20T07:55:45.326 INFO:teuthology.run_tasks:Running task print... 2024-02-20T07:55:45.337 INFO:teuthology.task.print:**** done client 2024-02-20T07:55:45.337 INFO:teuthology.run_tasks:Running task parallel... 2024-02-20T07:55:45.477 INFO:teuthology.task.parallel:starting parallel... 2024-02-20T07:55:45.478 INFO:teuthology.task.parallel:In parallel, running task sequential... 2024-02-20T07:55:45.478 INFO:teuthology.task.sequential:In sequential, running task cephadm.shell... 2024-02-20T07:55:45.479 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi133.front.sepia.ceph.com 2024-02-20T07:55:45.479 DEBUG:teuthology.orchestra.run.smithi133:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4537327c-cfc4-11ee-95bb-87774f69a715 -e sha1=91ff50e903dcc99ee3ecf9523050878eb452da5c -- bash -c 'ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force' 2024-02-20T07:55:45.480 INFO:teuthology.task.parallel:In parallel, running task sequential... 2024-02-20T07:55:45.480 INFO:teuthology.task.sequential:In sequential, running task workunit... 2024-02-20T07:55:45.484 INFO:tasks.workunit:Pulling workunits from ref 3d72bcfd7affbb4a923c191d212814b4934cfeaa 2024-02-20T07:55:45.484 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-02-20T07:55:45.484 INFO:tasks.workunit:timeout=3h 2024-02-20T07:55:45.484 INFO:tasks.workunit:cleanup=True 2024-02-20T07:55:45.485 DEBUG:teuthology.orchestra.run.smithi133:> stat -- /home/ubuntu/cephtest/mnt.0 2024-02-20T07:55:45.504 INFO:teuthology.orchestra.run.smithi133.stdout: File: /home/ubuntu/cephtest/mnt.0 2024-02-20T07:55:45.504 INFO:teuthology.orchestra.run.smithi133.stdout: Size: 0 Blocks: 0 IO Block: 65536 directory 2024-02-20T07:55:45.504 INFO:teuthology.orchestra.run.smithi133.stdout:Device: 7ah/122d Inode: 1 Links: 2 2024-02-20T07:55:45.505 INFO:teuthology.orchestra.run.smithi133.stdout:Access: (1777/drwxrwxrwt) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-20T07:55:45.505 INFO:teuthology.orchestra.run.smithi133.stdout:Context: system_u:object_r:unlabeled_t:s0 2024-02-20T07:55:45.505 INFO:teuthology.orchestra.run.smithi133.stdout:Access: 1970-01-01 00:00:00.000000000 +0000 2024-02-20T07:55:45.505 INFO:teuthology.orchestra.run.smithi133.stdout:Modify: 2024-02-20 07:55:13.713501794 +0000 2024-02-20T07:55:45.505 INFO:teuthology.orchestra.run.smithi133.stdout:Change: 2024-02-20 07:55:40.773943406 +0000 2024-02-20T07:55:45.505 INFO:teuthology.orchestra.run.smithi133.stdout: Birth: 2024-02-20 07:55:13.713501794 +0000 2024-02-20T07:55:45.505 INFO:tasks.workunit:Did not need to create dir /home/ubuntu/cephtest/mnt.0 2024-02-20T07:55:45.505 DEBUG:teuthology.orchestra.run.smithi133:> cd -- /home/ubuntu/cephtest/mnt.0 && sudo install -d -m 0755 --owner=ubuntu -- client.0 2024-02-20T07:55:45.578 DEBUG:teuthology.orchestra.run.smithi183:> stat -- /home/ubuntu/cephtest/mnt.1 2024-02-20T07:55:45.594 INFO:teuthology.orchestra.run.smithi183.stdout: File: /home/ubuntu/cephtest/mnt.1 2024-02-20T07:55:45.594 INFO:teuthology.orchestra.run.smithi183.stdout: Size: 1 Blocks: 0 IO Block: 65536 directory 2024-02-20T07:55:45.595 INFO:teuthology.orchestra.run.smithi183.stdout:Device: 73h/115d Inode: 1 Links: 3 2024-02-20T07:55:45.595 INFO:teuthology.orchestra.run.smithi183.stdout:Access: (1777/drwxrwxrwt) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-20T07:55:45.595 INFO:teuthology.orchestra.run.smithi183.stdout:Context: system_u:object_r:unlabeled_t:s0 2024-02-20T07:55:45.595 INFO:teuthology.orchestra.run.smithi183.stdout:Access: 1970-01-01 00:00:00.000000000 +0000 2024-02-20T07:55:45.595 INFO:teuthology.orchestra.run.smithi183.stdout:Modify: 2024-02-20 07:55:45.571856293 +0000 2024-02-20T07:55:45.595 INFO:teuthology.orchestra.run.smithi183.stdout:Change: 2024-02-20 07:55:45.571856293 +0000 2024-02-20T07:55:45.595 INFO:teuthology.orchestra.run.smithi183.stdout: Birth: 2024-02-20 07:55:13.713501794 +0000 2024-02-20T07:55:45.595 INFO:tasks.workunit:Did not need to create dir /home/ubuntu/cephtest/mnt.1 2024-02-20T07:55:45.595 DEBUG:teuthology.orchestra.run.smithi183:> cd -- /home/ubuntu/cephtest/mnt.1 && sudo install -d -m 0755 --owner=ubuntu -- client.1 2024-02-20T07:55:45.670 DEBUG:teuthology.orchestra.run.smithi133:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://github.com/batrick/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout 3d72bcfd7affbb4a923c191d212814b4934cfeaa 2024-02-20T07:55:45.670 DEBUG:teuthology.orchestra.run.smithi183:> rm -rf /home/ubuntu/cephtest/clone.client.1 && git clone https://github.com/batrick/ceph.git /home/ubuntu/cephtest/clone.client.1 && cd /home/ubuntu/cephtest/clone.client.1 && git checkout 3d72bcfd7affbb4a923c191d212814b4934cfeaa 2024-02-20T07:55:45.685 INFO:tasks.workunit.client.0.smithi133.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-02-20T07:55:45.725 INFO:tasks.workunit.client.1.smithi183.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.1'... 2024-02-20T07:55:45.759 INFO:teuthology.orchestra.run.smithi133.stderr:Inferring config /var/lib/ceph/4537327c-cfc4-11ee-95bb-87774f69a715/mon.smithi133/config 2024-02-20T07:55:46.991 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:55:46 smithi133 ceph-mon[29117]: pgmap v195: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.8 KiB/s rd, 255 B/s wr, 3 op/s 2024-02-20T07:55:47.047 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:55:46 smithi183 ceph-mon[35832]: pgmap v195: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.8 KiB/s rd, 255 B/s wr, 3 op/s 2024-02-20T07:55:47.483 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:47.477+0000 7fd72acba700 1 -- 172.21.15.133:0/2197865738 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fd7240fbe50 msgr2=0x7fd7240fc260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:55:47.483 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:47.477+0000 7fd72acba700 1 --2- 172.21.15.133:0/2197865738 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fd7240fbe50 0x7fd7240fc260 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7fd718004940 tx=0x7fd718009e30 comp rx=0 tx=0).stop 2024-02-20T07:55:47.483 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:47.477+0000 7fd72acba700 1 -- 172.21.15.133:0/2197865738 shutdown_connections 2024-02-20T07:55:47.484 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:47.477+0000 7fd72acba700 1 --2- 172.21.15.133:0/2197865738 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd7240fcfc0 0x7fd7240fd430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:55:47.484 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:47.477+0000 7fd72acba700 1 --2- 172.21.15.133:0/2197865738 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fd7240fbe50 0x7fd7240fc260 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:55:47.484 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:47.477+0000 7fd72acba700 1 -- 172.21.15.133:0/2197865738 >> 172.21.15.133:0/2197865738 conn(0x7fd7240f7510 msgr2=0x7fd7240f9940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:55:47.484 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:47.477+0000 7fd72acba700 1 -- 172.21.15.133:0/2197865738 shutdown_connections 2024-02-20T07:55:47.484 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:47.477+0000 7fd72acba700 1 -- 172.21.15.133:0/2197865738 wait complete. 2024-02-20T07:55:47.484 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:47.478+0000 7fd72acba700 1 Processor -- start 2024-02-20T07:55:47.484 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:47.478+0000 7fd72acba700 1 -- start start 2024-02-20T07:55:47.484 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:47.478+0000 7fd72acba700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fd7240fbe50 0x7fd72410d8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:55:47.484 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:47.478+0000 7fd72acba700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd7240fcfc0 0x7fd72410ddf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:55:47.484 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:47.478+0000 7fd72acba700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd72410e3f0 con 0x7fd7240fbe50 2024-02-20T07:55:47.484 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:47.478+0000 7fd72acba700 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd72410ec90 con 0x7fd7240fcfc0 2024-02-20T07:55:47.485 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:47.478+0000 7fd728a56700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fd7240fbe50 0x7fd72410d8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:55:47.485 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:47.478+0000 7fd728a56700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fd7240fbe50 0x7fd72410d8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.133:43148/0 (socket says 172.21.15.133:43148) 2024-02-20T07:55:47.485 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:47.478+0000 7fd728a56700 1 -- 172.21.15.133:0/88238148 learned_addr learned my addr 172.21.15.133:0/88238148 (peer_addr_for_me v2:172.21.15.133:0/0) 2024-02-20T07:55:47.485 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:47.478+0000 7fd723fff700 1 --2- 172.21.15.133:0/88238148 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd7240fcfc0 0x7fd72410ddf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:55:47.485 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:47.478+0000 7fd728a56700 1 -- 172.21.15.133:0/88238148 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd7240fcfc0 msgr2=0x7fd72410ddf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:55:47.485 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:47.478+0000 7fd728a56700 1 --2- 172.21.15.133:0/88238148 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd7240fcfc0 0x7fd72410ddf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:55:47.485 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:47.478+0000 7fd728a56700 1 -- 172.21.15.133:0/88238148 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd718005040 con 0x7fd7240fbe50 2024-02-20T07:55:47.485 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:47.479+0000 7fd728a56700 1 --2- 172.21.15.133:0/88238148 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fd7240fbe50 0x7fd72410d8b0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fd718000c00 tx=0x7fd718009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:55:47.485 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:47.479+0000 7fd7217fa700 1 -- 172.21.15.133:0/88238148 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd718021070 con 0x7fd7240fbe50 2024-02-20T07:55:47.485 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:47.479+0000 7fd7217fa700 1 -- 172.21.15.133:0/88238148 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd71801c400 con 0x7fd7240fbe50 2024-02-20T07:55:47.485 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:47.479+0000 7fd72acba700 1 -- 172.21.15.133:0/88238148 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd72410ef10 con 0x7fd7240fbe50 2024-02-20T07:55:47.485 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:47.479+0000 7fd7217fa700 1 -- 172.21.15.133:0/88238148 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd71800e070 con 0x7fd7240fbe50 2024-02-20T07:55:47.485 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:47.479+0000 7fd72acba700 1 -- 172.21.15.133:0/88238148 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd72410f3f0 con 0x7fd7240fbe50 2024-02-20T07:55:47.485 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:47.480+0000 7fd72acba700 1 -- 172.21.15.133:0/88238148 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd72404bed0 con 0x7fd7240fbe50 2024-02-20T07:55:47.485 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:47.480+0000 7fd7217fa700 1 -- 172.21.15.133:0/88238148 <== mon.0 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7fd71801ca50 con 0x7fd7240fbe50 2024-02-20T07:55:47.486 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:47.481+0000 7fd7217fa700 1 --2- 172.21.15.133:0/88238148 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7fd7140692a0 0x7fd71406b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:55:47.486 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:47.481+0000 7fd7217fa700 1 -- 172.21.15.133:0/88238148 <== mon.0 v2:172.21.15.133:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5550+0+0 (secure 0 0 0) 0x7fd718088480 con 0x7fd7240fbe50 2024-02-20T07:55:47.486 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:47.481+0000 7fd723fff700 1 --2- 172.21.15.133:0/88238148 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7fd7140692a0 0x7fd71406b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:55:47.486 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:47.482+0000 7fd723fff700 1 --2- 172.21.15.133:0/88238148 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7fd7140692a0 0x7fd71406b750 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fd710009b80 tx=0x7fd71000b040 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:55:47.487 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:47.486+0000 7fd7217fa700 1 -- 172.21.15.133:0/88238148 <== mon.0 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7fd718058150 con 0x7fd7240fbe50 2024-02-20T07:55:47.683 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:47.681+0000 7fd72acba700 1 -- 172.21.15.133:0/88238148 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command([{prefix=config set, name=mon_warn_on_insecure_global_id_reclaim}] v 0) v1 -- 0x7fd724104940 con 0x7fd7240fbe50 2024-02-20T07:55:47.683 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:47.681+0000 7fd7217fa700 1 -- 172.21.15.133:0/88238148 <== mon.0 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_warn_on_insecure_global_id_reclaim}]=0 v21)=0 v21) v1 ==== 155+0+0 (secure 0 0 0) 0x7fd718057ce0 con 0x7fd7240fbe50 2024-02-20T07:55:47.685 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:47.683+0000 7fd72acba700 1 -- 172.21.15.133:0/88238148 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7fd7140692a0 msgr2=0x7fd71406b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:55:47.685 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:47.683+0000 7fd72acba700 1 --2- 172.21.15.133:0/88238148 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7fd7140692a0 0x7fd71406b750 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fd710009b80 tx=0x7fd71000b040 comp rx=0 tx=0).stop 2024-02-20T07:55:47.685 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:47.683+0000 7fd72acba700 1 -- 172.21.15.133:0/88238148 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fd7240fbe50 msgr2=0x7fd72410d8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:55:47.685 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:47.683+0000 7fd72acba700 1 --2- 172.21.15.133:0/88238148 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fd7240fbe50 0x7fd72410d8b0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fd718000c00 tx=0x7fd718009e30 comp rx=0 tx=0).stop 2024-02-20T07:55:47.685 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:47.684+0000 7fd72acba700 1 -- 172.21.15.133:0/88238148 shutdown_connections 2024-02-20T07:55:47.685 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:47.684+0000 7fd72acba700 1 --2- 172.21.15.133:0/88238148 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fd7240fbe50 0x7fd72410d8b0 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:55:47.685 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:47.684+0000 7fd72acba700 1 --2- 172.21.15.133:0/88238148 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7fd7140692a0 0x7fd71406b750 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:55:47.685 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:47.684+0000 7fd72acba700 1 --2- 172.21.15.133:0/88238148 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd7240fcfc0 0x7fd72410ddf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:55:47.686 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:47.684+0000 7fd72acba700 1 -- 172.21.15.133:0/88238148 >> 172.21.15.133:0/88238148 conn(0x7fd7240f7510 msgr2=0x7fd724065f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:55:47.686 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:47.684+0000 7fd72acba700 1 -- 172.21.15.133:0/88238148 shutdown_connections 2024-02-20T07:55:47.686 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:47.684+0000 7fd72acba700 1 -- 172.21.15.133:0/88238148 wait complete. 2024-02-20T07:55:48.394 DEBUG:teuthology.orchestra.run.smithi133:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4537327c-cfc4-11ee-95bb-87774f69a715 -e sha1=91ff50e903dcc99ee3ecf9523050878eb452da5c -- bash -c 'ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force' 2024-02-20T07:55:48.692 INFO:teuthology.orchestra.run.smithi133.stderr:Inferring config /var/lib/ceph/4537327c-cfc4-11ee-95bb-87774f69a715/mon.smithi133/config 2024-02-20T07:55:48.963 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:55:48 smithi133 ceph-mon[29117]: pgmap v196: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.2 KiB/s rd, 341 B/s wr, 4 op/s 2024-02-20T07:55:49.047 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:55:48 smithi183 ceph-mon[35832]: pgmap v196: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.2 KiB/s rd, 341 B/s wr, 4 op/s 2024-02-20T07:55:50.403 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:50.400+0000 7f850a8e8700 1 -- 172.21.15.133:0/1893676693 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f85041003a0 msgr2=0x7f8504102780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:55:50.403 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:50.400+0000 7f850a8e8700 1 --2- 172.21.15.133:0/1893676693 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f85041003a0 0x7f8504102780 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f84f8004940 tx=0x7f84f8009e30 comp rx=0 tx=0).stop 2024-02-20T07:55:50.403 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:50.400+0000 7f850a8e8700 1 -- 172.21.15.133:0/1893676693 shutdown_connections 2024-02-20T07:55:50.403 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:50.400+0000 7f850a8e8700 1 --2- 172.21.15.133:0/1893676693 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f85041003a0 0x7f8504102780 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:55:50.403 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:50.400+0000 7f850a8e8700 1 --2- 172.21.15.133:0/1893676693 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f85040fda80 0x7f85040ffe60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:55:50.404 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:50.400+0000 7f850a8e8700 1 -- 172.21.15.133:0/1893676693 >> 172.21.15.133:0/1893676693 conn(0x7f85040f74e0 msgr2=0x7f85040f9930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:55:50.404 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:50.400+0000 7f850a8e8700 1 -- 172.21.15.133:0/1893676693 shutdown_connections 2024-02-20T07:55:50.404 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:50.400+0000 7f850a8e8700 1 -- 172.21.15.133:0/1893676693 wait complete. 2024-02-20T07:55:50.404 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:50.401+0000 7f850a8e8700 1 Processor -- start 2024-02-20T07:55:50.404 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:50.401+0000 7f850a8e8700 1 -- start start 2024-02-20T07:55:50.404 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:50.401+0000 7f850a8e8700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f85040fda80 0x7f85040fd510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:55:50.404 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:50.401+0000 7f850a8e8700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f85041003a0 0x7f85040fb990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:55:50.404 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:50.401+0000 7f850a8e8700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85040fbf90 con 0x7f85040fda80 2024-02-20T07:55:50.404 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:50.401+0000 7f850a8e8700 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85040fc100 con 0x7f85041003a0 2024-02-20T07:55:50.404 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:50.401+0000 7f8503fff700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f85040fda80 0x7f85040fd510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:55:50.404 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:50.401+0000 7f8503fff700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f85040fda80 0x7f85040fd510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.133:43168/0 (socket says 172.21.15.133:43168) 2024-02-20T07:55:50.404 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:50.401+0000 7f8503fff700 1 -- 172.21.15.133:0/887791628 learned_addr learned my addr 172.21.15.133:0/887791628 (peer_addr_for_me v2:172.21.15.133:0/0) 2024-02-20T07:55:50.404 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:50.401+0000 7f85037fe700 1 --2- 172.21.15.133:0/887791628 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f85041003a0 0x7f85040fb990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:55:50.405 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:50.401+0000 7f8503fff700 1 -- 172.21.15.133:0/887791628 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f85041003a0 msgr2=0x7f85040fb990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:55:50.405 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:50.402+0000 7f8503fff700 1 --2- 172.21.15.133:0/887791628 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f85041003a0 0x7f85040fb990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:55:50.405 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:50.402+0000 7f8503fff700 1 -- 172.21.15.133:0/887791628 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84f8005040 con 0x7f85040fda80 2024-02-20T07:55:50.405 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:50.402+0000 7f8503fff700 1 --2- 172.21.15.133:0/887791628 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f85040fda80 0x7f85040fd510 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f84f400eb40 tx=0x7f84f400b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:55:50.405 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:50.402+0000 7f8500ff9700 1 -- 172.21.15.133:0/887791628 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f84f4007cc0 con 0x7f85040fda80 2024-02-20T07:55:50.406 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:50.402+0000 7f8500ff9700 1 -- 172.21.15.133:0/887791628 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f84f400f040 con 0x7f85040fda80 2024-02-20T07:55:50.406 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:50.402+0000 7f850a8e8700 1 -- 172.21.15.133:0/887791628 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f85040fc380 con 0x7f85040fda80 2024-02-20T07:55:50.406 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:50.402+0000 7f8500ff9700 1 -- 172.21.15.133:0/887791628 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f84f400ca10 con 0x7f85040fda80 2024-02-20T07:55:50.406 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:50.402+0000 7f850a8e8700 1 -- 172.21.15.133:0/887791628 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8504115910 con 0x7f85040fda80 2024-02-20T07:55:50.406 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:50.403+0000 7f850a8e8700 1 -- 172.21.15.133:0/887791628 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8504107a50 con 0x7f85040fda80 2024-02-20T07:55:50.406 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:50.404+0000 7f8500ff9700 1 -- 172.21.15.133:0/887791628 <== mon.0 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7f84f400cb70 con 0x7f85040fda80 2024-02-20T07:55:50.406 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:50.404+0000 7f8500ff9700 1 --2- 172.21.15.133:0/887791628 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7f84ec0692a0 0x7f84ec06b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:55:50.406 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:50.404+0000 7f8500ff9700 1 -- 172.21.15.133:0/887791628 <== mon.0 v2:172.21.15.133:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f84f4086cf0 con 0x7f85040fda80 2024-02-20T07:55:50.406 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:50.404+0000 7f85037fe700 1 --2- 172.21.15.133:0/887791628 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7f84ec0692a0 0x7f84ec06b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:55:50.406 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:50.405+0000 7f85037fe700 1 --2- 172.21.15.133:0/887791628 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7f84ec0692a0 0x7f84ec06b750 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f84f8009b30 tx=0x7f84f8009e30 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:55:50.410 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:50.408+0000 7f8500ff9700 1 -- 172.21.15.133:0/887791628 <== mon.0 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f84f404dbd0 con 0x7f85040fda80 2024-02-20T07:55:50.603 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:50.601+0000 7f850a8e8700 1 -- 172.21.15.133:0/887791628 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command([{prefix=config set, name=mon_warn_on_insecure_global_id_reclaim_allowed}] v 0) v1 -- 0x7f850404bed0 con 0x7f85040fda80 2024-02-20T07:55:50.603 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:50.602+0000 7f8500ff9700 1 -- 172.21.15.133:0/887791628 <== mon.0 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_warn_on_insecure_global_id_reclaim_allowed}]=0 v21)=0 v21) v1 ==== 163+0+0 (secure 0 0 0) 0x7f84f4056880 con 0x7f85040fda80 2024-02-20T07:55:50.606 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:50.604+0000 7f850a8e8700 1 -- 172.21.15.133:0/887791628 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7f84ec0692a0 msgr2=0x7f84ec06b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:55:50.606 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:50.604+0000 7f850a8e8700 1 --2- 172.21.15.133:0/887791628 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7f84ec0692a0 0x7f84ec06b750 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f84f8009b30 tx=0x7f84f8009e30 comp rx=0 tx=0).stop 2024-02-20T07:55:50.606 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:50.604+0000 7f850a8e8700 1 -- 172.21.15.133:0/887791628 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f85040fda80 msgr2=0x7f85040fd510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:55:50.606 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:50.604+0000 7f850a8e8700 1 --2- 172.21.15.133:0/887791628 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f85040fda80 0x7f85040fd510 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f84f400eb40 tx=0x7f84f400b040 comp rx=0 tx=0).stop 2024-02-20T07:55:50.606 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:50.604+0000 7f850a8e8700 1 -- 172.21.15.133:0/887791628 shutdown_connections 2024-02-20T07:55:50.606 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:50.604+0000 7f850a8e8700 1 --2- 172.21.15.133:0/887791628 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f85040fda80 0x7f85040fd510 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:55:50.606 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:50.604+0000 7f850a8e8700 1 --2- 172.21.15.133:0/887791628 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7f84ec0692a0 0x7f84ec06b750 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:55:50.606 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:50.604+0000 7f850a8e8700 1 --2- 172.21.15.133:0/887791628 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f85041003a0 0x7f85040fb990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:55:50.606 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:50.604+0000 7f850a8e8700 1 -- 172.21.15.133:0/887791628 >> 172.21.15.133:0/887791628 conn(0x7f85040f74e0 msgr2=0x7f85040f9930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:55:50.606 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:50.604+0000 7f850a8e8700 1 -- 172.21.15.133:0/887791628 shutdown_connections 2024-02-20T07:55:50.606 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:50.604+0000 7f850a8e8700 1 -- 172.21.15.133:0/887791628 wait complete. 2024-02-20T07:55:50.984 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:55:50 smithi133 ceph-mon[29117]: pgmap v197: 65 pgs: 65 active+clean; 586 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.8 KiB/s rd, 938 B/s wr, 5 op/s 2024-02-20T07:55:51.047 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:55:50 smithi183 ceph-mon[35832]: pgmap v197: 65 pgs: 65 active+clean; 586 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.8 KiB/s rd, 938 B/s wr, 5 op/s 2024-02-20T07:55:51.317 DEBUG:teuthology.orchestra.run.smithi133:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4537327c-cfc4-11ee-95bb-87774f69a715 -e sha1=91ff50e903dcc99ee3ecf9523050878eb452da5c -- bash -c 'ceph config set global log_to_journald false --force' 2024-02-20T07:55:51.580 INFO:teuthology.orchestra.run.smithi133.stderr:Inferring config /var/lib/ceph/4537327c-cfc4-11ee-95bb-87774f69a715/mon.smithi133/config 2024-02-20T07:55:52.991 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:55:52 smithi133 ceph-mon[29117]: pgmap v198: 65 pgs: 65 active+clean; 586 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.8 KiB/s rd, 938 B/s wr, 4 op/s 2024-02-20T07:55:53.047 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:55:52 smithi183 ceph-mon[35832]: pgmap v198: 65 pgs: 65 active+clean; 586 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.8 KiB/s rd, 938 B/s wr, 4 op/s 2024-02-20T07:55:53.272 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:53.268+0000 7f5c5521e700 1 -- 172.21.15.133:0/2562022424 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f5c501003a0 msgr2=0x7f5c50102780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:55:53.272 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:53.268+0000 7f5c5521e700 1 --2- 172.21.15.133:0/2562022424 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f5c501003a0 0x7f5c50102780 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f5c44004940 tx=0x7f5c44009e30 comp rx=0 tx=0).stop 2024-02-20T07:55:53.272 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:53.268+0000 7f5c5521e700 1 -- 172.21.15.133:0/2562022424 shutdown_connections 2024-02-20T07:55:53.272 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:53.268+0000 7f5c5521e700 1 --2- 172.21.15.133:0/2562022424 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f5c501003a0 0x7f5c50102780 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:55:53.272 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:53.268+0000 7f5c5521e700 1 --2- 172.21.15.133:0/2562022424 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5c500fda80 0x7f5c500ffe60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:55:53.272 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:53.268+0000 7f5c5521e700 1 -- 172.21.15.133:0/2562022424 >> 172.21.15.133:0/2562022424 conn(0x7f5c500f74e0 msgr2=0x7f5c500f9930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:55:53.272 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:53.268+0000 7f5c5521e700 1 -- 172.21.15.133:0/2562022424 shutdown_connections 2024-02-20T07:55:53.272 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:53.268+0000 7f5c5521e700 1 -- 172.21.15.133:0/2562022424 wait complete. 2024-02-20T07:55:53.272 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:53.269+0000 7f5c5521e700 1 Processor -- start 2024-02-20T07:55:53.272 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:53.269+0000 7f5c5521e700 1 -- start start 2024-02-20T07:55:53.272 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:53.269+0000 7f5c5521e700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5c500fda80 0x7f5c5010d8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:55:53.272 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:53.269+0000 7f5c5521e700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f5c501003a0 0x7f5c5010de20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:55:53.273 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:53.269+0000 7f5c5521e700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c5010e420 con 0x7f5c501003a0 2024-02-20T07:55:53.273 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:53.269+0000 7f5c5521e700 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c5010ecc0 con 0x7f5c500fda80 2024-02-20T07:55:53.273 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:53.269+0000 7f5c4e59c700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f5c501003a0 0x7f5c5010de20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:55:53.273 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:53.269+0000 7f5c4e59c700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f5c501003a0 0x7f5c5010de20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.133:43184/0 (socket says 172.21.15.133:43184) 2024-02-20T07:55:53.273 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:53.269+0000 7f5c4e59c700 1 -- 172.21.15.133:0/1816258961 learned_addr learned my addr 172.21.15.133:0/1816258961 (peer_addr_for_me v2:172.21.15.133:0/0) 2024-02-20T07:55:53.273 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:53.269+0000 7f5c4ed9d700 1 --2- 172.21.15.133:0/1816258961 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5c500fda80 0x7f5c5010d8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:55:53.273 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:53.270+0000 7f5c4e59c700 1 -- 172.21.15.133:0/1816258961 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5c500fda80 msgr2=0x7f5c5010d8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:55:53.273 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:53.270+0000 7f5c4e59c700 1 --2- 172.21.15.133:0/1816258961 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5c500fda80 0x7f5c5010d8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:55:53.273 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:53.270+0000 7f5c4e59c700 1 -- 172.21.15.133:0/1816258961 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c44005040 con 0x7f5c501003a0 2024-02-20T07:55:53.273 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:53.270+0000 7f5c4e59c700 1 --2- 172.21.15.133:0/1816258961 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f5c501003a0 0x7f5c5010de20 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f5c44000c00 tx=0x7f5c44009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:55:53.273 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:53.270+0000 7f5c377fe700 1 -- 172.21.15.133:0/1816258961 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5c44021070 con 0x7f5c501003a0 2024-02-20T07:55:53.273 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:53.270+0000 7f5c5521e700 1 -- 172.21.15.133:0/1816258961 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5c5010ef40 con 0x7f5c501003a0 2024-02-20T07:55:53.273 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:53.270+0000 7f5c377fe700 1 -- 172.21.15.133:0/1816258961 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5c4401c400 con 0x7f5c501003a0 2024-02-20T07:55:53.273 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:53.270+0000 7f5c377fe700 1 -- 172.21.15.133:0/1816258961 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5c4400e070 con 0x7f5c501003a0 2024-02-20T07:55:53.274 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:53.270+0000 7f5c5521e700 1 -- 172.21.15.133:0/1816258961 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5c5010f420 con 0x7f5c501003a0 2024-02-20T07:55:53.274 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:53.271+0000 7f5c5521e700 1 -- 172.21.15.133:0/1816258961 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5c50107a50 con 0x7f5c501003a0 2024-02-20T07:55:53.274 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:53.271+0000 7f5c377fe700 1 -- 172.21.15.133:0/1816258961 <== mon.0 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7f5c44013af0 con 0x7f5c501003a0 2024-02-20T07:55:53.275 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:53.272+0000 7f5c377fe700 1 --2- 172.21.15.133:0/1816258961 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7f5c380692b0 0x7f5c3806b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:55:53.275 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:53.272+0000 7f5c377fe700 1 -- 172.21.15.133:0/1816258961 <== mon.0 v2:172.21.15.133:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f5c44088440 con 0x7f5c501003a0 2024-02-20T07:55:53.275 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:53.272+0000 7f5c4ed9d700 1 --2- 172.21.15.133:0/1816258961 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7f5c380692b0 0x7f5c3806b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:55:53.275 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:53.272+0000 7f5c4ed9d700 1 --2- 172.21.15.133:0/1816258961 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7f5c380692b0 0x7f5c3806b760 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f5c40009b80 tx=0x7f5c4000b040 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:55:53.277 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:53.276+0000 7f5c377fe700 1 -- 172.21.15.133:0/1816258961 <== mon.0 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f5c44058150 con 0x7f5c501003a0 2024-02-20T07:55:53.472 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:53.470+0000 7f5c5521e700 1 -- 172.21.15.133:0/1816258961 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command([{prefix=config set, name=log_to_journald}] v 0) v1 -- 0x7f5c5004bed0 con 0x7f5c501003a0 2024-02-20T07:55:53.472 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:53.470+0000 7f5c377fe700 1 -- 172.21.15.133:0/1816258961 <== mon.0 v2:172.21.15.133:3300/0 7 ==== mon_command_ack([{prefix=config set, name=log_to_journald}]=0 v21)=0 v21) v1 ==== 135+0+0 (secure 0 0 0) 0x7f5c44057ce0 con 0x7f5c501003a0 2024-02-20T07:55:53.474 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:53.472+0000 7f5c5521e700 1 -- 172.21.15.133:0/1816258961 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7f5c380692b0 msgr2=0x7f5c3806b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:55:53.474 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:53.472+0000 7f5c5521e700 1 --2- 172.21.15.133:0/1816258961 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7f5c380692b0 0x7f5c3806b760 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f5c40009b80 tx=0x7f5c4000b040 comp rx=0 tx=0).stop 2024-02-20T07:55:53.474 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:53.472+0000 7f5c5521e700 1 -- 172.21.15.133:0/1816258961 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f5c501003a0 msgr2=0x7f5c5010de20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:55:53.474 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:53.472+0000 7f5c5521e700 1 --2- 172.21.15.133:0/1816258961 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f5c501003a0 0x7f5c5010de20 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f5c44000c00 tx=0x7f5c44009e30 comp rx=0 tx=0).stop 2024-02-20T07:55:53.475 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:53.472+0000 7f5c5521e700 1 -- 172.21.15.133:0/1816258961 shutdown_connections 2024-02-20T07:55:53.475 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:53.472+0000 7f5c5521e700 1 --2- 172.21.15.133:0/1816258961 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7f5c380692b0 0x7f5c3806b760 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:55:53.475 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:53.472+0000 7f5c5521e700 1 --2- 172.21.15.133:0/1816258961 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5c500fda80 0x7f5c5010d8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:55:53.475 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:53.473+0000 7f5c5521e700 1 --2- 172.21.15.133:0/1816258961 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f5c501003a0 0x7f5c5010de20 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:55:53.475 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:53.473+0000 7f5c5521e700 1 -- 172.21.15.133:0/1816258961 >> 172.21.15.133:0/1816258961 conn(0x7f5c500f74e0 msgr2=0x7f5c500f9930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:55:53.475 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:53.473+0000 7f5c5521e700 1 -- 172.21.15.133:0/1816258961 shutdown_connections 2024-02-20T07:55:53.475 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:53.473+0000 7f5c5521e700 1 -- 172.21.15.133:0/1816258961 wait complete. 2024-02-20T07:55:53.990 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:55:53 smithi183 ceph-mon[35832]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-20T07:55:53.991 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:55:53 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-20T07:55:54.218 DEBUG:teuthology.orchestra.run.smithi133:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4537327c-cfc4-11ee-95bb-87774f69a715 -e sha1=91ff50e903dcc99ee3ecf9523050878eb452da5c -- bash -c 'ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 --daemon-types mgr' 2024-02-20T07:55:54.486 INFO:teuthology.orchestra.run.smithi133.stderr:Inferring config /var/lib/ceph/4537327c-cfc4-11ee-95bb-87774f69a715/mon.smithi133/config 2024-02-20T07:55:54.785 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:55:54 smithi133 ceph-mon[29117]: pgmap v199: 65 pgs: 65 active+clean; 586 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.8 KiB/s rd, 1.1 KiB/s wr, 6 op/s 2024-02-20T07:55:55.047 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:55:54 smithi183 ceph-mon[35832]: pgmap v199: 65 pgs: 65 active+clean; 586 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.8 KiB/s rd, 1.1 KiB/s wr, 6 op/s 2024-02-20T07:55:56.229 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:56.226+0000 7f456ae2e700 1 -- 172.21.15.133:0/1607900101 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f45640fbe40 msgr2=0x7f45640fc250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:55:56.230 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:56.226+0000 7f456ae2e700 1 --2- 172.21.15.133:0/1607900101 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f45640fbe40 0x7f45640fc250 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f4558004940 tx=0x7f4558009e30 comp rx=0 tx=0).stop 2024-02-20T07:55:56.230 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:56.226+0000 7f456ae2e700 1 -- 172.21.15.133:0/1607900101 shutdown_connections 2024-02-20T07:55:56.230 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:56.226+0000 7f456ae2e700 1 --2- 172.21.15.133:0/1607900101 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f45640fcfb0 0x7f45640fd420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:55:56.230 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:56.226+0000 7f456ae2e700 1 --2- 172.21.15.133:0/1607900101 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f45640fbe40 0x7f45640fc250 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:55:56.230 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:56.226+0000 7f456ae2e700 1 -- 172.21.15.133:0/1607900101 >> 172.21.15.133:0/1607900101 conn(0x7f45640f74e0 msgr2=0x7f45640f9930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:55:56.230 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:56.226+0000 7f456ae2e700 1 -- 172.21.15.133:0/1607900101 shutdown_connections 2024-02-20T07:55:56.230 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:56.226+0000 7f456ae2e700 1 -- 172.21.15.133:0/1607900101 wait complete. 2024-02-20T07:55:56.230 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:56.226+0000 7f456ae2e700 1 Processor -- start 2024-02-20T07:55:56.230 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:56.227+0000 7f456ae2e700 1 -- start start 2024-02-20T07:55:56.230 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:56.227+0000 7f456ae2e700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f45640fbe40 0x7f456410fab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:55:56.230 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:56.227+0000 7f456ae2e700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f45640fcfb0 0x7f456410fff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:55:56.230 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:56.227+0000 7f456ae2e700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45641105f0 con 0x7f45640fbe40 2024-02-20T07:55:56.230 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:56.227+0000 7f456ae2e700 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4564110e90 con 0x7f45640fcfb0 2024-02-20T07:55:56.230 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:56.227+0000 7f4568bca700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f45640fbe40 0x7f456410fab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:55:56.230 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:56.227+0000 7f4568bca700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f45640fbe40 0x7f456410fab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.133:43208/0 (socket says 172.21.15.133:43208) 2024-02-20T07:55:56.231 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:56.227+0000 7f4568bca700 1 -- 172.21.15.133:0/3305947218 learned_addr learned my addr 172.21.15.133:0/3305947218 (peer_addr_for_me v2:172.21.15.133:0/0) 2024-02-20T07:55:56.231 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:56.227+0000 7f4563fff700 1 --2- 172.21.15.133:0/3305947218 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f45640fcfb0 0x7f456410fff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:55:56.231 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:56.227+0000 7f4568bca700 1 -- 172.21.15.133:0/3305947218 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f45640fcfb0 msgr2=0x7f456410fff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:55:56.231 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:56.227+0000 7f4568bca700 1 --2- 172.21.15.133:0/3305947218 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f45640fcfb0 0x7f456410fff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:55:56.231 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:56.227+0000 7f4568bca700 1 -- 172.21.15.133:0/3305947218 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4558005040 con 0x7f45640fbe40 2024-02-20T07:55:56.231 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:56.228+0000 7f4568bca700 1 --2- 172.21.15.133:0/3305947218 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f45640fbe40 0x7f456410fab0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f45580093b0 tx=0x7f4558009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:55:56.231 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:56.228+0000 7f45617fa700 1 -- 172.21.15.133:0/3305947218 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f455800e040 con 0x7f45640fbe40 2024-02-20T07:55:56.231 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:56.228+0000 7f456ae2e700 1 -- 172.21.15.133:0/3305947218 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4564111110 con 0x7f45640fbe40 2024-02-20T07:55:56.232 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:56.228+0000 7f45617fa700 1 -- 172.21.15.133:0/3305947218 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f455800bcc0 con 0x7f45640fbe40 2024-02-20T07:55:56.232 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:56.228+0000 7f45617fa700 1 -- 172.21.15.133:0/3305947218 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f455801c630 con 0x7f45640fbe40 2024-02-20T07:55:56.232 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:56.228+0000 7f456ae2e700 1 -- 172.21.15.133:0/3305947218 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4564111650 con 0x7f45640fbe40 2024-02-20T07:55:56.233 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:56.229+0000 7f456ae2e700 1 -- 172.21.15.133:0/3305947218 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f456404bed0 con 0x7f45640fbe40 2024-02-20T07:55:56.233 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:56.229+0000 7f45617fa700 1 -- 172.21.15.133:0/3305947218 <== mon.0 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7f4558013980 con 0x7f45640fbe40 2024-02-20T07:55:56.233 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:56.230+0000 7f45617fa700 1 --2- 172.21.15.133:0/3305947218 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7f45540692a0 0x7f455406b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:55:56.233 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:56.230+0000 7f45617fa700 1 -- 172.21.15.133:0/3305947218 <== mon.0 v2:172.21.15.133:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f4558088590 con 0x7f45640fbe40 2024-02-20T07:55:56.233 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:56.230+0000 7f4563fff700 1 --2- 172.21.15.133:0/3305947218 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7f45540692a0 0x7f455406b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:55:56.233 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:56.230+0000 7f4563fff700 1 --2- 172.21.15.133:0/3305947218 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7f45540692a0 0x7f455406b750 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f4550009fd0 tx=0x7f455000b040 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:55:56.235 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:56.234+0000 7f45617fa700 1 -- 172.21.15.133:0/3305947218 <== mon.0 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f4558058300 con 0x7f45640fbe40 2024-02-20T07:55:56.445 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:55:56.443+0000 7f456ae2e700 1 -- 172.21.15.133:0/3305947218 --> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] -- mgr_command(tid 0: {"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:91ff50e903dcc99ee3ecf9523050878eb452da5c", "daemon_types": "mgr", "target": ["mon-mgr", ""]}) v1 -- 0x7f4564065dd0 con 0x7f45540692a0 2024-02-20T07:55:56.991 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:55:56 smithi133 ceph-mon[29117]: pgmap v200: 65 pgs: 65 active+clean; 586 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.2 KiB/s rd, 853 B/s wr, 4 op/s 2024-02-20T07:55:57.047 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:55:56 smithi183 ceph-mon[35832]: pgmap v200: 65 pgs: 65 active+clean; 586 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.2 KiB/s rd, 853 B/s wr, 4 op/s 2024-02-20T07:55:57.991 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:55:57 smithi133 ceph-mon[29117]: from='client.14556 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:91ff50e903dcc99ee3ecf9523050878eb452da5c", "daemon_types": "mgr", "target": ["mon-mgr", ""]}]: dispatch 2024-02-20T07:55:58.047 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:55:57 smithi183 ceph-mon[35832]: from='client.14556 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:91ff50e903dcc99ee3ecf9523050878eb452da5c", "daemon_types": "mgr", "target": ["mon-mgr", ""]}]: dispatch 2024-02-20T07:55:58.991 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:55:58 smithi133 ceph-mon[29117]: pgmap v201: 65 pgs: 65 active+clean; 586 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.7 KiB/s rd, 853 B/s wr, 4 op/s 2024-02-20T07:55:59.047 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:55:58 smithi183 ceph-mon[35832]: pgmap v201: 65 pgs: 65 active+clean; 586 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.7 KiB/s rd, 853 B/s wr, 4 op/s 2024-02-20T07:56:00.991 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:56:00 smithi133 ceph-mon[29117]: pgmap v202: 65 pgs: 65 active+clean; 586 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.7 KiB/s rd, 767 B/s wr, 5 op/s 2024-02-20T07:56:01.047 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:56:00 smithi183 ceph-mon[35832]: pgmap v202: 65 pgs: 65 active+clean; 586 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.7 KiB/s rd, 767 B/s wr, 5 op/s 2024-02-20T07:56:02.991 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:56:02 smithi133 ceph-mon[29117]: pgmap v203: 65 pgs: 65 active+clean; 586 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 170 B/s wr, 4 op/s 2024-02-20T07:56:03.047 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:56:02 smithi183 ceph-mon[35832]: pgmap v203: 65 pgs: 65 active+clean; 586 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 170 B/s wr, 4 op/s 2024-02-20T07:56:04.991 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:56:04 smithi133 ceph-mon[29117]: pgmap v204: 65 pgs: 65 active+clean; 586 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 170 B/s wr, 4 op/s 2024-02-20T07:56:05.048 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:56:04 smithi183 ceph-mon[35832]: pgmap v204: 65 pgs: 65 active+clean; 586 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 170 B/s wr, 4 op/s 2024-02-20T07:56:06.991 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:56:06 smithi133 ceph-mon[29117]: pgmap v205: 65 pgs: 65 active+clean; 586 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-02-20T07:56:07.047 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:56:06 smithi183 ceph-mon[35832]: pgmap v205: 65 pgs: 65 active+clean; 586 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-02-20T07:56:08.991 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:56:08 smithi133 ceph-mon[29117]: pgmap v206: 65 pgs: 65 active+clean; 586 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-02-20T07:56:08.992 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:56:08 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-20T07:56:09.047 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:56:08 smithi183 ceph-mon[35832]: pgmap v206: 65 pgs: 65 active+clean; 586 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-02-20T07:56:09.047 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:56:08 smithi183 ceph-mon[35832]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-20T07:56:10.991 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:56:10 smithi133 ceph-mon[29117]: pgmap v207: 65 pgs: 65 active+clean; 586 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-02-20T07:56:11.047 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:56:10 smithi183 ceph-mon[35832]: pgmap v207: 65 pgs: 65 active+clean; 586 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-02-20T07:56:12.991 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:56:12 smithi133 ceph-mon[29117]: pgmap v208: 65 pgs: 65 active+clean; 586 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-02-20T07:56:13.047 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:56:12 smithi183 ceph-mon[35832]: pgmap v208: 65 pgs: 65 active+clean; 586 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-02-20T07:56:15.047 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:56:14 smithi183 ceph-mon[35832]: pgmap v209: 65 pgs: 65 active+clean; 586 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-02-20T07:56:15.241 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:56:14 smithi133 ceph-mon[29117]: pgmap v209: 65 pgs: 65 active+clean; 586 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-02-20T07:56:17.047 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:56:16 smithi183 ceph-mon[35832]: pgmap v210: 65 pgs: 65 active+clean; 586 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-02-20T07:56:17.241 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:56:16 smithi133 ceph-mon[29117]: pgmap v210: 65 pgs: 65 active+clean; 586 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-02-20T07:56:19.047 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:56:18 smithi183 ceph-mon[35832]: pgmap v211: 65 pgs: 65 active+clean; 586 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-02-20T07:56:19.241 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:56:18 smithi133 ceph-mon[29117]: pgmap v211: 65 pgs: 65 active+clean; 586 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-02-20T07:56:21.047 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:56:20 smithi183 ceph-mon[35832]: pgmap v212: 65 pgs: 65 active+clean; 587 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 255 B/s wr, 4 op/s 2024-02-20T07:56:21.241 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:56:20 smithi133 ceph-mon[29117]: pgmap v212: 65 pgs: 65 active+clean; 587 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 255 B/s wr, 4 op/s 2024-02-20T07:56:23.047 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:56:22 smithi183 ceph-mon[35832]: pgmap v213: 65 pgs: 65 active+clean; 587 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 255 B/s wr, 3 op/s 2024-02-20T07:56:23.241 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:56:22 smithi133 ceph-mon[29117]: pgmap v213: 65 pgs: 65 active+clean; 587 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 255 B/s wr, 3 op/s 2024-02-20T07:56:24.047 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:56:23 smithi183 ceph-mon[35832]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-20T07:56:24.241 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:56:23 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-20T07:56:25.047 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:56:24 smithi183 ceph-mon[35832]: pgmap v214: 65 pgs: 65 active+clean; 589 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.4 KiB/s rd, 511 B/s wr, 5 op/s 2024-02-20T07:56:25.084 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:56:24 smithi133 ceph-mon[29117]: pgmap v214: 65 pgs: 65 active+clean; 589 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.4 KiB/s rd, 511 B/s wr, 5 op/s 2024-02-20T07:56:25.834 INFO:tasks.workunit.client.1.smithi183.stderr:Note: switching to '3d72bcfd7affbb4a923c191d212814b4934cfeaa'. 2024-02-20T07:56:25.834 INFO:tasks.workunit.client.1.smithi183.stderr: 2024-02-20T07:56:25.834 INFO:tasks.workunit.client.1.smithi183.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-02-20T07:56:25.834 INFO:tasks.workunit.client.1.smithi183.stderr:changes and commit them, and you can discard any commits you make in this 2024-02-20T07:56:25.834 INFO:tasks.workunit.client.1.smithi183.stderr:state without impacting any branches by switching back to a branch. 2024-02-20T07:56:25.835 INFO:tasks.workunit.client.1.smithi183.stderr: 2024-02-20T07:56:25.835 INFO:tasks.workunit.client.1.smithi183.stderr:If you want to create a new branch to retain commits you create, you may 2024-02-20T07:56:25.835 INFO:tasks.workunit.client.1.smithi183.stderr:do so (now or later) by using -c with the switch command. Example: 2024-02-20T07:56:25.835 INFO:tasks.workunit.client.1.smithi183.stderr: 2024-02-20T07:56:25.835 INFO:tasks.workunit.client.1.smithi183.stderr: git switch -c 2024-02-20T07:56:25.835 INFO:tasks.workunit.client.1.smithi183.stderr: 2024-02-20T07:56:25.835 INFO:tasks.workunit.client.1.smithi183.stderr:Or undo this operation with: 2024-02-20T07:56:25.835 INFO:tasks.workunit.client.1.smithi183.stderr: 2024-02-20T07:56:25.835 INFO:tasks.workunit.client.1.smithi183.stderr: git switch - 2024-02-20T07:56:25.835 INFO:tasks.workunit.client.1.smithi183.stderr: 2024-02-20T07:56:25.835 INFO:tasks.workunit.client.1.smithi183.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-02-20T07:56:25.835 INFO:tasks.workunit.client.1.smithi183.stderr: 2024-02-20T07:56:25.835 INFO:tasks.workunit.client.1.smithi183.stderr:HEAD is now at 3d72bcfd7af test 2024-02-20T07:56:25.839 DEBUG:teuthology.orchestra.run.smithi183:> cd -- /home/ubuntu/cephtest/clone.client.1/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.1 2024-02-20T07:56:25.906 INFO:tasks.workunit.client.1.smithi183.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-02-20T07:56:25.908 INFO:tasks.workunit.client.1.smithi183.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.1/qa/workunits/direct_io' 2024-02-20T07:56:25.908 INFO:tasks.workunit.client.1.smithi183.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-02-20T07:56:26.153 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:56:26.153+0000 7f45617fa700 1 -- 172.21.15.133:0/3305947218 <== mgr.14208 v2:172.21.15.133:6800/2703586938 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+89 (secure 0 0 0) 0x7f4564065dd0 con 0x7f45540692a0 2024-02-20T07:56:26.154 INFO:teuthology.orchestra.run.smithi133.stdout:Initiating upgrade to quay.ceph.io/ceph-ci/ceph:91ff50e903dcc99ee3ecf9523050878eb452da5c 2024-02-20T07:56:26.155 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:56:26.155+0000 7f456ae2e700 1 -- 172.21.15.133:0/3305947218 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7f45540692a0 msgr2=0x7f455406b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:56:26.156 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:56:26.155+0000 7f456ae2e700 1 --2- 172.21.15.133:0/3305947218 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7f45540692a0 0x7f455406b750 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f4550009fd0 tx=0x7f455000b040 comp rx=0 tx=0).stop 2024-02-20T07:56:26.156 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:56:26.155+0000 7f456ae2e700 1 -- 172.21.15.133:0/3305947218 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f45640fbe40 msgr2=0x7f456410fab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:56:26.156 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:56:26.155+0000 7f456ae2e700 1 --2- 172.21.15.133:0/3305947218 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f45640fbe40 0x7f456410fab0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f45580093b0 tx=0x7f4558009e30 comp rx=0 tx=0).stop 2024-02-20T07:56:26.156 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:56:26.155+0000 7f456ae2e700 1 -- 172.21.15.133:0/3305947218 shutdown_connections 2024-02-20T07:56:26.156 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:56:26.155+0000 7f456ae2e700 1 --2- 172.21.15.133:0/3305947218 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f45640fbe40 0x7f456410fab0 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:56:26.156 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:56:26.155+0000 7f456ae2e700 1 --2- 172.21.15.133:0/3305947218 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7f45540692a0 0x7f455406b750 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:56:26.156 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:56:26.155+0000 7f456ae2e700 1 --2- 172.21.15.133:0/3305947218 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f45640fcfb0 0x7f456410fff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:56:26.156 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:56:26.155+0000 7f456ae2e700 1 -- 172.21.15.133:0/3305947218 >> 172.21.15.133:0/3305947218 conn(0x7f45640f74e0 msgr2=0x7f45640656c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:56:26.156 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:56:26.155+0000 7f456ae2e700 1 -- 172.21.15.133:0/3305947218 shutdown_connections 2024-02-20T07:56:26.156 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:56:26.155+0000 7f456ae2e700 1 -- 172.21.15.133:0/3305947218 wait complete. 2024-02-20T07:56:26.833 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:56:26 smithi133 ceph-mon[29117]: pgmap v215: 65 pgs: 65 active+clean; 589 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.0 KiB/s rd, 511 B/s wr, 3 op/s 2024-02-20T07:56:26.833 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:56:26 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:56:26.833 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:56:26 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-20T07:56:26.852 INFO:tasks.workunit.client.1.smithi183.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-02-20T07:56:26.857 DEBUG:teuthology.orchestra.run.smithi133:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4537327c-cfc4-11ee-95bb-87774f69a715 -e sha1=91ff50e903dcc99ee3ecf9523050878eb452da5c -- bash -c 'while ceph orch upgrade status | jq '"'"'.in_progress'"'"' | grep true && ! ceph orch upgrade status | jq '"'"'.message'"'"' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; sleep 30 ; done' 2024-02-20T07:56:26.920 INFO:tasks.workunit.client.1.smithi183.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-02-20T07:56:26.960 INFO:tasks.workunit.client.1.smithi183.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.1/qa/workunits/direct_io' 2024-02-20T07:56:26.961 INFO:tasks.workunit.client.1.smithi183.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.1/qa/workunits/fs' 2024-02-20T07:56:26.961 INFO:tasks.workunit.client.1.smithi183.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-02-20T07:56:27.001 INFO:tasks.workunit.client.1.smithi183.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.1/qa/workunits/fs' 2024-02-20T07:56:27.004 DEBUG:teuthology.orchestra.run.smithi183:> set -ex 2024-02-20T07:56:27.004 DEBUG:teuthology.orchestra.run.smithi183:> dd if=/home/ubuntu/cephtest/workunits.list.client.1 of=/dev/stdout 2024-02-20T07:56:27.047 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:56:26 smithi183 ceph-mon[35832]: pgmap v215: 65 pgs: 65 active+clean; 589 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.0 KiB/s rd, 511 B/s wr, 3 op/s 2024-02-20T07:56:27.047 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:56:26 smithi183 ceph-mon[35832]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:56:27.047 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:56:26 smithi183 ceph-mon[35832]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-20T07:56:27.060 INFO:tasks.workunit:Running workunits matching suites/fsstress.sh on client.1... 2024-02-20T07:56:27.061 INFO:tasks.workunit:Running workunit suites/fsstress.sh... 2024-02-20T07:56:27.062 DEBUG:teuthology.orchestra.run.smithi183:workunit test suites/fsstress.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && cd -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=3d72bcfd7affbb4a923c191d212814b4934cfeaa TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="1" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.1 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.1 CEPH_MNT=/home/ubuntu/cephtest/mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.1/qa/workunits/suites/fsstress.sh 2024-02-20T07:56:27.126 INFO:tasks.workunit.client.1.smithi183.stderr:+ mkdir -p fsstress 2024-02-20T07:56:27.128 INFO:tasks.workunit.client.1.smithi183.stderr:+ pushd fsstress 2024-02-20T07:56:27.128 INFO:tasks.workunit.client.1.smithi183.stdout:~/cephtest/mnt.1/client.1/tmp/fsstress ~/cephtest/mnt.1/client.1/tmp 2024-02-20T07:56:27.129 INFO:tasks.workunit.client.1.smithi183.stderr:+ wget -q -O ltp-full.tgz http://download.ceph.com/qa/ltp-full-20091231.tgz 2024-02-20T07:56:27.154 INFO:teuthology.orchestra.run.smithi133.stderr:Inferring config /var/lib/ceph/4537327c-cfc4-11ee-95bb-87774f69a715/mon.smithi133/config 2024-02-20T07:56:27.657 INFO:tasks.workunit.client.1.smithi183.stderr:+ tar xzf ltp-full.tgz 2024-02-20T07:56:27.991 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:56:27 smithi133 ceph-mon[29117]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:91ff50e903dcc99ee3ecf9523050878eb452da5c 2024-02-20T07:56:27.992 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:56:27 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-20T07:56:27.992 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:56:27 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-20T07:56:27.992 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:56:27 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:56:28.047 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:56:27 smithi183 ceph-mon[35832]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:91ff50e903dcc99ee3ecf9523050878eb452da5c 2024-02-20T07:56:28.047 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:56:27 smithi183 ceph-mon[35832]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-20T07:56:28.048 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:56:27 smithi183 ceph-mon[35832]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-20T07:56:28.048 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:56:27 smithi183 ceph-mon[35832]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:56:29.147 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:56:29.146+0000 7f0579808700 1 -- 172.21.15.133:0/1489498424 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f05740fbe40 msgr2=0x7f05740fc250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:56:29.147 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:56:29.146+0000 7f0579808700 1 --2- 172.21.15.133:0/1489498424 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f05740fbe40 0x7f05740fc250 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f0564004940 tx=0x7f0564009e30 comp rx=0 tx=0).stop 2024-02-20T07:56:29.147 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:56:29.146+0000 7f0579808700 1 -- 172.21.15.133:0/1489498424 shutdown_connections 2024-02-20T07:56:29.147 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:56:29.146+0000 7f0579808700 1 --2- 172.21.15.133:0/1489498424 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f05740fcfb0 0x7f05740fd420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:56:29.147 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:56:29.146+0000 7f0579808700 1 --2- 172.21.15.133:0/1489498424 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f05740fbe40 0x7f05740fc250 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:56:29.147 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:56:29.146+0000 7f0579808700 1 -- 172.21.15.133:0/1489498424 >> 172.21.15.133:0/1489498424 conn(0x7f05740f74e0 msgr2=0x7f05740f9930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:56:29.147 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:56:29.146+0000 7f0579808700 1 -- 172.21.15.133:0/1489498424 shutdown_connections 2024-02-20T07:56:29.147 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:56:29.146+0000 7f0579808700 1 -- 172.21.15.133:0/1489498424 wait complete. 2024-02-20T07:56:29.149 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:56:29.147+0000 7f0579808700 1 Processor -- start 2024-02-20T07:56:29.149 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:56:29.147+0000 7f0579808700 1 -- start start 2024-02-20T07:56:29.149 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:56:29.147+0000 7f0579808700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f05740fbe40 0x7f05740fff00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:56:29.149 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:56:29.147+0000 7f0579808700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f05740fcfb0 0x7f05740fe380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:56:29.149 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:56:29.147+0000 7f0579808700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05740fe980 con 0x7f05740fbe40 2024-02-20T07:56:29.149 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:56:29.147+0000 7f0579808700 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05740feaf0 con 0x7f05740fcfb0 2024-02-20T07:56:29.149 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:56:29.147+0000 7f0572ffd700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f05740fbe40 0x7f05740fff00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:56:29.149 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:56:29.147+0000 7f0572ffd700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f05740fbe40 0x7f05740fff00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.133:43312/0 (socket says 172.21.15.133:43312) 2024-02-20T07:56:29.150 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:56:29.147+0000 7f0572ffd700 1 -- 172.21.15.133:0/1664442234 learned_addr learned my addr 172.21.15.133:0/1664442234 (peer_addr_for_me v2:172.21.15.133:0/0) 2024-02-20T07:56:29.150 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:56:29.147+0000 7f05727fc700 1 --2- 172.21.15.133:0/1664442234 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f05740fcfb0 0x7f05740fe380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:56:29.150 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:56:29.148+0000 7f0572ffd700 1 -- 172.21.15.133:0/1664442234 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f05740fcfb0 msgr2=0x7f05740fe380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:56:29.150 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:56:29.148+0000 7f0572ffd700 1 --2- 172.21.15.133:0/1664442234 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f05740fcfb0 0x7f05740fe380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:56:29.150 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:56:29.148+0000 7f0572ffd700 1 -- 172.21.15.133:0/1664442234 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0564005040 con 0x7f05740fbe40 2024-02-20T07:56:29.150 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:56:29.148+0000 7f05727fc700 1 --2- 172.21.15.133:0/1664442234 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f05740fcfb0 0x7f05740fe380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-20T07:56:29.150 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:56:29.148+0000 7f0572ffd700 1 --2- 172.21.15.133:0/1664442234 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f05740fbe40 0x7f05740fff00 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f05640093b0 tx=0x7f0564009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:56:29.150 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:56:29.148+0000 7f055b7fe700 1 -- 172.21.15.133:0/1664442234 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f056400e040 con 0x7f05740fbe40 2024-02-20T07:56:29.150 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:56:29.148+0000 7f055b7fe700 1 -- 172.21.15.133:0/1664442234 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f056400bcc0 con 0x7f05740fbe40 2024-02-20T07:56:29.150 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:56:29.148+0000 7f0579808700 1 -- 172.21.15.133:0/1664442234 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f05740fed70 con 0x7f05740fbe40 2024-02-20T07:56:29.150 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:56:29.148+0000 7f055b7fe700 1 -- 172.21.15.133:0/1664442234 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f056401c630 con 0x7f05740fbe40 2024-02-20T07:56:29.150 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:56:29.148+0000 7f0579808700 1 -- 172.21.15.133:0/1664442234 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f05741158e0 con 0x7f05740fbe40 2024-02-20T07:56:29.151 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:56:29.149+0000 7f0579808700 1 -- 172.21.15.133:0/1664442234 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f057404bed0 con 0x7f05740fbe40 2024-02-20T07:56:29.151 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:56:29.150+0000 7f055b7fe700 1 -- 172.21.15.133:0/1664442234 <== mon.0 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7f056401c8a0 con 0x7f05740fbe40 2024-02-20T07:56:29.151 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:56:29.150+0000 7f055b7fe700 1 --2- 172.21.15.133:0/1664442234 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7f055c0692a0 0x7f055c06b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:56:29.151 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:56:29.150+0000 7f055b7fe700 1 -- 172.21.15.133:0/1664442234 <== mon.0 v2:172.21.15.133:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f0564088570 con 0x7f05740fbe40 2024-02-20T07:56:29.152 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:56:29.150+0000 7f05727fc700 1 --2- 172.21.15.133:0/1664442234 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7f055c0692a0 0x7f055c06b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:56:29.152 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:56:29.151+0000 7f05727fc700 1 --2- 172.21.15.133:0/1664442234 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7f055c0692a0 0x7f055c06b750 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f0568009fd0 tx=0x7f056800b040 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:56:29.155 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:56:29.155+0000 7f055b7fe700 1 -- 172.21.15.133:0/1664442234 <== mon.0 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f0564058350 con 0x7f05740fbe40 2024-02-20T07:56:29.417 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:56:29.416+0000 7f0579808700 1 -- 172.21.15.133:0/1664442234 --> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f0574065dd0 con 0x7f055c0692a0 2024-02-20T07:56:29.417 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:56:29 smithi133 ceph-mon[29117]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:91ff50e903dcc99ee3ecf9523050878eb452da5c 2024-02-20T07:56:29.417 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:56:29 smithi133 ceph-mon[29117]: pgmap v216: 65 pgs: 65 active+clean; 589 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.4 KiB/s rd, 511 B/s wr, 4 op/s 2024-02-20T07:56:29.418 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:56:29.418+0000 7f055b7fe700 1 -- 172.21.15.133:0/1664442234 <== mgr.14208 v2:172.21.15.133:6800/2703586938 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+351 (secure 0 0 0) 0x7f0574065dd0 con 0x7f055c0692a0 2024-02-20T07:56:29.421 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:56:29.420+0000 7f0579808700 1 -- 172.21.15.133:0/1664442234 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7f055c0692a0 msgr2=0x7f055c06b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:56:29.421 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:56:29.420+0000 7f0579808700 1 --2- 172.21.15.133:0/1664442234 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7f055c0692a0 0x7f055c06b750 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f0568009fd0 tx=0x7f056800b040 comp rx=0 tx=0).stop 2024-02-20T07:56:29.421 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:56:29.420+0000 7f0579808700 1 -- 172.21.15.133:0/1664442234 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f05740fbe40 msgr2=0x7f05740fff00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:56:29.421 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:56:29.420+0000 7f0579808700 1 --2- 172.21.15.133:0/1664442234 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f05740fbe40 0x7f05740fff00 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f05640093b0 tx=0x7f0564009e30 comp rx=0 tx=0).stop 2024-02-20T07:56:29.421 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:56:29.420+0000 7f0579808700 1 -- 172.21.15.133:0/1664442234 shutdown_connections 2024-02-20T07:56:29.421 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:56:29.420+0000 7f0579808700 1 --2- 172.21.15.133:0/1664442234 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f05740fbe40 0x7f05740fff00 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:56:29.421 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:56:29.420+0000 7f0579808700 1 --2- 172.21.15.133:0/1664442234 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7f055c0692a0 0x7f055c06b750 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:56:29.421 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:56:29.420+0000 7f0579808700 1 --2- 172.21.15.133:0/1664442234 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f05740fcfb0 0x7f05740fe380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:56:29.421 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:56:29.420+0000 7f0579808700 1 -- 172.21.15.133:0/1664442234 >> 172.21.15.133:0/1664442234 conn(0x7f05740f74e0 msgr2=0x7f05740656c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:56:29.421 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:56:29.420+0000 7f0579808700 1 -- 172.21.15.133:0/1664442234 shutdown_connections 2024-02-20T07:56:29.421 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:56:29.420+0000 7f0579808700 1 -- 172.21.15.133:0/1664442234 wait complete. 2024-02-20T07:56:29.432 INFO:teuthology.orchestra.run.smithi133.stdout:true 2024-02-20T07:56:29.544 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:56:29.543+0000 7f075eb00700 1 -- 172.21.15.133:0/1194572363 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f07580fe2b0 msgr2=0x7f07580fe700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:56:29.545 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:56:29.543+0000 7f075eb00700 1 --2- 172.21.15.133:0/1194572363 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f07580fe2b0 0x7f07580fe700 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f0748004940 tx=0x7f0748009e30 comp rx=0 tx=0).stop 2024-02-20T07:56:29.545 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:56:29.543+0000 7f075eb00700 1 -- 172.21.15.133:0/1194572363 shutdown_connections 2024-02-20T07:56:29.545 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:56:29.543+0000 7f075eb00700 1 --2- 172.21.15.133:0/1194572363 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f07580fe2b0 0x7f07580fe700 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:56:29.545 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:56:29.543+0000 7f075eb00700 1 --2- 172.21.15.133:0/1194572363 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f07580fb220 0x7f07580fb630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:56:29.545 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:56:29.543+0000 7f075eb00700 1 -- 172.21.15.133:0/1194572363 >> 172.21.15.133:0/1194572363 conn(0x7f07580f6ca0 msgr2=0x7f07580f90d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:56:29.545 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:56:29.543+0000 7f075eb00700 1 -- 172.21.15.133:0/1194572363 shutdown_connections 2024-02-20T07:56:29.545 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:56:29.543+0000 7f075eb00700 1 -- 172.21.15.133:0/1194572363 wait complete. 2024-02-20T07:56:29.545 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:56:29.544+0000 7f075eb00700 1 Processor -- start 2024-02-20T07:56:29.545 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:56:29.544+0000 7f075eb00700 1 -- start start 2024-02-20T07:56:29.546 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:56:29.545+0000 7f075eb00700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f07580fb220 0x7f075810fbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:56:29.547 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:56:29.545+0000 7f075eb00700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f07580fe2b0 0x7f0758110130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:56:29.547 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:56:29.545+0000 7f075eb00700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0758110730 con 0x7f07580fe2b0 2024-02-20T07:56:29.547 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:56:29.545+0000 7f075eb00700 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0758110fd0 con 0x7f07580fb220 2024-02-20T07:56:29.547 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:56:29.545+0000 7f075c89c700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f07580fb220 0x7f075810fbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:56:29.547 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:56:29.545+0000 7f075c89c700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f07580fb220 0x7f075810fbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.133:39430/0 (socket says 172.21.15.133:39430) 2024-02-20T07:56:29.547 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:56:29.545+0000 7f075c89c700 1 -- 172.21.15.133:0/3166017686 learned_addr learned my addr 172.21.15.133:0/3166017686 (peer_addr_for_me v2:172.21.15.133:0/0) 2024-02-20T07:56:29.547 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:56:29.545+0000 7f0757fff700 1 --2- 172.21.15.133:0/3166017686 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f07580fe2b0 0x7f0758110130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:56:29.547 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:56:29.545+0000 7f075c89c700 1 -- 172.21.15.133:0/3166017686 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f07580fe2b0 msgr2=0x7f0758110130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:56:29.547 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:56:29.545+0000 7f075c89c700 1 --2- 172.21.15.133:0/3166017686 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f07580fe2b0 0x7f0758110130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:56:29.547 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:56:29.545+0000 7f075c89c700 1 -- 172.21.15.133:0/3166017686 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0748005040 con 0x7f07580fb220 2024-02-20T07:56:29.547 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:56:29.545+0000 7f075c89c700 1 --2- 172.21.15.133:0/3166017686 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f07580fb220 0x7f075810fbf0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f075000cb10 tx=0x7f075000ce20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:56:29.547 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:56:29.546+0000 7f07557fa700 1 -- 172.21.15.133:0/3166017686 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f075000a980 con 0x7f07580fb220 2024-02-20T07:56:29.547 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:56:29.546+0000 7f07557fa700 1 -- 172.21.15.133:0/3166017686 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0750012420 con 0x7f07580fb220 2024-02-20T07:56:29.548 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:56:29.546+0000 7f07557fa700 1 -- 172.21.15.133:0/3166017686 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f07500115c0 con 0x7f07580fb220 2024-02-20T07:56:29.548 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:56:29.546+0000 7f075eb00700 1 -- 172.21.15.133:0/3166017686 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f07581112b0 con 0x7f07580fb220 2024-02-20T07:56:29.548 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:56:29.546+0000 7f075eb00700 1 -- 172.21.15.133:0/3166017686 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f07581117a0 con 0x7f07580fb220 2024-02-20T07:56:29.548 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:56:29 smithi183 ceph-mon[35832]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:91ff50e903dcc99ee3ecf9523050878eb452da5c 2024-02-20T07:56:29.548 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:56:29 smithi183 ceph-mon[35832]: pgmap v216: 65 pgs: 65 active+clean; 589 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.4 KiB/s rd, 511 B/s wr, 4 op/s 2024-02-20T07:56:29.549 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:56:29.547+0000 7f075eb00700 1 -- 172.21.15.133:0/3166017686 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f075804bed0 con 0x7f07580fb220 2024-02-20T07:56:29.549 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:56:29.548+0000 7f07557fa700 1 -- 172.21.15.133:0/3166017686 <== mon.1 v2:172.21.15.183:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7f075000aae0 con 0x7f07580fb220 2024-02-20T07:56:29.549 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:56:29.548+0000 7f07557fa700 1 --2- 172.21.15.133:0/3166017686 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7f07440692b0 0x7f074406b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:56:29.549 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:56:29.548+0000 7f07557fa700 1 -- 172.21.15.133:0/3166017686 <== mon.1 v2:172.21.15.183:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f0750087810 con 0x7f07580fb220 2024-02-20T07:56:29.549 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:56:29.548+0000 7f0757fff700 1 --2- 172.21.15.133:0/3166017686 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7f07440692b0 0x7f074406b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:56:29.549 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:56:29.549+0000 7f0757fff700 1 --2- 172.21.15.133:0/3166017686 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7f07440692b0 0x7f074406b760 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f074800b3b0 tx=0x7f0748009e30 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:56:29.553 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:56:29.553+0000 7f07557fa700 1 -- 172.21.15.133:0/3166017686 <== mon.1 v2:172.21.15.183:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f0750057580 con 0x7f07580fb220 2024-02-20T07:56:29.865 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:56:29.865+0000 7f075eb00700 1 -- 172.21.15.133:0/3166017686 --> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f0758111ca0 con 0x7f07440692b0 2024-02-20T07:56:29.866 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:56:29.866+0000 7f07557fa700 1 -- 172.21.15.133:0/3166017686 <== mgr.14208 v2:172.21.15.133:6800/2703586938 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+351 (secure 0 0 0) 0x7f0758111ca0 con 0x7f07440692b0 2024-02-20T07:56:29.869 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:56:29.868+0000 7f075eb00700 1 -- 172.21.15.133:0/3166017686 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7f07440692b0 msgr2=0x7f074406b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:56:29.869 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:56:29.868+0000 7f075eb00700 1 --2- 172.21.15.133:0/3166017686 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7f07440692b0 0x7f074406b760 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f074800b3b0 tx=0x7f0748009e30 comp rx=0 tx=0).stop 2024-02-20T07:56:29.869 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:56:29.869+0000 7f075eb00700 1 -- 172.21.15.133:0/3166017686 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f07580fb220 msgr2=0x7f075810fbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:56:29.870 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:56:29.869+0000 7f075eb00700 1 --2- 172.21.15.133:0/3166017686 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f07580fb220 0x7f075810fbf0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f075000cb10 tx=0x7f075000ce20 comp rx=0 tx=0).stop 2024-02-20T07:56:29.870 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:56:29.869+0000 7f075eb00700 1 -- 172.21.15.133:0/3166017686 shutdown_connections 2024-02-20T07:56:29.870 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:56:29.869+0000 7f075eb00700 1 --2- 172.21.15.133:0/3166017686 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7f07440692b0 0x7f074406b760 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:56:29.870 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:56:29.869+0000 7f075eb00700 1 --2- 172.21.15.133:0/3166017686 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f07580fb220 0x7f075810fbf0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:56:29.870 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:56:29.869+0000 7f075eb00700 1 --2- 172.21.15.133:0/3166017686 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f07580fe2b0 0x7f0758110130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:56:29.870 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:56:29.869+0000 7f075eb00700 1 -- 172.21.15.133:0/3166017686 >> 172.21.15.133:0/3166017686 conn(0x7f07580f6ca0 msgr2=0x7f07580f9050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:56:29.870 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:56:29.869+0000 7f075eb00700 1 -- 172.21.15.133:0/3166017686 shutdown_connections 2024-02-20T07:56:29.870 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:56:29.869+0000 7f075eb00700 1 -- 172.21.15.133:0/3166017686 wait complete. 2024-02-20T07:56:30.022 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:56:30.021+0000 7fb5440ab700 1 -- 172.21.15.133:0/3485056901 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb53c0fbe50 msgr2=0x7fb53c0fc260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:56:30.022 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:56:30.021+0000 7fb5440ab700 1 --2- 172.21.15.133:0/3485056901 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb53c0fbe50 0x7fb53c0fc260 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fb52c004880 tx=0x7fb52c009e30 comp rx=0 tx=0).stop 2024-02-20T07:56:30.025 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:56:30.022+0000 7fb5440ab700 1 -- 172.21.15.133:0/3485056901 shutdown_connections 2024-02-20T07:56:30.025 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:56:30.022+0000 7fb5440ab700 1 --2- 172.21.15.133:0/3485056901 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb53c0fcfc0 0x7fb53c0fd430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:56:30.025 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:56:30.022+0000 7fb5440ab700 1 --2- 172.21.15.133:0/3485056901 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb53c0fbe50 0x7fb53c0fc260 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:56:30.025 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:56:30.022+0000 7fb5440ab700 1 -- 172.21.15.133:0/3485056901 >> 172.21.15.133:0/3485056901 conn(0x7fb53c0f7510 msgr2=0x7fb53c0f9940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:56:30.025 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:56:30.022+0000 7fb5440ab700 1 -- 172.21.15.133:0/3485056901 shutdown_connections 2024-02-20T07:56:30.025 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:56:30.022+0000 7fb5440ab700 1 -- 172.21.15.133:0/3485056901 wait complete. 2024-02-20T07:56:30.025 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:56:30.022+0000 7fb5440ab700 1 Processor -- start 2024-02-20T07:56:30.026 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:56:30.022+0000 7fb5440ab700 1 -- start start 2024-02-20T07:56:30.026 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:56:30.023+0000 7fb5440ab700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb53c0fbe50 0x7fb53c113f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:56:30.026 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:56:30.023+0000 7fb5440ab700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb53c0fcfc0 0x7fb53c1144c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:56:30.026 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:56:30.023+0000 7fb5440ab700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb53c114ac0 con 0x7fb53c0fbe50 2024-02-20T07:56:30.026 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:56:30.023+0000 7fb5440ab700 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb53c115360 con 0x7fb53c0fcfc0 2024-02-20T07:56:30.026 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:56:30.023+0000 7fb541e47700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb53c0fbe50 0x7fb53c113f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:56:30.026 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:56:30.023+0000 7fb541646700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb53c0fcfc0 0x7fb53c1144c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:56:30.026 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:56:30.023+0000 7fb541646700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb53c0fcfc0 0x7fb53c1144c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.133:39442/0 (socket says 172.21.15.133:39442) 2024-02-20T07:56:30.026 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:56:30.023+0000 7fb541646700 1 -- 172.21.15.133:0/79281601 learned_addr learned my addr 172.21.15.133:0/79281601 (peer_addr_for_me v2:172.21.15.133:0/0) 2024-02-20T07:56:30.026 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:56:30.023+0000 7fb541646700 1 -- 172.21.15.133:0/79281601 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb53c0fbe50 msgr2=0x7fb53c113f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:56:30.026 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:56:30.023+0000 7fb541646700 1 --2- 172.21.15.133:0/79281601 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb53c0fbe50 0x7fb53c113f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:56:30.027 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:56:30.023+0000 7fb541646700 1 -- 172.21.15.133:0/79281601 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb538005040 con 0x7fb53c0fcfc0 2024-02-20T07:56:30.027 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:56:30.023+0000 7fb541646700 1 --2- 172.21.15.133:0/79281601 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb53c0fcfc0 0x7fb53c1144c0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fb5380096b0 tx=0x7fb538009e30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:56:30.027 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:56:30.024+0000 7fb5327fc700 1 -- 172.21.15.133:0/79281601 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb538017070 con 0x7fb53c0fcfc0 2024-02-20T07:56:30.027 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:56:30.024+0000 7fb5327fc700 1 -- 172.21.15.133:0/79281601 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb53800f040 con 0x7fb53c0fcfc0 2024-02-20T07:56:30.027 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:56:30.024+0000 7fb5440ab700 1 -- 172.21.15.133:0/79281601 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb52c005040 con 0x7fb53c0fcfc0 2024-02-20T07:56:30.027 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:56:30.024+0000 7fb5440ab700 1 -- 172.21.15.133:0/79281601 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb53c115990 con 0x7fb53c0fcfc0 2024-02-20T07:56:30.027 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:56:30.024+0000 7fb5327fc700 1 -- 172.21.15.133:0/79281601 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb538013be0 con 0x7fb53c0fcfc0 2024-02-20T07:56:30.028 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:56:30.026+0000 7fb5327fc700 1 -- 172.21.15.133:0/79281601 <== mon.1 v2:172.21.15.183:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7fb53801c410 con 0x7fb53c0fcfc0 2024-02-20T07:56:30.028 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:56:30.026+0000 7fb5440ab700 1 -- 172.21.15.133:0/79281601 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb5200052e0 con 0x7fb53c0fcfc0 2024-02-20T07:56:30.028 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:56:30.026+0000 7fb5327fc700 1 --2- 172.21.15.133:0/79281601 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7fb5280692b0 0x7fb52806b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:56:30.029 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:56:30.026+0000 7fb5327fc700 1 -- 172.21.15.133:0/79281601 <== mon.1 v2:172.21.15.183:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5550+0+0 (secure 0 0 0) 0x7fb538087fe0 con 0x7fb53c0fcfc0 2024-02-20T07:56:30.029 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:56:30.026+0000 7fb541e47700 1 --2- 172.21.15.133:0/79281601 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7fb5280692b0 0x7fb52806b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:56:30.029 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:56:30.027+0000 7fb541e47700 1 --2- 172.21.15.133:0/79281601 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7fb5280692b0 0x7fb52806b760 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fb52c0137c0 tx=0x7fb52c009e30 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:56:30.031 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:56:30.031+0000 7fb5327fc700 1 -- 172.21.15.133:0/79281601 <== mon.1 v2:172.21.15.183:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7fb538057d50 con 0x7fb53c0fcfc0 2024-02-20T07:56:30.438 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:56:30.437+0000 7fb5440ab700 1 -- 172.21.15.133:0/79281601 --> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fb520000bf0 con 0x7fb5280692b0 2024-02-20T07:56:30.457 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:56:30.457+0000 7fb5327fc700 1 -- 172.21.15.133:0/79281601 <== mgr.14208 v2:172.21.15.133:6800/2703586938 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3366 (secure 0 0 0) 0x7fb520000bf0 con 0x7fb5280692b0 2024-02-20T07:56:30.458 INFO:teuthology.orchestra.run.smithi133.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-20T07:56:30.458 INFO:teuthology.orchestra.run.smithi133.stdout:alertmanager.smithi133 smithi133 *:9093,9094 running (5m) 51s ago 6m 24.9M - 0.25.0 c8568f914cd2 26689e669607 2024-02-20T07:56:30.458 INFO:teuthology.orchestra.run.smithi133.stdout:crash.smithi133 smithi133 running (6m) 51s ago 6m 6660k - 17.2.7-521-gc80e7168 10b0c032c913 ba7610707152 2024-02-20T07:56:30.458 INFO:teuthology.orchestra.run.smithi133.stdout:crash.smithi183 smithi183 running (5m) 56s ago 5m 6669k - 17.2.7-521-gc80e7168 10b0c032c913 55659076790f 2024-02-20T07:56:30.458 INFO:teuthology.orchestra.run.smithi133.stdout:grafana.smithi133 smithi133 *:3000 running (5m) 51s ago 6m 85.3M - 9.4.7 954c08fa6188 d35d6b4393e9 2024-02-20T07:56:30.458 INFO:teuthology.orchestra.run.smithi133.stdout:mds.cephfs.smithi133.cmofvy smithi133 running (71s) 51s ago 70s 17.1M - 17.2.7-521-gc80e7168 10b0c032c913 218633eb7f9e 2024-02-20T07:56:30.458 INFO:teuthology.orchestra.run.smithi133.stdout:mds.cephfs.smithi133.itlmbm smithi133 running (78s) 51s ago 77s 11.4M - 17.2.7-521-gc80e7168 10b0c032c913 a24a27419574 2024-02-20T07:56:30.458 INFO:teuthology.orchestra.run.smithi133.stdout:mds.cephfs.smithi183.cwfunk smithi183 running (68s) 56s ago 68s 13.3M - 17.2.7-521-gc80e7168 10b0c032c913 96e2b71b668b 2024-02-20T07:56:30.458 INFO:teuthology.orchestra.run.smithi133.stdout:mds.cephfs.smithi183.eipfta smithi183 running (75s) 56s ago 74s 16.9M - 17.2.7-521-gc80e7168 10b0c032c913 45f0b3247e7a 2024-02-20T07:56:30.458 INFO:teuthology.orchestra.run.smithi133.stdout:mgr.smithi133.mrvcbp smithi133 *:9283 running (8m) 51s ago 8m 480M - 17.2.7-521-gc80e7168 10b0c032c913 58ed54a355b1 2024-02-20T07:56:30.458 INFO:teuthology.orchestra.run.smithi133.stdout:mgr.smithi183.hebfng smithi183 *:8443,9283 running (5m) 56s ago 5m 434M - 17.2.7-521-gc80e7168 10b0c032c913 a0dc319ab2bd 2024-02-20T07:56:30.458 INFO:teuthology.orchestra.run.smithi133.stdout:mon.smithi133 smithi133 running (8m) 51s ago 8m 44.2M 2048M 17.2.7-521-gc80e7168 10b0c032c913 fe7046a20c87 2024-02-20T07:56:30.459 INFO:teuthology.orchestra.run.smithi133.stdout:mon.smithi183 smithi183 running (5m) 56s ago 5m 36.3M 2048M 17.2.7-521-gc80e7168 10b0c032c913 971a48e1d959 2024-02-20T07:56:30.459 INFO:teuthology.orchestra.run.smithi133.stdout:node-exporter.smithi133 smithi133 *:9100 running (6m) 51s ago 6m 16.0M - 1.5.0 0da6a335fe13 71e70059b12f 2024-02-20T07:56:30.459 INFO:teuthology.orchestra.run.smithi133.stdout:node-exporter.smithi183 smithi183 *:9100 running (5m) 56s ago 5m 16.2M - 1.5.0 0da6a335fe13 c567d872acb6 2024-02-20T07:56:30.459 INFO:teuthology.orchestra.run.smithi133.stdout:osd.0 smithi133 running (4m) 51s ago 4m 47.7M 4288M 17.2.7-521-gc80e7168 10b0c032c913 eee5e0d5322f 2024-02-20T07:56:30.459 INFO:teuthology.orchestra.run.smithi133.stdout:osd.1 smithi133 running (4m) 51s ago 4m 46.0M 4288M 17.2.7-521-gc80e7168 10b0c032c913 aa50c25fdf11 2024-02-20T07:56:30.459 INFO:teuthology.orchestra.run.smithi133.stdout:osd.2 smithi133 running (4m) 51s ago 4m 43.9M 4288M 17.2.7-521-gc80e7168 10b0c032c913 91e34ab8741f 2024-02-20T07:56:30.459 INFO:teuthology.orchestra.run.smithi133.stdout:osd.3 smithi183 running (3m) 56s ago 3m 47.1M 5312M 17.2.7-521-gc80e7168 10b0c032c913 4e2ae9566662 2024-02-20T07:56:30.459 INFO:teuthology.orchestra.run.smithi133.stdout:osd.4 smithi183 running (3m) 56s ago 3m 45.2M 5312M 17.2.7-521-gc80e7168 10b0c032c913 79015466a807 2024-02-20T07:56:30.459 INFO:teuthology.orchestra.run.smithi133.stdout:osd.5 smithi183 running (2m) 56s ago 2m 42.4M 5312M 17.2.7-521-gc80e7168 10b0c032c913 f85f43119a0b 2024-02-20T07:56:30.459 INFO:teuthology.orchestra.run.smithi133.stdout:prometheus.smithi133 smithi133 *:9095 running (5m) 51s ago 6m 54.4M - 2.43.0 a07b618ecd1d 568c27a3eecc 2024-02-20T07:56:30.461 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:56:30.461+0000 7fb5440ab700 1 -- 172.21.15.133:0/79281601 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7fb5280692b0 msgr2=0x7fb52806b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:56:30.462 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:56:30.461+0000 7fb5440ab700 1 --2- 172.21.15.133:0/79281601 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7fb5280692b0 0x7fb52806b760 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fb52c0137c0 tx=0x7fb52c009e30 comp rx=0 tx=0).stop 2024-02-20T07:56:30.462 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:56:30.461+0000 7fb5440ab700 1 -- 172.21.15.133:0/79281601 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb53c0fcfc0 msgr2=0x7fb53c1144c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:56:30.462 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:56:30.461+0000 7fb5440ab700 1 --2- 172.21.15.133:0/79281601 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb53c0fcfc0 0x7fb53c1144c0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fb5380096b0 tx=0x7fb538009e30 comp rx=0 tx=0).stop 2024-02-20T07:56:30.462 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:56:30.461+0000 7fb5440ab700 1 -- 172.21.15.133:0/79281601 shutdown_connections 2024-02-20T07:56:30.462 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:56:30.461+0000 7fb5440ab700 1 --2- 172.21.15.133:0/79281601 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7fb53c0fbe50 0x7fb53c113f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:56:30.462 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:56:30.461+0000 7fb5440ab700 1 --2- 172.21.15.133:0/79281601 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7fb5280692b0 0x7fb52806b760 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:56:30.462 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:56:30.461+0000 7fb5440ab700 1 --2- 172.21.15.133:0/79281601 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb53c0fcfc0 0x7fb53c1144c0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:56:30.462 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:56:30.461+0000 7fb5440ab700 1 -- 172.21.15.133:0/79281601 >> 172.21.15.133:0/79281601 conn(0x7fb53c0f7510 msgr2=0x7fb53c065f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:56:30.462 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:56:30.461+0000 7fb5440ab700 1 -- 172.21.15.133:0/79281601 shutdown_connections 2024-02-20T07:56:30.462 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:56:30.461+0000 7fb5440ab700 1 -- 172.21.15.133:0/79281601 wait complete. 2024-02-20T07:56:30.590 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:56:30.589+0000 7f3b68777700 1 -- 172.21.15.133:0/3043967755 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f3b601003a0 msgr2=0x7f3b60102780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:56:30.590 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:56:30.589+0000 7f3b68777700 1 --2- 172.21.15.133:0/3043967755 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f3b601003a0 0x7f3b60102780 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f3b50004940 tx=0x7f3b50009e30 comp rx=0 tx=0).stop 2024-02-20T07:56:30.590 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:56:30.589+0000 7f3b68777700 1 -- 172.21.15.133:0/3043967755 shutdown_connections 2024-02-20T07:56:30.590 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:56:30.589+0000 7f3b68777700 1 --2- 172.21.15.133:0/3043967755 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f3b601003a0 0x7f3b60102780 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:56:30.590 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:56:30.589+0000 7f3b68777700 1 --2- 172.21.15.133:0/3043967755 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f3b600fda80 0x7f3b600ffe60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:56:30.590 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:56:30.589+0000 7f3b68777700 1 -- 172.21.15.133:0/3043967755 >> 172.21.15.133:0/3043967755 conn(0x7f3b600f7520 msgr2=0x7f3b600f9930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:56:30.590 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:56:30.589+0000 7f3b68777700 1 -- 172.21.15.133:0/3043967755 shutdown_connections 2024-02-20T07:56:30.591 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:56:30.589+0000 7f3b68777700 1 -- 172.21.15.133:0/3043967755 wait complete. 2024-02-20T07:56:30.595 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:56:30.590+0000 7f3b68777700 1 Processor -- start 2024-02-20T07:56:30.595 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:56:30.590+0000 7f3b68777700 1 -- start start 2024-02-20T07:56:30.595 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:56:30.590+0000 7f3b68777700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f3b600fda80 0x7f3b6010d8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:56:30.595 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:56:30.590+0000 7f3b68777700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f3b601003a0 0x7f3b6010de20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:56:30.595 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:56:30.590+0000 7f3b68777700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b6010e420 con 0x7f3b600fda80 2024-02-20T07:56:30.595 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:56:30.590+0000 7f3b68777700 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b6010ecc0 con 0x7f3b601003a0 2024-02-20T07:56:30.595 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:56:30.590+0000 7f3b65d12700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f3b601003a0 0x7f3b6010de20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:56:30.596 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:56:30.591+0000 7f3b65d12700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f3b601003a0 0x7f3b6010de20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.133:39466/0 (socket says 172.21.15.133:39466) 2024-02-20T07:56:30.596 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:56:30.591+0000 7f3b65d12700 1 -- 172.21.15.133:0/4153205754 learned_addr learned my addr 172.21.15.133:0/4153205754 (peer_addr_for_me v2:172.21.15.133:0/0) 2024-02-20T07:56:30.596 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:56:30.591+0000 7f3b66513700 1 --2- 172.21.15.133:0/4153205754 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f3b600fda80 0x7f3b6010d8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:56:30.596 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:56:30.591+0000 7f3b65d12700 1 -- 172.21.15.133:0/4153205754 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f3b600fda80 msgr2=0x7f3b6010d8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:56:30.596 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:56:30.591+0000 7f3b65d12700 1 --2- 172.21.15.133:0/4153205754 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f3b600fda80 0x7f3b6010d8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:56:30.596 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:56:30.591+0000 7f3b65d12700 1 -- 172.21.15.133:0/4153205754 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b50005040 con 0x7f3b601003a0 2024-02-20T07:56:30.596 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:56:30.591+0000 7f3b65d12700 1 --2- 172.21.15.133:0/4153205754 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f3b601003a0 0x7f3b6010de20 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f3b50009fd0 tx=0x7f3b50009e30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:56:30.596 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:56:30.591+0000 7f3b56ffd700 1 -- 172.21.15.133:0/4153205754 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3b50021070 con 0x7f3b601003a0 2024-02-20T07:56:30.596 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:56:30.591+0000 7f3b56ffd700 1 -- 172.21.15.133:0/4153205754 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3b5001c400 con 0x7f3b601003a0 2024-02-20T07:56:30.596 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:56:30.592+0000 7f3b68777700 1 -- 172.21.15.133:0/4153205754 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3b6010ef40 con 0x7f3b601003a0 2024-02-20T07:56:30.596 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:56:30.592+0000 7f3b68777700 1 -- 172.21.15.133:0/4153205754 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3b6010f420 con 0x7f3b601003a0 2024-02-20T07:56:30.596 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:56:30.592+0000 7f3b56ffd700 1 -- 172.21.15.133:0/4153205754 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3b5000e070 con 0x7f3b601003a0 2024-02-20T07:56:30.596 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:56:30.593+0000 7f3b68777700 1 -- 172.21.15.133:0/4153205754 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3b440052e0 con 0x7f3b601003a0 2024-02-20T07:56:30.596 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:56:30.593+0000 7f3b56ffd700 1 -- 172.21.15.133:0/4153205754 <== mon.1 v2:172.21.15.183:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7f3b5000b650 con 0x7f3b601003a0 2024-02-20T07:56:30.597 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:56:30.594+0000 7f3b56ffd700 1 --2- 172.21.15.133:0/4153205754 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7f3b4c0692a0 0x7f3b4c06b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:56:30.597 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:56:30.594+0000 7f3b56ffd700 1 -- 172.21.15.133:0/4153205754 <== mon.1 v2:172.21.15.183:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f3b50088390 con 0x7f3b601003a0 2024-02-20T07:56:30.597 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:56:30.594+0000 7f3b66513700 1 --2- 172.21.15.133:0/4153205754 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7f3b4c0692a0 0x7f3b4c06b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:56:30.597 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:56:30.595+0000 7f3b66513700 1 --2- 172.21.15.133:0/4153205754 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7f3b4c0692a0 0x7f3b4c06b750 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f3b5c009b80 tx=0x7f3b5c00b040 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:56:30.598 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:56:30.598+0000 7f3b56ffd700 1 -- 172.21.15.133:0/4153205754 <== mon.1 v2:172.21.15.183:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f3b500580d0 con 0x7f3b601003a0 2024-02-20T07:56:30.951 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:56:30.950+0000 7f3b68777700 1 -- 172.21.15.133:0/4153205754 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f3b44005da0 con 0x7f3b601003a0 2024-02-20T07:56:30.951 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:56:30.951+0000 7f3b56ffd700 1 -- 172.21.15.133:0/4153205754 <== mon.1 v2:172.21.15.183:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+638 (secure 0 0 0) 0x7f3b50057c60 con 0x7f3b601003a0 2024-02-20T07:56:30.952 INFO:teuthology.orchestra.run.smithi133.stdout:{ 2024-02-20T07:56:30.952 INFO:teuthology.orchestra.run.smithi133.stdout: "mon": { 2024-02-20T07:56:30.952 INFO:teuthology.orchestra.run.smithi133.stdout: "ceph version 17.2.7-521-gc80e7168 (c80e7168938c48b49e5ce2e002f48bf150b5f4a4) quincy (stable)": 2 2024-02-20T07:56:30.952 INFO:teuthology.orchestra.run.smithi133.stdout: }, 2024-02-20T07:56:30.952 INFO:teuthology.orchestra.run.smithi133.stdout: "mgr": { 2024-02-20T07:56:30.952 INFO:teuthology.orchestra.run.smithi133.stdout: "ceph version 17.2.7-521-gc80e7168 (c80e7168938c48b49e5ce2e002f48bf150b5f4a4) quincy (stable)": 2 2024-02-20T07:56:30.952 INFO:teuthology.orchestra.run.smithi133.stdout: }, 2024-02-20T07:56:30.952 INFO:teuthology.orchestra.run.smithi133.stdout: "osd": { 2024-02-20T07:56:30.952 INFO:teuthology.orchestra.run.smithi133.stdout: "ceph version 17.2.7-521-gc80e7168 (c80e7168938c48b49e5ce2e002f48bf150b5f4a4) quincy (stable)": 6 2024-02-20T07:56:30.952 INFO:teuthology.orchestra.run.smithi133.stdout: }, 2024-02-20T07:56:30.952 INFO:teuthology.orchestra.run.smithi133.stdout: "mds": { 2024-02-20T07:56:30.952 INFO:teuthology.orchestra.run.smithi133.stdout: "ceph version 17.2.7-521-gc80e7168 (c80e7168938c48b49e5ce2e002f48bf150b5f4a4) quincy (stable)": 4 2024-02-20T07:56:30.953 INFO:teuthology.orchestra.run.smithi133.stdout: }, 2024-02-20T07:56:30.953 INFO:teuthology.orchestra.run.smithi133.stdout: "overall": { 2024-02-20T07:56:30.953 INFO:teuthology.orchestra.run.smithi133.stdout: "ceph version 17.2.7-521-gc80e7168 (c80e7168938c48b49e5ce2e002f48bf150b5f4a4) quincy (stable)": 14 2024-02-20T07:56:30.953 INFO:teuthology.orchestra.run.smithi133.stdout: } 2024-02-20T07:56:30.953 INFO:teuthology.orchestra.run.smithi133.stdout:} 2024-02-20T07:56:30.954 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:56:30.954+0000 7f3b68777700 1 -- 172.21.15.133:0/4153205754 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7f3b4c0692a0 msgr2=0x7f3b4c06b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:56:30.954 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:56:30.954+0000 7f3b68777700 1 --2- 172.21.15.133:0/4153205754 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7f3b4c0692a0 0x7f3b4c06b750 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f3b5c009b80 tx=0x7f3b5c00b040 comp rx=0 tx=0).stop 2024-02-20T07:56:30.955 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:56:30.954+0000 7f3b68777700 1 -- 172.21.15.133:0/4153205754 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f3b601003a0 msgr2=0x7f3b6010de20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:56:30.955 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:56:30.954+0000 7f3b68777700 1 --2- 172.21.15.133:0/4153205754 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f3b601003a0 0x7f3b6010de20 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f3b50009fd0 tx=0x7f3b50009e30 comp rx=0 tx=0).stop 2024-02-20T07:56:30.955 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:56:30.954+0000 7f3b68777700 1 -- 172.21.15.133:0/4153205754 shutdown_connections 2024-02-20T07:56:30.955 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:56:30.954+0000 7f3b68777700 1 --2- 172.21.15.133:0/4153205754 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7f3b600fda80 0x7f3b6010d8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:56:30.955 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:56:30.954+0000 7f3b68777700 1 --2- 172.21.15.133:0/4153205754 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7f3b4c0692a0 0x7f3b4c06b750 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:56:30.955 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:56:30.954+0000 7f3b68777700 1 --2- 172.21.15.133:0/4153205754 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f3b601003a0 0x7f3b6010de20 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:56:30.955 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:56:30.954+0000 7f3b68777700 1 -- 172.21.15.133:0/4153205754 >> 172.21.15.133:0/4153205754 conn(0x7f3b600f7520 msgr2=0x7f3b600f9900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:56:30.955 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:56:30.954+0000 7f3b68777700 1 -- 172.21.15.133:0/4153205754 shutdown_connections 2024-02-20T07:56:30.955 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:56:30.954+0000 7f3b68777700 1 -- 172.21.15.133:0/4153205754 wait complete. 2024-02-20T07:56:31.086 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:56:31.085+0000 7ff14c336700 1 -- 172.21.15.133:0/1516286606 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ff1440fbe40 msgr2=0x7ff1440fc250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:56:31.086 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:56:31.085+0000 7ff14c336700 1 --2- 172.21.15.133:0/1516286606 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ff1440fbe40 0x7ff1440fc250 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7ff134004940 tx=0x7ff134009e30 comp rx=0 tx=0).stop 2024-02-20T07:56:31.086 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:56:31.086+0000 7ff14c336700 1 -- 172.21.15.133:0/1516286606 shutdown_connections 2024-02-20T07:56:31.086 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:56:31.086+0000 7ff14c336700 1 --2- 172.21.15.133:0/1516286606 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff1440fcfb0 0x7ff1440fd420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:56:31.087 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:56:31.086+0000 7ff14c336700 1 --2- 172.21.15.133:0/1516286606 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ff1440fbe40 0x7ff1440fc250 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:56:31.087 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:56:31.086+0000 7ff14c336700 1 -- 172.21.15.133:0/1516286606 >> 172.21.15.133:0/1516286606 conn(0x7ff1440f7500 msgr2=0x7ff1440f9930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:56:31.087 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:56:31.086+0000 7ff14c336700 1 -- 172.21.15.133:0/1516286606 shutdown_connections 2024-02-20T07:56:31.087 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:56:31.087+0000 7ff14c336700 1 -- 172.21.15.133:0/1516286606 wait complete. 2024-02-20T07:56:31.087 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:56:31.087+0000 7ff14c336700 1 Processor -- start 2024-02-20T07:56:31.091 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:56:31.087+0000 7ff14c336700 1 -- start start 2024-02-20T07:56:31.091 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:56:31.088+0000 7ff14c336700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ff1440fbe40 0x7ff1440fff00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:56:31.091 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:56:31.088+0000 7ff14c336700 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff1440fcfb0 0x7ff1440fe380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:56:31.091 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:56:31.088+0000 7ff14c336700 1 -- --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff144111630 con 0x7ff1440fbe40 2024-02-20T07:56:31.092 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:56:31.088+0000 7ff14c336700 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1441117a0 con 0x7ff1440fcfb0 2024-02-20T07:56:31.092 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:56:31.088+0000 7ff14a0d2700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ff1440fbe40 0x7ff1440fff00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:56:31.092 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:56:31.088+0000 7ff14a0d2700 1 --2- >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ff1440fbe40 0x7ff1440fff00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.133:3300/0 says I am v2:172.21.15.133:43372/0 (socket says 172.21.15.133:43372) 2024-02-20T07:56:31.092 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:56:31.088+0000 7ff14a0d2700 1 -- 172.21.15.133:0/2480311097 learned_addr learned my addr 172.21.15.133:0/2480311097 (peer_addr_for_me v2:172.21.15.133:0/0) 2024-02-20T07:56:31.093 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:56:31.088+0000 7ff14a0d2700 1 -- 172.21.15.133:0/2480311097 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff1440fcfb0 msgr2=0x7ff1440fe380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:56:31.093 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:56:31.088+0000 7ff14a0d2700 1 --2- 172.21.15.133:0/2480311097 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff1440fcfb0 0x7ff1440fe380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:56:31.093 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:56:31.088+0000 7ff14a0d2700 1 -- 172.21.15.133:0/2480311097 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff134005040 con 0x7ff1440fbe40 2024-02-20T07:56:31.093 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:56:31.089+0000 7ff14a0d2700 1 --2- 172.21.15.133:0/2480311097 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ff1440fbe40 0x7ff1440fff00 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7ff13400b3b0 tx=0x7ff134009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:56:31.093 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:56:31.089+0000 7ff13affd700 1 -- 172.21.15.133:0/2480311097 <== mon.0 v2:172.21.15.133:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff13400e030 con 0x7ff1440fbe40 2024-02-20T07:56:31.093 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:56:31.089+0000 7ff13affd700 1 -- 172.21.15.133:0/2480311097 <== mon.0 v2:172.21.15.133:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff134004290 con 0x7ff1440fbe40 2024-02-20T07:56:31.093 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:56:31.089+0000 7ff13affd700 1 -- 172.21.15.133:0/2480311097 <== mon.0 v2:172.21.15.133:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff13401c640 con 0x7ff1440fbe40 2024-02-20T07:56:31.093 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:56:31.091+0000 7ff14c336700 1 -- 172.21.15.133:0/2480311097 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff1440fe980 con 0x7ff1440fbe40 2024-02-20T07:56:31.093 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:56:31.091+0000 7ff14c336700 1 -- 172.21.15.133:0/2480311097 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff1440fee60 con 0x7ff1440fbe40 2024-02-20T07:56:31.094 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:56:31.092+0000 7ff14c336700 1 -- 172.21.15.133:0/2480311097 --> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff14404bed0 con 0x7ff1440fbe40 2024-02-20T07:56:31.098 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:56:31.097+0000 7ff13affd700 1 -- 172.21.15.133:0/2480311097 <== mon.0 v2:172.21.15.133:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7ff13400bbf0 con 0x7ff1440fbe40 2024-02-20T07:56:31.098 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:56:31.097+0000 7ff13affd700 1 --2- 172.21.15.133:0/2480311097 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7ff1300693c0 0x7ff13006b870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-20T07:56:31.098 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:56:31.097+0000 7ff1498d1700 1 --2- 172.21.15.133:0/2480311097 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7ff1300693c0 0x7ff13006b870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-20T07:56:31.098 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:56:31.098+0000 7ff13affd700 1 -- 172.21.15.133:0/2480311097 <== mon.0 v2:172.21.15.133:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5550+0+0 (secure 0 0 0) 0x7ff134088840 con 0x7ff1440fbe40 2024-02-20T07:56:31.099 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:56:31.099+0000 7ff1498d1700 1 --2- 172.21.15.133:0/2480311097 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7ff1300693c0 0x7ff13006b870 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7ff140004d00 tx=0x7ff140009ce0 comp rx=0 tx=0).ready entity=mgr.14208 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-20T07:56:31.102 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:56:31.102+0000 7ff13affd700 1 -- 172.21.15.133:0/2480311097 <== mon.0 v2:172.21.15.133:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7ff134058590 con 0x7ff1440fbe40 2024-02-20T07:56:31.241 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:56:31 smithi133 ceph-mon[29117]: from='client.14560 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-20T07:56:31.241 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:56:31 smithi133 ceph-mon[29117]: pgmap v217: 65 pgs: 65 active+clean; 1.9 MiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 3.6 KiB/s rd, 111 KiB/s wr, 5 op/s 2024-02-20T07:56:31.242 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:56:31 smithi133 ceph-mon[29117]: from='client.24325 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-20T07:56:31.242 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:56:31 smithi133 ceph-mon[29117]: from='client.? 172.21.15.133:0/4153205754' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-20T07:56:31.382 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:56:31.382+0000 7ff14c336700 1 -- 172.21.15.133:0/2480311097 --> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7ff144066660 con 0x7ff1300693c0 2024-02-20T07:56:31.383 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:56:31.383+0000 7ff13affd700 1 -- 172.21.15.133:0/2480311097 <== mgr.14208 v2:172.21.15.133:6800/2703586938 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+351 (secure 0 0 0) 0x7ff144066660 con 0x7ff1300693c0 2024-02-20T07:56:31.384 INFO:teuthology.orchestra.run.smithi133.stdout:{ 2024-02-20T07:56:31.384 INFO:teuthology.orchestra.run.smithi133.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:91ff50e903dcc99ee3ecf9523050878eb452da5c", 2024-02-20T07:56:31.384 INFO:teuthology.orchestra.run.smithi133.stdout: "in_progress": true, 2024-02-20T07:56:31.384 INFO:teuthology.orchestra.run.smithi133.stdout: "which": "Upgrading daemons of type(s) mgr", 2024-02-20T07:56:31.384 INFO:teuthology.orchestra.run.smithi133.stdout: "services_complete": [], 2024-02-20T07:56:31.384 INFO:teuthology.orchestra.run.smithi133.stdout: "progress": "", 2024-02-20T07:56:31.384 INFO:teuthology.orchestra.run.smithi133.stdout: "message": "Doing first pull of quay.ceph.io/ceph-ci/ceph:91ff50e903dcc99ee3ecf9523050878eb452da5c image", 2024-02-20T07:56:31.384 INFO:teuthology.orchestra.run.smithi133.stdout: "is_paused": false 2024-02-20T07:56:31.384 INFO:teuthology.orchestra.run.smithi133.stdout:} 2024-02-20T07:56:31.387 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:56:31.386+0000 7ff14c336700 1 -- 172.21.15.133:0/2480311097 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7ff1300693c0 msgr2=0x7ff13006b870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:56:31.387 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:56:31.386+0000 7ff14c336700 1 --2- 172.21.15.133:0/2480311097 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7ff1300693c0 0x7ff13006b870 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7ff140004d00 tx=0x7ff140009ce0 comp rx=0 tx=0).stop 2024-02-20T07:56:31.387 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:56:31.386+0000 7ff14c336700 1 -- 172.21.15.133:0/2480311097 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ff1440fbe40 msgr2=0x7ff1440fff00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-20T07:56:31.387 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:56:31.386+0000 7ff14c336700 1 --2- 172.21.15.133:0/2480311097 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ff1440fbe40 0x7ff1440fff00 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7ff13400b3b0 tx=0x7ff134009e30 comp rx=0 tx=0).stop 2024-02-20T07:56:31.387 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:56:31.386+0000 7ff14c336700 1 -- 172.21.15.133:0/2480311097 shutdown_connections 2024-02-20T07:56:31.387 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:56:31.386+0000 7ff14c336700 1 --2- 172.21.15.133:0/2480311097 >> [v2:172.21.15.133:3300/0,v1:172.21.15.133:6789/0] conn(0x7ff1440fbe40 0x7ff1440fff00 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:56:31.387 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:56:31.386+0000 7ff14c336700 1 --2- 172.21.15.133:0/2480311097 >> [v2:172.21.15.133:6800/2703586938,v1:172.21.15.133:6801/2703586938] conn(0x7ff1300693c0 0x7ff13006b870 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:56:31.387 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:56:31.386+0000 7ff14c336700 1 --2- 172.21.15.133:0/2480311097 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff1440fcfb0 0x7ff1440fe380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-20T07:56:31.387 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:56:31.386+0000 7ff14c336700 1 -- 172.21.15.133:0/2480311097 >> 172.21.15.133:0/2480311097 conn(0x7ff1440f7500 msgr2=0x7ff144065f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-20T07:56:31.387 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:56:31.386+0000 7ff14c336700 1 -- 172.21.15.133:0/2480311097 shutdown_connections 2024-02-20T07:56:31.387 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T07:56:31.386+0000 7ff14c336700 1 -- 172.21.15.133:0/2480311097 wait complete. 2024-02-20T07:56:31.547 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:56:31 smithi183 ceph-mon[35832]: from='client.14560 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-20T07:56:31.547 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:56:31 smithi183 ceph-mon[35832]: pgmap v217: 65 pgs: 65 active+clean; 1.9 MiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 3.6 KiB/s rd, 111 KiB/s wr, 5 op/s 2024-02-20T07:56:31.547 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:56:31 smithi183 ceph-mon[35832]: from='client.24325 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-20T07:56:31.547 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:56:31 smithi183 ceph-mon[35832]: from='client.? 172.21.15.133:0/4153205754' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-20T07:56:32.448 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:56:32 smithi183 ceph-mon[35832]: from='client.24329 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-20T07:56:32.449 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:56:32 smithi183 ceph-mon[35832]: from='client.14576 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-20T07:56:32.449 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:56:32 smithi183 ceph-mon[35832]: pgmap v218: 65 pgs: 65 active+clean; 1.9 MiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 3.6 KiB/s rd, 111 KiB/s wr, 4 op/s 2024-02-20T07:56:32.449 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:56:32 smithi183 ceph-mon[35832]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:56:32.449 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:56:32 smithi183 ceph-mon[35832]: Upgrade: Target is version 19.0.0-1587-g91ff50e9 (unknown) 2024-02-20T07:56:32.449 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:56:32 smithi183 ceph-mon[35832]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:0a951893099c32f5b1000756a8c82990a9c3171c476da2a0cf233c4bae101024, digests ['quay.ceph.io/ceph-ci/ceph@sha256:0a951893099c32f5b1000756a8c82990a9c3171c476da2a0cf233c4bae101024'] 2024-02-20T07:56:32.449 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:56:32 smithi183 ceph-mon[35832]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-20T07:56:32.449 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:56:32 smithi183 ceph-mon[35832]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:56:32.449 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:56:32 smithi183 ceph-mon[35832]: Upgrade: Need to upgrade myself (mgr.smithi133.mrvcbp) 2024-02-20T07:56:32.449 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:56:32 smithi183 ceph-mon[35832]: Upgrade: Pulling quay.ceph.io/ceph-ci/ceph@sha256:0a951893099c32f5b1000756a8c82990a9c3171c476da2a0cf233c4bae101024 on smithi183 2024-02-20T07:56:32.492 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:56:32 smithi133 ceph-mon[29117]: from='client.24329 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-20T07:56:32.492 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:56:32 smithi133 ceph-mon[29117]: from='client.14576 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-20T07:56:32.492 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:56:32 smithi133 ceph-mon[29117]: pgmap v218: 65 pgs: 65 active+clean; 1.9 MiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 3.6 KiB/s rd, 111 KiB/s wr, 4 op/s 2024-02-20T07:56:32.492 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:56:32 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:56:32.492 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:56:32 smithi133 ceph-mon[29117]: Upgrade: Target is version 19.0.0-1587-g91ff50e9 (unknown) 2024-02-20T07:56:32.492 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:56:32 smithi133 ceph-mon[29117]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:0a951893099c32f5b1000756a8c82990a9c3171c476da2a0cf233c4bae101024, digests ['quay.ceph.io/ceph-ci/ceph@sha256:0a951893099c32f5b1000756a8c82990a9c3171c476da2a0cf233c4bae101024'] 2024-02-20T07:56:32.492 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:56:32 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-20T07:56:32.492 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:56:32 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' 2024-02-20T07:56:32.492 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:56:32 smithi133 ceph-mon[29117]: Upgrade: Need to upgrade myself (mgr.smithi133.mrvcbp) 2024-02-20T07:56:32.492 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:56:32 smithi133 ceph-mon[29117]: Upgrade: Pulling quay.ceph.io/ceph-ci/ceph@sha256:0a951893099c32f5b1000756a8c82990a9c3171c476da2a0cf233c4bae101024 on smithi183 2024-02-20T07:56:34.991 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:56:34 smithi133 ceph-mon[29117]: pgmap v219: 65 pgs: 65 active+clean; 23 MiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 784 KiB/s rd, 2.0 MiB/s wr, 116 op/s 2024-02-20T07:56:35.047 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:56:34 smithi183 ceph-mon[35832]: pgmap v219: 65 pgs: 65 active+clean; 23 MiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 784 KiB/s rd, 2.0 MiB/s wr, 116 op/s 2024-02-20T07:56:36.991 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:56:36 smithi133 ceph-mon[29117]: pgmap v220: 65 pgs: 65 active+clean; 25 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 784 KiB/s rd, 2.2 MiB/s wr, 156 op/s 2024-02-20T07:56:36.991 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:56:36 smithi133 ceph-mon[29117]: fsmap cephfs:2 {0=cephfs.smithi133.cmofvy=up:active,1=cephfs.smithi183.eipfta=up:active} 2 up:standby-replay 2024-02-20T07:56:37.047 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:56:36 smithi183 ceph-mon[35832]: pgmap v220: 65 pgs: 65 active+clean; 25 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 784 KiB/s rd, 2.2 MiB/s wr, 156 op/s 2024-02-20T07:56:37.047 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:56:36 smithi183 ceph-mon[35832]: fsmap cephfs:2 {0=cephfs.smithi133.cmofvy=up:active,1=cephfs.smithi183.eipfta=up:active} 2 up:standby-replay 2024-02-20T07:56:38.992 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:56:38 smithi133 ceph-mon[29117]: pgmap v221: 65 pgs: 65 active+clean; 47 MiB data, 1.9 GiB used, 535 GiB / 536 GiB avail; 1.1 MiB/s rd, 4.1 MiB/s wr, 386 op/s 2024-02-20T07:56:38.992 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:56:38 smithi133 ceph-mon[29117]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-20T07:56:39.047 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:56:38 smithi183 ceph-mon[35832]: pgmap v221: 65 pgs: 65 active+clean; 47 MiB data, 1.9 GiB used, 535 GiB / 536 GiB avail; 1.1 MiB/s rd, 4.1 MiB/s wr, 386 op/s 2024-02-20T07:56:39.047 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:56:38 smithi183 ceph-mon[35832]: from='mgr.14208 172.21.15.133:0/3399870080' entity='mgr.smithi133.mrvcbp' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-20T07:56:40.957 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:56:40 smithi133 ceph-mon[29117]: pgmap v222: 65 pgs: 65 active+clean; 56 MiB data, 1.9 GiB used, 534 GiB / 536 GiB avail; 1.8 MiB/s rd, 4.9 MiB/s wr, 422 op/s 2024-02-20T07:56:41.047 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:56:40 smithi183 ceph-mon[35832]: pgmap v222: 65 pgs: 65 active+clean; 56 MiB data, 1.9 GiB used, 534 GiB / 536 GiB avail; 1.8 MiB/s rd, 4.9 MiB/s wr, 422 op/s 2024-02-20T07:56:42.991 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:56:42 smithi133 ceph-mon[29117]: pgmap v223: 65 pgs: 65 active+clean; 60 MiB data, 1.9 GiB used, 534 GiB / 536 GiB avail; 2.1 MiB/s rd, 5.1 MiB/s wr, 425 op/s 2024-02-20T07:56:43.047 INFO:journalctl@ceph.mon.smithi183.smithi183.stdout:Feb 20 07:56:42 smithi183 ceph-mon[35832]: pgmap v223: 65 pgs: 65 active+clean; 60 MiB data, 1.9 GiB used, 534 GiB / 536 GiB avail; 2.1 MiB/s rd, 5.1 MiB/s wr, 425 op/s 2024-02-20T07:56:43.803 INFO:tasks.workunit.client.1.smithi183.stderr:+ pushd ltp-full-20091231/testcases/kernel/fs/fsstress 2024-02-20T07:56:43.803 INFO:tasks.workunit.client.1.smithi183.stdout:~/cephtest/mnt.1/client.1/tmp/fsstress/ltp-full-20091231/testcases/kernel/fs/fsstress ~/cephtest/mnt.1/client.1/tmp/fsstress ~/cephtest/mnt.1/client.1/tmp 2024-02-20T07:56:43.803 INFO:tasks.workunit.client.1.smithi183.stderr:+ make 2024-02-20T07:56:43.899 INFO:tasks.workunit.client.1.smithi183.stdout:cc -DNO_XFS -I/home/ubuntu/cephtest/mnt.1/client.1/tmp/fsstress/ltp-full-20091231/testcases/kernel/fs/fsstress -D_LARGEFILE64_SOURCE -D_GNU_SOURCE -I../../../../include -I../../../../include -L../../../../lib fsstress.c -o fsstress 2024-02-20T07:57:02.858 INFO:tasks.workunit.client.0.smithi133.stderr:Note: switching to '3d72bcfd7affbb4a923c191d212814b4934cfeaa'. 2024-02-20T07:57:02.858 INFO:tasks.workunit.client.0.smithi133.stderr: 2024-02-20T07:57:02.858 INFO:tasks.workunit.client.0.smithi133.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-02-20T07:57:02.858 INFO:tasks.workunit.client.0.smithi133.stderr:changes and commit them, and you can discard any commits you make in this 2024-02-20T07:57:02.858 INFO:tasks.workunit.client.0.smithi133.stderr:state without impacting any branches by switching back to a branch. 2024-02-20T07:57:02.859 INFO:tasks.workunit.client.0.smithi133.stderr: 2024-02-20T07:57:02.859 INFO:tasks.workunit.client.0.smithi133.stderr:If you want to create a new branch to retain commits you create, you may 2024-02-20T07:57:02.859 INFO:tasks.workunit.client.0.smithi133.stderr:do so (now or later) by using -c with the switch command. Example: 2024-02-20T07:57:02.859 INFO:tasks.workunit.client.0.smithi133.stderr: 2024-02-20T07:57:02.859 INFO:tasks.workunit.client.0.smithi133.stderr: git switch -c 2024-02-20T07:57:02.859 INFO:tasks.workunit.client.0.smithi133.stderr: 2024-02-20T07:57:02.859 INFO:tasks.workunit.client.0.smithi133.stderr:Or undo this operation with: 2024-02-20T07:57:02.859 INFO:tasks.workunit.client.0.smithi133.stderr: 2024-02-20T07:57:02.859 INFO:tasks.workunit.client.0.smithi133.stderr: git switch - 2024-02-20T07:57:02.859 INFO:tasks.workunit.client.0.smithi133.stderr: 2024-02-20T07:57:02.859 INFO:tasks.workunit.client.0.smithi133.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-02-20T07:57:02.859 INFO:tasks.workunit.client.0.smithi133.stderr: 2024-02-20T07:57:02.859 INFO:tasks.workunit.client.0.smithi133.stderr:HEAD is now at 3d72bcfd7af test 2024-02-20T07:57:02.863 DEBUG:teuthology.orchestra.run.smithi133:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-02-20T07:57:02.891 INFO:tasks.workunit.client.0.smithi133.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-02-20T07:57:02.893 INFO:tasks.workunit.client.0.smithi133.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-02-20T07:57:02.893 INFO:tasks.workunit.client.0.smithi133.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-02-20T07:57:03.705 INFO:tasks.workunit.client.0.smithi133.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-02-20T07:57:03.760 INFO:tasks.workunit.client.0.smithi133.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-02-20T07:57:03.800 INFO:tasks.workunit.client.0.smithi133.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-02-20T07:57:03.801 INFO:tasks.workunit.client.0.smithi133.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-02-20T07:57:03.801 INFO:tasks.workunit.client.0.smithi133.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-02-20T07:57:03.839 INFO:tasks.workunit.client.0.smithi133.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-02-20T07:57:03.842 DEBUG:teuthology.orchestra.run.smithi133:> set -ex 2024-02-20T07:57:03.843 DEBUG:teuthology.orchestra.run.smithi133:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-02-20T07:57:03.899 INFO:tasks.workunit:Running workunits matching suites/fsstress.sh on client.0... 2024-02-20T07:57:03.900 INFO:tasks.workunit:Running workunit suites/fsstress.sh... 2024-02-20T07:57:03.900 DEBUG:teuthology.orchestra.run.smithi133:workunit test suites/fsstress.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=3d72bcfd7affbb4a923c191d212814b4934cfeaa TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/fsstress.sh 2024-02-20T07:57:05.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:57:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:57:05.646+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:57:05.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:57:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:57:05.646+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:57:05.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:57:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:57:05.646+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:57:06.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:57:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:57:06.672+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:57:06.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:57:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:57:06.672+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:57:06.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:57:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:57:06.672+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:57:06.992 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:57:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:57:06.656+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:57:06.992 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:57:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:57:06.656+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:57:06.992 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:57:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:57:06.656+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:57:07.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:57:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:57:07.652+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:57:07.992 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:57:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:57:07.652+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:57:07.992 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:57:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:57:07.652+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:57:07.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:57:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:57:07.676+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:57:07.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:57:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:57:07.676+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:57:07.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:57:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:57:07.676+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:57:08.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:57:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:57:08.682+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:57:08.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:57:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:57:08.682+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:57:08.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:57:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:57:08.682+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:57:08.992 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:57:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:57:08.695+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:57:08.992 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:57:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:57:08.695+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:57:08.992 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:57:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:57:08.695+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:57:09.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:57:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:57:09.732+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:57:09.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:57:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:57:09.732+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:57:09.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:57:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:57:09.732+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:57:09.992 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:57:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:57:09.646+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:57:09.992 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:57:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:57:09.646+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:57:09.992 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:57:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:57:09.646+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:57:10.491 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:57:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:57:10.114+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:57:10.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:57:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:57:10.114+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:57:10.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:57:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:57:10.114+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:57:10.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:57:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:57:10.596+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:57:10.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:57:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:57:10.596+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:57:10.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:57:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:57:10.596+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:57:10.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:57:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:57:10.763+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:57:10.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:57:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:57:10.763+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:57:10.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:57:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:57:10.763+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:57:11.491 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:57:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:57:11.084+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:57:11.491 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:57:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:57:11.084+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:57:11.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:57:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:57:11.084+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:57:11.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:57:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:57:11.646+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:57:11.992 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:57:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:57:11.646+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:57:11.992 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:57:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:57:11.646+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:57:11.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:57:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:57:11.748+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:57:11.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:57:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:57:11.748+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:57:11.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:57:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:57:11.748+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:57:12.491 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:57:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:57:12.036+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:57:12.491 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:57:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:57:12.036+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:57:12.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:57:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:57:12.036+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:57:12.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:57:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:57:12.628+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:57:12.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:57:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:57:12.628+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:57:12.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:57:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:57:12.628+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:57:12.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:57:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:57:12.764+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:57:12.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:57:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:57:12.764+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:57:12.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:57:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:57:12.764+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:57:13.491 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:57:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:57:13.073+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:57:13.491 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:57:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:57:13.073+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:57:13.491 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:57:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:57:13.073+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:57:13.796 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:57:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:57:13.597+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:57:13.796 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:57:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:57:13.597+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:57:13.796 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:57:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:57:13.597+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:57:14.117 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:57:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:57:13.795+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:57:14.117 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:57:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:57:13.795+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:57:14.117 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:57:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:57:13.795+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:57:14.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:57:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:57:14.116+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:57:14.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:57:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:57:14.116+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:57:14.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:57:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:57:14.116+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:57:14.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:57:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:57:14.823+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:57:14.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:57:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:57:14.823+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:57:14.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:57:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:57:14.823+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:57:14.992 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:57:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:57:14.625+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:57:14.992 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:57:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:57:14.625+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:57:14.992 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:57:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:57:14.625+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:57:15.491 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:57:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:57:15.149+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:57:15.491 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:57:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:57:15.149+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:57:15.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:57:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:57:15.149+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:57:15.867 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:57:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:57:15.605+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:57:15.867 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:57:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:57:15.605+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:57:15.867 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:57:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:57:15.605+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:57:16.182 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:57:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:57:15.867+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:57:16.182 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:57:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:57:15.867+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:57:16.182 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:57:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:57:15.867+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:57:16.491 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:57:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:57:16.181+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:57:16.491 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:57:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:57:16.181+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:57:16.491 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:57:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:57:16.181+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:57:16.896 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:57:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:57:16.626+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:57:16.896 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:57:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:57:16.626+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:57:16.896 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:57:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:57:16.626+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:57:17.160 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:57:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:57:16.896+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:57:17.160 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:57:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:57:16.896+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:57:17.160 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:57:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:57:16.896+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:57:17.491 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:57:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:57:17.159+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:57:17.491 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:57:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:57:17.159+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:57:17.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:57:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:57:17.159+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:57:17.880 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:57:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:57:17.611+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:57:17.880 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:57:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:57:17.611+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:57:17.880 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:57:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:57:17.611+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:57:18.154 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:57:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:57:17.879+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:57:18.154 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:57:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:57:17.879+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:57:18.155 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:57:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:57:17.879+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:57:18.491 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:57:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:57:18.154+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:57:18.599 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:57:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:57:18.154+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:57:18.599 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:57:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:57:18.154+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:57:18.915 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:57:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:57:18.605+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:57:18.916 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:57:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:57:18.605+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:57:18.916 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:57:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:57:18.605+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:57:19.241 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:57:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:57:19.123+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:57:19.241 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:57:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:57:19.123+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:57:19.241 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:57:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:57:19.123+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:57:19.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:57:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:57:18.915+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:57:19.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:57:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:57:18.915+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:57:19.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:57:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:57:18.915+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:57:19.885 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:57:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:57:19.595+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:57:19.885 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:57:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:57:19.595+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:57:19.886 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:57:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:57:19.595+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:57:20.241 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:57:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:57:20.123+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:57:20.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:57:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:57:20.123+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:57:20.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:57:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:57:20.123+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:57:20.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:57:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:57:19.914+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:57:20.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:57:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:57:19.914+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:57:20.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:57:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:57:19.914+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:57:20.896 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:57:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:57:20.642+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:57:20.896 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:57:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:57:20.642+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:57:20.896 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:57:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:57:20.642+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:57:21.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:57:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:57:20.895+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:57:21.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:57:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:57:20.895+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:57:21.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:57:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:57:20.895+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:57:21.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:57:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:57:21.138+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:57:21.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:57:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:57:21.138+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:57:21.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:57:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:57:21.138+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:57:21.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:57:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:57:21.857+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:57:21.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:57:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:57:21.857+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:57:21.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:57:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:57:21.857+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:57:21.992 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:57:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:57:21.641+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:57:21.992 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:57:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:57:21.641+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:57:21.992 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:57:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:57:21.641+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:57:22.491 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:57:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:57:22.154+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:57:22.491 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:57:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:57:22.154+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:57:22.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:57:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:57:22.154+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:57:22.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:57:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:57:22.854+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:57:22.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:57:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:57:22.854+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:57:22.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:57:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:57:22.854+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:57:22.992 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:57:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:57:22.645+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:57:22.992 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:57:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:57:22.645+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:57:22.992 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:57:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:57:22.645+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:57:23.491 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:57:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:57:23.140+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:57:23.491 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:57:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:57:23.140+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:57:23.491 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:57:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:57:23.140+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:57:23.844 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:57:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:57:23.681+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:57:23.845 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:57:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:57:23.681+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:57:23.845 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:57:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:57:23.681+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:57:24.130 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:57:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:57:23.844+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:57:24.130 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:57:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:57:23.844+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:57:24.131 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:57:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:57:23.844+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:57:24.491 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:57:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:57:24.130+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:57:24.491 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:57:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:57:24.130+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:57:24.491 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:57:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:57:24.130+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:57:24.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:57:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:57:24.706+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:57:24.992 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:57:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:57:24.706+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:57:24.992 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:57:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:57:24.706+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:57:24.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:57:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:57:24.843+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:57:24.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:57:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:57:24.843+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:57:24.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:57:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:57:24.843+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:57:25.491 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:57:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:57:25.174+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:57:25.491 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:57:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:57:25.174+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:57:25.491 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:57:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:57:25.174+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:57:25.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:57:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:57:25.817+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:57:25.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:57:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:57:25.817+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:57:25.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:57:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:57:25.817+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:57:25.992 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:57:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:57:25.737+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:57:25.992 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:57:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:57:25.737+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:57:25.992 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:57:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:57:25.737+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:57:26.491 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:57:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:57:26.173+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:57:26.491 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:57:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:57:26.173+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:57:26.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:57:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:57:26.173+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:57:26.992 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:57:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:57:26.763+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:57:26.992 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:57:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:57:26.763+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:57:26.992 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:57:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:57:26.763+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:57:26.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:57:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:57:26.767+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:57:26.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:57:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:57:26.767+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:57:26.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:57:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:57:26.767+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:57:27.491 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:57:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:57:27.128+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:57:27.491 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:57:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:57:27.128+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:57:27.491 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:57:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:57:27.128+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:57:27.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:57:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:57:27.732+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:57:27.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:57:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:57:27.732+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:57:27.992 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:57:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:57:27.732+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:57:27.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:57:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:57:27.787+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:57:27.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:57:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:57:27.787+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:57:27.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:57:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:57:27.787+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:57:28.491 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:57:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:57:28.094+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:57:28.491 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:57:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:57:28.094+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:57:28.491 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:57:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:57:28.094+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:57:28.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:57:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:57:28.815+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:57:28.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:57:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:57:28.815+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:57:28.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:57:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:57:28.815+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:57:28.992 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:57:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:57:28.737+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:57:28.992 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:57:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:57:28.737+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:57:28.992 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:57:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:57:28.737+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:57:29.491 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:57:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:57:29.066+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:57:29.491 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:57:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:57:29.066+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:57:29.491 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:57:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:57:29.066+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:57:30.070 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:57:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:57:29.788+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:57:30.071 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:57:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:57:29.788+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:57:30.071 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:57:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:57:29.788+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:57:30.071 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:57:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:57:29.776+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:57:30.071 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:57:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:57:29.776+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:57:30.071 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:57:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:57:29.776+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:57:30.491 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:57:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:57:30.069+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:57:30.491 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:57:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:57:30.069+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:57:30.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:57:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:57:30.069+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:57:30.992 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:57:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:57:30.737+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:57:30.992 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:57:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:57:30.737+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:57:30.992 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:57:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:57:30.737+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:57:30.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:57:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:57:30.825+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:57:30.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:57:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:57:30.825+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:57:30.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:57:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:57:30.825+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:57:31.491 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:57:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:57:31.043+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:57:31.491 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:57:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:57:31.043+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:57:31.491 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:57:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:57:31.043+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:57:31.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:57:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:57:31.822+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:57:31.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:57:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:57:31.822+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:57:31.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:57:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:57:31.822+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:57:31.992 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:57:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:57:31.767+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:57:31.992 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:57:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:57:31.767+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:57:31.992 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:57:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:57:31.767+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:57:32.491 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:57:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:57:32.075+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:57:32.491 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:57:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:57:32.075+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:57:32.491 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:57:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:57:32.075+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:57:33.101 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:57:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:57:32.746+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:57:33.101 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:57:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:57:32.746+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:57:33.101 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:57:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:57:32.746+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:57:33.102 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:57:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:57:32.849+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:57:33.102 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:57:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:57:32.849+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:57:33.102 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:57:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:57:32.849+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:57:33.491 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:57:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:57:33.099+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:57:33.491 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:57:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:57:33.099+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:57:33.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:57:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:57:33.099+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:57:34.065 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:57:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:57:33.809+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:57:34.065 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:57:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:57:33.809+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:57:34.065 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:57:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:57:33.809+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:57:34.066 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:57:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:57:33.784+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:57:34.066 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:57:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:57:33.784+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:57:34.066 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:57:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:57:33.784+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:57:34.491 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:57:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:57:34.064+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:57:34.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:57:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:57:34.064+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:57:34.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:57:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:57:34.064+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:57:35.028 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:57:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:57:34.825+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:57:35.028 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:57:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:57:34.825+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:57:35.028 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:57:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:57:34.825+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:57:35.029 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:57:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:57:34.765+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:57:35.029 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:57:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:57:34.765+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:57:35.029 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:57:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:57:34.765+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:57:35.491 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:57:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:57:35.027+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:57:35.491 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:57:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:57:35.027+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:57:35.491 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:57:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:57:35.027+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:57:36.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:57:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:57:35.807+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:57:36.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:57:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:57:35.807+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:57:36.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:57:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:57:35.807+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:57:36.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:57:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:57:36.005+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:57:36.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:57:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:57:36.005+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:57:36.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:57:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:57:36.005+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:57:36.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:57:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:57:35.845+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:57:36.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:57:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:57:35.845+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:57:36.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:57:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:57:35.845+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:57:36.989 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:57:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:57:36.843+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:57:36.989 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:57:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:57:36.843+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:57:36.989 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:57:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:57:36.843+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:57:36.989 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:57:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:57:36.876+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:57:36.990 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:57:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:57:36.876+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:57:36.990 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:57:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:57:36.876+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:57:37.241 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:57:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:57:36.987+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:57:37.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:57:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:57:36.987+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:57:37.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:57:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:57:36.987+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:57:38.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:57:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:57:37.840+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:57:38.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:57:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:57:37.840+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:57:38.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:57:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:57:37.840+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:57:38.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:57:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:57:37.973+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:57:38.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:57:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:57:37.973+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:57:38.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:57:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:57:37.973+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:57:38.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:57:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:57:37.880+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:57:38.245 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:57:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:57:37.880+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:57:38.245 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:57:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:57:37.880+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:57:39.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:57:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:57:38.812+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:57:39.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:57:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:57:38.812+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:57:39.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:57:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:57:38.812+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:57:39.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:57:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:57:39.015+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:57:39.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:57:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:57:39.015+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:57:39.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:57:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:57:39.015+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:57:39.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:57:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:57:38.889+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:57:39.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:57:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:57:38.889+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:57:39.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:57:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:57:38.889+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:57:40.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:57:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:57:39.805+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:57:40.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:57:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:57:39.805+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:57:40.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:57:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:57:39.805+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:57:40.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:57:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:57:39.987+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:57:40.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:57:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:57:39.987+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:57:40.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:57:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:57:39.987+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:57:40.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:57:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:57:39.888+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:57:40.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:57:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:57:39.888+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:57:40.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:57:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:57:39.888+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:57:41.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:57:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:57:40.845+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:57:41.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:57:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:57:40.845+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:57:41.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:57:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:57:40.845+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:57:41.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:57:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:57:41.020+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:57:41.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:57:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:57:41.020+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:57:41.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:57:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:57:41.020+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:57:41.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:57:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:57:40.918+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:57:41.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:57:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:57:40.918+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:57:41.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:57:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:57:40.918+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:57:41.940 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:57:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:57:41.871+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:57:41.940 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:57:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:57:41.871+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:57:41.940 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:57:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:57:41.871+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:57:42.241 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:57:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:57:42.000+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:57:42.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:57:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:57:42.000+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:57:42.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:57:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:57:42.000+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:57:42.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:57:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:57:41.938+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:57:42.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:57:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:57:41.938+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:57:42.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:57:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:57:41.938+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:57:43.241 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:57:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:57:42.955+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:57:43.241 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:57:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:57:42.955+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:57:43.241 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:57:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:57:42.955+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:57:43.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:57:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:57:42.950+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:57:43.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:57:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:57:42.950+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:57:43.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:57:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:57:42.950+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:57:43.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:57:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:57:42.895+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:57:43.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:57:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:57:42.895+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:57:43.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:57:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:57:42.895+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:57:44.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:57:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:57:43.938+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:57:44.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:57:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:57:43.938+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:57:44.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:57:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:57:43.938+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:57:44.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:57:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:57:43.975+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:57:44.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:57:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:57:43.975+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:57:44.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:57:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:57:43.975+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:57:44.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:57:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:57:43.991+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:57:44.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:57:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:57:43.991+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:57:44.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:57:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:57:43.991+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:57:45.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:57:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:57:44.935+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:57:45.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:57:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:57:44.935+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:57:45.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:57:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:57:44.935+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:57:45.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:57:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:57:45.003+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:57:45.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:57:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:57:45.003+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:57:45.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:57:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:57:45.003+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:57:45.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:57:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:57:44.973+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:57:45.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:57:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:57:44.973+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:57:45.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:57:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:57:44.973+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:57:46.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:57:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:57:45.931+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:57:46.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:57:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:57:45.931+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:57:46.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:57:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:57:45.931+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:57:46.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:57:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:57:45.972+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:57:46.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:57:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:57:45.972+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:57:46.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:57:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:57:45.972+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:57:46.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:57:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:57:46.003+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:57:46.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:57:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:57:46.003+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:57:46.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:57:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:57:46.003+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:57:46.969 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:57:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:57:46.918+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:57:46.970 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:57:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:57:46.918+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:57:46.970 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:57:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:57:46.918+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:57:47.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:57:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:57:47.011+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:57:47.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:57:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:57:47.011+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:57:47.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:57:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:57:47.011+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:57:47.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:57:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:57:46.968+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:57:47.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:57:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:57:46.968+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:57:47.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:57:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:57:46.968+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:57:48.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:57:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:57:47.914+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:57:48.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:57:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:57:47.914+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:57:48.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:57:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:57:47.914+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:57:48.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:57:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:57:47.995+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:57:48.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:57:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:57:47.995+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:57:48.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:57:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:57:47.995+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:57:48.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:57:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:57:47.923+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:57:48.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:57:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:57:47.923+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:57:48.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:57:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:57:47.923+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:57:49.176 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:57:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:57:48.917+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:57:49.176 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:57:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:57:48.917+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:57:49.176 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:57:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:57:48.917+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:57:49.176 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:57:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:57:49.000+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:57:49.176 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:57:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:57:49.000+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:57:49.176 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:57:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:57:49.000+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:57:49.177 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:57:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:57:48.974+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:57:49.177 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:57:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:57:48.974+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:57:49.177 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:57:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:57:48.974+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:57:50.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:57:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:57:49.915+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:57:50.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:57:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:57:49.915+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:57:50.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:57:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:57:49.915+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:57:50.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:57:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:57:49.971+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:57:50.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:57:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:57:49.971+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:57:50.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:57:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:57:49.971+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:57:50.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:57:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:57:49.993+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:57:50.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:57:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:57:49.993+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:57:50.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:57:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:57:49.993+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:57:51.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:57:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:57:50.950+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:57:51.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:57:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:57:50.950+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:57:51.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:57:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:57:50.950+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:57:51.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:57:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:57:50.966+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:57:51.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:57:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:57:50.966+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:57:51.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:57:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:57:50.966+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:57:51.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:57:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:57:50.975+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:57:51.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:57:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:57:50.975+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:57:51.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:57:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:57:50.975+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:57:52.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:57:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:57:51.968+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:57:52.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:57:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:57:51.968+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:57:52.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:57:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:57:51.968+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:57:52.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:57:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:57:51.967+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:57:52.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:57:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:57:51.967+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:57:52.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:57:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:57:51.967+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:57:52.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:57:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:57:51.982+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:57:52.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:57:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:57:51.982+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:57:52.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:57:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:57:51.982+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:57:53.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:57:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:57:52.927+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:57:53.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:57:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:57:52.927+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:57:53.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:57:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:57:52.927+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:57:53.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:57:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:57:52.937+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:57:53.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:57:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:57:52.937+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:57:53.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:57:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:57:52.937+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:57:53.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:57:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:57:52.965+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:57:53.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:57:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:57:52.965+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:57:53.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:57:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:57:52.965+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:57:54.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:57:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:57:53.890+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:57:54.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:57:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:57:53.890+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:57:54.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:57:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:57:53.890+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:57:54.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:57:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:57:53.950+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:57:54.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:57:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:57:53.950+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:57:54.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:57:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:57:53.950+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:57:54.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:57:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:57:53.999+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:57:54.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:57:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:57:53.999+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:57:54.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:57:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:57:53.999+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:57:55.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:57:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:57:54.862+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:57:55.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:57:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:57:54.862+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:57:55.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:57:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:57:54.862+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:57:55.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:57:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:57:54.948+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:57:55.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:57:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:57:54.948+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:57:55.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:57:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:57:54.948+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:57:55.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:57:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:57:55.030+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:57:55.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:57:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:57:55.030+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:57:55.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:57:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:57:55.030+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:57:56.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:57:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:57:55.868+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:57:56.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:57:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:57:55.868+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:57:56.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:57:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:57:55.868+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:57:56.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:57:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:57:55.934+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:57:56.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:57:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:57:55.934+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:57:56.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:57:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:57:55.934+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:57:56.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:57:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:57:56.057+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:57:56.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:57:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:57:56.057+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:57:56.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:57:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:57:56.057+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:57:56.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:57:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:57:56.835+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:57:56.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:57:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:57:56.835+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:57:56.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:57:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:57:56.835+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:57:56.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:57:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:57:56.910+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:57:56.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:57:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:57:56.910+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:57:56.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:57:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:57:56.910+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:57:57.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:57:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:57:57.060+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:57:57.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:57:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:57:57.060+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:57:57.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:57:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:57:57.060+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:57:58.048 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:57:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:57:57.789+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:57:58.048 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:57:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:57:57.789+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:57:58.048 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:57:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:57:57.789+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:57:58.048 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:57:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:57:57.868+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:57:58.048 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:57:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:57:57.868+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:57:58.048 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:57:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:57:57.868+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:57:58.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:57:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:57:58.046+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:57:58.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:57:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:57:58.046+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:57:58.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:57:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:57:58.046+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:57:59.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:57:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:57:58.819+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:57:59.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:57:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:57:58.819+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:57:59.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:57:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:57:58.819+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:57:59.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:57:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:57:58.839+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:57:59.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:57:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:57:58.839+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:57:59.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:57:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:57:58.839+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:57:59.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:57:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:57:59.060+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:57:59.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:57:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:57:59.060+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:57:59.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:57:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:57:59.060+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:58:00.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:57:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:57:59.837+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:58:00.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:57:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:57:59.837+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:58:00.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:57:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:57:59.837+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:58:00.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:57:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:57:59.805+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:58:00.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:57:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:57:59.805+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:58:00.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:57:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:57:59.805+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:58:00.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:58:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:58:00.026+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:58:00.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:58:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:58:00.026+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:58:00.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:58:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:58:00.026+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:58:01.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:58:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:58:00.834+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:58:01.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:58:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:58:00.834+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:58:01.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:58:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:58:00.834+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:58:01.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:58:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:58:00.801+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:58:01.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:58:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:58:00.801+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:58:01.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:58:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:58:00.801+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:58:01.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:58:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:58:01.011+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:58:01.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:58:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:58:01.011+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:58:01.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:58:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:58:01.011+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:58:01.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:58:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:58:01.880+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:58:01.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:58:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:58:01.880+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:58:01.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:58:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:58:01.880+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:58:01.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:58:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:58:01.825+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:58:01.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:58:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:58:01.825+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:58:01.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:58:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:58:01.825+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:58:02.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:58:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:58:02.046+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:58:02.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:58:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:58:02.046+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:58:02.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:58:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:58:02.046+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:58:03.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:58:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:58:02.848+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:58:03.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:58:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:58:02.848+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:58:03.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:58:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:58:02.848+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:58:03.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:58:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:58:02.865+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:58:03.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:58:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:58:02.865+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:58:03.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:58:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:58:02.865+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:58:03.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:58:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:58:03.082+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:58:03.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:58:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:58:03.082+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:58:03.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:58:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:58:03.082+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:58:04.126 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:58:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:58:03.886+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:58:04.127 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:58:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:58:03.886+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:58:04.127 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:58:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:58:03.886+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:58:04.127 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:58:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:58:03.832+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:58:04.127 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:58:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:58:03.832+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:58:04.127 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:58:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:58:03.832+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:58:04.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:58:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:58:04.126+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:58:04.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:58:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:58:04.126+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:58:04.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:58:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:58:04.126+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:58:05.174 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:58:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:58:04.905+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:58:05.174 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:58:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:58:04.905+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:58:05.174 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:58:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:58:04.905+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:58:05.175 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:58:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:58:04.880+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:58:05.175 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:58:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:58:04.880+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:58:05.175 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:58:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:58:04.880+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:58:05.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:58:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:58:05.173+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:58:05.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:58:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:58:05.173+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:58:05.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:58:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:58:05.173+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:58:06.241 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:58:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:58:05.904+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:58:06.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:58:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:58:05.904+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:58:06.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:58:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:58:05.904+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:58:06.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:58:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:58:06.129+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:58:06.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:58:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:58:06.129+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:58:06.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:58:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:58:06.129+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:58:06.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:58:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:58:05.892+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:58:06.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:58:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:58:05.892+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:58:06.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:58:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:58:05.892+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:58:06.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:58:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:58:06.888+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:58:06.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:58:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:58:06.888+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:58:06.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:58:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:58:06.888+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:58:06.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:58:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:58:06.887+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:58:06.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:58:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:58:06.887+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:58:06.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:58:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:58:06.887+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:58:07.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:58:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:58:07.137+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:58:07.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:58:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:58:07.137+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:58:07.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:58:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:58:07.137+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:58:08.241 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:58:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:58:07.923+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:58:08.241 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:58:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:58:07.923+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:58:08.241 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:58:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:58:07.923+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:58:08.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:58:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:58:07.896+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:58:08.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:58:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:58:07.897+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:58:08.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:58:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:58:07.897+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:58:08.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:58:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:58:08.090+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:58:08.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:58:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:58:08.090+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:58:08.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:58:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:58:08.090+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:58:09.241 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:58:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:58:08.893+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:58:09.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:58:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:58:08.893+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:58:09.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:58:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:58:08.893+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:58:09.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:58:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:58:08.937+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:58:09.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:58:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:58:08.937+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:58:09.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:58:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:58:08.937+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:58:09.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:58:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:58:09.101+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:58:09.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:58:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:58:09.101+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:58:09.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:58:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:58:09.101+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:58:10.241 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:58:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:58:09.933+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:58:10.241 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:58:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:58:09.933+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:58:10.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:58:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:58:09.933+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:58:10.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:58:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:58:09.897+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:58:10.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:58:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:58:09.897+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:58:10.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:58:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:58:09.897+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:58:10.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:58:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:58:10.091+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:58:10.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:58:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:58:10.091+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:58:10.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:58:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:58:10.091+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:58:11.241 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:58:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:58:10.916+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:58:11.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:58:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:58:10.916+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:58:11.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:58:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:58:10.916+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:58:11.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:58:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:58:10.885+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:58:11.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:58:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:58:10.885+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:58:11.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:58:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:58:10.885+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:58:11.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:58:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:58:11.057+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:58:11.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:58:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:58:11.057+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:58:11.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:58:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:58:11.057+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:58:11.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:58:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:58:11.913+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:58:11.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:58:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:58:11.913+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:58:11.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:58:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:58:11.913+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:58:11.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:58:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:58:11.915+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:58:11.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:58:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:58:11.915+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:58:11.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:58:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:58:11.915+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:58:12.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:58:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:58:12.051+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:58:12.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:58:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:58:12.051+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:58:12.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:58:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:58:12.051+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:58:13.241 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:58:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:58:12.890+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:58:13.241 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:58:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:58:12.890+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:58:13.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:58:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:58:12.890+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:58:13.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:58:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:58:12.879+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:58:13.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:58:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:58:12.879+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:58:13.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:58:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:58:12.879+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:58:13.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:58:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:58:13.020+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:58:13.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:58:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:58:13.020+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:58:13.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:58:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:58:13.020+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:58:14.241 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:58:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:58:13.880+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:58:14.241 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:58:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:58:13.880+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:58:14.241 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:58:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:58:13.880+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:58:14.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:58:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:58:13.843+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:58:14.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:58:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:58:13.843+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:58:14.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:58:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:58:13.843+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:58:14.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:58:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:58:14.053+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:58:14.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:58:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:58:14.053+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:58:14.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:58:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:58:14.053+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:58:15.066 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:58:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:58:14.796+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:58:15.067 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:58:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:58:14.796+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:58:15.067 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:58:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:58:14.796+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:58:15.067 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:58:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:58:14.907+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:58:15.067 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:58:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:58:14.907+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:58:15.067 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:58:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:58:14.907+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:58:15.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:58:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:58:15.066+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:58:15.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:58:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:58:15.066+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:58:15.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:58:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:58:15.066+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:58:16.241 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:58:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:58:15.933+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:58:16.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:58:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:58:15.933+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:58:16.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:58:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:58:15.933+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:58:16.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:58:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:58:15.792+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:58:16.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:58:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:58:15.792+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:58:16.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:58:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:58:15.792+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:58:16.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:58:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:58:16.038+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:58:16.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:58:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:58:16.038+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:58:16.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:58:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:58:16.038+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:58:16.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:58:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:58:16.805+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:58:16.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:58:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:58:16.805+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:58:16.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:58:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:58:16.805+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:58:16.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:58:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:58:16.918+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:58:16.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:58:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:58:16.918+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:58:16.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:58:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:58:16.918+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:58:17.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:58:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:58:16.996+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:58:17.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:58:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:58:16.996+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:58:17.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:58:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:58:16.996+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:58:18.241 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:58:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:58:17.966+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:58:18.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:58:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:58:17.966+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:58:18.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:58:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:58:17.966+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:58:18.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:58:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:58:17.756+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:58:18.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:58:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:58:17.756+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:58:18.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:58:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:58:17.756+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:58:18.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:58:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:58:17.947+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:58:18.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:58:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:58:17.947+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:58:18.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:58:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:58:17.947+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:58:19.241 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:58:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:58:18.969+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:58:19.241 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:58:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:58:18.969+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:58:19.241 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:58:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:58:18.969+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:58:19.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:58:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:58:18.754+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:58:19.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:58:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:58:18.754+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:58:19.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:58:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:58:18.754+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:58:19.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:58:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:58:18.944+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:58:19.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:58:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:58:18.944+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:58:19.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:58:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:58:18.944+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:58:20.241 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:58:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:58:19.931+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:58:20.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:58:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:58:19.931+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:58:20.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:58:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:58:19.931+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:58:20.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:58:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:58:19.757+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:58:20.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:58:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:58:19.758+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:58:20.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:58:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:58:19.758+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:58:20.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:58:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:58:19.941+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:58:20.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:58:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:58:19.941+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:58:20.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:58:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:58:19.941+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:58:21.241 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:58:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:58:20.958+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:58:21.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:58:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:58:20.958+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:58:21.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:58:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:58:20.958+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:58:21.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:58:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:58:20.778+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:58:21.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:58:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:58:20.778+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:58:21.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:58:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:58:20.778+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:58:21.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:58:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:58:20.956+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:58:21.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:58:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:58:20.956+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:58:21.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:58:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:58:20.956+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:58:21.919 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:58:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:58:21.752+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:58:21.919 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:58:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:58:21.752+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:58:21.920 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:58:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:58:21.752+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:58:22.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:58:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:58:21.953+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:58:22.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:58:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:58:21.953+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:58:22.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:58:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:58:21.953+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:58:22.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:58:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:58:21.918+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:58:22.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:58:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:58:21.918+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:58:22.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:58:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:58:21.918+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:58:23.144 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:58:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:58:22.782+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:58:23.144 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:58:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:58:22.782+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:58:23.144 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:58:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:58:22.782+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:58:23.145 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:58:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:58:22.923+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:58:23.145 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:58:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:58:22.924+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:58:23.145 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:58:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:58:22.924+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:58:23.145 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:58:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:58:22.874+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:58:23.145 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:58:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:58:22.874+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:58:23.145 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:58:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:58:22.874+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:58:23.906 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:58:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:58:23.760+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:58:23.906 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:58:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:58:23.760+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:58:23.906 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:58:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:58:23.760+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:58:24.241 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:58:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:58:23.904+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:58:24.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:58:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:58:23.904+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:58:24.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:58:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:58:23.904+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:58:24.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:58:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:58:23.923+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:58:24.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:58:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:58:23.923+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:58:24.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:58:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:58:23.923+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:58:25.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:58:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:58:24.744+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:58:25.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:58:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:58:24.744+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:58:25.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:58:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:58:24.744+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:58:25.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:58:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:58:24.900+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:58:25.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:58:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:58:24.900+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:58:25.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:58:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:58:24.900+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:58:25.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:58:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:58:24.931+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:58:25.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:58:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:58:24.931+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:58:25.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:58:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:58:24.931+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:58:26.040 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:58:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:58:25.782+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:58:26.040 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:58:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:58:25.782+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:58:26.040 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:58:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:58:25.782+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:58:26.040 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:58:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:58:25.942+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:58:26.040 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:58:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:58:25.942+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:58:26.041 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:58:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:58:25.942+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:58:26.041 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:58:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:58:25.944+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:58:26.041 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:58:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:58:25.944+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:58:26.041 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:58:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:58:25.944+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:58:26.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:58:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:58:26.787+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:58:26.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:58:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:58:26.787+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:58:26.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:58:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:58:26.787+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:58:26.992 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:58:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:58:26.908+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:58:26.992 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:58:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:58:26.908+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:58:26.992 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:58:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:58:26.908+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:58:27.491 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:58:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:58:26.990+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:58:27.491 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:58:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:58:26.990+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:58:27.491 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:58:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:58:26.990+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:58:28.034 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:58:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:58:27.765+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:58:28.035 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:58:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:58:27.765+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:58:28.035 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:58:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:58:27.765+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:58:28.035 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:58:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:58:27.913+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:58:28.035 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:58:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:58:27.913+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:58:28.035 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:58:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:58:27.913+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:58:28.491 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:58:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:58:28.033+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:58:28.491 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:58:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:58:28.033+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:58:28.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:58:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:58:28.033+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:58:29.081 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:58:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:58:28.804+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:58:29.081 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:58:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:58:28.804+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:58:29.081 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:58:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:58:28.804+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:58:29.081 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:58:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:58:28.942+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:58:29.081 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:58:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:58:28.942+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:58:29.081 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:58:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:58:28.942+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:58:29.491 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:58:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:58:29.079+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:58:29.491 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:58:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:58:29.079+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:58:29.491 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:58:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:58:29.079+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:58:30.098 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:58:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:58:29.982+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:58:30.098 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:58:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:58:29.982+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:58:30.099 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:58:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:58:29.982+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:58:30.099 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:58:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:58:29.817+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:58:30.099 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:58:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:58:29.817+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:58:30.099 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:58:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:58:29.817+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:58:30.491 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:58:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:58:30.097+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:58:30.491 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:58:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:58:30.097+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:58:30.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:58:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:58:30.097+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:58:31.119 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:58:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:58:30.943+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:58:31.119 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:58:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:58:30.943+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:58:31.119 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:58:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:58:30.943+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:58:31.120 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:58:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:58:30.828+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:58:31.120 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:58:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:58:30.828+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:58:31.120 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:58:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:58:30.828+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:58:31.491 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:58:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:58:31.115+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:58:31.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:58:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:58:31.115+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:58:31.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:58:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:58:31.115+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:58:31.933 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:58:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:58:31.813+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:58:31.933 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:58:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:58:31.813+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:58:31.933 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:58:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:58:31.813+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:58:32.184 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:58:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:58:31.931+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:58:32.184 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:58:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:58:31.931+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:58:32.184 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:58:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:58:31.931+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:58:32.184 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:58:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:58:32.079+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:58:32.184 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:58:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:58:32.079+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:58:32.184 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:58:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:58:32.079+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:58:33.056 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:58:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:58:32.796+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:58:33.056 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:58:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:58:32.796+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:58:33.056 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:58:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:58:32.796+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:58:33.056 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:58:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:58:32.938+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:58:33.056 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:58:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:58:32.938+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:58:33.057 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:58:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:58:32.938+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:58:33.491 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:58:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:58:33.054+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:58:33.491 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:58:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:58:33.054+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:58:33.491 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:58:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:58:33.054+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:58:33.987 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:58:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:58:33.758+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:58:33.988 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:58:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:58:33.758+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:58:33.988 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:58:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:58:33.758+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:58:34.241 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:58:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:58:34.040+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:58:34.241 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:58:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:58:34.040+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:58:34.241 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:58:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:58:34.040+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:58:34.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:58:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:58:33.986+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:58:34.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:58:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:58:33.986+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:58:34.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:58:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:58:33.986+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:58:34.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:58:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:58:34.714+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:58:34.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:58:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:58:34.714+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:58:34.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:58:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:58:34.715+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:58:35.491 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:58:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:58:35.077+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:58:35.491 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:58:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:58:35.077+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:58:35.491 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:58:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:58:35.077+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:58:35.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:58:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:58:35.033+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:58:35.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:58:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:58:35.033+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:58:35.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:58:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:58:35.033+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:58:35.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:58:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:58:35.721+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:58:35.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:58:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:58:35.721+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:58:35.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:58:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:58:35.721+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:58:36.491 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:58:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:58:36.068+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:58:36.491 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:58:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:58:36.068+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:58:36.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:58:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:58:36.068+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:58:36.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:58:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:58:36.070+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:58:36.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:58:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:58:36.070+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:58:36.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:58:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:58:36.070+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:58:37.098 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:58:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:58:36.745+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:58:37.098 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:58:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:58:36.745+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:58:37.098 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:58:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:58:36.745+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:58:37.491 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:58:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:58:37.096+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:58:37.491 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:58:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:58:37.096+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:58:37.491 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:58:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:58:37.096+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:58:37.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:58:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:58:37.118+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:58:37.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:58:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:58:37.118+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:58:37.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:58:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:58:37.118+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:58:38.063 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:58:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:58:37.759+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:58:38.063 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:58:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:58:37.759+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:58:38.063 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:58:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:58:37.759+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:58:38.491 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:58:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:58:38.061+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:58:38.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:58:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:58:38.061+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:58:38.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:58:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:58:38.061+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:58:38.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:58:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:58:38.145+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:58:38.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:58:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:58:38.145+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:58:38.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:58:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:58:38.145+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:58:39.115 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:58:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:58:38.806+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:58:39.115 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:58:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:58:38.806+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:58:39.115 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:58:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:58:38.806+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:58:39.115 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:58:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:58:39.048+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:58:39.115 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:58:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:58:39.048+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:58:39.115 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:58:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:58:39.048+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:58:39.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:58:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:58:39.113+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:58:39.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:58:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:58:39.113+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:58:39.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:58:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:58:39.113+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:58:40.142 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:58:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:58:39.825+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:58:40.142 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:58:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:58:39.825+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:58:40.142 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:58:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:58:39.825+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:58:40.142 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:58:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:58:40.013+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:58:40.142 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:58:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:58:40.013+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:58:40.142 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:58:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:58:40.013+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:58:40.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:58:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:58:40.138+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:58:40.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:58:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:58:40.138+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:58:40.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:58:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:58:40.138+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:58:41.107 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:58:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:58:40.829+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:58:41.108 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:58:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:58:40.829+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:58:41.108 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:58:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:58:40.829+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:58:41.108 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:58:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:58:41.022+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:58:41.108 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:58:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:58:41.022+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:58:41.108 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:58:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:58:41.022+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:58:41.400 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:58:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:58:41.105+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:58:41.400 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:58:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:58:41.105+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:58:41.400 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:58:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:58:41.105+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:58:41.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:58:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:58:41.855+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:58:41.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:58:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:58:41.855+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:58:41.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:58:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:58:41.855+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:58:42.491 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:58:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:58:42.015+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:58:42.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:58:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:58:42.015+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:58:42.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:58:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:58:42.015+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:58:42.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:58:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:58:42.078+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:58:42.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:58:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:58:42.078+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:58:42.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:58:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:58:42.078+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:58:43.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:58:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:58:42.883+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:58:43.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:58:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:58:42.883+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:58:43.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:58:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:58:42.883+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:58:43.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:58:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:58:43.001+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:58:43.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:58:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:58:43.001+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:58:43.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:58:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:58:43.001+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:58:43.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:58:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:58:43.106+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:58:43.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:58:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:58:43.106+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:58:43.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:58:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:58:43.106+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:58:44.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:58:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:58:43.858+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:58:44.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:58:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:58:43.858+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:58:44.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:58:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:58:43.858+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:58:44.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:58:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:58:43.987+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:58:44.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:58:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:58:43.987+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:58:44.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:58:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:58:43.987+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:58:44.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:58:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:58:44.103+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:58:44.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:58:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:58:44.103+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:58:44.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:58:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:58:44.103+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:58:45.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:58:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:58:44.901+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:58:45.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:58:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:58:44.901+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:58:45.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:58:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:58:44.901+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:58:45.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:58:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:58:44.946+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:58:45.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:58:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:58:44.946+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:58:45.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:58:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:58:44.946+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:58:45.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:58:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:58:45.144+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:58:45.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:58:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:58:45.144+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:58:45.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:58:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:58:45.144+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:58:46.241 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:58:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:58:45.938+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:58:46.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:58:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:58:45.938+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:58:46.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:58:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:58:45.938+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:58:46.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:58:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:58:45.930+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:58:46.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:58:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:58:45.930+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:58:46.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:58:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:58:45.930+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:58:46.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:58:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:58:46.121+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:58:46.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:58:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:58:46.121+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:58:46.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:58:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:58:46.121+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:58:46.978 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:58:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T07:58:46.683+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 1 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T07:58:46.979 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:58:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:58:46.912+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:58:46.979 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:58:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:58:46.912+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:58:46.979 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:58:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:58:46.912+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:58:47.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:58:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:58:47.162+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:58:47.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:58:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:58:47.162+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:58:47.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:58:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:58:47.162+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:58:47.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:58:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:58:46.978+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:58:47.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:58:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:58:46.978+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:58:47.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:58:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:58:46.978+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:58:48.207 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:58:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:58:47.872+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:58:48.207 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:58:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:58:47.872+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:58:48.207 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:58:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:58:47.872+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:58:48.207 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:58:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:58:47.931+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:58:48.207 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:58:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:58:47.931+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:58:48.207 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:58:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:58:47.931+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:58:48.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:58:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:58:48.206+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:58:48.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:58:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:58:48.206+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:58:48.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:58:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:58:48.206+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:58:49.218 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:58:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:58:48.907+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:58:49.218 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:58:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:58:48.907+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:58:49.219 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:58:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:58:48.907+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:58:49.219 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:58:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:58:48.851+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:58:49.219 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:58:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:58:48.851+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:58:49.219 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:58:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:58:48.851+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:58:49.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:58:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:58:49.216+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:58:49.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:58:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:58:49.216+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:58:49.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:58:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:58:49.216+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:58:50.216 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:58:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:58:49.889+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:58:50.216 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:58:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:58:49.889+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:58:50.216 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:58:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:58:49.889+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:58:50.216 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:58:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:58:49.878+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:58:50.217 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:58:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:58:49.878+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:58:50.217 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:58:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:58:49.878+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:58:50.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:58:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:58:50.215+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:58:50.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:58:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:58:50.215+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:58:50.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:58:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:58:50.215+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:58:51.189 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:58:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:58:50.854+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:58:51.189 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:58:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:58:50.854+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:58:51.189 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:58:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:58:50.854+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:58:51.190 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:58:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:58:50.879+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:58:51.190 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:58:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:58:50.879+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:58:51.190 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:58:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:58:50.879+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:58:51.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:58:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:58:51.188+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:58:51.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:58:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:58:51.188+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:58:51.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:58:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:58:51.188+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:58:51.887 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:58:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T07:58:51.684+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 8 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T07:58:52.080 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:58:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:58:51.831+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:58:52.080 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:58:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:58:51.831+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:58:52.080 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:58:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:58:51.831+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:58:52.184 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:58:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:58:51.886+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:58:52.184 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:58:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:58:51.886+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:58:52.184 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:58:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:58:51.886+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:58:52.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:58:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:58:52.183+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:58:52.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:58:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:58:52.183+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:58:52.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:58:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:58:52.183+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:58:53.196 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:58:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:58:52.886+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:58:53.196 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:58:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:58:52.886+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:58:53.196 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:58:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:58:52.886+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:58:53.197 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:58:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:58:52.818+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:58:53.197 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:58:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:58:52.818+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:58:53.197 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:58:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:58:52.818+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:58:53.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:58:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:58:53.195+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:58:53.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:58:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:58:53.195+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:58:53.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:58:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:58:53.195+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:58:54.226 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:58:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:58:53.873+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:58:54.226 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:58:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:58:53.873+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:58:54.226 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:58:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:58:53.873+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:58:54.226 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:58:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:58:53.816+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:58:54.226 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:58:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:58:53.816+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:58:54.226 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:58:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:58:53.816+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:58:54.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:58:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:58:54.225+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:58:54.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:58:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:58:54.225+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:58:54.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:58:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:58:54.225+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:58:55.241 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:58:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:58:54.790+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:58:55.241 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:58:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:58:54.790+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:58:55.241 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:58:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:58:54.790+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:58:55.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:58:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:58:54.868+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:58:55.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:58:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:58:54.868+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:58:55.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:58:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:58:54.868+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:58:55.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:58:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:58:55.250+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:58:55.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:58:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:58:55.250+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:58:55.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:58:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:58:55.250+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:58:56.241 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:58:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:58:55.778+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:58:56.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:58:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:58:55.778+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:58:56.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:58:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:58:55.778+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:58:56.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:58:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:58:55.865+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:58:56.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:58:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:58:55.865+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:58:56.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:58:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:58:55.865+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:58:56.686 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:58:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:58:56.244+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:58:56.687 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:58:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:58:56.244+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:58:56.687 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:58:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:58:56.244+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:58:56.991 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:58:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T07:58:56.686+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 21 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T07:58:56.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:58:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:58:56.768+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:58:56.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:58:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:58:56.769+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:58:56.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:58:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:58:56.769+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:58:56.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:58:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:58:56.821+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:58:56.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:58:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:58:56.821+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:58:56.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:58:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:58:56.821+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:58:57.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:58:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:58:57.238+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:58:57.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:58:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:58:57.238+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:58:57.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:58:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:58:57.238+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:58:58.237 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:58:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:58:57.780+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:58:58.237 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:58:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:58:57.780+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:58:58.237 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:58:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:58:57.780+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:58:58.238 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:58:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:58:57.769+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:58:58.238 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:58:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:58:57.769+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:58:58.238 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:58:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:58:57.769+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:58:58.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:58:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:58:58.236+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:58:58.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:58:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:58:58.236+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:58:58.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:58:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:58:58.236+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:58:59.235 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:58:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:58:58.829+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:58:59.236 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:58:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:58:58.829+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:58:59.236 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:58:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:58:58.829+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:58:59.236 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:58:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:58:58.817+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:58:59.236 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:58:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:58:58.817+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:58:59.236 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:58:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:58:58.817+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:58:59.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:58:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:58:59.235+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:58:59.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:58:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:58:59.235+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:58:59.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:58:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:58:59.235+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:59:00.218 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:58:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:58:59.787+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:59:00.218 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:58:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:58:59.787+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:59:00.218 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:58:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:58:59.787+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:59:00.218 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:58:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:58:59.796+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:59:00.219 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:58:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:58:59.796+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:59:00.219 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:58:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:58:59.796+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:59:00.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:59:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:59:00.217+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:59:00.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:59:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:59:00.217+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:59:00.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:59:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:59:00.217+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:59:01.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:59:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:59:00.819+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:59:01.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:59:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:59:00.819+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:59:01.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:59:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:59:00.819+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:59:01.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:59:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:59:00.767+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:59:01.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:59:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:59:00.767+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:59:01.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:59:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:59:00.767+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:59:01.630 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:59:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:59:01.254+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:59:01.630 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:59:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:59:01.254+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:59:01.630 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:59:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:59:01.254+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:59:01.991 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:59:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T07:59:01.686+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 30 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T07:59:01.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:59:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:59:01.843+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:59:01.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:59:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:59:01.843+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:59:01.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:59:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:59:01.843+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:59:01.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:59:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:59:01.816+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:59:01.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:59:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:59:01.816+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:59:01.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:59:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:59:01.816+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:59:02.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:59:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:59:02.276+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:59:02.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:59:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:59:02.276+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:59:02.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:59:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:59:02.276+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:59:03.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:59:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:59:02.877+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:59:03.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:59:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:59:02.877+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:59:03.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:59:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:59:02.877+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:59:03.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:59:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:59:02.794+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:59:03.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:59:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:59:02.794+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:59:03.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:59:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:59:02.794+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:59:03.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:59:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:59:03.289+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:59:03.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:59:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:59:03.289+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:59:03.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:59:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:59:03.289+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:59:04.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:59:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:59:03.838+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:59:04.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:59:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:59:03.838+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:59:04.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:59:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:59:03.838+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:59:04.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:59:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:59:03.795+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:59:04.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:59:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:59:03.795+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:59:04.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:59:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:59:03.795+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:59:04.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:59:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:59:04.277+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:59:04.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:59:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:59:04.277+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:59:04.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:59:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:59:04.277+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:59:05.239 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:59:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:59:04.821+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:59:05.239 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:59:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:59:04.821+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:59:05.239 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:59:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:59:04.821+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:59:05.240 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:59:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:59:04.752+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:59:05.240 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:59:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:59:04.752+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:59:05.240 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:59:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:59:04.752+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:59:05.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:59:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:59:05.239+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:59:05.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:59:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:59:05.239+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:59:05.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:59:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:59:05.239+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:59:06.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:59:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:59:05.819+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:59:06.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:59:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:59:05.819+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:59:06.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:59:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:59:05.819+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:59:06.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:59:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:59:05.753+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:59:06.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:59:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:59:05.753+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:59:06.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:59:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:59:05.753+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:59:06.688 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:59:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:59:06.276+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:59:06.688 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:59:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:59:06.276+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:59:06.688 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:59:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:59:06.276+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:59:06.991 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:59:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T07:59:06.687+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 57 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T07:59:06.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:59:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:59:06.719+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:59:06.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:59:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:59:06.719+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:59:06.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:59:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:59:06.719+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:59:06.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:59:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:59:06.775+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:59:06.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:59:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:59:06.775+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:59:06.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:59:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:59:06.775+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:59:07.732 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:59:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:59:07.287+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:59:07.732 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:59:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:59:07.287+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:59:07.732 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:59:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:59:07.287+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:59:07.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:59:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:59:07.805+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:59:07.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:59:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:59:07.805+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:59:07.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:59:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:59:07.805+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:59:07.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:59:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:59:07.731+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:59:07.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:59:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:59:07.731+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:59:07.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:59:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:59:07.731+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:59:08.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:59:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:59:08.337+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:59:08.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:59:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:59:08.337+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:59:08.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:59:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:59:08.337+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:59:09.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:59:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:59:08.801+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:59:09.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:59:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:59:08.801+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:59:09.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:59:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:59:08.801+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:59:09.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:59:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:59:08.773+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:59:09.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:59:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:59:08.773+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:59:09.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:59:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:59:08.773+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:59:09.731 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:59:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:59:09.300+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:59:09.731 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:59:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:59:09.300+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:59:09.731 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:59:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:59:09.300+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:59:09.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:59:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:59:09.730+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:59:09.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:59:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:59:09.730+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:59:09.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:59:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:59:09.730+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:59:09.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:59:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:59:09.796+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:59:09.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:59:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:59:09.796+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:59:09.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:59:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:59:09.796+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:59:10.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:59:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:59:10.258+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:59:10.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:59:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:59:10.258+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:59:10.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:59:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:59:10.258+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:59:11.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:59:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:59:10.776+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:59:11.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:59:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:59:10.776+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:59:11.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:59:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:59:10.776+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:59:11.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:59:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:59:10.773+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:59:11.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:59:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:59:10.773+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:59:11.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:59:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:59:10.773+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:59:11.630 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:59:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:59:11.264+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:59:11.630 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:59:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:59:11.264+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:59:11.631 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:59:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:59:11.264+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:59:11.991 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:59:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T07:59:11.687+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 86 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T07:59:11.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:59:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:59:11.785+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:59:11.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:59:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:59:11.785+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:59:11.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:59:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:59:11.785+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:59:11.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:59:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:59:11.822+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:59:11.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:59:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:59:11.822+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:59:11.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:59:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:59:11.822+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:59:12.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:59:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:59:12.294+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:59:12.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:59:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:59:12.294+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:59:12.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:59:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:59:12.294+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:59:13.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:59:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:59:12.826+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:59:13.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:59:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:59:12.826+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:59:13.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:59:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:59:12.826+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:59:13.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:59:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:59:12.837+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:59:13.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:59:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:59:12.837+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:59:13.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:59:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:59:12.837+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:59:13.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:59:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:59:13.312+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:59:13.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:59:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:59:13.312+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:59:13.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:59:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:59:13.312+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:59:14.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:59:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:59:13.864+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:59:14.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:59:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:59:13.864+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:59:14.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:59:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:59:13.864+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:59:14.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:59:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:59:13.861+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:59:14.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:59:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:59:13.861+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:59:14.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:59:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:59:13.861+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:59:14.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:59:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:59:14.265+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:59:14.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:59:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:59:14.265+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:59:14.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:59:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:59:14.265+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:59:15.192 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:59:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:59:14.850+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:59:15.192 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:59:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:59:14.850+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:59:15.192 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:59:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:59:14.850+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:59:15.192 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:59:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:59:14.857+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:59:15.192 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:59:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:59:14.857+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:59:15.192 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:59:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:59:14.857+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:59:15.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:59:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:59:15.240+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:59:15.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:59:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:59:15.240+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:59:15.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:59:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:59:15.240+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:59:16.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:59:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:59:15.827+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:59:16.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:59:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:59:15.827+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:59:16.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:59:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:59:15.827+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:59:16.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:59:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:59:15.903+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:59:16.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:59:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:59:15.903+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:59:16.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:59:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:59:15.903+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:59:16.689 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:59:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:59:16.253+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:59:16.689 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:59:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:59:16.253+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:59:16.689 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:59:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:59:16.253+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:59:16.991 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:59:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T07:59:16.687+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 114 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T07:59:16.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:59:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:59:16.883+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:59:16.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:59:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:59:16.883+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:59:16.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:59:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:59:16.883+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:59:16.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:59:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:59:16.866+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:59:16.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:59:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:59:16.866+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:59:16.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:59:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:59:16.866+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:59:17.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:59:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:59:17.231+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:59:17.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:59:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:59:17.231+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:59:17.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:59:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:59:17.231+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:59:18.215 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:59:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:59:17.913+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:59:18.216 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:59:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:59:17.913+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:59:18.216 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:59:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:59:17.913+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:59:18.216 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:59:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:59:17.917+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:59:18.216 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:59:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:59:17.917+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:59:18.216 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:59:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:59:17.917+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:59:18.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:59:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:59:18.214+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:59:18.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:59:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:59:18.214+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:59:18.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:59:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:59:18.214+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:59:19.229 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:59:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:59:18.955+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:59:19.230 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:59:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:59:18.955+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:59:19.230 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:59:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:59:18.955+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:59:19.230 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:59:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:59:18.952+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:59:19.230 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:59:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:59:18.952+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:59:19.230 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:59:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:59:18.952+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:59:19.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:59:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:59:19.228+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:59:19.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:59:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:59:19.228+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:59:19.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:59:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:59:19.228+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:59:20.228 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:59:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:59:19.954+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:59:20.228 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:59:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:59:19.954+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:59:20.228 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:59:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:59:19.954+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:59:20.228 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:59:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:59:19.988+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:59:20.228 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:59:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:59:19.988+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:59:20.228 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:59:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:59:19.988+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:59:20.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:59:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:59:20.226+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:59:20.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:59:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:59:20.226+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:59:20.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:59:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:59:20.226+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:59:21.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:59:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:59:20.946+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:59:21.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:59:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:59:20.946+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:59:21.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:59:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:59:20.946+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:59:21.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:59:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:59:20.958+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:59:21.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:59:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:59:20.958+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:59:21.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:59:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:59:20.958+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:59:21.631 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:59:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:59:21.272+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:59:21.631 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:59:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:59:21.272+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:59:21.631 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:59:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:59:21.272+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:59:21.904 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:59:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T07:59:21.688+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 141 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T07:59:22.241 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:59:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:59:21.934+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:59:22.241 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:59:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:59:21.934+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:59:22.241 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:59:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:59:21.934+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:59:22.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:59:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:59:21.902+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:59:22.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:59:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:59:21.902+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:59:22.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:59:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:59:21.902+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:59:22.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:59:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:59:22.293+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:59:22.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:59:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:59:22.293+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:59:22.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:59:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:59:22.293+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:59:23.241 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:59:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:59:22.896+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:59:23.241 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:59:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:59:22.896+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:59:23.241 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:59:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:59:22.896+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:59:23.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:59:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:59:22.859+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:59:23.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:59:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:59:22.859+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:59:23.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:59:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:59:22.859+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:59:23.519 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:59:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:59:23.264+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:59:23.519 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:59:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:59:23.264+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:59:23.519 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:59:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:59:23.264+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:59:24.241 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:59:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:59:23.943+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:59:24.241 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:59:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:59:23.943+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:59:24.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:59:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:59:23.943+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:59:24.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:59:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:59:23.839+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:59:24.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:59:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:59:23.839+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:59:24.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:59:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:59:23.839+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:59:24.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:59:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:59:24.270+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:59:24.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:59:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:59:24.270+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:59:24.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:59:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:59:24.270+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:59:25.241 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:59:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:59:24.895+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:59:25.241 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:59:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:59:24.895+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:59:25.241 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:59:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:59:24.895+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:59:25.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:59:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:59:24.861+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:59:25.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:59:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:59:24.861+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:59:25.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:59:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:59:24.861+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:59:25.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:59:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:59:25.294+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:59:25.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:59:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:59:25.294+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:59:25.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:59:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:59:25.294+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:59:26.241 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:59:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:59:25.864+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:59:26.241 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:59:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:59:25.864+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:59:26.241 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:59:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:59:25.864+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:59:26.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:59:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:59:25.878+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:59:26.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:59:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:59:25.878+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:59:26.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:59:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:59:25.878+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:59:26.690 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:59:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:59:26.247+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:59:26.690 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:59:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:59:26.247+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:59:26.690 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:59:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:59:26.247+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:59:26.991 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:59:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T07:59:26.688+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 168 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T07:59:26.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:59:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:59:26.846+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:59:26.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:59:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:59:26.846+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:59:26.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:59:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:59:26.846+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:59:26.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:59:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:59:26.878+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:59:26.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:59:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:59:26.878+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:59:26.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:59:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:59:26.878+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:59:27.480 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:59:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:59:27.205+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:59:27.480 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:59:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:59:27.205+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:59:27.480 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:59:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:59:27.205+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:59:28.185 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:59:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:59:27.917+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:59:28.185 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:59:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:59:27.917+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:59:28.185 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:59:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:59:27.917+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:59:28.185 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:59:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:59:27.801+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:59:28.185 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:59:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:59:27.801+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:59:28.186 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:59:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:59:27.801+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:59:28.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:59:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:59:28.184+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:59:28.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:59:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:59:28.184+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:59:28.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:59:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:59:28.184+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:59:29.203 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:59:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:59:28.954+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:59:29.203 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:59:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:59:28.954+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:59:29.203 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:59:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:59:28.954+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:59:29.203 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:59:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:59:28.767+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:59:29.203 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:59:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:59:28.767+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:59:29.203 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:59:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:59:28.767+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:59:29.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:59:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:59:29.201+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:59:29.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:59:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:59:29.201+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:59:29.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:59:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:59:29.201+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:59:30.220 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:59:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:59:30.000+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:59:30.221 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:59:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:59:30.000+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:59:30.221 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:59:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:59:30.000+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:59:30.221 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:59:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:59:29.780+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:59:30.221 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:59:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:59:29.780+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:59:30.221 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:59:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:59:29.780+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:59:30.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:59:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:59:30.219+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:59:30.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:59:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:59:30.219+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:59:30.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:59:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:59:30.219+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:59:30.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:59:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:59:30.798+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:59:30.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:59:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:59:30.798+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:59:30.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:59:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:59:30.798+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:59:31.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:59:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:59:31.048+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:59:31.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:59:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:59:31.048+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:59:31.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:59:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:59:31.048+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:59:31.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:59:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:59:31.182+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:59:31.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:59:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:59:31.182+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:59:31.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:59:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:59:31.182+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:59:31.991 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:59:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T07:59:31.689+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 196 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T07:59:31.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:59:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:59:31.779+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:59:31.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:59:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:59:31.779+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:59:31.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:59:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:59:31.779+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:59:32.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:59:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:59:32.078+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:59:32.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:59:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:59:32.078+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:59:32.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:59:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:59:32.078+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:59:32.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:59:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:59:32.194+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:59:32.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:59:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:59:32.194+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:59:32.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:59:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:59:32.194+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:59:33.060 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:59:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:59:32.766+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:59:33.060 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:59:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:59:32.766+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:59:33.060 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:59:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:59:32.766+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:59:33.492 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:59:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:59:33.059+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:59:33.492 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:59:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:59:33.059+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:59:33.492 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:59:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:59:33.059+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:59:33.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:59:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:59:33.216+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:59:33.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:59:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:59:33.216+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:59:33.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:59:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:59:33.216+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:59:34.077 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:59:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:59:33.789+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:59:34.077 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:59:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:59:33.789+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:59:34.077 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:59:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:59:33.789+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:59:34.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:59:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:59:34.076+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:59:34.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:59:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:59:34.076+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:59:34.492 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:59:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:59:34.076+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:59:34.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:59:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:59:34.224+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:59:34.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:59:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:59:34.224+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:59:34.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:59:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:59:34.224+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:59:35.084 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:59:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:59:34.783+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:59:35.084 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:59:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:59:34.783+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:59:35.084 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:59:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:59:34.783+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:59:35.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:59:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:59:35.083+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:59:35.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:59:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:59:35.083+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:59:35.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:59:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:59:35.083+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:59:35.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:59:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:59:35.269+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:59:35.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:59:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:59:35.269+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:59:35.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:59:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:59:35.269+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:59:36.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:59:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:59:36.066+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:59:36.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:59:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:59:36.066+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:59:36.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:59:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:59:36.066+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:59:36.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:59:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:59:35.825+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:59:36.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:59:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:59:35.825+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:59:36.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:59:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:59:35.825+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:59:36.691 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:59:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:59:36.275+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:59:36.691 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:59:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:59:36.275+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:59:36.691 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:59:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:59:36.275+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:59:36.991 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:59:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T07:59:36.690+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 224 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T07:59:36.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:59:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:59:36.806+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:59:36.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:59:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:59:36.806+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:59:36.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:59:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:59:36.806+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:59:37.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:59:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:59:37.295+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:59:37.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:59:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:59:37.295+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:59:37.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:59:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:59:37.295+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:59:37.492 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:59:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:59:37.109+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:59:37.492 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:59:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:59:37.109+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:59:37.492 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:59:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:59:37.109+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:59:38.122 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:59:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:59:37.842+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:59:38.122 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:59:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:59:37.842+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:59:38.123 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:59:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:59:37.842+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:59:38.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:59:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:59:38.288+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:59:38.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:59:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:59:38.288+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:59:38.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:59:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:59:38.288+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:59:38.492 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:59:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:59:38.121+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:59:38.492 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:59:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:59:38.121+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:59:38.492 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:59:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:59:38.121+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:59:39.240 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:59:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:59:39.085+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:59:39.240 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:59:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:59:39.085+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:59:39.240 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:59:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:59:39.085+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:59:39.240 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:59:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:59:38.850+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:59:39.240 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:59:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:59:38.850+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:59:39.241 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:59:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:59:38.850+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:59:39.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:59:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:59:39.239+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:59:39.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:59:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:59:39.239+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:59:39.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:59:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:59:39.239+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:59:40.199 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:59:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:59:40.086+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:59:40.199 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:59:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:59:40.086+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:59:40.199 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:59:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:59:40.086+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:59:40.199 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:59:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:59:39.876+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:59:40.199 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:59:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:59:39.876+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:59:40.199 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:59:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:59:39.876+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:59:40.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:59:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:59:40.197+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:59:40.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:59:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:59:40.197+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:59:40.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:59:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:59:40.197+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:59:41.177 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:59:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:59:41.085+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:59:41.177 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:59:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:59:41.085+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:59:41.177 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:59:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:59:41.085+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:59:41.177 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:59:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:59:40.858+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:59:41.177 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:59:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:59:40.858+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:59:41.177 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:59:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:59:40.858+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:59:41.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:59:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:59:41.175+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:59:41.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:59:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:59:41.175+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:59:41.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:59:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:59:41.175+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:59:41.992 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:59:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T07:59:41.691+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 256 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T07:59:41.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:59:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:59:41.836+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:59:41.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:59:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:59:41.836+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:59:41.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:59:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:59:41.836+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:59:42.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:59:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:59:42.089+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:59:42.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:59:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:59:42.089+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:59:42.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:59:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:59:42.089+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:59:42.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:59:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:59:42.189+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:59:42.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:59:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:59:42.189+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:59:42.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:59:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:59:42.189+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:59:43.078 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:59:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:59:42.789+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:59:43.079 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:59:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:59:42.789+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:59:43.079 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:59:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:59:42.789+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:59:43.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:59:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:59:43.164+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:59:43.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:59:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:59:43.164+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:59:43.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:59:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:59:43.164+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:59:43.492 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:59:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:59:43.078+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:59:43.492 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:59:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:59:43.078+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:59:43.492 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:59:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:59:43.078+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:59:43.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:59:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:59:43.758+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:59:43.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:59:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:59:43.758+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:59:43.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:59:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:59:43.758+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:59:44.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:59:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:59:44.042+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:59:44.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:59:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:59:44.042+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:59:44.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:59:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:59:44.042+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:59:44.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:59:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:59:44.148+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:59:44.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:59:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:59:44.148+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:59:44.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:59:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:59:44.148+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:59:45.121 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:59:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:59:44.995+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:59:45.121 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:59:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:59:44.995+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:59:45.121 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:59:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:59:44.995+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:59:45.121 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:59:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:59:44.745+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:59:45.122 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:59:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:59:44.745+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:59:45.122 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:59:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:59:44.745+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:59:45.122 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:59:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:59:44.745+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T07:59:45.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:59:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:59:45.120+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:59:45.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:59:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:59:45.120+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:59:45.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:59:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:59:45.120+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:59:45.970 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:59:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:59:45.712+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:59:45.971 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:59:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:59:45.712+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:59:45.971 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:59:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:59:45.712+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:59:45.971 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:59:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:59:45.712+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T07:59:46.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:59:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:59:45.970+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:59:46.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:59:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:59:45.970+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:59:46.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:59:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:59:45.970+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:59:46.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:59:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:59:46.122+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:59:46.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:59:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:59:46.122+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:59:46.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:59:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:59:46.122+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:59:46.986 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:59:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T07:59:46.691+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 283 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T07:59:46.987 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:59:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:59:46.685+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:59:46.987 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:59:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:59:46.685+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:59:46.987 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:59:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:59:46.685+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:59:46.987 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:59:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:59:46.685+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T07:59:47.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:59:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:59:47.163+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:59:47.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:59:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:59:47.163+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:59:47.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:59:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:59:47.163+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:59:47.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:59:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:59:46.985+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:59:47.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:59:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:59:46.985+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:59:47.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:59:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:59:46.985+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:59:47.978 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:59:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:59:47.640+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:59:47.979 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:59:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:59:47.640+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:59:47.979 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:59:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:59:47.640+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:59:47.979 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:59:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:59:47.640+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T07:59:48.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:59:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:59:47.978+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:59:48.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:59:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:59:47.978+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:59:48.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:59:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:59:47.978+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:59:48.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:59:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:59:48.183+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:59:48.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:59:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:59:48.183+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:59:48.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:59:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:59:48.183+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:59:48.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:59:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:59:48.596+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:59:48.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:59:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:59:48.596+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:59:48.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:59:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:59:48.596+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:59:48.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:59:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:59:48.596+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T07:59:49.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:59:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:59:49.010+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:59:49.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:59:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:59:49.010+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:59:49.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:59:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:59:49.010+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:59:49.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:59:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:59:49.174+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:59:49.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:59:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:59:49.174+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:59:49.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:59:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:59:49.174+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:59:49.885 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:59:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:59:49.578+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:59:49.885 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:59:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:59:49.578+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:59:49.885 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:59:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:59:49.578+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:59:49.885 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:59:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:59:49.578+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T07:59:50.202 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:59:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:59:49.969+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:59:50.203 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:59:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:59:49.969+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:59:50.203 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:59:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:59:49.969+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:59:50.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:59:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:59:50.202+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:59:50.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:59:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:59:50.202+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:59:50.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:59:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:59:50.202+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:59:50.959 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:59:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:59:50.544+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:59:50.959 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:59:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:59:50.544+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:59:50.959 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:59:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:59:50.544+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:59:50.959 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:59:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:59:50.545+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T07:59:51.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:59:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:59:50.958+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:59:51.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:59:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:59:50.958+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:59:51.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:59:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:59:50.958+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:59:51.516 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:59:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:59:51.242+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:59:51.516 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:59:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:59:51.242+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:59:51.517 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:59:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:59:51.242+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:59:51.818 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:59:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T07:59:51.693+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 311 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T07:59:51.819 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:59:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:59:51.515+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:59:51.819 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:59:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:59:51.515+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:59:51.819 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:59:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:59:51.515+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:59:51.819 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:59:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:59:51.515+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T07:59:52.234 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:59:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:59:51.940+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:59:52.234 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:59:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:59:51.940+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:59:52.234 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:59:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:59:51.940+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:59:52.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:59:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:59:52.233+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:59:52.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:59:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:59:52.233+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:59:52.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:59:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:59:52.233+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:59:52.910 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:59:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:59:52.521+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:59:52.910 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:59:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:59:52.521+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:59:52.910 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:59:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:59:52.521+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:59:52.910 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:59:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:59:52.521+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T07:59:53.198 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:59:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:59:52.910+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:59:53.198 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:59:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:59:52.910+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:59:53.198 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:59:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:59:52.910+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:59:53.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:59:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:59:53.197+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:59:53.562 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:59:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:59:53.197+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:59:53.562 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:59:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:59:53.197+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:59:53.929 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:59:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:59:53.493+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:59:53.929 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:59:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:59:53.493+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:59:53.930 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:59:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:59:53.493+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:59:53.930 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:59:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:59:53.493+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T07:59:54.204 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:59:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:59:53.928+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:59:54.205 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:59:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:59:53.928+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:59:54.205 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:59:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:59:53.928+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:59:54.456 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:59:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:59:54.203+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:59:54.456 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:59:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:59:54.203+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:59:54.456 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:59:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:59:54.203+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:59:54.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:59:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:59:54.456+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:59:54.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:59:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:59:54.456+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:59:54.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:59:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:59:54.456+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:59:54.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:59:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:59:54.456+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T07:59:55.190 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:59:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:59:54.889+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:59:55.190 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:59:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:59:54.889+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:59:55.190 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:59:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:59:54.889+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:59:55.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:59:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:59:55.190+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:59:55.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:59:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:59:55.190+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:59:55.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:59:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:59:55.190+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:59:55.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:59:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:59:55.414+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:59:55.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:59:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:59:55.414+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:59:55.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:59:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:59:55.414+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:59:55.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:59:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:59:55.414+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T07:59:56.172 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:59:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:59:55.858+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:59:56.172 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:59:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:59:55.858+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:59:56.172 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:59:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:59:55.858+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:59:56.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:59:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:59:56.171+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:59:56.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:59:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:59:56.171+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:59:56.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:59:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:59:56.171+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:59:56.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:59:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:59:56.373+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:59:56.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:59:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:59:56.373+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:59:56.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:59:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:59:56.373+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:59:56.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:59:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:59:56.373+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T07:59:56.991 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 07:59:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T07:59:56.693+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 339 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T07:59:56.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:59:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:59:56.842+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:59:56.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:59:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:59:56.842+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:59:56.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:59:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:59:56.842+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:59:57.379 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:59:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:59:57.128+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:59:57.379 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:59:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:59:57.128+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:59:57.379 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:59:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:59:57.128+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:59:57.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:59:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:59:57.378+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:59:57.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:59:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:59:57.378+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:59:57.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:59:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:59:57.378+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:59:57.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:59:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:59:57.378+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T07:59:58.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:59:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:59:57.880+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:59:58.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:59:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:59:57.880+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:59:58.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:59:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:59:57.880+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:59:58.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:59:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:59:58.125+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:59:58.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:59:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:59:58.125+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:59:58.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:59:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:59:58.125+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:59:58.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:59:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:59:58.346+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:59:58.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:59:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:59:58.346+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:59:58.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:59:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:59:58.346+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:59:58.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:59:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:59:58.346+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T07:59:59.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:59:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:59:58.894+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:59:59.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:59:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:59:58.894+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:59:59.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:59:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:59:58.894+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T07:59:59.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:59:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:59:59.126+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:59:59.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:59:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:59:59.126+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:59:59.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 07:59:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T07:59:59.126+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T07:59:59.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:59:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:59:59.392+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:59:59.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:59:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:59:59.392+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:59:59.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:59:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:59:59.392+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T07:59:59.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 07:59:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T07:59:59.392+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:00:00.162 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:59:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:59:59.937+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:00:00.162 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:59:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:59:59.937+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:00:00.162 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 07:59:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T07:59:59.937+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:00:00.433 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:00:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:00:00.161+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:00:00.433 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:00:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:00:00.161+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:00:00.434 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:00:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:00:00.161+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:00:00.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:00:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:00:00.433+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:00:00.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:00:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:00:00.433+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:00:00.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:00:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:00:00.433+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:00:00.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:00:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:00:00.433+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:00:01.173 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:00:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:00:00.899+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:00:01.173 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:00:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:00:00.899+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:00:01.174 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:00:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:00:00.899+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:00:01.428 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:00:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:00:01.173+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:00:01.429 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:00:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:00:01.173+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:00:01.429 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:00:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:00:01.173+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:00:01.695 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:00:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:00:01.427+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:00:01.695 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:00:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:00:01.427+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:00:01.695 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:00:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:00:01.427+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:00:01.695 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:00:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:00:01.427+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:00:01.991 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:00:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:00:01.694+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 366 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:00:01.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:00:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:00:01.880+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:00:01.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:00:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:00:01.880+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:00:01.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:00:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:00:01.880+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:00:02.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:00:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:00:02.211+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:00:02.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:00:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:00:02.211+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:00:02.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:00:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:00:02.211+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:00:02.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:00:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:00:02.412+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:00:02.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:00:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:00:02.412+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:00:02.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:00:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:00:02.412+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:00:02.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:00:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:00:02.412+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:00:03.235 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:00:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:00:02.887+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:00:03.236 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:00:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:00:02.887+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:00:03.236 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:00:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:00:02.887+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:00:03.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:00:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:00:03.235+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:00:03.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:00:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:00:03.235+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:00:03.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:00:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:00:03.235+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:00:03.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:00:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:00:03.389+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:00:03.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:00:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:00:03.389+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:00:03.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:00:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:00:03.389+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:00:03.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:00:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:00:03.389+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:00:04.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:00:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:00:03.842+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:00:04.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:00:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:00:03.842+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:00:04.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:00:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:00:03.842+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:00:04.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:00:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:00:04.277+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:00:04.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:00:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:00:04.277+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:00:04.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:00:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:00:04.277+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:00:04.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:00:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:00:04.348+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:00:04.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:00:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:00:04.348+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:00:04.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:00:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:00:04.348+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:00:04.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:00:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:00:04.348+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:00:05.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:00:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:00:04.882+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:00:05.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:00:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:00:04.882+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:00:05.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:00:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:00:04.882+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:00:05.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:00:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:00:05.305+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:00:05.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:00:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:00:05.305+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:00:05.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:00:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:00:05.305+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:00:05.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:00:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:00:05.363+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:00:05.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:00:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:00:05.363+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:00:05.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:00:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:00:05.363+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:00:05.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:00:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:00:05.363+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:00:06.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:00:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:00:05.913+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:00:06.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:00:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:00:05.913+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:00:06.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:00:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:00:05.913+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:00:06.696 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:00:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:00:06.412+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:00:06.696 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:00:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:00:06.412+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:00:06.696 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:00:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:00:06.412+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:00:06.696 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:00:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:00:06.412+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:00:06.696 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:00:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:00:06.275+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:00:06.696 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:00:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:00:06.275+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:00:06.696 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:00:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:00:06.275+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:00:06.991 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:00:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:00:06.695+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 395 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:00:06.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:00:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:00:06.865+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:00:06.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:00:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:00:06.865+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:00:06.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:00:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:00:06.865+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:00:07.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:00:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:00:07.249+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:00:07.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:00:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:00:07.249+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:00:07.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:00:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:00:07.249+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:00:07.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:00:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:00:07.367+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:00:07.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:00:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:00:07.367+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:00:07.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:00:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:00:07.367+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:00:07.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:00:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:00:07.367+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:00:08.232 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:00:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:00:07.822+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:00:08.232 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:00:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:00:07.822+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:00:08.232 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:00:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:00:07.822+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:00:08.491 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:00:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:00:08.328+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:00:08.491 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:00:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:00:08.328+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:00:08.491 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:00:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:00:08.328+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:00:08.491 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:00:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:00:08.328+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:00:08.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:00:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:00:08.232+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:00:08.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:00:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:00:08.232+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:00:08.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:00:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:00:08.232+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:00:09.216 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:00:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:00:08.814+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:00:09.216 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:00:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:00:08.814+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:00:09.216 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:00:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:00:08.814+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:00:09.491 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:00:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:00:09.304+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:00:09.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:00:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:00:09.304+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:00:09.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:00:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:00:09.304+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:00:09.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:00:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:00:09.304+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:00:09.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:00:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:00:09.215+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:00:09.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:00:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:00:09.215+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:00:09.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:00:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:00:09.215+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:00:10.239 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:00:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:00:09.841+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:00:10.239 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:00:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:00:09.841+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:00:10.240 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:00:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:00:09.841+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:00:10.491 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:00:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:00:10.279+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:00:10.491 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:00:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:00:10.279+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:00:10.491 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:00:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:00:10.279+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:00:10.491 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:00:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:00:10.279+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:00:10.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:00:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:00:10.237+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:00:10.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:00:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:00:10.237+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:00:10.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:00:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:00:10.237+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:00:11.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:00:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:00:10.878+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:00:11.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:00:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:00:10.878+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:00:11.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:00:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:00:10.878+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:00:11.634 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:00:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:00:11.262+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:00:11.634 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:00:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:00:11.262+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:00:11.634 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:00:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:00:11.262+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:00:11.635 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:00:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:00:11.254+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:00:11.635 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:00:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:00:11.254+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:00:11.635 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:00:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:00:11.254+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:00:11.635 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:00:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:00:11.254+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:00:11.921 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:00:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:00:11.695+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 423 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:00:12.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:00:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:00:11.919+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:00:12.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:00:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:00:11.919+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:00:12.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:00:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:00:11.919+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:00:12.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:00:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:00:12.259+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:00:12.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:00:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:00:12.259+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:00:12.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:00:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:00:12.259+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:00:12.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:00:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:00:12.259+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:00:12.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:00:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:00:12.274+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:00:12.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:00:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:00:12.274+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:00:12.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:00:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:00:12.274+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:00:13.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:00:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:00:12.963+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:00:13.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:00:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:00:12.963+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:00:13.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:00:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:00:12.963+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:00:13.522 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:00:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:00:13.324+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:00:13.523 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:00:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:00:13.324+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:00:13.523 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:00:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:00:13.324+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:00:13.523 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:00:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:00:13.260+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:00:13.523 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:00:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:00:13.260+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:00:13.523 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:00:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:00:13.260+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:00:13.523 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:00:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:00:13.260+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:00:14.279 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:00:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:00:13.992+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:00:14.279 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:00:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:00:13.992+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:00:14.279 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:00:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:00:13.992+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:00:14.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:00:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:00:14.277+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:00:14.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:00:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:00:14.277+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:00:14.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:00:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:00:14.277+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:00:14.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:00:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:00:14.277+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:00:14.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:00:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:00:14.332+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:00:14.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:00:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:00:14.332+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:00:14.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:00:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:00:14.332+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:00:15.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:00:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:00:14.945+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:00:15.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:00:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:00:14.945+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:00:15.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:00:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:00:14.945+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:00:15.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:00:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:00:15.307+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:00:15.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:00:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:00:15.307+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:00:15.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:00:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:00:15.307+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:00:15.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:00:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:00:15.280+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:00:15.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:00:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:00:15.280+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:00:15.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:00:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:00:15.280+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:00:15.745 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:00:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:00:15.280+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:00:16.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:00:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:00:15.925+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:00:16.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:00:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:00:15.925+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:00:16.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:00:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:00:15.925+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:00:16.697 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:00:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:00:16.318+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:00:16.697 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:00:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:00:16.318+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:00:16.697 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:00:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:00:16.318+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:00:16.697 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:00:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:00:16.329+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:00:16.697 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:00:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:00:16.329+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:00:16.697 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:00:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:00:16.329+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:00:16.698 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:00:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:00:16.330+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:00:16.963 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:00:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:00:16.695+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 451 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:00:17.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:00:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:00:16.962+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:00:17.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:00:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:00:16.962+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:00:17.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:00:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:00:16.962+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:00:17.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:00:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:00:17.325+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:00:17.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:00:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:00:17.325+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:00:17.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:00:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:00:17.325+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:00:17.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:00:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:00:17.284+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:00:17.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:00:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:00:17.284+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:00:17.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:00:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:00:17.284+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:00:17.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:00:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:00:17.284+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:00:18.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:00:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:00:17.980+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:00:18.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:00:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:00:17.980+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:00:18.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:00:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:00:17.980+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:00:18.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:00:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:00:18.324+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:00:18.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:00:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:00:18.324+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:00:18.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:00:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:00:18.324+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:00:18.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:00:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:00:18.324+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:00:18.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:00:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:00:18.288+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:00:18.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:00:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:00:18.288+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:00:18.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:00:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:00:18.288+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:00:19.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:00:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:00:18.938+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:00:19.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:00:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:00:18.938+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:00:19.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:00:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:00:18.938+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:00:19.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:00:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:00:19.315+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:00:19.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:00:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:00:19.315+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:00:19.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:00:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:00:19.315+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:00:19.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:00:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:00:19.316+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:00:19.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:00:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:00:19.325+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:00:19.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:00:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:00:19.325+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:00:19.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:00:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:00:19.325+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:00:20.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:00:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:00:19.966+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:00:20.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:00:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:00:19.966+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:00:20.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:00:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:00:19.966+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:00:20.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:00:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:00:20.356+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:00:20.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:00:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:00:20.356+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:00:20.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:00:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:00:20.356+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:00:20.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:00:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:00:20.280+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:00:20.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:00:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:00:20.280+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:00:20.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:00:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:00:20.280+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:00:20.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:00:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:00:20.280+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:00:21.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:00:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:00:20.961+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:00:21.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:00:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:00:20.961+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:00:21.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:00:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:00:20.961+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:00:21.635 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:00:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:00:21.322+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:00:21.636 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:00:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:00:21.322+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:00:21.636 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:00:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:00:21.322+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:00:21.636 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:00:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:00:21.322+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:00:21.636 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:00:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:00:21.352+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:00:21.636 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:00:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:00:21.352+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:00:21.636 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:00:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:00:21.352+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:00:21.988 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:00:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:00:21.696+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 480 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:00:22.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:00:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:00:21.987+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:00:22.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:00:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:00:21.987+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:00:22.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:00:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:00:21.987+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:00:22.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:00:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:00:22.314+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:00:22.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:00:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:00:22.314+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:00:22.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:00:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:00:22.314+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:00:22.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:00:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:00:22.314+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:00:22.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:00:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:00:22.387+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:00:22.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:00:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:00:22.387+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:00:22.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:00:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:00:22.387+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:00:23.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:00:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:00:22.949+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:00:23.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:00:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:00:22.949+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:00:23.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:00:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:00:22.949+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:00:23.524 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:00:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:00:23.266+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:00:23.524 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:00:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:00:23.266+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:00:23.524 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:00:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:00:23.266+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:00:23.524 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:00:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:00:23.266+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:00:23.524 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:00:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:00:23.392+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:00:23.524 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:00:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:00:23.392+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:00:23.524 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:00:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:00:23.392+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:00:24.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:00:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:00:23.989+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:00:24.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:00:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:00:23.989+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:00:24.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:00:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:00:23.989+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:00:24.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:00:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:00:24.394+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:00:24.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:00:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:00:24.394+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:00:24.745 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:00:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:00:24.394+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:00:24.745 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:00:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:00:24.269+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:00:24.745 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:00:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:00:24.269+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:00:24.745 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:00:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:00:24.269+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:00:24.745 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:00:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:00:24.269+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:00:25.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:00:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:00:24.979+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:00:25.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:00:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:00:24.979+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:00:25.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:00:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:00:24.979+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:00:25.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:00:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:00:25.219+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:00:25.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:00:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:00:25.219+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:00:25.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:00:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:00:25.219+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:00:25.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:00:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:00:25.220+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:00:25.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:00:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:00:25.417+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:00:25.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:00:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:00:25.417+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:00:25.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:00:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:00:25.417+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:00:26.463 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:00:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:00:26.026+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:00:26.463 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:00:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:00:26.026+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:00:26.463 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:00:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:00:26.026+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:00:26.464 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:00:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:00:26.184+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:00:26.464 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:00:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:00:26.184+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:00:26.464 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:00:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:00:26.184+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:00:26.464 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:00:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:00:26.184+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:00:26.719 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:00:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:00:26.696+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 510 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:00:26.719 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:00:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:00:26.462+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:00:26.719 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:00:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:00:26.462+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:00:26.719 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:00:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:00:26.462+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:00:27.435 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:00:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:00:27.229+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:00:27.436 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:00:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:00:27.229+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:00:27.436 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:00:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:00:27.229+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:00:27.436 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:00:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:00:27.229+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:00:27.436 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:00:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:00:27.011+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:00:27.436 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:00:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:00:27.011+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:00:27.436 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:00:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:00:27.011+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:00:27.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:00:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:00:27.434+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:00:27.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:00:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:00:27.434+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:00:27.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:00:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:00:27.434+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:00:28.280 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:00:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:00:28.006+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:00:28.280 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:00:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:00:28.006+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:00:28.280 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:00:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:00:28.006+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:00:28.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:00:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:00:28.279+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:00:28.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:00:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:00:28.279+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:00:28.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:00:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:00:28.279+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:00:28.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:00:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:00:28.279+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:00:28.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:00:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:00:28.398+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:00:28.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:00:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:00:28.398+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:00:28.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:00:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:00:28.398+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:00:29.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:00:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:00:28.974+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:00:29.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:00:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:00:28.974+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:00:29.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:00:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:00:28.974+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:00:29.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:00:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:00:29.437+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:00:29.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:00:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:00:29.437+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:00:29.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:00:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:00:29.437+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:00:29.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:00:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:00:29.299+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:00:29.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:00:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:00:29.299+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:00:29.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:00:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:00:29.299+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:00:29.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:00:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:00:29.299+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:00:30.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:00:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:00:30.002+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:00:30.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:00:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:00:30.002+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:00:30.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:00:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:00:30.002+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:00:30.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:00:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:00:30.345+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:00:30.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:00:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:00:30.345+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:00:30.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:00:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:00:30.345+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:00:30.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:00:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:00:30.345+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:00:30.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:00:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:00:30.479+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:00:30.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:00:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:00:30.479+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:00:30.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:00:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:00:30.479+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:00:31.310 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:00:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:00:30.994+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:00:31.310 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:00:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:00:30.994+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:00:31.310 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:00:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:00:30.994+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:00:31.635 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:00:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:00:31.445+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:00:31.636 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:00:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:00:31.445+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:00:31.636 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:00:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:00:31.445+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:00:31.636 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:00:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:00:31.309+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:00:31.636 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:00:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:00:31.309+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:00:31.636 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:00:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:00:31.309+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:00:31.636 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:00:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:00:31.309+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:00:31.991 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:00:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:00:31.697+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 538 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:00:32.266 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:00:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:00:32.034+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:00:32.266 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:00:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:00:32.034+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:00:32.266 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:00:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:00:32.034+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:00:32.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:00:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:00:32.463+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:00:32.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:00:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:00:32.463+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:00:32.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:00:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:00:32.463+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:00:32.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:00:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:00:32.320+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:00:32.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:00:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:00:32.320+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:00:32.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:00:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:00:32.320+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:00:32.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:00:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:00:32.320+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:00:33.315 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:00:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:00:33.033+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:00:33.315 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:00:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:00:33.033+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:00:33.315 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:00:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:00:33.033+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:00:33.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:00:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:00:33.415+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:00:33.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:00:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:00:33.415+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:00:33.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:00:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:00:33.415+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:00:33.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:00:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:00:33.314+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:00:33.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:00:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:00:33.314+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:00:33.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:00:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:00:33.314+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:00:33.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:00:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:00:33.314+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:00:34.378 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:00:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:00:34.068+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:00:34.379 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:00:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:00:34.068+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:00:34.379 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:00:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:00:34.068+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:00:34.379 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:00:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:00:34.302+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:00:34.379 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:00:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:00:34.302+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:00:34.379 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:00:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:00:34.302+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:00:34.379 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:00:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:00:34.302+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:00:34.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:00:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:00:34.378+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:00:34.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:00:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:00:34.378+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:00:34.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:00:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:00:34.378+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:00:35.334 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:00:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:00:35.033+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:00:35.334 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:00:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:00:35.033+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:00:35.334 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:00:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:00:35.033+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:00:35.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:00:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:00:35.422+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:00:35.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:00:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:00:35.422+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:00:35.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:00:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:00:35.422+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:00:35.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:00:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:00:35.333+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:00:35.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:00:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:00:35.333+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:00:35.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:00:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:00:35.333+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:00:35.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:00:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:00:35.333+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:00:36.345 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:00:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:00:36.035+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:00:36.345 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:00:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:00:36.035+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:00:36.346 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:00:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:00:36.035+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:00:36.699 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:00:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:00:36.345+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:00:36.700 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:00:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:00:36.345+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:00:36.700 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:00:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:00:36.345+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:00:36.700 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:00:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:00:36.345+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:00:36.700 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:00:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:00:36.392+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:00:36.700 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:00:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:00:36.392+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:00:36.700 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:00:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:00:36.392+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:00:36.991 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:00:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:00:36.698+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 571 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:00:37.315 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:00:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:00:37.028+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:00:37.316 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:00:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:00:37.028+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:00:37.316 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:00:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:00:37.028+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:00:37.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:00:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:00:37.372+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:00:37.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:00:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:00:37.372+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:00:37.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:00:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:00:37.372+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:00:37.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:00:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:00:37.315+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:00:37.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:00:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:00:37.315+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:00:37.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:00:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:00:37.315+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:00:37.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:00:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:00:37.315+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:00:38.388 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:00:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:00:38.050+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:00:38.389 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:00:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:00:38.050+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:00:38.389 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:00:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:00:38.050+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:00:38.389 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:00:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:00:38.300+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:00:38.389 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:00:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:00:38.300+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:00:38.389 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:00:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:00:38.300+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:00:38.389 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:00:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:00:38.300+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:00:38.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:00:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:00:38.387+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:00:38.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:00:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:00:38.387+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:00:38.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:00:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:00:38.387+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:00:39.302 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:00:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:00:39.035+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:00:39.302 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:00:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:00:39.035+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:00:39.302 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:00:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:00:39.035+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:00:39.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:00:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:00:39.357+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:00:39.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:00:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:00:39.357+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:00:39.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:00:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:00:39.357+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:00:39.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:00:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:00:39.300+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:00:39.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:00:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:00:39.300+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:00:39.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:00:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:00:39.300+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:00:39.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:00:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:00:39.300+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:00:40.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:00:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:00:39.991+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:00:40.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:00:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:00:39.991+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:00:40.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:00:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:00:39.991+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:00:40.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:00:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:00:40.324+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:00:40.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:00:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:00:40.324+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:00:40.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:00:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:00:40.324+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:00:40.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:00:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:00:40.348+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:00:40.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:00:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:00:40.348+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:00:40.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:00:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:00:40.348+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:00:40.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:00:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:00:40.348+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:00:41.316 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:00:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:00:41.008+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:00:41.316 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:00:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:00:41.008+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:00:41.316 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:00:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:00:41.008+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:00:41.636 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:00:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:00:41.315+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:00:41.636 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:00:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:00:41.315+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:00:41.636 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:00:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:00:41.315+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:00:41.637 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:00:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:00:41.398+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:00:41.637 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:00:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:00:41.398+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:00:41.637 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:00:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:00:41.398+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:00:41.637 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:00:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:00:41.398+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:00:41.991 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:00:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:00:41.700+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 603 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:00:42.267 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:00:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:00:41.996+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:00:42.267 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:00:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:00:41.996+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:00:42.268 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:00:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:00:41.996+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:00:42.692 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:00:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:00:42.267+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:00:42.692 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:00:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:00:42.267+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:00:42.692 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:00:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:00:42.267+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:00:42.692 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:00:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:00:42.385+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:00:42.693 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:00:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:00:42.385+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:00:42.693 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:00:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:00:42.385+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:00:42.693 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:00:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:00:42.385+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:00:43.260 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:00:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:00:43.231+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:00:43.260 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:00:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:00:43.231+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:00:43.260 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:00:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:00:43.231+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:00:43.260 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:00:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:00:43.006+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:00:43.260 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:00:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:00:43.006+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:00:43.260 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:00:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:00:43.006+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:00:43.525 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:00:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:00:43.431+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:00:43.525 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:00:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:00:43.431+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:00:43.525 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:00:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:00:43.431+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:00:43.525 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:00:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:00:43.431+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:00:44.280 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:00:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:00:44.186+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:00:44.280 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:00:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:00:44.186+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:00:44.280 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:00:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:00:44.186+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:00:44.281 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:00:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:00:44.010+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:00:44.281 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:00:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:00:44.010+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:00:44.281 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:00:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:00:44.010+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:00:44.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:00:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:00:44.409+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:00:44.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:00:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:00:44.409+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:00:44.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:00:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:00:44.409+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:00:44.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:00:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:00:44.409+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:00:45.391 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:00:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:00:45.151+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:00:45.392 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:00:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:00:45.151+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:00:45.392 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:00:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:00:45.151+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:00:45.392 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:00:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:00:45.019+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:00:45.392 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:00:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:00:45.019+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:00:45.392 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:00:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:00:45.019+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:00:45.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:00:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:00:45.391+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:00:45.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:00:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:00:45.391+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:00:45.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:00:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:00:45.391+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:00:45.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:00:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:00:45.391+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:00:46.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:00:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:00:45.978+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:00:46.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:00:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:00:45.978+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:00:46.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:00:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:00:45.978+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:00:46.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:00:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:00:46.104+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:00:46.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:00:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:00:46.104+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:00:46.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:00:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:00:46.104+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:00:46.701 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:00:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:00:46.429+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:00:46.701 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:00:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:00:46.429+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:00:46.701 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:00:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:00:46.429+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:00:46.701 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:00:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:00:46.429+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:00:46.959 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:00:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:00:46.700+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 630 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:00:47.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:00:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:00:46.959+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:00:47.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:00:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:00:46.959+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:00:47.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:00:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:00:46.959+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:00:47.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:00:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:00:47.069+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:00:47.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:00:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:00:47.069+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:00:47.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:00:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:00:47.069+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:00:47.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:00:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:00:47.436+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:00:47.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:00:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:00:47.436+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:00:47.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:00:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:00:47.436+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:00:47.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:00:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:00:47.436+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:00:48.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:00:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:00:47.960+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:00:48.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:00:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:00:47.960+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:00:48.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:00:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:00:47.960+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:00:48.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:00:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:00:48.112+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:00:48.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:00:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:00:48.112+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:00:48.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:00:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:00:48.112+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:00:48.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:00:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:00:48.402+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:00:48.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:00:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:00:48.402+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:00:48.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:00:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:00:48.402+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:00:48.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:00:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:00:48.402+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:00:49.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:00:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:00:48.945+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:00:49.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:00:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:00:48.945+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:00:49.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:00:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:00:48.945+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:00:49.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:00:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:00:49.157+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:00:49.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:00:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:00:49.157+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:00:49.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:00:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:00:49.157+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:00:49.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:00:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:00:49.387+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:00:49.778 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:00:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:00:49.387+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:00:49.778 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:00:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:00:49.387+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:00:49.779 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:00:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:00:49.387+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:00:50.176 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:00:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:00:49.896+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:00:50.176 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:00:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:00:49.896+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:00:50.177 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:00:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:00:49.896+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:00:50.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:00:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:00:50.176+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:00:50.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:00:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:00:50.176+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:00:50.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:00:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:00:50.176+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:00:50.491 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:00:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:00:50.348+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:00:50.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:00:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:00:50.348+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:00:50.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:00:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:00:50.348+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:00:50.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:00:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:00:50.348+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:00:51.209 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:00:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:00:50.917+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:00:51.209 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:00:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:00:50.917+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:00:51.209 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:00:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:00:50.917+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:00:51.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:00:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:00:51.208+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:00:51.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:00:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:00:51.208+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:00:51.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:00:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:00:51.208+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:00:51.491 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:00:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:00:51.329+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:00:51.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:00:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:00:51.329+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:00:51.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:00:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:00:51.329+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:00:51.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:00:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:00:51.329+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:00:51.917 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:00:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:00:51.701+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 658 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:00:52.173 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:00:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:00:51.917+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:00:52.173 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:00:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:00:51.917+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:00:52.173 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:00:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:00:51.917+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:00:52.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:00:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:00:52.173+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:00:52.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:00:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:00:52.173+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:00:52.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:00:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:00:52.173+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:00:52.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:00:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:00:52.310+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:00:52.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:00:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:00:52.311+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:00:52.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:00:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:00:52.311+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:00:52.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:00:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:00:52.311+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:00:53.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:00:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:00:53.141+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:00:53.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:00:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:00:53.141+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:00:53.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:00:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:00:53.141+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:00:53.245 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:00:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:00:52.923+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:00:53.245 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:00:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:00:52.923+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:00:53.245 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:00:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:00:52.923+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:00:53.525 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:00:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:00:53.270+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:00:53.526 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:00:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:00:53.270+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:00:53.526 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:00:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:00:53.270+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:00:53.526 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:00:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:00:53.270+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:00:54.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:00:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:00:54.124+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:00:54.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:00:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:00:54.124+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:00:54.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:00:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:00:54.124+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:00:54.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:00:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:00:53.946+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:00:54.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:00:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:00:53.946+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:00:54.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:00:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:00:53.946+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:00:54.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:00:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:00:54.291+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:00:54.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:00:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:00:54.291+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:00:54.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:00:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:00:54.291+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:00:54.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:00:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:00:54.291+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:00:55.336 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:00:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:00:55.114+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:00:55.336 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:00:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:00:55.114+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:00:55.336 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:00:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:00:55.114+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:00:55.337 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:00:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:00:54.996+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:00:55.337 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:00:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:00:54.996+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:00:55.337 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:00:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:00:54.996+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:00:55.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:00:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:00:55.335+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:00:55.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:00:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:00:55.335+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:00:55.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:00:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:00:55.335+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:00:55.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:00:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:00:55.336+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:00:56.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:00:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:00:56.106+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:00:56.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:00:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:00:56.106+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:00:56.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:00:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:00:56.106+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:00:56.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:00:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:00:55.988+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:00:56.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:00:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:00:55.988+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:00:56.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:00:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:00:55.988+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:00:56.703 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:00:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:00:56.343+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:00:56.704 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:00:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:00:56.343+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:00:56.704 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:00:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:00:56.343+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:00:56.704 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:00:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:00:56.343+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:00:56.960 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:00:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:00:56.702+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 686 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:00:57.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:00:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:00:56.960+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:00:57.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:00:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:00:56.960+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:00:57.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:00:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:00:56.960+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:00:57.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:00:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:00:57.150+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:00:57.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:00:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:00:57.150+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:00:57.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:00:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:00:57.150+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:00:57.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:00:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:00:57.391+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:00:57.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:00:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:00:57.391+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:00:57.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:00:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:00:57.391+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:00:57.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:00:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:00:57.391+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:00:58.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:00:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:00:57.987+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:00:58.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:00:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:00:57.987+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:00:58.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:00:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:00:57.987+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:00:58.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:00:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:00:58.181+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:00:58.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:00:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:00:58.181+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:00:58.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:00:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:00:58.181+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:00:58.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:00:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:00:58.420+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:00:58.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:00:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:00:58.420+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:00:58.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:00:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:00:58.420+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:00:58.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:00:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:00:58.421+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:00:59.402 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:00:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:00:59.199+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:00:59.402 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:00:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:00:59.199+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:00:59.402 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:00:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:00:59.199+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:00:59.402 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:00:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:00:58.998+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:00:59.403 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:00:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:00:58.998+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:00:59.403 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:00:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:00:58.998+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:00:59.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:00:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:00:59.402+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:00:59.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:00:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:00:59.402+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:00:59.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:00:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:00:59.402+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:00:59.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:00:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:00:59.402+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:01:00.218 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:00:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:00:59.964+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:01:00.218 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:00:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:00:59.964+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:01:00.218 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:00:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:00:59.964+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:01:00.491 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:01:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:01:00.356+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:01:00.491 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:01:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:01:00.356+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:01:00.491 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:01:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:01:00.356+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:01:00.491 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:01:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:01:00.356+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:01:00.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:01:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:01:00.217+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:01:00.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:01:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:01:00.217+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:01:00.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:01:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:01:00.217+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:01:01.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:01:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:01:00.961+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:01:01.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:01:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:01:00.961+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:01:01.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:01:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:01:00.961+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:01:01.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:01:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:01:01.199+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:01:01.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:01:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:01:01.199+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:01:01.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:01:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:01:01.199+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:01:01.637 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:01:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:01:01.378+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:01:01.638 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:01:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:01:01.378+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:01:01.638 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:01:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:01:01.378+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:01:01.638 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:01:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:01:01.378+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:01:01.978 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:01:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:01:01.702+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 713 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:01:02.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:01:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:01:01.977+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:01:02.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:01:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:01:01.977+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:01:02.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:01:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:01:01.977+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:01:02.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:01:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:01:02.215+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:01:02.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:01:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:01:02.215+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:01:02.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:01:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:01:02.215+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:01:02.711 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:01:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:01:02.418+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:01:02.712 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:01:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:01:02.418+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:01:02.712 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:01:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:01:02.418+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:01:02.712 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:01:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:01:02.418+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:01:03.372 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:01:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:01:03.021+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:01:03.372 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:01:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:01:03.021+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:01:03.372 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:01:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:01:03.021+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:01:03.372 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:01:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:01:03.198+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:01:03.372 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:01:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:01:03.198+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:01:03.372 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:01:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:01:03.198+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:01:03.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:01:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:01:03.370+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:01:03.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:01:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:01:03.370+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:01:03.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:01:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:01:03.370+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:01:03.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:01:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:01:03.370+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:01:04.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:01:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:01:03.977+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:01:04.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:01:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:01:03.977+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:01:04.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:01:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:01:03.977+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:01:04.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:01:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:01:04.162+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:01:04.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:01:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:01:04.162+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:01:04.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:01:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:01:04.162+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:01:04.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:01:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:01:04.369+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:01:04.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:01:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:01:04.369+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:01:04.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:01:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:01:04.369+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:01:04.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:01:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:01:04.369+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:01:05.330 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:01:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:01:05.021+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:01:05.330 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:01:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:01:05.021+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:01:05.331 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:01:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:01:05.021+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:01:05.331 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:01:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:01:05.112+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:01:05.331 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:01:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:01:05.112+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:01:05.331 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:01:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:01:05.112+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:01:05.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:01:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:01:05.329+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:01:05.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:01:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:01:05.329+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:01:05.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:01:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:01:05.329+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:01:05.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:01:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:01:05.329+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:01:06.148 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:01:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:01:05.982+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:01:06.149 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:01:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:01:05.982+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:01:06.149 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:01:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:01:05.982+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:01:06.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:01:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:01:06.147+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:01:06.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:01:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:01:06.147+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:01:06.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:01:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:01:06.147+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:01:06.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:01:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:01:06.348+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:01:06.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:01:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:01:06.348+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:01:06.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:01:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:01:06.348+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:01:06.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:01:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:01:06.348+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:01:06.991 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:01:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:01:06.702+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 740 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:01:07.326 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:01:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:01:07.027+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:01:07.326 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:01:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:01:07.027+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:01:07.326 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:01:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:01:07.027+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:01:07.327 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:01:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:01:07.121+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:01:07.327 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:01:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:01:07.121+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:01:07.327 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:01:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:01:07.121+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:01:07.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:01:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:01:07.324+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:01:07.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:01:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:01:07.324+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:01:07.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:01:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:01:07.324+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:01:07.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:01:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:01:07.325+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:01:08.304 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:01:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:01:08.043+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:01:08.305 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:01:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:01:08.043+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:01:08.305 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:01:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:01:08.043+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:01:08.305 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:01:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:01:08.150+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:01:08.305 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:01:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:01:08.150+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:01:08.305 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:01:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:01:08.150+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:01:08.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:01:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:01:08.303+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:01:08.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:01:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:01:08.303+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:01:08.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:01:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:01:08.303+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:01:08.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:01:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:01:08.303+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:01:09.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:01:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:01:09.105+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:01:09.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:01:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:01:09.105+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:01:09.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:01:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:01:09.105+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:01:09.492 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:01:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:01:09.081+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:01:09.492 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:01:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:01:09.081+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:01:09.492 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:01:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:01:09.081+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:01:09.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:01:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:01:09.295+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:01:09.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:01:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:01:09.295+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:01:09.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:01:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:01:09.295+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:01:09.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:01:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:01:09.295+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:01:10.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:01:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:01:10.090+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:01:10.251 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:01:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:01:10.090+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:01:10.251 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:01:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:01:10.090+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:01:10.251 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:01:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:01:10.043+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:01:10.251 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:01:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:01:10.043+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:01:10.251 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:01:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:01:10.043+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:01:10.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:01:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:01:10.294+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:01:10.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:01:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:01:10.294+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:01:10.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:01:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:01:10.294+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:01:10.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:01:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:01:10.294+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:01:11.279 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:01:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:01:11.093+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:01:11.279 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:01:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:01:11.093+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:01:11.279 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:01:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:01:11.093+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:01:11.280 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:01:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:01:11.000+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:01:11.280 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:01:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:01:11.000+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:01:11.280 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:01:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:01:11.000+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:01:11.638 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:01:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:01:11.278+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:01:11.638 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:01:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:01:11.278+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:01:11.638 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:01:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:01:11.278+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:01:11.639 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:01:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:01:11.278+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:01:11.927 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:01:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:01:11.703+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 768 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:01:12.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:01:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:01:12.130+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:01:12.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:01:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:01:12.130+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:01:12.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:01:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:01:12.130+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:01:12.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:01:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:01:11.981+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:01:12.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:01:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:01:11.981+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:01:12.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:01:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:01:11.981+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:01:12.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:01:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:01:12.287+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:01:12.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:01:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:01:12.287+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:01:12.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:01:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:01:12.287+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:01:12.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:01:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:01:12.287+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:01:13.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:01:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:01:13.134+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:01:13.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:01:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:01:13.134+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:01:13.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:01:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:01:13.134+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:01:13.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:01:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:01:12.958+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:01:13.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:01:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:01:12.958+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:01:13.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:01:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:01:12.958+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:01:13.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:01:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:01:13.336+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:01:13.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:01:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:01:13.336+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:01:13.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:01:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:01:13.336+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:01:13.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:01:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:01:13.336+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:01:14.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:01:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:01:14.095+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:01:14.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:01:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:01:14.095+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:01:14.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:01:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:01:14.095+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:01:14.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:01:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:01:13.910+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:01:14.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:01:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:01:13.910+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:01:14.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:01:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:01:13.910+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:01:14.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:01:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:01:14.352+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:01:14.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:01:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:01:14.352+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:01:14.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:01:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:01:14.352+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:01:14.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:01:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:01:14.352+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:01:15.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:01:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:01:14.951+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:01:15.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:01:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:01:14.951+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:01:15.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:01:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:01:14.951+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:01:15.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:01:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:01:15.132+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:01:15.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:01:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:01:15.132+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:01:15.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:01:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:01:15.132+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:01:15.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:01:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:01:15.353+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:01:15.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:01:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:01:15.353+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:01:15.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:01:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:01:15.353+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:01:15.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:01:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:01:15.353+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:01:16.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:01:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:01:15.951+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:01:16.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:01:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:01:15.951+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:01:16.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:01:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:01:15.951+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:01:16.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:01:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:01:16.113+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:01:16.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:01:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:01:16.113+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:01:16.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:01:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:01:16.113+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:01:16.705 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:01:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:01:16.366+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:01:16.705 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:01:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:01:16.366+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:01:16.705 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:01:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:01:16.366+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:01:16.705 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:01:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:01:16.366+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:01:16.991 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:01:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:01:16.703+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 795 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:01:16.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:01:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:01:16.915+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:01:16.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:01:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:01:16.915+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:01:16.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:01:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:01:16.915+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:01:17.491 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:01:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:01:17.323+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:01:17.491 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:01:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:01:17.323+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:01:17.491 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:01:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:01:17.323+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:01:17.491 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:01:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:01:17.323+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:01:17.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:01:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:01:17.125+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:01:17.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:01:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:01:17.126+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:01:17.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:01:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:01:17.126+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:01:18.137 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:01:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:01:17.879+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:01:18.138 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:01:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:01:17.879+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:01:18.138 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:01:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:01:17.879+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:01:18.491 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:01:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:01:18.354+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:01:18.491 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:01:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:01:18.354+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:01:18.491 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:01:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:01:18.354+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:01:18.491 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:01:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:01:18.354+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:01:18.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:01:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:01:18.136+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:01:18.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:01:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:01:18.136+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:01:18.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:01:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:01:18.136+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:01:19.145 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:01:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:01:18.862+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:01:19.145 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:01:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:01:18.862+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:01:19.145 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:01:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:01:18.862+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:01:19.491 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:01:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:01:19.388+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:01:19.491 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:01:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:01:19.388+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:01:19.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:01:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:01:19.388+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:01:19.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:01:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:01:19.388+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:01:19.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:01:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:01:19.144+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:01:19.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:01:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:01:19.144+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:01:19.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:01:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:01:19.144+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:01:20.099 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:01:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:01:19.813+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:01:20.100 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:01:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:01:19.813+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:01:20.100 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:01:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:01:19.813+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:01:20.380 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:01:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:01:20.098+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:01:20.380 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:01:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:01:20.098+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:01:20.380 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:01:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:01:20.098+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:01:20.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:01:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:01:20.378+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:01:20.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:01:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:01:20.378+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:01:20.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:01:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:01:20.378+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:01:20.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:01:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:01:20.378+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:01:21.085 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:01:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:01:20.790+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:01:21.085 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:01:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:01:20.790+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:01:21.086 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:01:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:01:20.790+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:01:21.400 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:01:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:01:21.084+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:01:21.401 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:01:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:01:21.084+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:01:21.401 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:01:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:01:21.084+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:01:21.705 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:01:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:01:21.400+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:01:21.706 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:01:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:01:21.400+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:01:21.706 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:01:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:01:21.400+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:01:21.706 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:01:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:01:21.400+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:01:21.991 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:01:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:01:21.704+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 827 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:01:21.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:01:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:01:21.802+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:01:21.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:01:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:01:21.802+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:01:21.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:01:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:01:21.802+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:01:22.451 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:01:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:01:22.108+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:01:22.451 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:01:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:01:22.108+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:01:22.451 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:01:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:01:22.108+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:01:22.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:01:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:01:22.450+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:01:22.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:01:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:01:22.450+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:01:22.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:01:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:01:22.450+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:01:22.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:01:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:01:22.450+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:01:23.063 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:01:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:01:22.776+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:01:23.063 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:01:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:01:22.776+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:01:23.063 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:01:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:01:22.776+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:01:23.438 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:01:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:01:23.062+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:01:23.439 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:01:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:01:23.062+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:01:23.439 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:01:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:01:23.062+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:01:23.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:01:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:01:23.437+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:01:23.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:01:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:01:23.437+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:01:23.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:01:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:01:23.437+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:01:23.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:01:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:01:23.437+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:01:24.054 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:01:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:01:23.775+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:01:24.054 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:01:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:01:23.775+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:01:24.054 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:01:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:01:23.775+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:01:24.488 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:01:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:01:24.053+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:01:24.489 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:01:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:01:24.053+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:01:24.489 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:01:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:01:24.053+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:01:24.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:01:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:01:24.487+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:01:24.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:01:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:01:24.487+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:01:24.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:01:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:01:24.487+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:01:24.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:01:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:01:24.487+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:01:25.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:01:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:01:24.790+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:01:25.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:01:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:01:24.790+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:01:25.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:01:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:01:24.790+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:01:25.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:01:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:01:25.011+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:01:25.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:01:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:01:25.011+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:01:25.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:01:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:01:25.011+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:01:25.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:01:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:01:25.750+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:01:25.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:01:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:01:25.750+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:01:25.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:01:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:01:25.750+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:01:25.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:01:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:01:25.512+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:01:25.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:01:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:01:25.512+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:01:25.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:01:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:01:25.512+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:01:25.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:01:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:01:25.512+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:01:26.487 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:01:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:01:26.044+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:01:26.487 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:01:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:01:26.044+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:01:26.487 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:01:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:01:26.044+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:01:26.741 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:01:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:01:26.705+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 865 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:01:26.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:01:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:01:26.726+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:01:26.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:01:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:01:26.726+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:01:26.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:01:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:01:26.726+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:01:26.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:01:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:01:26.485+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:01:26.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:01:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:01:26.485+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:01:26.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:01:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:01:26.485+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:01:26.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:01:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:01:26.486+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:01:27.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:01:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:01:27.022+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:01:27.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:01:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:01:27.022+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:01:27.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:01:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:01:27.022+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:01:27.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:01:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:01:27.446+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:01:27.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:01:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:01:27.446+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:01:27.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:01:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:01:27.446+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:01:27.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:01:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:01:27.446+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:01:28.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:01:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:01:27.984+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:01:28.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:01:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:01:27.984+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:01:28.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:01:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:01:27.984+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:01:28.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:01:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:01:27.771+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:01:28.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:01:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:01:27.771+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:01:28.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:01:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:01:27.771+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:01:28.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:01:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:01:28.466+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:01:28.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:01:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:01:28.466+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:01:28.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:01:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:01:28.466+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:01:28.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:01:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:01:28.466+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:01:29.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:01:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:01:28.797+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:01:29.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:01:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:01:28.797+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:01:29.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:01:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:01:28.797+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:01:29.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:01:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:01:29.000+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:01:29.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:01:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:01:29.000+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:01:29.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:01:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:01:29.000+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:01:29.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:01:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:01:29.456+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:01:29.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:01:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:01:29.456+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:01:29.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:01:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:01:29.456+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:01:29.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:01:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:01:29.456+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:01:30.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:01:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:01:29.829+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:01:30.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:01:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:01:29.829+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:01:30.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:01:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:01:29.829+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:01:30.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:01:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:01:30.028+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:01:30.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:01:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:01:30.028+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:01:30.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:01:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:01:30.028+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:01:30.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:01:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:01:30.475+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:01:30.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:01:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:01:30.475+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:01:30.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:01:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:01:30.475+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:01:30.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:01:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:01:30.475+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:01:31.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:01:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:01:31.021+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:01:31.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:01:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:01:31.021+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:01:31.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:01:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:01:31.021+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:01:31.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:01:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:01:30.784+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:01:31.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:01:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:01:30.784+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:01:31.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:01:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:01:30.784+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:01:31.767 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:01:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:01:31.706+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 894 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:01:31.767 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:01:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:01:31.510+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:01:31.767 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:01:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:01:31.510+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:01:31.767 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:01:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:01:31.510+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:01:31.768 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:01:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:01:31.510+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:01:32.027 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:01:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:01:31.766+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:01:32.027 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:01:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:01:31.766+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:01:32.027 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:01:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:01:31.766+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:01:32.489 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:01:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:01:32.026+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:01:32.489 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:01:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:01:32.026+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:01:32.489 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:01:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:01:32.026+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:01:32.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:01:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:01:32.730+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:01:32.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:01:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:01:32.730+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:01:32.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:01:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:01:32.730+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:01:32.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:01:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:01:32.488+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:01:32.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:01:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:01:32.488+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:01:32.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:01:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:01:32.488+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:01:32.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:01:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:01:32.488+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:01:33.432 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:01:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:01:33.009+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:01:33.432 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:01:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:01:33.009+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:01:33.432 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:01:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:01:33.009+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:01:33.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:01:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:01:33.461+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:01:33.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:01:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:01:33.461+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:01:33.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:01:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:01:33.461+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:01:33.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:01:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:01:33.461+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:01:34.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:01:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:01:33.770+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:01:34.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:01:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:01:33.770+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:01:34.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:01:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:01:33.770+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:01:34.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:01:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:01:33.976+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:01:34.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:01:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:01:33.976+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:01:34.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:01:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:01:33.976+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:01:34.778 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:01:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:01:34.492+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:01:34.778 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:01:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:01:34.492+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:01:34.779 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:01:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:01:34.492+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:01:34.779 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:01:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:01:34.492+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:01:35.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:01:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:01:34.778+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:01:35.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:01:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:01:34.778+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:01:35.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:01:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:01:34.778+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:01:35.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:01:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:01:35.020+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:01:35.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:01:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:01:35.020+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:01:35.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:01:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:01:35.020+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:01:35.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:01:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:01:35.465+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:01:35.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:01:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:01:35.465+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:01:35.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:01:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:01:35.465+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:01:35.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:01:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:01:35.465+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:01:36.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:01:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:01:35.795+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:01:36.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:01:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:01:35.795+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:01:36.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:01:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:01:35.795+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:01:36.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:01:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:01:36.009+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:01:36.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:01:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:01:36.009+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:01:36.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:01:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:01:36.009+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:01:36.503 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:01:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:01:36.432+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:01:36.503 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:01:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:01:36.432+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:01:36.504 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:01:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:01:36.432+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:01:36.504 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:01:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:01:36.432+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:01:36.757 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:01:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:01:36.707+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 921 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:01:37.048 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:01:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:01:36.757+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:01:37.048 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:01:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:01:36.757+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:01:37.048 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:01:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:01:36.757+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:01:37.467 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:01:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:01:37.047+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:01:37.467 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:01:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:01:37.047+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:01:37.467 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:01:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:01:37.048+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:01:37.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:01:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:01:37.467+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:01:37.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:01:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:01:37.467+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:01:37.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:01:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:01:37.467+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:01:37.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:01:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:01:37.467+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:01:38.091 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:01:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:01:37.759+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:01:38.091 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:01:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:01:37.759+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:01:38.092 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:01:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:01:37.759+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:01:38.478 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:01:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:01:38.090+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:01:38.478 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:01:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:01:38.090+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:01:38.478 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:01:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:01:38.090+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:01:38.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:01:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:01:38.478+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:01:38.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:01:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:01:38.478+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:01:38.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:01:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:01:38.478+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:01:38.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:01:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:01:38.478+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:01:39.084 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:01:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:01:38.793+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:01:39.084 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:01:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:01:38.793+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:01:39.084 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:01:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:01:38.793+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:01:39.430 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:01:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:01:39.083+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:01:39.430 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:01:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:01:39.084+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:01:39.430 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:01:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:01:39.084+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:01:39.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:01:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:01:39.429+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:01:39.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:01:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:01:39.429+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:01:39.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:01:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:01:39.429+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:01:39.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:01:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:01:39.430+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:01:40.059 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:01:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:01:39.789+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:01:40.059 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:01:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:01:39.789+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:01:40.059 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:01:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:01:39.789+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:01:40.402 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:01:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:01:40.058+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:01:40.402 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:01:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:01:40.058+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:01:40.402 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:01:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:01:40.058+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:01:40.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:01:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:01:40.402+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:01:40.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:01:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:01:40.402+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:01:40.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:01:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:01:40.402+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:01:40.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:01:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:01:40.402+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:01:41.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:01:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:01:41.032+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:01:41.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:01:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:01:41.032+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:01:41.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:01:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:01:41.032+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:01:41.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:01:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:01:40.804+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:01:41.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:01:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:01:40.804+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:01:41.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:01:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:01:40.804+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:01:41.709 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:01:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:01:41.400+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:01:41.709 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:01:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:01:41.400+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:01:41.709 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:01:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:01:41.400+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:01:41.709 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:01:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:01:41.400+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:01:41.991 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:01:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:01:41.708+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 949 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:01:41.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:01:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:01:41.848+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:01:41.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:01:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:01:41.848+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:01:41.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:01:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:01:41.848+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:01:42.435 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:01:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:01:42.073+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:01:42.435 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:01:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:01:42.073+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:01:42.436 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:01:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:01:42.073+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:01:42.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:01:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:01:42.435+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:01:42.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:01:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:01:42.435+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:01:42.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:01:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:01:42.435+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:01:42.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:01:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:01:42.435+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:01:43.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:01:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:01:42.845+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:01:43.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:01:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:01:42.845+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:01:43.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:01:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:01:42.845+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:01:43.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:01:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:01:43.073+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:01:43.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:01:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:01:43.073+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:01:43.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:01:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:01:43.073+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:01:43.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:01:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:01:43.459+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:01:43.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:01:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:01:43.459+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:01:43.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:01:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:01:43.459+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:01:43.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:01:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:01:43.459+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:01:44.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:01:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:01:43.849+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:01:44.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:01:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:01:43.849+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:01:44.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:01:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:01:43.849+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:01:44.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:01:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:01:44.061+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:01:44.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:01:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:01:44.061+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:01:44.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:01:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:01:44.061+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:01:44.859 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:01:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:01:44.505+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:01:44.859 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:01:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:01:44.505+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:01:44.860 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:01:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:01:44.505+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:01:44.860 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:01:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:01:44.505+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:01:45.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:01:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:01:44.859+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:01:45.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:01:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:01:44.859+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:01:45.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:01:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:01:44.859+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:01:45.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:01:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:01:45.101+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:01:45.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:01:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:01:45.101+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:01:45.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:01:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:01:45.101+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:01:45.897 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:01:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:01:45.544+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:01:45.897 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:01:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:01:45.544+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:01:45.898 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:01:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:01:45.544+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:01:45.898 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:01:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:01:45.544+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:01:46.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:01:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:01:45.897+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:01:46.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:01:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:01:45.897+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:01:46.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:01:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:01:45.897+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:01:46.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:01:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:01:46.070+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:01:46.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:01:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:01:46.070+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:01:46.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:01:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:01:46.070+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:01:46.833 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:01:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:01:46.709+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 978 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:01:46.833 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:01:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:01:46.536+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:01:46.834 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:01:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:01:46.536+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:01:46.834 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:01:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:01:46.536+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:01:46.834 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:01:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:01:46.536+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:01:47.101 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:01:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:01:46.947+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:01:47.102 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:01:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:01:46.947+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:01:47.102 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:01:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:01:46.947+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:01:47.448 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:01:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:01:47.101+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:01:47.448 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:01:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:01:47.101+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:01:47.448 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:01:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:01:47.101+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:01:47.930 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:01:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:01:47.580+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:01:47.930 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:01:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:01:47.580+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:01:47.930 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:01:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:01:47.580+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:01:47.930 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:01:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:01:47.580+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:01:48.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:01:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:01:47.929+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:01:48.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:01:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:01:47.929+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:01:48.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:01:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:01:47.929+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:01:48.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:01:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:01:48.150+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:01:48.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:01:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:01:48.150+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:01:48.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:01:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:01:48.150+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:01:48.884 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:01:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:01:48.594+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:01:48.884 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:01:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:01:48.594+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:01:48.884 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:01:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:01:48.594+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:01:48.884 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:01:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:01:48.594+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:01:49.196 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:01:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:01:48.882+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:01:49.196 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:01:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:01:48.882+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:01:49.196 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:01:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:01:48.882+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:01:49.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:01:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:01:49.195+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:01:49.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:01:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:01:49.195+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:01:49.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:01:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:01:49.195+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:01:49.851 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:01:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:01:49.599+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:01:49.851 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:01:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:01:49.599+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:01:49.851 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:01:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:01:49.599+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:01:49.851 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:01:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:01:49.599+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:01:50.219 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:01:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:01:49.851+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:01:50.219 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:01:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:01:49.851+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:01:50.219 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:01:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:01:49.851+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:01:50.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:01:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:01:50.218+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:01:50.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:01:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:01:50.218+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:01:50.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:01:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:01:50.218+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:01:50.872 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:01:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:01:50.594+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:01:50.872 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:01:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:01:50.594+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:01:50.872 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:01:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:01:50.594+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:01:50.872 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:01:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:01:50.594+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:01:51.176 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:01:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:01:50.871+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:01:51.199 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:01:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:01:50.871+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:01:51.199 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:01:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:01:50.871+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:01:51.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:01:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:01:51.176+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:01:51.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:01:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:01:51.176+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:01:51.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:01:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:01:51.176+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:01:51.834 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:01:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:01:51.710+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 1012 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:01:51.834 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:01:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:01:51.573+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:01:51.834 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:01:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:01:51.573+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:01:51.834 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:01:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:01:51.573+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:01:51.834 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:01:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:01:51.573+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:01:52.170 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:01:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:01:51.857+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:01:52.170 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:01:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:01:51.857+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:01:52.170 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:01:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:01:51.857+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:01:52.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:01:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:01:52.170+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:01:52.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:01:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:01:52.170+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:01:52.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:01:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:01:52.170+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:01:52.884 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:01:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:01:52.611+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:01:52.884 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:01:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:01:52.611+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:01:52.884 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:01:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:01:52.611+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:01:52.884 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:01:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:01:52.611+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:01:53.142 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:01:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:01:52.883+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:01:53.142 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:01:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:01:52.883+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:01:53.142 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:01:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:01:52.883+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:01:53.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:01:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:01:53.142+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:01:53.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:01:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:01:53.142+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:01:53.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:01:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:01:53.142+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:01:53.842 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:01:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:01:53.653+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:01:53.842 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:01:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:01:53.653+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:01:53.842 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:01:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:01:53.653+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:01:53.842 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:01:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:01:53.653+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:01:54.184 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:01:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:01:53.841+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:01:54.184 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:01:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:01:53.841+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:01:54.184 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:01:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:01:53.841+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:01:54.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:01:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:01:54.183+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:01:54.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:01:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:01:54.183+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:01:54.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:01:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:01:54.183+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:01:54.935 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:01:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:01:54.866+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:01:54.936 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:01:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:01:54.866+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:01:54.936 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:01:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:01:54.866+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:01:54.936 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:01:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:01:54.679+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:01:54.936 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:01:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:01:54.679+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:01:54.936 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:01:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:01:54.679+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:01:54.936 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:01:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:01:54.679+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:01:55.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:01:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:01:55.164+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:01:55.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:01:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:01:55.164+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:01:55.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:01:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:01:55.164+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:01:55.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:01:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:01:55.833+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:01:55.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:01:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:01:55.833+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:01:55.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:01:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:01:55.833+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:01:55.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:01:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:01:55.671+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:01:55.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:01:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:01:55.671+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:01:55.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:01:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:01:55.671+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:01:55.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:01:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:01:55.671+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:01:56.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:01:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:01:56.162+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:01:56.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:01:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:01:56.162+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:01:56.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:01:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:01:56.162+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:01:56.991 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:01:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:01:56.710+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 1044 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:01:56.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:01:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:01:56.836+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:01:56.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:01:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:01:56.836+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:01:56.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:01:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:01:56.836+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:01:56.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:01:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:01:56.631+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:01:56.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:01:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:01:56.631+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:01:56.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:01:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:01:56.631+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:01:56.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:01:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:01:56.631+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:01:57.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:01:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:01:57.175+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:01:57.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:01:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:01:57.175+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:01:57.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:01:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:01:57.175+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:01:57.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:01:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:01:57.824+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:01:57.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:01:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:01:57.824+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:01:57.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:01:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:01:57.824+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:01:57.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:01:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:01:57.660+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:01:57.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:01:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:01:57.660+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:01:57.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:01:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:01:57.660+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:01:57.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:01:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:01:57.660+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:01:58.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:01:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:01:58.160+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:01:58.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:01:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:01:58.160+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:01:58.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:01:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:01:58.160+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:01:58.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:01:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:01:58.826+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:01:58.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:01:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:01:58.826+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:01:58.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:01:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:01:58.826+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:01:58.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:01:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:01:58.637+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:01:58.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:01:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:01:58.637+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:01:58.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:01:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:01:58.637+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:01:58.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:01:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:01:58.637+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:01:59.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:01:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:01:59.136+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:01:59.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:01:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:01:59.136+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:01:59.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:01:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:01:59.136+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:01:59.884 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:01:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:01:59.595+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:01:59.884 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:01:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:01:59.595+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:01:59.884 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:01:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:01:59.595+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:01:59.884 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:01:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:01:59.595+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:01:59.885 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:01:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:01:59.799+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:01:59.885 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:01:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:01:59.799+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:01:59.885 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:01:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:01:59.799+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:02:00.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:02:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:02:00.120+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:02:00.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:02:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:02:00.120+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:02:00.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:02:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:02:00.120+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:02:00.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:02:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:02:00.828+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:02:00.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:02:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:02:00.828+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:02:00.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:02:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:02:00.828+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:02:00.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:02:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:02:00.608+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:02:00.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:02:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:02:00.608+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:02:00.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:02:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:02:00.608+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:02:00.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:02:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:02:00.608+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:02:01.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:02:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:02:01.163+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:02:01.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:02:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:02:01.163+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:02:01.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:02:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:02:01.163+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:02:01.500 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T08:02:01.498+0000 7f0958495700 0 monclient(hunting): authenticate timed out after 300 2024-02-20T08:02:01.501 INFO:teuthology.orchestra.run.smithi133.stderr:[errno 110] RADOS timed out (error connecting to the cluster) 2024-02-20T08:02:01.836 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:02:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:02:01.710+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 1077 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:02:01.836 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:02:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:02:01.643+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:02:01.836 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:02:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:02:01.643+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:02:01.836 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:02:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:02:01.643+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:02:01.836 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:02:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:02:01.643+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:02:02.197 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:02:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:02:01.835+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:02:02.197 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:02:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:02:01.835+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:02:02.197 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:02:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:02:01.835+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:02:02.364 DEBUG:teuthology.orchestra.run.smithi133:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4537327c-cfc4-11ee-95bb-87774f69a715 -e sha1=91ff50e903dcc99ee3ecf9523050878eb452da5c -- bash -c 'ceph versions | jq -e '"'"'.mgr | length == 1'"'"'' 2024-02-20T08:02:02.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:02:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:02:02.195+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:02:02.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:02:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:02:02.195+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:02:02.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:02:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:02:02.195+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:02:02.631 INFO:teuthology.orchestra.run.smithi133.stderr:Inferring config /var/lib/ceph/4537327c-cfc4-11ee-95bb-87774f69a715/mon.smithi133/config 2024-02-20T08:02:02.926 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:02:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:02:02.792+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:02:02.926 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:02:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:02:02.792+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:02:02.927 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:02:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:02:02.792+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:02:02.927 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:02:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:02:02.669+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:02:02.927 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:02:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:02:02.669+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:02:02.927 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:02:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:02:02.669+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:02:02.927 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:02:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:02:02.669+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:02:03.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:02:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:02:03.167+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:02:03.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:02:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:02:03.167+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:02:03.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:02:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:02:03.167+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:02:03.837 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:02:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:02:03.673+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:02:03.837 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:02:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:02:03.673+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:02:03.837 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:02:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:02:03.673+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:02:03.837 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:02:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:02:03.673+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:02:04.088 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:02:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:02:03.835+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:02:04.088 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:02:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:02:03.835+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:02:04.088 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:02:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:02:03.835+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:02:04.456 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:02:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:02:04.169+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:02:04.456 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:02:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:02:04.169+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:02:04.456 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:02:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:02:04.169+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:02:04.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:02:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:02:04.665+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:02:04.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:02:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:02:04.665+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:02:04.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:02:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:02:04.665+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:02:04.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:02:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:02:04.665+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:02:05.215 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:02:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:02:04.874+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:02:05.215 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:02:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:02:04.874+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:02:05.215 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:02:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:02:04.874+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:02:05.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:02:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:02:05.213+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:02:05.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:02:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:02:05.213+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:02:05.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:02:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:02:05.213+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:02:05.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:02:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:02:05.914+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:02:05.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:02:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:02:05.914+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:02:05.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:02:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:02:05.914+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:02:05.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:02:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:02:05.672+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:02:05.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:02:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:02:05.672+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:02:05.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:02:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:02:05.672+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:02:05.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:02:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:02:05.672+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:02:06.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:02:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:02:06.198+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:02:06.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:02:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:02:06.198+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:02:06.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:02:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:02:06.198+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:02:06.768 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:02:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:02:06.711+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 1108 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:02:06.768 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:02:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:02:06.644+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:02:06.769 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:02:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:02:06.644+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:02:06.769 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:02:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:02:06.644+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:02:06.769 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:02:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:02:06.644+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:02:07.227 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:02:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:02:06.909+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:02:07.227 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:02:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:02:06.909+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:02:07.227 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:02:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:02:06.909+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:02:07.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:02:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:02:07.225+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:02:07.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:02:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:02:07.225+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:02:07.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:02:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:02:07.225+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:02:07.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:02:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:02:07.676+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:02:07.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:02:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:02:07.676+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:02:07.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:02:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:02:07.676+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:02:07.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:02:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:02:07.676+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:02:07.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:02:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:02:07.860+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:02:07.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:02:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:02:07.860+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:02:07.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:02:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:02:07.860+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:02:08.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:02:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:02:08.193+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:02:08.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:02:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:02:08.193+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:02:08.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:02:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:02:08.193+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:02:08.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:02:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:02:08.687+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:02:08.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:02:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:02:08.687+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:02:08.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:02:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:02:08.687+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:02:08.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:02:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:02:08.687+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:02:08.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:02:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:02:08.867+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:02:08.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:02:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:02:08.867+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:02:08.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:02:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:02:08.867+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:02:09.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:02:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:02:09.217+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:02:09.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:02:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:02:09.217+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:02:09.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:02:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:02:09.217+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:02:09.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:02:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:02:09.840+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:02:09.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:02:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:02:09.840+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:02:09.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:02:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:02:09.840+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:02:09.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:02:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:02:09.674+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:02:09.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:02:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:02:09.674+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:02:09.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:02:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:02:09.674+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:02:09.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:02:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:02:09.674+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:02:10.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:02:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:02:10.191+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:02:10.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:02:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:02:10.191+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:02:10.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:02:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:02:10.191+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:02:10.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:02:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:02:10.888+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:02:10.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:02:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:02:10.888+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:02:10.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:02:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:02:10.888+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:02:10.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:02:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:02:10.717+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:02:10.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:02:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:02:10.717+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:02:10.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:02:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:02:10.717+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:02:10.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:02:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:02:10.717+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:02:11.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:02:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:02:11.166+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:02:11.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:02:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:02:11.166+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:02:11.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:02:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:02:11.166+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:02:11.991 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:02:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:02:11.712+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 1142 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:02:11.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:02:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:02:11.864+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:02:11.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:02:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:02:11.864+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:02:11.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:02:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:02:11.864+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:02:11.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:02:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:02:11.710+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:02:11.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:02:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:02:11.710+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:02:11.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:02:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:02:11.710+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:02:11.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:02:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:02:11.710+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:02:12.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:02:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:02:12.156+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:02:12.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:02:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:02:12.156+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:02:12.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:02:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:02:12.156+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:02:12.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:02:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:02:12.879+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:02:12.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:02:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:02:12.879+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:02:12.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:02:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:02:12.879+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:02:12.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:02:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:02:12.703+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:02:12.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:02:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:02:12.703+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:02:12.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:02:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:02:12.703+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:02:12.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:02:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:02:12.703+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:02:13.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:02:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:02:13.187+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:02:13.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:02:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:02:13.187+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:02:13.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:02:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:02:13.187+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:02:13.911 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:02:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:02:13.717+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:02:13.911 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:02:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:02:13.717+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:02:13.911 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:02:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:02:13.717+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:02:13.911 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:02:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:02:13.717+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:02:14.167 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:02:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:02:13.910+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:02:14.167 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:02:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:02:13.910+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:02:14.167 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:02:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:02:13.910+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:02:14.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:02:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:02:14.166+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:02:14.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:02:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:02:14.166+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:02:14.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:02:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:02:14.166+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:02:14.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:02:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:02:14.726+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:02:14.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:02:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:02:14.726+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:02:14.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:02:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:02:14.726+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:02:14.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:02:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:02:14.727+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:02:14.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:02:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:02:14.956+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:02:14.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:02:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:02:14.956+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:02:14.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:02:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:02:14.956+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:02:15.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:02:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:02:15.153+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:02:15.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:02:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:02:15.153+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:02:15.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:02:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:02:15.153+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:02:15.985 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:02:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:02:15.726+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:02:15.985 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:02:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:02:15.726+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:02:15.985 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:02:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:02:15.726+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:02:15.985 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:02:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:02:15.726+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:02:16.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:02:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:02:16.135+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:02:16.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:02:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:02:16.135+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:02:16.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:02:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:02:16.135+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:02:16.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:02:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:02:15.983+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:02:16.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:02:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:02:15.983+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:02:16.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:02:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:02:15.983+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:02:16.991 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:02:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:02:16.713+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 1174 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:02:16.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:02:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:02:16.708+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:02:16.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:02:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:02:16.708+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:02:16.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:02:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:02:16.708+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:02:16.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:02:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:02:16.708+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:02:17.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:02:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:02:17.184+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:02:17.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:02:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:02:17.184+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:02:17.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:02:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:02:17.184+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:02:17.492 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:02:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:02:16.993+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:02:17.492 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:02:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:02:16.993+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:02:17.492 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:02:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:02:16.993+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:02:17.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:02:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:02:17.666+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:02:17.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:02:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:02:17.666+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:02:17.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:02:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:02:17.666+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:02:17.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:02:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:02:17.666+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:02:18.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:02:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:02:18.181+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:02:18.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:02:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:02:18.181+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:02:18.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:02:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:02:18.181+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:02:18.492 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:02:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:02:18.002+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:02:18.492 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:02:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:02:18.002+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:02:18.492 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:02:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:02:18.002+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:02:18.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:02:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:02:18.663+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:02:18.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:02:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:02:18.663+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:02:18.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:02:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:02:18.663+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:02:18.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:02:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:02:18.663+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:02:19.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:02:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:02:19.014+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:02:19.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:02:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:02:19.014+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:02:19.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:02:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:02:19.014+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:02:19.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:02:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:02:19.208+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:02:19.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:02:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:02:19.208+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:02:19.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:02:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:02:19.208+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:02:19.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:02:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:02:19.635+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:02:19.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:02:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:02:19.635+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:02:19.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:02:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:02:19.635+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:02:19.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:02:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:02:19.635+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:02:20.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:02:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:02:20.057+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:02:20.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:02:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:02:20.057+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:02:20.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:02:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:02:20.057+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:02:20.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:02:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:02:20.222+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:02:20.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:02:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:02:20.223+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:02:20.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:02:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:02:20.223+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:02:20.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:02:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:02:20.609+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:02:20.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:02:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:02:20.609+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:02:20.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:02:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:02:20.609+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:02:20.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:02:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:02:20.609+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:02:21.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:02:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:02:21.042+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:02:21.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:02:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:02:21.042+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:02:21.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:02:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:02:21.042+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:02:21.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:02:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:02:21.227+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:02:21.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:02:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:02:21.227+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:02:21.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:02:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:02:21.227+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:02:21.841 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:02:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:02:21.713+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 1207 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:02:21.841 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:02:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:02:21.562+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:02:21.842 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:02:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:02:21.562+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:02:21.842 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:02:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:02:21.562+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:02:21.842 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:02:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:02:21.562+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:02:22.235 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:02:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:02:22.088+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:02:22.235 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:02:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:02:22.088+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:02:22.235 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:02:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:02:22.088+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:02:22.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:02:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:02:22.234+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:02:22.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:02:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:02:22.234+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:02:22.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:02:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:02:22.234+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:02:22.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:02:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:02:22.553+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:02:22.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:02:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:02:22.553+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:02:22.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:02:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:02:22.553+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:02:22.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:02:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:02:22.553+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:02:23.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:02:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:02:23.093+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:02:23.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:02:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:02:23.093+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:02:23.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:02:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:02:23.093+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:02:23.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:02:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:02:23.260+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:02:23.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:02:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:02:23.260+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:02:23.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:02:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:02:23.260+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:02:23.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:02:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:02:23.532+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:02:23.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:02:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:02:23.532+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:02:23.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:02:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:02:23.532+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:02:23.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:02:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:02:23.532+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:02:24.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:02:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:02:24.128+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:02:24.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:02:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:02:24.128+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:02:24.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:02:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:02:24.128+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:02:24.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:02:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:02:24.240+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:02:24.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:02:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:02:24.240+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:02:24.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:02:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:02:24.240+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:02:24.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:02:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:02:24.530+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:02:24.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:02:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:02:24.530+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:02:24.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:02:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:02:24.530+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:02:24.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:02:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:02:24.530+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:02:25.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:02:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:02:25.142+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:02:25.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:02:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:02:25.142+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:02:25.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:02:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:02:25.142+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:02:25.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:02:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:02:25.198+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:02:25.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:02:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:02:25.198+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:02:25.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:02:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:02:25.198+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:02:25.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:02:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:02:25.564+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:02:25.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:02:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:02:25.564+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:02:25.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:02:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:02:25.564+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:02:25.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:02:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:02:25.564+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:02:26.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:02:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:02:26.166+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:02:26.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:02:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:02:26.166+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:02:26.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:02:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:02:26.166+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:02:26.492 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:02:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:02:26.150+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:02:26.492 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:02:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:02:26.150+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:02:26.492 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:02:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:02:26.150+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:02:26.842 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:02:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:02:26.714+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 1238 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:02:26.843 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:02:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:02:26.558+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:02:26.843 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:02:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:02:26.558+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:02:26.843 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:02:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:02:26.558+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:02:26.843 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:02:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:02:26.558+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:02:27.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:02:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:02:27.180+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:02:27.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:02:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:02:27.180+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:02:27.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:02:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:02:27.180+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:02:27.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:02:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:02:27.197+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:02:27.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:02:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:02:27.197+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:02:27.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:02:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:02:27.197+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:02:27.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:02:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:02:27.546+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:02:27.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:02:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:02:27.546+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:02:27.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:02:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:02:27.546+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:02:27.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:02:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:02:27.546+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:02:28.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:02:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:02:28.165+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:02:28.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:02:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:02:28.165+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:02:28.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:02:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:02:28.165+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:02:28.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:02:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:02:28.236+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:02:28.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:02:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:02:28.236+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:02:28.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:02:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:02:28.236+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:02:28.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:02:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:02:28.528+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:02:28.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:02:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:02:28.528+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:02:28.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:02:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:02:28.528+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:02:28.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:02:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:02:28.528+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:02:29.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:02:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:02:29.170+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:02:29.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:02:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:02:29.170+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:02:29.492 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:02:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:02:29.170+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:02:29.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:02:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:02:29.207+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:02:29.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:02:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:02:29.207+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:02:29.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:02:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:02:29.207+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:02:29.884 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:02:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:02:29.502+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:02:29.884 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:02:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:02:29.502+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:02:29.884 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:02:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:02:29.502+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:02:29.885 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:02:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:02:29.502+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:02:30.169 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:02:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:02:30.121+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:02:30.169 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:02:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:02:30.121+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:02:30.170 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:02:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:02:30.121+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:02:30.460 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:02:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:02:30.169+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:02:30.460 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:02:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:02:30.169+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:02:30.460 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:02:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:02:30.169+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:02:30.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:02:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:02:30.459+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:02:30.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:02:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:02:30.459+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:02:30.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:02:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:02:30.459+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:02:30.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:02:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:02:30.459+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:02:31.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:02:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:02:31.174+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:02:31.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:02:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:02:31.174+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:02:31.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:02:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:02:31.174+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:02:31.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:02:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:02:31.169+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:02:31.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:02:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:02:31.169+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:02:31.492 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:02:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:02:31.169+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:02:31.800 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:02:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:02:31.716+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 1272 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:02:31.800 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:02:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:02:31.491+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:02:31.800 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:02:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:02:31.491+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:02:31.800 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:02:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:02:31.491+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:02:31.800 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:02:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:02:31.491+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:02:32.475 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:02:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:02:32.125+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:02:32.475 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:02:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:02:32.125+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:02:32.475 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:02:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:02:32.125+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:02:32.475 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:02:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:02:32.222+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:02:32.475 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:02:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:02:32.222+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:02:32.475 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:02:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:02:32.222+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:02:32.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:02:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:02:32.473+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:02:32.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:02:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:02:32.473+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:02:32.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:02:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:02:32.473+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:02:32.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:02:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:02:32.473+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:02:33.433 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:02:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:02:33.159+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:02:33.433 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:02:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:02:33.159+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:02:33.433 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:02:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:02:33.159+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:02:33.433 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:02:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:02:33.251+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:02:33.433 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:02:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:02:33.251+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:02:33.433 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:02:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:02:33.251+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:02:33.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:02:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:02:33.432+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:02:33.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:02:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:02:33.432+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:02:33.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:02:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:02:33.432+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:02:33.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:02:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:02:33.432+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:02:34.478 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:02:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:02:34.155+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:02:34.478 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:02:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:02:34.155+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:02:34.478 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:02:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:02:34.155+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:02:34.478 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:02:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:02:34.217+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:02:34.478 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:02:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:02:34.217+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:02:34.479 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:02:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:02:34.217+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:02:34.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:02:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:02:34.477+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:02:34.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:02:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:02:34.477+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:02:34.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:02:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:02:34.477+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:02:34.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:02:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:02:34.477+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:02:35.486 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:02:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:02:35.198+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:02:35.486 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:02:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:02:35.198+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:02:35.486 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:02:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:02:35.198+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:02:35.486 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:02:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:02:35.265+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:02:35.486 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:02:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:02:35.265+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:02:35.486 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:02:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:02:35.265+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:02:35.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:02:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:02:35.485+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:02:35.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:02:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:02:35.485+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:02:35.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:02:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:02:35.485+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:02:35.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:02:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:02:35.486+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:02:36.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:02:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:02:36.230+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:02:36.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:02:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:02:36.230+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:02:36.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:02:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:02:36.230+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:02:36.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:02:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:02:36.219+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:02:36.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:02:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:02:36.219+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:02:36.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:02:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:02:36.219+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:02:36.841 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:02:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:02:36.717+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 1304 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:02:36.841 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:02:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:02:36.516+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:02:36.841 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:02:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:02:36.516+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:02:36.841 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:02:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:02:36.516+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:02:36.841 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:02:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:02:36.516+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:02:37.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:02:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:02:37.254+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:02:37.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:02:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:02:37.254+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:02:37.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:02:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:02:37.254+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:02:37.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:02:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:02:37.230+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:02:37.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:02:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:02:37.230+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:02:37.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:02:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:02:37.230+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:02:37.944 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:02:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:02:37.550+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:02:37.944 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:02:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:02:37.550+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:02:37.944 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:02:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:02:37.550+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:02:37.944 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:02:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:02:37.550+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:02:38.527 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:02:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:02:38.247+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:02:38.528 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:02:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:02:38.247+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:02:38.528 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:02:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:02:38.247+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:02:38.528 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:02:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:02:38.262+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:02:38.528 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:02:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:02:38.262+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:02:38.528 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:02:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:02:38.262+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:02:38.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:02:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:02:38.527+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:02:38.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:02:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:02:38.527+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:02:38.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:02:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:02:38.527+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:02:38.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:02:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:02:38.527+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:02:39.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:02:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:02:39.296+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:02:39.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:02:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:02:39.296+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:02:39.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:02:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:02:39.296+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:02:39.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:02:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:02:39.222+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:02:39.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:02:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:02:39.222+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:02:39.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:02:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:02:39.222+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:02:39.885 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:02:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:02:39.550+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:02:39.885 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:02:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:02:39.550+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:02:39.885 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:02:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:02:39.550+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:02:39.885 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:02:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:02:39.550+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:02:40.520 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:02:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:02:40.264+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:02:40.520 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:02:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:02:40.264+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:02:40.520 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:02:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:02:40.264+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:02:40.521 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:02:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:02:40.342+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:02:40.521 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:02:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:02:40.342+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:02:40.521 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:02:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:02:40.342+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:02:40.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:02:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:02:40.520+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:02:40.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:02:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:02:40.520+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:02:40.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:02:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:02:40.520+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:02:40.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:02:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:02:40.520+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:02:41.306 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:02:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:02:41.216+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:02:41.306 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:02:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:02:41.216+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:02:41.306 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:02:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:02:41.216+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:02:41.560 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:02:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:02:41.305+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:02:41.561 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:02:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:02:41.305+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:02:41.561 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:02:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:02:41.305+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:02:41.844 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:02:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:02:41.718+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 1337 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:02:41.845 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:02:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:02:41.560+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:02:41.845 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:02:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:02:41.560+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:02:41.845 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:02:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:02:41.560+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:02:41.845 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:02:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:02:41.560+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:02:42.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:02:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:02:42.353+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:02:42.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:02:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:02:42.353+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:02:42.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:02:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:02:42.353+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:02:42.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:02:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:02:42.186+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:02:42.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:02:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:02:42.186+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:02:42.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:02:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:02:42.186+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:02:42.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:02:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:02:42.526+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:02:42.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:02:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:02:42.526+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:02:42.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:02:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:02:42.526+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:02:42.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:02:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:02:42.526+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:02:43.388 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:02:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:02:43.152+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:02:43.388 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:02:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:02:43.152+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:02:43.388 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:02:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:02:43.152+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:02:43.388 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:02:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:02:43.386+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:02:43.388 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:02:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:02:43.386+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:02:43.388 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:02:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:02:43.386+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:02:43.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:02:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:02:43.546+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:02:43.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:02:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:02:43.546+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:02:43.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:02:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:02:43.546+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:02:43.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:02:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:02:43.546+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:02:44.412 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:02:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:02:44.176+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:02:44.412 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:02:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:02:44.176+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:02:44.413 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:02:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:02:44.176+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:02:44.742 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:02:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:02:44.412+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:02:44.742 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:02:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:02:44.412+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:02:44.742 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:02:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:02:44.412+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:02:44.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:02:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:02:44.576+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:02:44.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:02:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:02:44.576+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:02:44.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:02:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:02:44.576+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:02:44.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:02:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:02:44.576+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:02:45.449 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:02:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:02:45.160+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:02:45.449 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:02:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:02:45.161+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:02:45.449 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:02:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:02:45.161+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:02:45.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:02:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:02:45.449+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:02:45.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:02:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:02:45.449+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:02:45.742 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:02:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:02:45.449+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:02:45.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:02:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:02:45.596+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:02:45.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:02:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:02:45.596+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:02:45.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:02:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:02:45.596+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:02:45.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:02:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:02:45.596+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:02:46.490 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:02:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:02:46.186+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:02:46.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:02:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:02:46.186+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:02:46.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:02:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:02:46.186+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:02:46.741 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:02:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:02:46.719+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 1412 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:02:46.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:02:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:02:46.607+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:02:46.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:02:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:02:46.607+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:02:46.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:02:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:02:46.607+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:02:46.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:02:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:02:46.607+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:02:46.742 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:02:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:02:46.490+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:02:46.742 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:02:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:02:46.490+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:02:46.742 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:02:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:02:46.490+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:02:47.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:02:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:02:47.452+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:02:47.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:02:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:02:47.452+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:02:47.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:02:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:02:47.452+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:02:47.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:02:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:02:47.229+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:02:47.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:02:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:02:47.229+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:02:47.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:02:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:02:47.229+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:02:47.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:02:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:02:47.567+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:02:47.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:02:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:02:47.567+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:02:47.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:02:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:02:47.567+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:02:47.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:02:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:02:47.567+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:02:48.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:02:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:02:48.448+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:02:48.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:02:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:02:48.448+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:02:48.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:02:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:02:48.448+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:02:48.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:02:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:02:48.201+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:02:48.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:02:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:02:48.201+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:02:48.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:02:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:02:48.201+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:02:48.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:02:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:02:48.550+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:02:48.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:02:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:02:48.550+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:02:48.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:02:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:02:48.550+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:02:48.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:02:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:02:48.550+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:02:49.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:02:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:02:49.186+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:02:49.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:02:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:02:49.186+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:02:49.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:02:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:02:49.186+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:02:49.884 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:02:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:02:49.568+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:02:49.885 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:02:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:02:49.568+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:02:49.885 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:02:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:02:49.568+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:02:49.885 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:02:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:02:49.568+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:02:49.885 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:02:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:02:49.494+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:02:49.885 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:02:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:02:49.494+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:02:49.885 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:02:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:02:49.494+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:02:50.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:02:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:02:50.190+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:02:50.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:02:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:02:50.190+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:02:50.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:02:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:02:50.190+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:02:50.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:02:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:02:50.581+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:02:50.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:02:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:02:50.581+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:02:50.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:02:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:02:50.581+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:02:50.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:02:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:02:50.581+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:02:50.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:02:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:02:50.515+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:02:50.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:02:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:02:50.515+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:02:50.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:02:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:02:50.515+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:02:51.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:02:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:02:51.234+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:02:51.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:02:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:02:51.234+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:02:51.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:02:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:02:51.234+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:02:51.845 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:02:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:02:51.718+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 1496 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:02:51.846 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:02:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:02:51.505+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:02:51.846 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:02:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:02:51.505+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:02:51.846 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:02:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:02:51.505+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:02:51.846 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:02:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:02:51.572+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:02:51.846 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:02:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:02:51.572+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:02:51.846 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:02:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:02:51.572+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:02:51.846 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:02:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:02:51.572+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:02:52.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:02:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:02:52.228+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:02:52.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:02:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:02:52.228+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:02:52.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:02:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:02:52.228+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:02:52.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:02:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:02:52.548+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:02:52.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:02:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:02:52.548+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:02:52.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:02:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:02:52.548+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:02:52.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:02:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:02:52.539+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:02:52.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:02:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:02:52.539+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:02:52.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:02:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:02:52.539+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:02:52.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:02:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:02:52.539+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:02:53.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:02:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:02:53.235+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:02:53.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:02:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:02:53.235+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:02:53.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:02:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:02:53.235+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:02:53.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:02:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:02:53.553+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:02:53.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:02:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:02:53.553+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:02:53.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:02:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:02:53.553+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:02:53.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:02:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:02:53.492+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:02:53.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:02:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:02:53.492+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:02:53.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:02:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:02:53.492+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:02:53.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:02:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:02:53.493+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:02:54.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:02:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:02:54.237+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:02:54.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:02:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:02:54.237+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:02:54.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:02:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:02:54.237+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:02:54.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:02:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:02:54.485+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:02:54.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:02:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:02:54.485+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:02:54.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:02:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:02:54.485+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:02:54.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:02:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:02:54.485+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:02:54.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:02:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:02:54.533+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:02:54.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:02:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:02:54.533+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:02:54.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:02:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:02:54.533+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:02:55.546 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:02:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:02:55.263+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:02:55.546 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:02:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:02:55.263+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:02:55.546 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:02:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:02:55.263+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:02:55.547 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:02:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:02:55.444+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:02:55.547 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:02:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:02:55.444+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:02:55.547 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:02:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:02:55.444+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:02:55.547 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:02:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:02:55.444+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:02:55.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:02:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:02:55.545+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:02:55.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:02:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:02:55.545+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:02:55.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:02:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:02:55.545+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:02:56.574 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:02:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:02:56.301+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:02:56.574 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:02:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:02:56.301+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:02:56.575 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:02:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:02:56.301+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:02:56.575 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:02:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:02:56.419+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:02:56.575 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:02:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:02:56.419+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:02:56.575 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:02:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:02:56.419+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:02:56.575 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:02:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:02:56.419+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:02:56.845 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:02:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:02:56.719+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 1578 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:02:56.845 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:02:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:02:56.573+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:02:56.845 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:02:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:02:56.573+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:02:56.845 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:02:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:02:56.573+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:02:57.595 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:02:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:02:57.376+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:02:57.595 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:02:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:02:57.376+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:02:57.595 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:02:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:02:57.376+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:02:57.595 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:02:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:02:57.376+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:02:57.595 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:02:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:02:57.342+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:02:57.595 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:02:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:02:57.342+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:02:57.595 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:02:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:02:57.342+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:02:57.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:02:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:02:57.594+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:02:57.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:02:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:02:57.594+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:02:57.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:02:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:02:57.594+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:02:58.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:02:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:02:58.389+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:02:58.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:02:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:02:58.389+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:02:58.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:02:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:02:58.389+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:02:58.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:02:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:02:58.370+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:02:58.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:02:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:02:58.370+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:02:58.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:02:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:02:58.370+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:02:58.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:02:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:02:58.370+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:02:58.742 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:02:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:02:58.583+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:02:58.742 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:02:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:02:58.583+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:02:58.742 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:02:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:02:58.583+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:02:59.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:02:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:02:59.341+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:02:59.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:02:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:02:59.341+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:02:59.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:02:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:02:59.341+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:02:59.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:02:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:02:59.580+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:02:59.742 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:02:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:02:59.580+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:02:59.742 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:02:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:02:59.580+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:02:59.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:02:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:02:59.343+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:02:59.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:02:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:02:59.343+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:02:59.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:02:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:02:59.343+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:02:59.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:02:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:02:59.343+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:03:00.620 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:03:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:03:00.295+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:03:00.620 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:03:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:03:00.295+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:03:00.620 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:03:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:03:00.295+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:03:00.621 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:03:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:03:00.335+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:03:00.621 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:03:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:03:00.335+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:03:00.621 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:03:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:03:00.335+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:03:00.621 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:03:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:03:00.335+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:03:00.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:03:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:03:00.619+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:03:00.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:03:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:03:00.619+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:03:00.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:03:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:03:00.619+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:03:01.591 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:03:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:03:01.382+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:03:01.591 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:03:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:03:01.382+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:03:01.591 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:03:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:03:01.382+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:03:01.591 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:03:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:03:01.382+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:03:01.591 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:03:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:03:01.282+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:03:01.591 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:03:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:03:01.282+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:03:01.591 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:03:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:03:01.282+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:03:01.846 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:03:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:03:01.720+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 1661 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:03:01.846 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:03:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:03:01.590+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:03:01.846 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:03:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:03:01.590+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:03:01.846 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:03:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:03:01.590+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:03:02.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:03:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:03:02.543+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:03:02.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:03:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:03:02.543+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:03:02.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:03:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:03:02.543+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:03:02.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:03:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:03:02.322+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:03:02.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:03:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:03:02.322+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:03:02.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:03:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:03:02.322+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:03:02.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:03:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:03:02.413+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:03:02.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:03:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:03:02.413+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:03:02.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:03:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:03:02.413+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:03:02.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:03:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:03:02.413+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:03:03.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:03:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:03:03.503+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:03:03.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:03:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:03:03.503+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:03:03.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:03:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:03:03.503+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:03:03.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:03:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:03:03.345+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:03:03.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:03:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:03:03.345+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:03:03.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:03:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:03:03.345+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:03:03.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:03:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:03:03.464+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:03:03.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:03:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:03:03.464+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:03:03.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:03:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:03:03.464+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:03:03.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:03:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:03:03.464+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:03:04.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:03:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:03:04.474+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:03:04.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:03:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:03:04.474+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:03:04.742 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:03:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:03:04.474+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:03:04.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:03:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:03:04.341+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:03:04.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:03:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:03:04.341+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:03:04.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:03:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:03:04.341+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:03:04.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:03:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:03:04.432+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:03:04.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:03:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:03:04.432+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:03:04.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:03:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:03:04.432+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:03:04.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:03:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:03:04.432+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:03:05.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:03:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:03:05.341+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:03:05.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:03:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:03:05.341+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:03:05.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:03:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:03:05.341+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:03:05.742 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:03:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:03:05.462+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:03:05.742 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:03:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:03:05.462+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:03:05.742 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:03:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:03:05.462+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:03:05.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:03:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:03:05.478+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:03:05.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:03:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:03:05.478+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:03:05.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:03:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:03:05.478+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:03:05.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:03:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:03:05.478+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:03:06.643 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:03:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:03:06.465+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:03:06.643 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:03:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:03:06.465+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:03:06.643 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:03:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:03:06.465+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:03:06.643 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:03:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:03:06.350+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:03:06.643 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:03:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:03:06.350+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:03:06.643 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:03:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:03:06.350+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:03:06.644 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:03:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:03:06.520+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:03:06.644 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:03:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:03:06.520+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:03:06.644 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:03:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:03:06.520+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:03:06.644 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:03:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:03:06.520+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:03:06.991 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:03:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:03:06.721+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 1742 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:03:07.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:03:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:03:07.309+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:03:07.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:03:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:03:07.309+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:03:07.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:03:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:03:07.309+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:03:07.742 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:03:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:03:07.456+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:03:07.742 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:03:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:03:07.456+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:03:07.742 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:03:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:03:07.456+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:03:07.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:03:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:03:07.477+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:03:07.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:03:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:03:07.477+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:03:07.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:03:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:03:07.477+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:03:07.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:03:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:03:07.477+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:03:08.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:03:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:03:08.499+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:03:08.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:03:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:03:08.499+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:03:08.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:03:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:03:08.499+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:03:08.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:03:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:03:08.432+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:03:08.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:03:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:03:08.432+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:03:08.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:03:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:03:08.432+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:03:08.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:03:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:03:08.432+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:03:08.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:03:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:03:08.284+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:03:08.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:03:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:03:08.284+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:03:08.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:03:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:03:08.284+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:03:09.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:03:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:03:09.492+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:03:09.742 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:03:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:03:09.492+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:03:09.742 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:03:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:03:09.492+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:03:09.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:03:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:03:09.314+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:03:09.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:03:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:03:09.314+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:03:09.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:03:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:03:09.314+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:03:09.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:03:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:03:09.447+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:03:09.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:03:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:03:09.447+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:03:09.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:03:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:03:09.447+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:03:09.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:03:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:03:09.447+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:03:10.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:03:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:03:10.270+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:03:10.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:03:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:03:10.270+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:03:10.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:03:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:03:10.270+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:03:10.742 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:03:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:03:10.444+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:03:10.742 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:03:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:03:10.444+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:03:10.742 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:03:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:03:10.444+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:03:10.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:03:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:03:10.406+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:03:10.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:03:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:03:10.406+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:03:10.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:03:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:03:10.406+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:03:10.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:03:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:03:10.406+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:03:11.646 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:03:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:03:11.415+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:03:11.646 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:03:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:03:11.415+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:03:11.646 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:03:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:03:11.415+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:03:11.646 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:03:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:03:11.415+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:03:11.646 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:03:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:03:11.246+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:03:11.646 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:03:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:03:11.246+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:03:11.646 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:03:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:03:11.246+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:03:11.647 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:03:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:03:11.405+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:03:11.647 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:03:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:03:11.405+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:03:11.647 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:03:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:03:11.405+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:03:11.991 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:03:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:03:11.722+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 1826 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:03:12.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:03:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:03:12.386+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:03:12.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:03:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:03:12.386+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:03:12.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:03:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:03:12.386+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:03:12.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:03:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:03:12.282+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:03:12.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:03:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:03:12.282+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:03:12.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:03:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:03:12.282+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:03:12.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:03:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:03:12.390+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:03:12.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:03:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:03:12.390+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:03:12.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:03:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:03:12.390+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:03:12.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:03:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:03:12.391+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:03:13.534 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:03:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:03:13.281+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:03:13.534 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:03:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:03:13.281+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:03:13.535 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:03:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:03:13.281+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:03:13.535 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:03:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:03:13.391+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:03:13.535 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:03:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:03:13.391+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:03:13.535 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:03:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:03:13.391+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:03:13.535 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:03:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:03:13.376+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:03:13.535 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:03:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:03:13.376+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:03:13.535 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:03:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:03:13.376+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:03:13.536 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:03:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:03:13.376+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:03:14.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:03:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:03:14.373+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:03:14.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:03:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:03:14.373+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:03:14.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:03:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:03:14.373+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:03:14.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:03:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:03:14.373+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:03:14.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:03:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:03:14.324+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:03:14.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:03:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:03:14.324+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:03:14.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:03:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:03:14.324+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:03:14.742 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:03:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:03:14.411+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:03:14.742 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:03:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:03:14.411+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:03:14.742 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:03:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:03:14.411+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:03:15.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:03:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:03:15.396+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:03:15.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:03:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:03:15.396+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:03:15.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:03:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:03:15.396+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:03:15.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:03:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:03:15.396+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:03:15.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:03:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:03:15.308+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:03:15.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:03:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:03:15.308+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:03:15.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:03:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:03:15.308+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:03:15.742 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:03:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:03:15.366+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:03:15.742 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:03:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:03:15.366+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:03:15.742 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:03:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:03:15.366+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:03:16.725 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:03:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:03:16.356+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:03:16.725 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:03:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:03:16.356+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:03:16.725 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:03:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:03:16.356+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:03:16.725 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:03:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:03:16.385+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:03:16.725 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:03:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:03:16.385+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:03:16.725 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:03:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:03:16.385+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:03:16.726 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:03:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:03:16.385+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:03:16.726 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:03:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:03:16.330+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:03:16.726 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:03:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:03:16.330+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:03:16.726 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:03:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:03:16.330+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:03:16.991 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:03:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:03:16.723+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 1908 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:03:17.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:03:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:03:17.359+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:03:17.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:03:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:03:17.359+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:03:17.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:03:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:03:17.359+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:03:17.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:03:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:03:17.374+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:03:17.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:03:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:03:17.374+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:03:17.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:03:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:03:17.374+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:03:17.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:03:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:03:17.374+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:03:17.742 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:03:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:03:17.316+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:03:17.742 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:03:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:03:17.316+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:03:17.742 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:03:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:03:17.316+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:03:18.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:03:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:03:18.312+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:03:18.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:03:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:03:18.312+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:03:18.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:03:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:03:18.312+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:03:18.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:03:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:03:18.399+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:03:18.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:03:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:03:18.399+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:03:18.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:03:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:03:18.399+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:03:18.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:03:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:03:18.401+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:03:18.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:03:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:03:18.401+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:03:18.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:03:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:03:18.401+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:03:18.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:03:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:03:18.401+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:03:19.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:03:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:03:19.347+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:03:19.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:03:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:03:19.347+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:03:19.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:03:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:03:19.347+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:03:19.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:03:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:03:19.419+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:03:19.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:03:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:03:19.419+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:03:19.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:03:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:03:19.419+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:03:19.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:03:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:03:19.419+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:03:19.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:03:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:03:19.363+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:03:19.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:03:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:03:19.363+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:03:19.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:03:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:03:19.363+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:03:20.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:03:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:03:20.388+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:03:20.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:03:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:03:20.388+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:03:20.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:03:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:03:20.388+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:03:20.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:03:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:03:20.463+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:03:20.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:03:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:03:20.463+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:03:20.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:03:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:03:20.463+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:03:20.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:03:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:03:20.463+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:03:20.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:03:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:03:20.354+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:03:20.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:03:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:03:20.354+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:03:20.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:03:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:03:20.354+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:03:21.646 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:03:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:03:21.368+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:03:21.646 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:03:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:03:21.368+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:03:21.646 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:03:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:03:21.368+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:03:21.647 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:03:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:03:21.355+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:03:21.647 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:03:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:03:21.355+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:03:21.647 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:03:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:03:21.355+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:03:21.647 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:03:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:03:21.482+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:03:21.647 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:03:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:03:21.482+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:03:21.647 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:03:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:03:21.482+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:03:21.647 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:03:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:03:21.482+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:03:21.991 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:03:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:03:21.724+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 1991 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:03:22.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:03:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:03:22.394+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:03:22.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:03:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:03:22.394+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:03:22.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:03:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:03:22.394+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:03:22.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:03:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:03:22.309+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:03:22.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:03:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:03:22.309+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:03:22.742 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:03:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:03:22.309+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:03:22.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:03:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:03:22.476+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:03:22.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:03:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:03:22.476+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:03:22.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:03:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:03:22.476+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:03:22.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:03:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:03:22.476+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:03:23.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:03:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:03:23.314+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:03:23.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:03:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:03:23.314+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:03:23.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:03:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:03:23.314+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:03:23.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:03:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:03:23.496+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:03:23.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:03:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:03:23.496+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:03:23.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:03:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:03:23.496+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:03:23.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:03:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:03:23.496+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:03:23.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:03:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:03:23.378+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:03:23.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:03:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:03:23.378+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:03:23.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:03:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:03:23.378+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:03:24.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:03:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:03:24.348+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:03:24.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:03:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:03:24.348+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:03:24.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:03:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:03:24.348+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:03:24.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:03:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:03:24.460+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:03:24.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:03:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:03:24.460+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:03:24.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:03:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:03:24.460+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:03:24.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:03:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:03:24.460+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:03:24.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:03:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:03:24.352+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:03:24.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:03:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:03:24.352+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:03:24.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:03:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:03:24.352+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:03:25.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:03:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:03:25.324+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:03:25.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:03:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:03:25.324+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:03:25.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:03:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:03:25.324+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:03:25.742 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:03:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:03:25.387+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:03:25.742 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:03:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:03:25.387+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:03:25.742 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:03:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:03:25.387+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:03:25.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:03:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:03:25.446+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:03:25.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:03:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:03:25.446+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:03:25.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:03:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:03:25.446+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:03:25.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:03:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:03:25.447+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:03:26.726 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:03:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:03:26.465+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:03:26.726 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:03:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:03:26.465+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:03:26.727 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:03:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:03:26.465+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:03:26.727 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:03:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:03:26.465+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:03:26.727 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:03:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:03:26.321+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:03:26.727 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:03:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:03:26.321+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:03:26.727 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:03:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:03:26.321+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:03:26.727 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:03:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:03:26.345+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:03:26.727 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:03:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:03:26.345+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:03:26.728 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:03:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:03:26.345+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:03:26.991 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:03:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:03:26.726+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 2073 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:03:27.488 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:03:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:03:27.306+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:03:27.488 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:03:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:03:27.306+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:03:27.488 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:03:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:03:27.306+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:03:27.488 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:03:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:03:27.333+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:03:27.488 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:03:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:03:27.333+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:03:27.488 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:03:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:03:27.333+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:03:27.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:03:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:03:27.487+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:03:27.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:03:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:03:27.487+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:03:27.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:03:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:03:27.487+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:03:27.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:03:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:03:27.487+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:03:28.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:03:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:03:28.262+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:03:28.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:03:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:03:28.262+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:03:28.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:03:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:03:28.262+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:03:28.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:03:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:03:28.458+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:03:28.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:03:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:03:28.458+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:03:28.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:03:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:03:28.458+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:03:28.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:03:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:03:28.458+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:03:28.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:03:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:03:28.380+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:03:28.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:03:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:03:28.380+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:03:28.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:03:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:03:28.380+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:03:29.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:03:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:03:29.221+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:03:29.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:03:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:03:29.221+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:03:29.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:03:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:03:29.221+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:03:29.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:03:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:03:29.427+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:03:29.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:03:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:03:29.427+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:03:29.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:03:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:03:29.427+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:03:29.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:03:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:03:29.427+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:03:29.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:03:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:03:29.352+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:03:29.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:03:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:03:29.352+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:03:29.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:03:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:03:29.352+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:03:30.491 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:03:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:03:30.408+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:03:30.491 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:03:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:03:30.408+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:03:30.491 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:03:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:03:30.408+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:03:30.491 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:03:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:03:30.408+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:03:30.492 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:03:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:03:30.175+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:03:30.492 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:03:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:03:30.175+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:03:30.492 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:03:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:03:30.175+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:03:30.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:03:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:03:30.329+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:03:30.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:03:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:03:30.329+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:03:30.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:03:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:03:30.329+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:03:31.442 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:03:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:03:31.136+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:03:31.442 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:03:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:03:31.136+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:03:31.442 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:03:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:03:31.136+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:03:31.443 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:03:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:03:31.357+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:03:31.443 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:03:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:03:31.357+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:03:31.443 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:03:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:03:31.357+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:03:31.727 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:03:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:03:31.442+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:03:31.727 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:03:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:03:31.442+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:03:31.727 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:03:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:03:31.442+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:03:31.727 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:03:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:03:31.442+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:03:31.991 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:03:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:03:31.726+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 2157 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:03:32.403 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:03:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:03:32.144+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:03:32.403 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:03:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:03:32.144+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:03:32.403 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:03:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:03:32.144+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:03:32.403 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:03:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:03:32.377+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:03:32.403 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:03:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:03:32.377+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:03:32.403 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:03:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:03:32.377+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:03:32.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:03:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:03:32.402+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:03:32.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:03:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:03:32.402+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:03:32.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:03:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:03:32.402+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:03:32.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:03:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:03:32.402+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:03:33.421 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:03:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:03:33.125+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:03:33.421 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:03:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:03:33.125+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:03:33.422 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:03:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:03:33.125+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:03:33.422 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:03:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:03:33.362+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:03:33.422 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:03:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:03:33.362+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:03:33.422 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:03:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:03:33.362+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:03:33.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:03:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:03:33.421+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:03:33.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:03:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:03:33.421+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:03:33.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:03:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:03:33.421+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:03:33.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:03:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:03:33.421+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:03:34.438 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:03:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:03:34.342+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:03:34.438 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:03:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:03:34.342+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:03:34.438 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:03:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:03:34.342+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:03:34.439 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:03:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:03:34.118+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:03:34.439 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:03:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:03:34.118+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:03:34.439 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:03:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:03:34.118+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:03:34.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:03:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:03:34.437+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:03:34.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:03:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:03:34.437+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:03:34.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:03:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:03:34.437+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:03:34.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:03:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:03:34.437+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:03:35.444 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:03:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:03:35.334+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:03:35.444 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:03:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:03:35.334+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:03:35.444 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:03:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:03:35.334+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:03:35.444 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:03:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:03:35.115+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:03:35.444 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:03:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:03:35.115+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:03:35.444 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:03:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:03:35.115+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:03:35.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:03:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:03:35.443+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:03:35.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:03:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:03:35.443+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:03:35.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:03:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:03:35.443+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:03:35.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:03:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:03:35.443+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:03:36.402 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:03:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:03:36.074+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:03:36.402 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:03:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:03:36.074+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:03:36.402 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:03:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:03:36.074+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:03:36.403 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:03:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:03:36.320+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:03:36.403 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:03:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:03:36.320+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:03:36.403 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:03:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:03:36.320+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:03:36.728 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:03:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:03:36.401+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:03:36.728 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:03:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:03:36.401+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:03:36.728 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:03:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:03:36.401+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:03:36.728 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:03:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:03:36.401+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:03:36.991 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:03:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:03:36.727+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 2239 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:03:37.364 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:03:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:03:37.111+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:03:37.364 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:03:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:03:37.111+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:03:37.364 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:03:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:03:37.111+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:03:37.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:03:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:03:37.363+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:03:37.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:03:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:03:37.364+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:03:37.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:03:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:03:37.364+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:03:37.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:03:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:03:37.421+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:03:37.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:03:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:03:37.421+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:03:37.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:03:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:03:37.421+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:03:37.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:03:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:03:37.421+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:03:38.439 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:03:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:03:38.105+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:03:38.439 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:03:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:03:38.105+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:03:38.440 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:03:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:03:38.105+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:03:38.440 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:03:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:03:38.324+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:03:38.440 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:03:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:03:38.324+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:03:38.440 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:03:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:03:38.324+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:03:38.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:03:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:03:38.439+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:03:38.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:03:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:03:38.439+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:03:38.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:03:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:03:38.439+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:03:38.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:03:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:03:38.439+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:03:39.384 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:03:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:03:39.121+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:03:39.384 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:03:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:03:39.121+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:03:39.384 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:03:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:03:39.121+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:03:39.384 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:03:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:03:39.285+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:03:39.384 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:03:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:03:39.285+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:03:39.385 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:03:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:03:39.285+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:03:39.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:03:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:03:39.481+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:03:39.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:03:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:03:39.481+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:03:39.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:03:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:03:39.481+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:03:39.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:03:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:03:39.481+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:03:40.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:03:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:03:40.092+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:03:40.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:03:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:03:40.092+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:03:40.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:03:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:03:40.092+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:03:40.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:03:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:03:40.333+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:03:40.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:03:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:03:40.333+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:03:40.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:03:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:03:40.333+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:03:40.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:03:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:03:40.527+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:03:40.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:03:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:03:40.527+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:03:40.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:03:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:03:40.527+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:03:40.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:03:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:03:40.527+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:03:41.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:03:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:03:41.138+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:03:41.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:03:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:03:41.138+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:03:41.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:03:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:03:41.138+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:03:41.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:03:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:03:41.337+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:03:41.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:03:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:03:41.337+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:03:41.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:03:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:03:41.337+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:03:41.852 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:03:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:03:41.728+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 2322 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:03:41.852 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:03:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:03:41.553+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:03:41.853 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:03:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:03:41.553+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:03:41.853 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:03:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:03:41.553+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:03:41.853 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:03:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:03:41.553+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:03:42.455 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:03:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:03:42.161+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:03:42.456 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:03:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:03:42.161+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:03:42.456 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:03:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:03:42.161+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:03:42.456 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:03:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:03:42.330+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:03:42.456 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:03:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:03:42.330+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:03:42.456 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:03:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:03:42.330+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:03:42.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:03:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:03:42.590+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:03:42.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:03:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:03:42.590+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:03:42.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:03:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:03:42.590+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:03:42.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:03:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:03:42.590+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:03:43.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:03:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:03:43.309+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:03:43.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:03:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:03:43.309+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:03:43.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:03:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:03:43.309+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:03:43.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:03:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:03:43.184+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:03:43.492 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:03:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:03:43.184+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:03:43.492 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:03:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:03:43.184+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:03:43.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:03:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:03:43.631+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:03:43.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:03:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:03:43.631+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:03:43.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:03:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:03:43.631+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:03:43.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:03:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:03:43.631+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:03:44.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:03:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:03:44.171+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:03:44.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:03:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:03:44.171+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:03:44.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:03:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:03:44.171+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:03:44.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:03:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:03:44.358+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:03:44.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:03:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:03:44.358+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:03:44.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:03:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:03:44.358+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:03:44.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:03:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:03:44.647+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:03:44.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:03:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:03:44.647+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:03:44.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:03:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:03:44.647+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:03:44.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:03:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:03:44.647+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:03:45.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:03:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:03:45.151+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:03:45.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:03:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:03:45.151+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:03:45.492 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:03:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:03:45.151+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:03:45.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:03:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:03:45.308+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:03:45.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:03:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:03:45.308+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:03:45.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:03:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:03:45.308+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:03:45.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:03:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:03:45.683+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:03:45.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:03:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:03:45.683+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:03:45.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:03:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:03:45.683+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:03:45.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:03:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:03:45.683+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:03:46.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:03:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:03:46.200+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:03:46.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:03:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:03:46.200+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:03:46.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:03:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:03:46.200+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:03:46.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:03:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:03:46.287+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:03:46.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:03:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:03:46.287+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:03:46.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:03:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:03:46.287+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:03:46.991 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:03:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:03:46.728+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 2403 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:03:46.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:03:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:03:46.688+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:03:46.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:03:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:03:46.688+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:03:46.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:03:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:03:46.688+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:03:46.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:03:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:03:46.688+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:03:47.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:03:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:03:47.204+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:03:47.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:03:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:03:47.204+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:03:47.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:03:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:03:47.204+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:03:47.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:03:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:03:47.329+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:03:47.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:03:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:03:47.329+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:03:47.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:03:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:03:47.330+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:03:47.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:03:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:03:47.695+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:03:47.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:03:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:03:47.695+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:03:47.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:03:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:03:47.695+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:03:47.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:03:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:03:47.695+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:03:48.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:03:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:03:48.197+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:03:48.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:03:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:03:48.197+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:03:48.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:03:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:03:48.197+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:03:48.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:03:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:03:48.353+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:03:48.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:03:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:03:48.353+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:03:48.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:03:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:03:48.353+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:03:48.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:03:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:03:48.663+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:03:48.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:03:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:03:48.663+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:03:48.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:03:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:03:48.663+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:03:48.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:03:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:03:48.663+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:03:49.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:03:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:03:49.224+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:03:49.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:03:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:03:49.224+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:03:49.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:03:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:03:49.224+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:03:49.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:03:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:03:49.322+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:03:49.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:03:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:03:49.322+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:03:49.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:03:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:03:49.322+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:03:49.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:03:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:03:49.706+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:03:49.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:03:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:03:49.706+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:03:49.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:03:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:03:49.706+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:03:49.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:03:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:03:49.706+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:03:50.671 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:03:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:03:50.300+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:03:50.671 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:03:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:03:50.300+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:03:50.672 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:03:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:03:50.300+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:03:50.672 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:03:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:03:50.259+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:03:50.672 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:03:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:03:50.259+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:03:50.672 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:03:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:03:50.259+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:03:50.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:03:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:03:50.670+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:03:50.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:03:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:03:50.670+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:03:50.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:03:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:03:50.670+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:03:50.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:03:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:03:50.670+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:03:51.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:03:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:03:51.221+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:03:51.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:03:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:03:51.221+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:03:51.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:03:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:03:51.221+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:03:51.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:03:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:03:51.332+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:03:51.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:03:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:03:51.332+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:03:51.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:03:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:03:51.332+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:03:51.991 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:03:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:03:51.729+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 2487 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:03:51.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:03:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:03:51.678+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:03:51.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:03:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:03:51.678+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:03:51.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:03:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:03:51.678+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:03:51.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:03:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:03:51.679+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:03:52.665 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:03:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:03:52.256+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:03:52.666 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:03:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:03:52.256+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:03:52.666 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:03:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:03:52.256+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:03:52.666 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:03:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:03:52.347+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:03:52.666 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:03:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:03:52.347+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:03:52.666 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:03:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:03:52.347+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:03:52.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:03:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:03:52.665+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:03:52.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:03:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:03:52.665+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:03:52.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:03:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:03:52.665+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:03:52.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:03:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:03:52.665+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:03:53.666 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:03:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:03:53.295+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:03:53.666 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:03:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:03:53.295+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:03:53.666 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:03:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:03:53.295+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:03:53.666 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:03:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:03:53.381+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:03:53.666 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:03:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:03:53.381+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:03:53.666 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:03:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:03:53.381+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:03:53.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:03:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:03:53.664+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:03:53.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:03:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:03:53.664+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:03:53.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:03:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:03:53.664+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:03:53.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:03:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:03:53.664+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:03:54.681 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:03:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:03:54.250+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:03:54.681 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:03:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:03:54.250+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:03:54.681 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:03:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:03:54.250+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:03:54.681 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:03:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:03:54.339+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:03:54.682 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:03:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:03:54.339+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:03:54.682 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:03:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:03:54.339+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:03:54.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:03:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:03:54.680+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:03:54.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:03:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:03:54.680+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:03:54.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:03:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:03:54.680+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:03:54.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:03:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:03:54.680+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:03:55.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:03:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:03:55.223+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:03:55.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:03:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:03:55.223+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:03:55.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:03:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:03:55.223+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:03:55.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:03:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:03:55.351+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:03:55.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:03:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:03:55.351+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:03:55.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:03:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:03:55.351+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:03:55.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:03:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:03:55.679+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:03:55.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:03:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:03:55.679+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:03:55.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:03:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:03:55.679+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:03:55.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:03:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:03:55.679+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:03:56.680 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:03:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:03:56.269+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:03:56.681 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:03:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:03:56.269+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:03:56.681 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:03:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:03:56.269+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:03:56.681 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:03:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:03:56.373+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:03:56.681 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:03:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:03:56.373+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:03:56.681 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:03:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:03:56.373+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:03:56.991 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:03:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:03:56.730+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 2569 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:03:56.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:03:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:03:56.679+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:03:56.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:03:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:03:56.679+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:03:56.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:03:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:03:56.679+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:03:56.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:03:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:03:56.679+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:03:57.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:03:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:03:57.220+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:03:57.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:03:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:03:57.220+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:03:57.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:03:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:03:57.220+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:03:57.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:03:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:03:57.412+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:03:57.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:03:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:03:57.412+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:03:57.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:03:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:03:57.412+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:03:57.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:03:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:03:57.693+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:03:57.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:03:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:03:57.693+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:03:57.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:03:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:03:57.693+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:03:57.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:03:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:03:57.693+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:03:58.462 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:03:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:03:58.182+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:03:58.462 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:03:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:03:58.182+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:03:58.462 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:03:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:03:58.182+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:03:58.713 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:03:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:03:58.461+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:03:58.713 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:03:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:03:58.461+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:03:58.713 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:03:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:03:58.461+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:03:58.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:03:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:03:58.712+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:03:58.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:03:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:03:58.712+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:03:58.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:03:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:03:58.712+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:03:58.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:03:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:03:58.712+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:03:59.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:03:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:03:59.169+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:03:59.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:03:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:03:59.169+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:03:59.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:03:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:03:59.169+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:03:59.758 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:03:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:03:59.493+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:03:59.758 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:03:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:03:59.493+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:03:59.759 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:03:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:03:59.493+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:04:00.202 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:03:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:03:59.757+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:04:00.202 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:03:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:03:59.757+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:04:00.202 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:03:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:03:59.757+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:04:00.202 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:03:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:03:59.757+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:04:00.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:04:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:04:00.201+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:04:00.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:04:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:04:00.201+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:04:00.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:04:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:04:00.201+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:04:00.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:04:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:04:00.743+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:04:00.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:04:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:04:00.743+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:04:00.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:04:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:04:00.743+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:04:00.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:04:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:04:00.743+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:04:00.992 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:04:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:04:00.534+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:04:00.992 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:04:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:04:00.534+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:04:00.992 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:04:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:04:00.534+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:04:01.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:04:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:04:01.191+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:04:01.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:04:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:04:01.191+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:04:01.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:04:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:04:01.191+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:04:01.857 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:04:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:04:01.731+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 2652 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:04:01.858 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:04:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:04:01.555+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:04:01.858 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:04:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:04:01.555+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:04:01.858 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:04:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:04:01.555+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:04:01.859 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:04:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:04:01.754+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:04:01.859 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:04:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:04:01.754+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:04:01.859 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:04:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:04:01.754+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:04:01.859 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:04:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:04:01.754+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:04:02.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:04:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:04:02.217+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:04:02.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:04:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:04:02.217+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:04:02.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:04:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:04:02.217+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:04:02.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:04:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:04:02.762+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:04:02.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:04:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:04:02.762+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:04:02.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:04:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:04:02.762+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:04:02.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:04:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:04:02.762+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:04:02.992 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:04:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:04:02.532+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:04:02.992 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:04:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:04:02.532+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:04:02.992 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:04:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:04:02.532+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:04:03.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:04:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:04:03.190+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:04:03.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:04:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:04:03.190+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:04:03.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:04:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:04:03.190+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:04:03.959 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:04:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:04:03.730+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:04:03.960 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:04:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:04:03.730+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:04:03.960 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:04:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:04:03.730+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:04:03.960 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:04:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:04:03.730+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:04:03.960 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:04:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:04:03.521+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:04:03.960 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:04:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:04:03.521+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:04:03.960 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:04:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:04:03.521+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:04:04.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:04:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:04:04.193+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:04:04.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:04:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:04:04.193+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:04:04.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:04:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:04:04.193+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:04:04.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:04:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:04:04.538+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:04:04.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:04:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:04:04.538+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:04:04.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:04:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:04:04.538+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:04:04.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:04:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:04:04.688+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:04:04.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:04:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:04:04.688+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:04:04.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:04:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:04:04.688+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:04:04.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:04:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:04:04.688+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:04:05.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:04:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:04:05.188+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:04:05.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:04:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:04:05.188+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:04:05.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:04:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:04:05.188+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:04:05.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:04:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:04:05.579+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:04:05.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:04:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:04:05.579+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:04:05.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:04:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:04:05.579+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:04:05.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:04:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:04:05.697+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:04:05.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:04:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:04:05.697+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:04:05.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:04:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:04:05.697+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:04:05.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:04:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:04:05.697+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:04:06.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:04:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:04:06.233+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:04:06.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:04:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:04:06.233+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:04:06.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:04:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:04:06.233+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:04:06.745 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:04:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:04:06.732+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 2733 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:04:06.746 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:04:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:04:06.576+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:04:06.746 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:04:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:04:06.576+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:04:06.746 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:04:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:04:06.576+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:04:06.746 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:04:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:04:06.651+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:04:06.746 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:04:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:04:06.651+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:04:06.746 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:04:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:04:06.651+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:04:06.746 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:04:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:04:06.651+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:04:07.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:04:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:04:07.223+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:04:07.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:04:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:04:07.223+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:04:07.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:04:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:04:07.223+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:04:07.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:04:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:04:07.621+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:04:07.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:04:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:04:07.621+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:04:07.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:04:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:04:07.621+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:04:07.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:04:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:04:07.621+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:04:07.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:04:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:04:07.564+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:04:07.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:04:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:04:07.564+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:04:07.992 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:04:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:04:07.564+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:04:08.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:04:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:04:08.182+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:04:08.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:04:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:04:08.182+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:04:08.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:04:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:04:08.182+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:04:08.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:04:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:04:08.627+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:04:08.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:04:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:04:08.627+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:04:08.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:04:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:04:08.627+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:04:08.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:04:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:04:08.627+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:04:08.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:04:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:04:08.594+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:04:08.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:04:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:04:08.594+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:04:08.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:04:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:04:08.594+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:04:09.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:04:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:04:09.142+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:04:09.492 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:04:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:04:09.142+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:04:09.492 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:04:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:04:09.142+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:04:09.885 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:04:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:04:09.610+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:04:09.885 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:04:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:04:09.610+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:04:09.886 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:04:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:04:09.610+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:04:09.886 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:04:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:04:09.592+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:04:09.886 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:04:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:04:09.592+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:04:09.886 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:04:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:04:09.592+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:04:09.886 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:04:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:04:09.592+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:04:10.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:04:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:04:10.101+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:04:10.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:04:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:04:10.101+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:04:10.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:04:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:04:10.101+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:04:10.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:04:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:04:10.629+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:04:10.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:04:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:04:10.629+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:04:10.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:04:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:04:10.629+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:04:10.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:04:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:04:10.577+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:04:10.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:04:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:04:10.577+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:04:10.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:04:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:04:10.577+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:04:10.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:04:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:04:10.577+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:04:11.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:04:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:04:11.151+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:04:11.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:04:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:04:11.151+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:04:11.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:04:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:04:11.151+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:04:11.860 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:04:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:04:11.733+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 2816 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:04:11.860 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:04:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:04:11.653+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:04:11.861 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:04:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:04:11.653+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:04:11.861 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:04:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:04:11.653+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:04:11.861 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:04:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:04:11.556+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:04:11.861 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:04:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:04:11.556+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:04:11.861 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:04:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:04:11.556+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:04:11.861 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:04:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:04:11.556+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:04:12.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:04:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:04:12.154+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:04:12.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:04:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:04:12.154+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:04:12.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:04:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:04:12.154+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:04:12.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:04:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:04:12.606+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:04:12.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:04:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:04:12.606+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:04:12.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:04:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:04:12.606+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:04:12.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:04:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:04:12.521+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:04:12.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:04:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:04:12.521+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:04:12.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:04:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:04:12.521+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:04:12.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:04:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:04:12.521+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:04:13.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:04:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:04:13.187+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:04:13.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:04:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:04:13.188+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:04:13.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:04:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:04:13.188+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:04:13.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:04:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:04:13.609+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:04:13.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:04:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:04:13.609+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:04:13.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:04:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:04:13.609+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:04:13.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:04:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:04:13.525+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:04:13.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:04:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:04:13.525+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:04:13.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:04:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:04:13.525+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:04:13.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:04:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:04:13.525+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:04:14.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:04:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:04:14.167+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:04:14.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:04:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:04:14.167+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:04:14.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:04:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:04:14.167+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:04:14.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:04:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:04:14.570+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:04:14.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:04:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:04:14.570+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:04:14.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:04:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:04:14.570+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:04:14.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:04:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:04:14.570+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:04:14.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:04:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:04:14.603+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:04:14.992 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:04:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:04:14.603+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:04:14.992 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:04:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:04:14.603+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:04:15.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:04:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:04:15.204+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:04:15.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:04:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:04:15.204+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:04:15.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:04:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:04:15.204+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:04:15.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:04:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:04:15.610+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:04:15.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:04:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:04:15.610+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:04:15.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:04:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:04:15.610+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:04:15.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:04:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:04:15.576+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:04:15.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:04:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:04:15.576+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:04:15.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:04:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:04:15.576+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:04:15.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:04:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:04:15.576+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:04:16.563 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:04:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:04:16.255+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:04:16.564 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:04:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:04:16.255+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:04:16.564 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:04:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:04:16.255+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:04:16.863 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:04:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:04:16.734+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 2899 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:04:16.864 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:04:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:04:16.563+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:04:16.864 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:04:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:04:16.563+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:04:16.864 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:04:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:04:16.563+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:04:16.864 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:04:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:04:16.563+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:04:16.864 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:04:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:04:16.563+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:04:16.864 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:04:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:04:16.563+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:04:16.864 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:04:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:04:16.563+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:04:17.553 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:04:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:04:17.302+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:04:17.554 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:04:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:04:17.302+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:04:17.554 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:04:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:04:17.302+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:04:17.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:04:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:04:17.553+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:04:17.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:04:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:04:17.553+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:04:17.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:04:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:04:17.553+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:04:17.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:04:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:04:17.556+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:04:17.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:04:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:04:17.556+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:04:17.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:04:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:04:17.556+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:04:17.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:04:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:04:17.556+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:04:18.580 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:04:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:04:18.280+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:04:18.580 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:04:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:04:18.280+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:04:18.580 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:04:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:04:18.280+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:04:18.580 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:04:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:04:18.509+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:04:18.580 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:04:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:04:18.509+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:04:18.580 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:04:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:04:18.509+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:04:18.580 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:04:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:04:18.509+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:04:18.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:04:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:04:18.579+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:04:18.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:04:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:04:18.579+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:04:18.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:04:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:04:18.579+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:04:19.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:04:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:04:19.239+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:04:19.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:04:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:04:19.239+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:04:19.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:04:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:04:19.239+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:04:19.884 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:04:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:04:19.597+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:04:19.884 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:04:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:04:19.597+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:04:19.884 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:04:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:04:19.597+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:04:19.885 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:04:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:04:19.544+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:04:19.885 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:04:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:04:19.544+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:04:19.885 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:04:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:04:19.544+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:04:19.885 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:04:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:04:19.544+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:04:20.590 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:04:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:04:20.285+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:04:20.591 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:04:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:04:20.285+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:04:20.591 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:04:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:04:20.285+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:04:20.591 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:04:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:04:20.494+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:04:20.591 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:04:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:04:20.494+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:04:20.591 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:04:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:04:20.495+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:04:20.591 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:04:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:04:20.495+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:04:20.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:04:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:04:20.590+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:04:20.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:04:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:04:20.590+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:04:20.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:04:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:04:20.590+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:04:21.619 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:04:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:04:21.469+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:04:21.619 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:04:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:04:21.469+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:04:21.619 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:04:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:04:21.469+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:04:21.619 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:04:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:04:21.469+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:04:21.619 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:04:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:04:21.291+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:04:21.619 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:04:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:04:21.291+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:04:21.619 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:04:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:04:21.291+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:04:21.991 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:04:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:04:21.736+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 2981 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:04:21.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:04:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:04:21.618+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:04:21.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:04:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:04:21.618+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:04:21.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:04:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:04:21.618+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:04:22.633 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:04:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:04:22.487+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:04:22.633 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:04:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:04:22.487+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:04:22.633 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:04:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:04:22.487+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:04:22.633 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:04:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:04:22.488+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:04:22.633 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:04:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:04:22.335+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:04:22.634 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:04:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:04:22.335+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:04:22.634 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:04:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:04:22.335+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:04:22.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:04:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:04:22.633+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:04:22.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:04:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:04:22.633+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:04:22.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:04:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:04:22.633+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:04:23.606 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:04:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:04:23.527+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:04:23.606 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:04:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:04:23.527+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:04:23.606 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:04:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:04:23.527+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:04:23.606 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:04:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:04:23.527+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:04:23.606 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:04:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:04:23.322+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:04:23.606 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:04:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:04:23.322+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:04:23.606 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:04:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:04:23.322+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:04:23.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:04:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:04:23.605+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:04:23.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:04:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:04:23.605+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:04:23.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:04:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:04:23.605+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:04:24.586 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:04:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:04:24.318+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:04:24.586 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:04:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:04:24.318+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:04:24.586 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:04:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:04:24.318+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:04:24.586 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:04:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:04:24.516+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:04:24.586 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:04:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:04:24.516+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:04:24.587 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:04:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:04:24.516+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:04:24.587 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:04:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:04:24.516+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:04:24.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:04:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:04:24.585+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:04:24.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:04:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:04:24.585+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:04:24.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:04:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:04:24.585+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:04:25.552 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:04:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:04:25.277+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:04:25.553 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:04:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:04:25.277+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:04:25.553 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:04:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:04:25.277+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:04:25.553 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:04:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:04:25.502+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:04:25.553 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:04:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:04:25.502+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:04:25.553 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:04:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:04:25.502+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:04:25.553 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:04:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:04:25.502+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:04:25.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:04:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:04:25.552+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:04:25.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:04:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:04:25.552+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:04:25.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:04:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:04:25.552+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:04:26.738 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:04:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:04:26.504+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:04:26.738 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:04:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:04:26.504+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:04:26.738 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:04:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:04:26.504+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:04:26.738 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:04:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:04:26.493+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:04:26.739 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:04:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:04:26.493+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:04:26.739 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:04:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:04:26.493+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:04:26.739 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:04:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:04:26.493+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:04:26.739 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:04:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:04:26.263+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:04:26.739 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:04:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:04:26.263+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:04:26.739 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:04:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:04:26.263+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:04:26.991 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:04:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:04:26.737+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 3064 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:04:27.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:04:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:04:27.290+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:04:27.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:04:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:04:27.290+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:04:27.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:04:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:04:27.290+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:04:27.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:04:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:04:27.493+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:04:27.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:04:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:04:27.493+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:04:27.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:04:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:04:27.493+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:04:27.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:04:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:04:27.450+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:04:27.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:04:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:04:27.450+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:04:27.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:04:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:04:27.450+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:04:27.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:04:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:04:27.450+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:04:28.536 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:04:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:04:28.252+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:04:28.536 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:04:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:04:28.252+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:04:28.536 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:04:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:04:28.252+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:04:28.536 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:04:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:04:28.494+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:04:28.536 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:04:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:04:28.494+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:04:28.536 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:04:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:04:28.494+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:04:28.537 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:04:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:04:28.401+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:04:28.537 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:04:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:04:28.401+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:04:28.537 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:04:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:04:28.401+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:04:28.537 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:04:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:04:28.401+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:04:29.539 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:04:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:04:29.268+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:04:29.540 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:04:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:04:29.268+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:04:29.540 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:04:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:04:29.268+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:04:29.540 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:04:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:04:29.434+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:04:29.540 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:04:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:04:29.434+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:04:29.540 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:04:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:04:29.434+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:04:29.540 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:04:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:04:29.435+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:04:29.885 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:04:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:04:29.539+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:04:29.937 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:04:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:04:29.539+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:04:29.937 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:04:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:04:29.539+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:04:30.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:04:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:04:30.224+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:04:30.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:04:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:04:30.224+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:04:30.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:04:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:04:30.224+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:04:30.491 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:04:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:04:30.409+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:04:30.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:04:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:04:30.409+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:04:30.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:04:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:04:30.409+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:04:30.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:04:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:04:30.409+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:04:30.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:04:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:04:30.504+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:04:30.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:04:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:04:30.504+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:04:30.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:04:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:04:30.504+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:04:31.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:04:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:04:31.235+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:04:31.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:04:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:04:31.235+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:04:31.492 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:04:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:04:31.235+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:04:31.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:04:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:04:31.443+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:04:31.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:04:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:04:31.443+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:04:31.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:04:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:04:31.443+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:04:31.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:04:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:04:31.443+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:04:31.866 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:04:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:04:31.739+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 3147 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:04:31.866 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:04:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:04:31.525+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:04:31.866 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:04:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:04:31.525+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:04:31.866 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:04:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:04:31.525+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:04:32.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:04:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:04:32.266+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:04:32.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:04:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:04:32.266+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:04:32.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:04:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:04:32.266+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:04:32.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:04:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:04:32.420+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:04:32.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:04:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:04:32.420+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:04:32.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:04:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:04:32.420+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:04:32.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:04:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:04:32.420+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:04:32.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:04:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:04:32.477+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:04:32.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:04:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:04:32.477+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:04:32.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:04:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:04:32.477+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:04:33.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:04:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:04:33.459+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:04:33.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:04:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:04:33.459+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:04:33.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:04:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:04:33.459+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:04:33.742 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:04:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:04:33.312+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:04:33.742 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:04:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:04:33.312+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:04:33.742 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:04:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:04:33.312+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:04:33.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:04:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:04:33.455+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:04:33.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:04:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:04:33.455+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:04:33.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:04:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:04:33.455+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:04:33.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:04:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:04:33.455+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:04:34.679 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:04:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:04:34.296+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:04:34.679 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:04:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:04:34.296+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:04:34.680 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:04:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:04:34.296+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:04:34.680 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:04:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:04:34.487+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:04:34.680 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:04:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:04:34.487+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:04:34.680 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:04:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:04:34.487+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:04:34.680 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:04:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:04:34.435+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:04:34.680 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:04:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:04:34.435+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:04:34.680 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:04:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:04:34.435+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:04:34.680 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:04:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:04:34.435+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:04:35.517 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:04:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:04:35.254+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:04:35.517 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:04:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:04:35.254+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:04:35.517 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:04:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:04:35.254+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:04:35.517 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:04:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:04:35.452+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:04:35.517 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:04:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:04:35.452+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:04:35.517 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:04:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:04:35.452+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:04:35.517 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:04:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:04:35.452+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:04:35.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:04:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:04:35.516+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:04:35.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:04:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:04:35.516+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:04:35.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:04:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:04:35.516+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:04:36.539 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:04:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:04:36.255+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:04:36.539 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:04:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:04:36.255+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:04:36.539 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:04:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:04:36.255+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:04:36.540 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:04:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:04:36.419+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:04:36.540 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:04:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:04:36.419+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:04:36.540 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:04:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:04:36.419+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:04:36.540 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:04:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:04:36.419+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:04:36.867 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:04:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:04:36.740+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 3230 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:04:36.867 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:04:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:04:36.539+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:04:36.867 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:04:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:04:36.539+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:04:36.867 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:04:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:04:36.539+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:04:37.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:04:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:04:37.237+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:04:37.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:04:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:04:37.237+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:04:37.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:04:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:04:37.237+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:04:37.491 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:04:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:04:37.409+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:04:37.491 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:04:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:04:37.409+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:04:37.491 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:04:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:04:37.409+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:04:37.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:04:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:04:37.409+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:04:37.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:04:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:04:37.511+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:04:37.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:04:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:04:37.511+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:04:37.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:04:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:04:37.511+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:04:38.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:04:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:04:38.221+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:04:38.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:04:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:04:38.221+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:04:38.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:04:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:04:38.221+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:04:38.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:04:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:04:38.445+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:04:38.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:04:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:04:38.445+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:04:38.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:04:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:04:38.445+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:04:38.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:04:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:04:38.445+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:04:38.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:04:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:04:38.509+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:04:38.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:04:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:04:38.509+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:04:38.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:04:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:04:38.509+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:04:39.542 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:04:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:04:39.255+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:04:39.542 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:04:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:04:39.255+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:04:39.542 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:04:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:04:39.255+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:04:39.542 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:04:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:04:39.486+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:04:39.542 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:04:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:04:39.486+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:04:39.543 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:04:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:04:39.486+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:04:39.543 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:04:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:04:39.486+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:04:39.884 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:04:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:04:39.541+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:04:39.884 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:04:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:04:39.541+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:04:39.884 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:04:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:04:39.541+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:04:40.503 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:04:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:04:40.250+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:04:40.503 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:04:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:04:40.250+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:04:40.503 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:04:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:04:40.250+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:04:40.824 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:04:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:04:40.532+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:04:40.824 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:04:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:04:40.532+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:04:40.824 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:04:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:04:40.532+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:04:40.825 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:04:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:04:40.502+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:04:40.825 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:04:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:04:40.502+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:04:40.825 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:04:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:04:40.502+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:04:40.825 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:04:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:04:40.502+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:04:41.651 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:04:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:04:41.487+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:04:41.652 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:04:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:04:41.487+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:04:41.652 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:04:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:04:41.487+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:04:41.652 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:04:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:04:41.291+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:04:41.652 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:04:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:04:41.291+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:04:41.652 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:04:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:04:41.291+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:04:41.652 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:04:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:04:41.503+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:04:41.652 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:04:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:04:41.503+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:04:41.652 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:04:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:04:41.503+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:04:41.653 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:04:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:04:41.503+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:04:41.991 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:04:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:04:41.740+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 3312 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:04:42.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:04:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:04:42.545+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:04:42.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:04:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:04:42.545+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:04:42.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:04:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:04:42.545+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:04:42.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:04:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:04:42.545+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:04:42.742 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:04:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:04:42.326+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:04:42.742 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:04:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:04:42.326+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:04:42.742 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:04:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:04:42.326+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:04:42.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:04:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:04:42.471+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:04:42.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:04:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:04:42.471+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:04:42.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:04:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:04:42.471+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:04:43.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:04:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:04:43.511+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:04:43.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:04:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:04:43.511+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:04:43.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:04:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:04:43.511+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:04:43.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:04:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:04:43.511+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:04:43.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:04:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:04:43.319+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:04:43.742 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:04:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:04:43.319+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:04:43.742 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:04:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:04:43.319+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:04:43.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:04:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:04:43.430+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:04:43.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:04:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:04:43.430+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:04:43.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:04:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:04:43.430+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:04:44.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:04:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:04:44.491+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:04:44.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:04:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:04:44.491+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:04:44.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:04:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:04:44.491+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:04:44.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:04:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:04:44.491+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:04:44.742 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:04:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:04:44.351+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:04:44.742 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:04:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:04:44.351+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:04:44.742 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:04:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:04:44.351+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:04:44.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:04:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:04:44.403+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:04:44.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:04:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:04:44.403+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:04:44.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:04:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:04:44.403+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:04:45.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:04:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:04:45.502+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:04:45.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:04:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:04:45.502+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:04:45.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:04:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:04:45.502+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:04:45.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:04:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:04:45.502+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:04:45.742 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:04:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:04:45.341+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:04:45.742 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:04:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:04:45.341+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:04:45.742 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:04:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:04:45.341+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:04:45.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:04:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:04:45.441+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:04:45.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:04:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:04:45.441+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:04:45.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:04:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:04:45.441+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:04:46.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:04:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:04:46.548+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:04:46.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:04:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:04:46.548+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:04:46.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:04:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:04:46.548+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:04:46.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:04:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:04:46.548+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:04:46.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:04:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:04:46.323+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:04:46.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:04:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:04:46.323+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:04:46.742 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:04:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:04:46.323+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:04:46.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:04:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:04:46.406+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:04:46.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:04:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:04:46.406+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:04:46.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:04:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:04:46.406+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:04:47.241 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:04:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:04:46.741+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 3394 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:04:47.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:04:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:04:47.572+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:04:47.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:04:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:04:47.572+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:04:47.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:04:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:04:47.572+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:04:47.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:04:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:04:47.572+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:04:47.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:04:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:04:47.339+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:04:47.742 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:04:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:04:47.339+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:04:47.742 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:04:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:04:47.339+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:04:47.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:04:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:04:47.408+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:04:47.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:04:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:04:47.408+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:04:47.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:04:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:04:47.408+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:04:48.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:04:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:04:48.371+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:04:48.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:04:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:04:48.371+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:04:48.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:04:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:04:48.371+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:04:48.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:04:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:04:48.448+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:04:48.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:04:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:04:48.448+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:04:48.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:04:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:04:48.448+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:04:48.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:04:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:04:48.589+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:04:48.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:04:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:04:48.589+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:04:48.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:04:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:04:48.589+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:04:48.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:04:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:04:48.589+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:04:49.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:04:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:04:49.347+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:04:49.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:04:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:04:49.347+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:04:49.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:04:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:04:49.347+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:04:49.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:04:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:04:49.431+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:04:49.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:04:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:04:49.431+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:04:49.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:04:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:04:49.431+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:04:49.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:04:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:04:49.591+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:04:49.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:04:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:04:49.591+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:04:49.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:04:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:04:49.591+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:04:49.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:04:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:04:49.591+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:04:50.605 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:04:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:04:50.448+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:04:50.606 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:04:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:04:50.448+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:04:50.606 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:04:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:04:50.448+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:04:50.606 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:04:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:04:50.339+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:04:50.606 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:04:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:04:50.339+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:04:50.606 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:04:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:04:50.339+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:04:50.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:04:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:04:50.605+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:04:50.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:04:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:04:50.605+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:04:50.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:04:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:04:50.605+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:04:50.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:04:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:04:50.605+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:04:51.652 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:04:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:04:51.580+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:04:51.652 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:04:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:04:51.580+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:04:51.652 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:04:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:04:51.580+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:04:51.653 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:04:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:04:51.580+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:04:51.653 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:04:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:04:51.451+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:04:51.653 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:04:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:04:51.451+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:04:51.653 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:04:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:04:51.451+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:04:51.653 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:04:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:04:51.352+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:04:51.653 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:04:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:04:51.352+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:04:51.653 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:04:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:04:51.352+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:04:51.991 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:04:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:04:51.742+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 3477 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:04:52.604 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:04:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:04:52.346+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:04:52.604 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:04:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:04:52.346+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:04:52.604 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:04:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:04:52.346+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:04:52.604 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:04:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:04:52.413+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:04:52.605 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:04:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:04:52.413+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:04:52.605 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:04:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:04:52.413+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:04:52.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:04:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:04:52.603+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:04:52.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:04:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:04:52.603+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:04:52.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:04:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:04:52.603+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:04:52.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:04:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:04:52.603+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:04:53.381 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:04:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:04:53.334+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:04:53.381 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:04:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:04:53.334+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:04:53.382 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:04:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:04:53.334+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:04:53.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:04:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:04:53.381+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:04:53.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:04:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:04:53.381+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:04:53.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:04:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:04:53.381+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:04:53.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:04:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:04:53.604+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:04:53.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:04:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:04:53.604+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:04:53.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:04:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:04:53.604+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:04:53.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:04:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:04:53.604+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:04:54.630 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:04:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:04:54.369+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:04:54.630 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:04:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:04:54.369+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:04:54.630 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:04:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:04:54.369+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:04:54.630 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:04:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:04:54.381+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:04:54.631 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:04:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:04:54.381+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:04:54.631 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:04:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:04:54.381+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:04:54.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:04:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:04:54.628+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:04:54.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:04:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:04:54.628+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:04:54.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:04:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:04:54.628+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:04:54.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:04:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:04:54.628+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:04:55.605 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:04:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:04:55.325+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:04:55.605 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:04:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:04:55.325+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:04:55.605 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:04:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:04:55.325+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:04:55.606 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:04:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:04:55.405+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:04:55.606 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:04:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:04:55.405+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:04:55.606 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:04:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:04:55.405+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:04:55.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:04:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:04:55.604+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:04:55.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:04:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:04:55.604+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:04:55.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:04:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:04:55.604+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:04:55.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:04:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:04:55.604+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:04:56.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:04:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:04:56.370+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:04:56.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:04:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:04:56.370+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:04:56.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:04:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:04:56.370+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:04:56.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:04:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:04:56.429+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:04:56.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:04:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:04:56.429+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:04:56.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:04:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:04:56.429+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:04:56.991 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:04:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:04:56.743+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 3560 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:04:56.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:04:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:04:56.642+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:04:56.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:04:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:04:56.642+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:04:56.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:04:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:04:56.642+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:04:56.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:04:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:04:56.642+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:04:57.633 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:04:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:04:57.374+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:04:57.633 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:04:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:04:57.374+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:04:57.633 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:04:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:04:57.374+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:04:57.633 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:04:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:04:57.460+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:04:57.633 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:04:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:04:57.460+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:04:57.633 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:04:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:04:57.460+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:04:57.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:04:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:04:57.631+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:04:57.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:04:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:04:57.631+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:04:57.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:04:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:04:57.631+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:04:57.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:04:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:04:57.631+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:04:58.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:04:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:04:58.408+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:04:58.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:04:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:04:58.408+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:04:58.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:04:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:04:58.408+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:04:58.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:04:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:04:58.410+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:04:58.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:04:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:04:58.411+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:04:58.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:04:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:04:58.411+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:04:58.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:04:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:04:58.647+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:04:58.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:04:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:04:58.647+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:04:58.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:04:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:04:58.647+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:04:58.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:04:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:04:58.647+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:04:59.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:04:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:04:59.621+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:04:59.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:04:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:04:59.621+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:04:59.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:04:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:04:59.621+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:04:59.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:04:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:04:59.621+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:04:59.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:04:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:04:59.372+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:04:59.742 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:04:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:04:59.372+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:04:59.742 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:04:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:04:59.372+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:04:59.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:04:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:04:59.387+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:04:59.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:04:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:04:59.387+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:04:59.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:04:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:04:59.387+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:05:00.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:05:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:05:00.368+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:05:00.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:05:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:05:00.368+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:05:00.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:05:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:05:00.368+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:05:00.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:05:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:05:00.595+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:05:00.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:05:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:05:00.595+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:05:00.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:05:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:05:00.595+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:05:00.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:05:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:05:00.595+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:05:00.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:05:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:05:00.430+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:05:00.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:05:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:05:00.430+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:05:00.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:05:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:05:00.430+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:05:01.652 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:05:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:05:01.393+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:05:01.653 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:05:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:05:01.393+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:05:01.653 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:05:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:05:01.393+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:05:01.653 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:05:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:05:01.579+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:05:01.653 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:05:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:05:01.579+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:05:01.653 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:05:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:05:01.579+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:05:01.653 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:05:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:05:01.579+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:05:01.653 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:05:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:05:01.395+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:05:01.653 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:05:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:05:01.395+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:05:01.654 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:05:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:05:01.395+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:05:01.991 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:05:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:05:01.744+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 3642 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:05:02.615 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:05:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:05:02.353+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:05:02.615 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:05:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:05:02.353+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:05:02.615 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:05:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:05:02.353+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:05:02.615 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:05:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:05:02.414+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:05:02.615 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:05:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:05:02.414+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:05:02.615 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:05:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:05:02.414+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:05:02.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:05:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:05:02.614+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:05:02.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:05:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:05:02.614+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:05:02.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:05:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:05:02.614+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:05:02.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:05:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:05:02.614+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:05:03.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:05:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:05:03.383+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:05:03.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:05:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:05:03.383+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:05:03.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:05:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:05:03.383+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:05:03.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:05:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:05:03.577+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:05:03.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:05:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:05:03.577+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:05:03.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:05:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:05:03.577+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:05:03.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:05:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:05:03.577+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:05:03.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:05:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:05:03.462+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:05:03.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:05:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:05:03.462+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:05:03.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:05:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:05:03.462+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:05:04.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:05:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:05:04.398+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:05:04.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:05:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:05:04.398+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:05:04.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:05:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:05:04.398+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:05:04.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:05:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:05:04.608+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:05:04.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:05:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:05:04.608+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:05:04.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:05:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:05:04.608+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:05:04.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:05:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:05:04.608+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:05:04.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:05:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:05:04.463+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:05:04.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:05:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:05:04.463+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:05:04.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:05:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:05:04.463+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:05:05.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:05:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:05:05.390+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:05:05.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:05:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:05:05.390+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:05:05.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:05:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:05:05.390+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:05:05.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:05:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:05:05.429+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:05:05.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:05:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:05:05.429+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:05:05.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:05:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:05:05.429+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:05:05.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:05:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:05:05.581+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:05:05.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:05:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:05:05.581+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:05:05.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:05:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:05:05.581+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:05:05.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:05:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:05:05.581+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:05:06.642 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:05:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:05:06.408+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:05:06.642 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:05:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:05:06.408+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:05:06.643 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:05:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:05:06.408+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:05:06.643 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:05:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:05:06.386+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:05:06.643 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:05:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:05:06.386+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:05:06.643 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:05:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:05:06.386+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:05:06.643 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:05:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:05:06.620+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:05:06.643 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:05:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:05:06.620+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:05:06.643 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:05:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:05:06.620+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:05:06.643 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:05:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:05:06.620+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:05:06.991 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:05:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:05:06.745+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 3724 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:05:07.632 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:05:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:05:07.356+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:05:07.632 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:05:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:05:07.356+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:05:07.632 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:05:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:05:07.356+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:05:07.633 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:05:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:05:07.435+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:05:07.633 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:05:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:05:07.435+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:05:07.633 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:05:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:05:07.435+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:05:07.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:05:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:05:07.632+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:05:07.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:05:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:05:07.632+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:05:07.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:05:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:05:07.632+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:05:07.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:05:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:05:07.632+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:05:08.620 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:05:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:05:08.365+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:05:08.620 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:05:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:05:08.365+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:05:08.620 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:05:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:05:08.365+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:05:08.620 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:05:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:05:08.484+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:05:08.621 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:05:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:05:08.484+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:05:08.621 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:05:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:05:08.484+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:05:08.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:05:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:05:08.619+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:05:08.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:05:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:05:08.619+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:05:08.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:05:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:05:08.619+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:05:08.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:05:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:05:08.619+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:05:09.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:05:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:05:09.471+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:05:09.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:05:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:05:09.471+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:05:09.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:05:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:05:09.471+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:05:09.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:05:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:05:09.639+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:05:09.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:05:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:05:09.639+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:05:09.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:05:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:05:09.639+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:05:09.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:05:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:05:09.639+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:05:09.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:05:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:05:09.410+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:05:09.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:05:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:05:09.410+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:05:09.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:05:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:05:09.410+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:05:10.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:05:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:05:10.431+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:05:10.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:05:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:05:10.431+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:05:10.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:05:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:05:10.431+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:05:10.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:05:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:05:10.598+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:05:10.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:05:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:05:10.598+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:05:10.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:05:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:05:10.598+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:05:10.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:05:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:05:10.598+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:05:10.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:05:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:05:10.372+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:05:10.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:05:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:05:10.372+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:05:10.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:05:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:05:10.372+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:05:11.608 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:05:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:05:11.354+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:05:11.608 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:05:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:05:11.354+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:05:11.608 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:05:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:05:11.354+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:05:11.608 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:05:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:05:11.475+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:05:11.609 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:05:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:05:11.475+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:05:11.609 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:05:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:05:11.475+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:05:11.873 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:05:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:05:11.747+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 3807 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:05:11.873 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:05:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:05:11.607+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:05:11.873 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:05:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:05:11.607+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:05:11.873 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:05:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:05:11.607+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:05:11.874 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:05:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:05:11.607+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:05:12.600 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:05:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:05:12.488+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:05:12.600 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:05:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:05:12.488+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:05:12.600 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:05:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:05:12.488+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:05:12.600 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:05:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:05:12.340+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:05:12.600 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:05:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:05:12.340+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:05:12.600 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:05:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:05:12.340+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:05:12.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:05:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:05:12.599+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:05:12.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:05:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:05:12.599+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:05:12.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:05:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:05:12.599+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:05:12.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:05:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:05:12.600+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:05:13.627 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:05:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:05:13.474+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:05:13.628 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:05:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:05:13.474+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:05:13.628 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:05:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:05:13.474+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:05:13.628 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:05:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:05:13.338+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:05:13.628 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:05:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:05:13.338+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:05:13.628 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:05:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:05:13.338+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:05:13.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:05:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:05:13.627+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:05:13.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:05:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:05:13.627+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:05:13.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:05:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:05:13.627+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:05:13.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:05:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:05:13.627+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:05:14.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:05:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:05:14.458+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:05:14.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:05:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:05:14.458+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:05:14.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:05:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:05:14.458+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:05:14.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:05:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:05:14.610+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:05:14.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:05:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:05:14.610+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:05:14.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:05:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:05:14.610+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:05:14.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:05:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:05:14.610+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:05:14.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:05:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:05:14.387+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:05:14.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:05:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:05:14.388+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:05:14.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:05:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:05:14.388+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:05:15.643 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:05:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:05:15.506+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:05:15.643 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:05:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:05:15.506+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:05:15.643 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:05:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:05:15.506+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:05:15.643 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:05:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:05:15.354+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:05:15.643 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:05:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:05:15.354+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:05:15.643 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:05:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:05:15.354+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:05:15.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:05:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:05:15.642+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:05:15.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:05:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:05:15.642+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:05:15.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:05:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:05:15.642+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:05:15.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:05:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:05:15.642+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:05:16.663 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:05:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:05:16.554+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:05:16.663 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:05:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:05:16.554+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:05:16.663 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:05:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:05:16.554+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:05:16.664 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:05:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:05:16.397+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:05:16.664 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:05:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:05:16.397+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:05:16.664 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:05:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:05:16.397+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:05:16.991 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:05:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:05:16.749+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 3890 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:05:16.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:05:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:05:16.663+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:05:16.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:05:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:05:16.663+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:05:16.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:05:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:05:16.663+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:05:16.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:05:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:05:16.663+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:05:17.625 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:05:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:05:17.529+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:05:17.625 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:05:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:05:17.529+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:05:17.625 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:05:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:05:17.529+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:05:17.626 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:05:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:05:17.366+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:05:17.626 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:05:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:05:17.366+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:05:17.626 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:05:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:05:17.366+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:05:17.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:05:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:05:17.625+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:05:17.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:05:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:05:17.625+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:05:17.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:05:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:05:17.625+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:05:17.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:05:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:05:17.625+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:05:18.619 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:05:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:05:18.514+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:05:18.620 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:05:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:05:18.514+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:05:18.620 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:05:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:05:18.514+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:05:18.620 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:05:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:05:18.367+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:05:18.620 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:05:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:05:18.367+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:05:18.620 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:05:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:05:18.367+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:05:18.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:05:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:05:18.619+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:05:18.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:05:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:05:18.619+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:05:18.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:05:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:05:18.619+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:05:18.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:05:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:05:18.619+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:05:19.666 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:05:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:05:19.516+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:05:19.666 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:05:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:05:19.516+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:05:19.666 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:05:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:05:19.516+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:05:19.666 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:05:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:05:19.319+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:05:19.667 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:05:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:05:19.319+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:05:19.667 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:05:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:05:19.320+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:05:19.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:05:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:05:19.666+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:05:19.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:05:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:05:19.666+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:05:19.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:05:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:05:19.666+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:05:19.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:05:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:05:19.666+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:05:20.638 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:05:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:05:20.537+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:05:20.639 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:05:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:05:20.537+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:05:20.639 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:05:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:05:20.537+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:05:20.639 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:05:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:05:20.307+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:05:20.639 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:05:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:05:20.307+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:05:20.639 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:05:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:05:20.307+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:05:20.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:05:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:05:20.638+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:05:20.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:05:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:05:20.638+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:05:20.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:05:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:05:20.638+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:05:20.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:05:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:05:20.638+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:05:21.654 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:05:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:05:21.351+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:05:21.654 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:05:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:05:21.351+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:05:21.654 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:05:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:05:21.351+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:05:21.654 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:05:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:05:21.582+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:05:21.654 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:05:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:05:21.582+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:05:21.655 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:05:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:05:21.582+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:05:21.991 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:05:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:05:21.750+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 3972 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:05:21.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:05:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:05:21.679+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:05:21.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:05:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:05:21.679+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:05:21.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:05:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:05:21.679+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:05:21.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:05:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:05:21.679+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:05:22.713 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:05:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:05:22.587+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:05:22.713 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:05:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:05:22.587+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:05:22.713 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:05:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:05:22.587+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:05:22.713 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:05:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:05:22.365+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:05:22.713 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:05:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:05:22.365+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:05:22.713 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:05:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:05:22.365+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:05:22.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:05:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:05:22.712+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:05:22.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:05:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:05:22.712+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:05:22.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:05:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:05:22.712+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:05:22.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:05:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:05:22.713+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:05:23.629 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:05:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:05:23.320+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:05:23.629 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:05:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:05:23.320+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:05:23.629 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:05:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:05:23.320+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:05:23.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:05:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:05:23.629+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:05:23.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:05:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:05:23.629+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:05:23.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:05:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:05:23.629+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:05:23.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:05:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:05:23.736+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:05:23.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:05:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:05:23.736+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:05:23.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:05:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:05:23.736+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:05:23.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:05:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:05:23.736+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:05:24.581 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:05:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:05:24.285+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:05:24.581 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:05:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:05:24.285+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:05:24.581 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:05:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:05:24.285+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:05:24.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:05:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:05:24.580+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:05:24.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:05:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:05:24.580+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:05:24.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:05:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:05:24.580+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:05:24.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:05:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:05:24.730+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:05:24.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:05:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:05:24.730+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:05:24.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:05:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:05:24.730+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:05:24.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:05:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:05:24.730+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:05:25.608 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:05:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:05:25.281+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:05:25.608 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:05:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:05:25.281+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:05:25.608 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:05:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:05:25.281+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:05:25.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:05:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:05:25.608+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:05:25.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:05:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:05:25.608+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:05:25.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:05:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:05:25.608+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:05:25.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:05:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:05:25.752+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:05:25.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:05:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:05:25.752+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:05:25.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:05:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:05:25.752+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:05:25.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:05:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:05:25.752+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:05:26.587 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:05:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:05:26.321+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:05:26.588 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:05:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:05:26.321+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:05:26.588 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:05:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:05:26.321+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:05:26.877 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:05:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:05:26.751+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 4058 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:05:26.877 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:05:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:05:26.587+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:05:26.877 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:05:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:05:26.587+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:05:26.878 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:05:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:05:26.587+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:05:26.878 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:05:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:05:26.742+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:05:26.878 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:05:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:05:26.742+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:05:26.878 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:05:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:05:26.742+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:05:26.878 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:05:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:05:26.742+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:05:27.637 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:05:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:05:27.282+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:05:27.637 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:05:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:05:27.282+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:05:27.638 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:05:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:05:27.282+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:05:27.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:05:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:05:27.637+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:05:27.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:05:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:05:27.637+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:05:27.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:05:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:05:27.637+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:05:27.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:05:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:05:27.701+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:05:27.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:05:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:05:27.701+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:05:27.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:05:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:05:27.701+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:05:27.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:05:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:05:27.702+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:05:28.671 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:05:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:05:28.323+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:05:28.672 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:05:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:05:28.323+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:05:28.672 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:05:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:05:28.323+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:05:28.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:05:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:05:28.671+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:05:28.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:05:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:05:28.671+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:05:28.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:05:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:05:28.671+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:05:28.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:05:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:05:28.702+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:05:28.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:05:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:05:28.702+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:05:28.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:05:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:05:28.702+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:05:28.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:05:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:05:28.702+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:05:29.667 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:05:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:05:29.363+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:05:29.667 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:05:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:05:29.363+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:05:29.667 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:05:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:05:29.363+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:05:29.976 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:05:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:05:29.667+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:05:29.976 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:05:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:05:29.667+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:05:29.976 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:05:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:05:29.667+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:05:29.976 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:05:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:05:29.706+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:05:29.977 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:05:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:05:29.706+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:05:29.977 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:05:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:05:29.706+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:05:29.977 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:05:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:05:29.706+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:05:30.659 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:05:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:05:30.380+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:05:30.659 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:05:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:05:30.380+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:05:30.659 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:05:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:05:30.380+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:05:30.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:05:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:05:30.658+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:05:30.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:05:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:05:30.658+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:05:30.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:05:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:05:30.658+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:05:30.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:05:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:05:30.664+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:05:30.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:05:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:05:30.664+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:05:30.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:05:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:05:30.664+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:05:30.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:05:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:05:30.664+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:05:31.617 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:05:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:05:31.339+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:05:31.617 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:05:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:05:31.339+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:05:31.618 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:05:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:05:31.339+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:05:31.877 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:05:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:05:31.752+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 4143 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:05:31.877 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:05:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:05:31.662+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:05:31.877 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:05:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:05:31.662+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:05:31.877 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:05:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:05:31.662+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:05:31.877 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:05:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:05:31.617+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:05:31.877 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:05:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:05:31.617+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:05:31.878 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:05:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:05:31.617+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:05:31.878 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:05:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:05:31.617+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:05:32.604 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:05:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:05:32.315+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:05:32.604 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:05:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:05:32.315+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:05:32.604 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:05:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:05:32.315+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:05:32.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:05:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:05:32.603+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:05:32.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:05:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:05:32.603+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:05:32.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:05:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:05:32.603+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:05:32.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:05:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:05:32.603+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:05:32.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:05:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:05:32.626+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:05:32.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:05:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:05:32.626+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:05:32.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:05:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:05:32.626+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:05:33.600 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:05:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:05:33.332+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:05:33.600 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:05:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:05:33.332+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:05:33.600 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:05:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:05:33.332+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:05:33.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:05:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:05:33.599+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:05:33.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:05:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:05:33.599+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:05:33.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:05:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:05:33.599+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:05:33.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:05:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:05:33.638+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:05:33.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:05:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:05:33.638+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:05:33.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:05:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:05:33.638+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:05:33.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:05:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:05:33.638+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:05:34.580 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:05:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:05:34.292+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:05:34.580 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:05:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:05:34.292+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:05:34.580 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:05:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:05:34.292+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:05:34.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:05:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:05:34.578+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:05:34.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:05:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:05:34.578+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:05:34.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:05:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:05:34.578+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:05:34.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:05:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:05:34.592+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:05:34.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:05:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:05:34.592+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:05:34.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:05:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:05:34.592+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:05:34.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:05:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:05:34.592+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:05:35.633 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:05:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:05:35.568+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:05:35.634 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:05:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:05:35.568+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:05:35.634 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:05:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:05:35.568+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:05:35.634 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:05:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:05:35.320+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:05:35.634 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:05:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:05:35.320+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:05:35.634 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:05:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:05:35.320+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:05:35.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:05:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:05:35.633+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:05:35.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:05:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:05:35.633+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:05:35.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:05:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:05:35.633+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:05:35.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:05:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:05:35.633+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:05:36.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:05:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:05:36.307+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:05:36.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:05:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:05:36.307+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:05:36.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:05:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:05:36.307+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:05:36.754 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:05:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:05:36.585+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:05:36.754 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:05:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:05:36.585+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:05:36.754 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:05:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:05:36.585+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:05:36.754 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:05:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:05:36.619+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:05:36.754 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:05:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:05:36.619+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:05:36.754 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:05:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:05:36.619+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:05:36.755 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:05:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:05:36.619+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:05:37.241 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:05:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:05:36.752+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 4232 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:05:37.555 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:05:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:05:37.285+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:05:37.555 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:05:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:05:37.285+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:05:37.555 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:05:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:05:37.285+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:05:37.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:05:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:05:37.612+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:05:37.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:05:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:05:37.612+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:05:37.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:05:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:05:37.612+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:05:37.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:05:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:05:37.612+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:05:37.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:05:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:05:37.554+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:05:37.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:05:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:05:37.554+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:05:37.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:05:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:05:37.554+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:05:38.569 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:05:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:05:38.248+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:05:38.569 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:05:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:05:38.248+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:05:38.569 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:05:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:05:38.248+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:05:38.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:05:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:05:38.569+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:05:38.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:05:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:05:38.569+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:05:38.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:05:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:05:38.569+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:05:38.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:05:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:05:38.569+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:05:38.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:05:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:05:38.567+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:05:38.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:05:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:05:38.567+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:05:38.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:05:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:05:38.568+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:05:39.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:05:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:05:39.256+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:05:39.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:05:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:05:39.256+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:05:39.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:05:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:05:39.256+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:05:39.885 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:05:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:05:39.521+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:05:39.885 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:05:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:05:39.521+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:05:39.885 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:05:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:05:39.521+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:05:39.885 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:05:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:05:39.544+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:05:39.885 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:05:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:05:39.544+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:05:39.885 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:05:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:05:39.544+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:05:39.885 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:05:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:05:39.544+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:05:40.545 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:05:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:05:40.274+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:05:40.546 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:05:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:05:40.274+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:05:40.546 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:05:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:05:40.274+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:05:40.546 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:05:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:05:40.482+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:05:40.546 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:05:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:05:40.482+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:05:40.546 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:05:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:05:40.482+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:05:40.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:05:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:05:40.545+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:05:40.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:05:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:05:40.545+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:05:40.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:05:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:05:40.545+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:05:40.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:05:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:05:40.545+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:05:41.567 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:05:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:05:41.506+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:05:41.568 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:05:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:05:41.506+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:05:41.568 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:05:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:05:41.506+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:05:41.568 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:05:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:05:41.271+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:05:41.568 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:05:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:05:41.271+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:05:41.568 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:05:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:05:41.271+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:05:41.879 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:05:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:05:41.754+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 4323 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:05:41.879 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:05:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:05:41.566+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:05:41.879 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:05:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:05:41.566+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:05:41.879 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:05:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:05:41.566+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:05:41.879 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:05:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:05:41.566+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:05:42.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:05:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:05:42.486+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:05:42.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:05:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:05:42.486+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:05:42.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:05:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:05:42.486+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:05:42.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:05:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:05:42.238+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:05:42.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:05:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:05:42.238+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:05:42.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:05:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:05:42.238+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:05:42.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:05:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:05:42.587+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:05:42.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:05:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:05:42.587+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:05:42.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:05:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:05:42.587+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:05:42.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:05:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:05:42.587+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:05:43.547 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:05:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:05:43.490+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:05:43.548 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:05:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:05:43.490+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:05:43.548 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:05:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:05:43.490+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:05:43.548 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:05:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:05:43.251+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:05:43.548 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:05:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:05:43.251+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:05:43.548 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:05:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:05:43.251+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:05:43.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:05:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:05:43.602+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:05:43.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:05:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:05:43.602+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:05:43.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:05:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:05:43.602+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:05:43.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:05:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:05:43.602+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:05:44.516 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:05:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:05:44.260+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:05:44.516 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:05:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:05:44.260+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:05:44.516 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:05:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:05:44.260+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:05:44.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:05:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:05:44.515+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:05:44.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:05:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:05:44.515+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:05:44.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:05:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:05:44.515+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:05:44.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:05:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:05:44.632+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:05:44.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:05:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:05:44.632+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:05:44.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:05:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:05:44.632+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:05:44.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:05:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:05:44.632+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:05:45.596 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:05:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:05:45.467+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:05:45.596 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:05:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:05:45.467+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:05:45.596 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:05:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:05:45.467+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:05:45.597 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:05:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:05:45.255+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:05:45.597 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:05:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:05:45.255+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:05:45.597 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:05:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:05:45.255+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:05:45.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:05:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:05:45.595+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:05:45.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:05:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:05:45.595+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:05:45.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:05:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:05:45.595+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:05:45.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:05:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:05:45.595+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:05:46.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:05:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:05:46.228+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:05:46.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:05:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:05:46.228+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:05:46.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:05:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:05:46.228+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:05:46.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:05:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:05:46.422+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:05:46.492 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:05:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:05:46.422+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:05:46.492 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:05:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:05:46.422+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:05:46.882 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:05:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:05:46.755+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 4405 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:05:46.882 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:05:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:05:46.587+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:05:46.883 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:05:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:05:46.587+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:05:46.883 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:05:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:05:46.587+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:05:46.883 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:05:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:05:46.587+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:05:47.606 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:05:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:05:47.268+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:05:47.606 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:05:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:05:47.268+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:05:47.607 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:05:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:05:47.268+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:05:47.607 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:05:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:05:47.383+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:05:47.607 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:05:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:05:47.383+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:05:47.607 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:05:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:05:47.383+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:05:47.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:05:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:05:47.605+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:05:47.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:05:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:05:47.605+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:05:47.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:05:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:05:47.605+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:05:47.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:05:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:05:47.605+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:05:48.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:05:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:05:48.393+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:05:48.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:05:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:05:48.393+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:05:48.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:05:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:05:48.393+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:05:48.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:05:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:05:48.229+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:05:48.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:05:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:05:48.229+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:05:48.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:05:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:05:48.229+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:05:48.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:05:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:05:48.595+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:05:48.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:05:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:05:48.595+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:05:48.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:05:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:05:48.595+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:05:48.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:05:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:05:48.595+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:05:49.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:05:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:05:49.396+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:05:49.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:05:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:05:49.396+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:05:49.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:05:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:05:49.396+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:05:49.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:05:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:05:49.233+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:05:49.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:05:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:05:49.233+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:05:49.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:05:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:05:49.233+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:05:49.885 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:05:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:05:49.569+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:05:49.885 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:05:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:05:49.569+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:05:49.885 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:05:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:05:49.569+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:05:49.885 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:05:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:05:49.569+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:05:50.583 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:05:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:05:50.377+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:05:50.583 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:05:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:05:50.377+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:05:50.583 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:05:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:05:50.377+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:05:50.584 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:05:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:05:50.272+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:05:50.584 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:05:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:05:50.272+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:05:50.584 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:05:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:05:50.272+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:05:50.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:05:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:05:50.582+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:05:50.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:05:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:05:50.582+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:05:50.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:05:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:05:50.582+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:05:50.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:05:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:05:50.582+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:05:51.588 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:05:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:05:51.425+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:05:51.588 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:05:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:05:51.425+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:05:51.588 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:05:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:05:51.425+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:05:51.588 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:05:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:05:51.257+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:05:51.588 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:05:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:05:51.257+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:05:51.588 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:05:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:05:51.257+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:05:51.884 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:05:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:05:51.756+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 4500 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:05:51.884 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:05:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:05:51.587+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:05:51.884 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:05:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:05:51.587+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:05:51.884 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:05:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:05:51.587+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:05:51.884 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:05:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:05:51.587+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:05:52.613 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:05:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:05:52.466+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:05:52.613 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:05:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:05:52.466+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:05:52.613 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:05:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:05:52.466+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:05:52.613 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:05:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:05:52.264+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:05:52.613 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:05:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:05:52.264+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:05:52.613 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:05:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:05:52.264+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:05:52.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:05:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:05:52.612+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:05:52.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:05:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:05:52.612+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:05:52.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:05:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:05:52.612+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:05:52.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:05:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:05:52.612+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:05:53.548 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:05:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:05:53.251+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:05:53.548 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:05:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:05:53.251+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:05:53.548 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:05:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:05:53.251+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:05:53.548 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:05:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:05:53.484+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:05:53.548 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:05:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:05:53.484+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:05:53.548 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:05:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:05:53.484+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:05:53.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:05:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:05:53.628+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:05:53.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:05:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:05:53.628+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:05:53.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:05:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:05:53.628+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:05:53.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:05:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:05:53.628+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:05:54.552 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:05:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:05:54.252+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:05:54.552 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:05:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:05:54.252+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:05:54.552 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:05:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:05:54.252+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:05:54.553 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:05:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:05:54.460+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:05:54.553 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:05:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:05:54.460+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:05:54.553 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:05:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:05:54.460+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:05:54.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:05:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:05:54.579+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:05:54.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:05:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:05:54.579+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:05:54.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:05:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:05:54.579+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:05:54.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:05:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:05:54.579+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:05:55.567 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:05:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:05:55.454+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:05:55.567 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:05:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:05:55.454+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:05:55.567 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:05:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:05:55.454+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:05:55.567 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:05:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:05:55.286+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:05:55.567 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:05:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:05:55.286+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:05:55.567 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:05:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:05:55.286+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:05:55.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:05:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:05:55.565+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:05:55.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:05:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:05:55.565+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:05:55.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:05:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:05:55.565+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:05:55.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:05:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:05:55.566+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:05:56.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:05:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:05:56.411+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:05:56.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:05:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:05:56.411+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:05:56.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:05:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:05:56.411+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:05:56.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:05:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:05:56.329+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:05:56.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:05:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:05:56.329+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:05:56.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:05:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:05:56.329+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:05:56.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:05:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:05:56.563+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:05:56.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:05:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:05:56.563+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:05:56.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:05:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:05:56.563+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:05:56.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:05:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:05:56.563+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:05:57.241 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:05:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:05:56.757+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 4583 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:05:57.592 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:05:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:05:57.436+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:05:57.592 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:05:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:05:57.436+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:05:57.592 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:05:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:05:57.436+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:05:57.593 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:05:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:05:57.331+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:05:57.593 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:05:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:05:57.331+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:05:57.593 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:05:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:05:57.331+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:05:57.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:05:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:05:57.592+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:05:57.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:05:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:05:57.592+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:05:57.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:05:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:05:57.592+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:05:57.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:05:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:05:57.592+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:05:58.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:05:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:05:58.376+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:05:58.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:05:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:05:58.376+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:05:58.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:05:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:05:58.376+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:05:58.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:05:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:05:58.406+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:05:58.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:05:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:05:58.406+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:05:58.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:05:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:05:58.406+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:05:58.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:05:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:05:58.560+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:05:58.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:05:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:05:58.560+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:05:58.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:05:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:05:58.560+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:05:58.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:05:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:05:58.560+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:05:59.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:05:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:05:59.381+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:05:59.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:05:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:05:59.381+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:05:59.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:05:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:05:59.381+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:05:59.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:05:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:05:59.408+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:05:59.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:05:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:05:59.408+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:05:59.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:05:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:05:59.408+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:05:59.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:05:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:05:59.534+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:05:59.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:05:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:05:59.534+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:05:59.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:05:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:05:59.534+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:05:59.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:05:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:05:59.534+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:06:00.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:06:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:06:00.377+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:06:00.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:06:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:06:00.377+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:06:00.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:06:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:06:00.377+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:06:00.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:06:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:06:00.521+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:06:00.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:06:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:06:00.521+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:06:00.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:06:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:06:00.521+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:06:00.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:06:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:06:00.521+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:06:00.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:06:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:06:00.397+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:06:00.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:06:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:06:00.397+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:06:00.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:06:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:06:00.397+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:06:01.657 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:06:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:06:01.377+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:06:01.657 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:06:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:06:01.377+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:06:01.657 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:06:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:06:01.377+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:06:01.657 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:06:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:06:01.377+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:06:01.658 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:06:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:06:01.377+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:06:01.658 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:06:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:06:01.377+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:06:01.658 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:06:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:06:01.549+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:06:01.658 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:06:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:06:01.549+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:06:01.658 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:06:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:06:01.549+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:06:01.658 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:06:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:06:01.549+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:06:01.991 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:06:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:06:01.758+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 4665 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:06:02.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:06:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:06:02.396+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:06:02.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:06:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:06:02.396+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:06:02.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:06:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:06:02.396+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:06:02.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:06:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:06:02.350+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:06:02.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:06:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:06:02.350+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:06:02.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:06:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:06:02.350+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:06:02.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:06:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:06:02.587+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:06:02.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:06:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:06:02.587+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:06:02.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:06:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:06:02.587+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:06:02.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:06:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:06:02.587+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:06:03.639 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:06:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:06:03.423+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:06:03.639 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:06:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:06:03.423+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:06:03.639 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:06:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:06:03.423+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:06:03.639 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:06:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:06:03.316+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:06:03.639 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:06:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:06:03.316+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:06:03.639 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:06:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:06:03.316+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:06:03.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:06:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:06:03.637+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:06:03.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:06:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:06:03.637+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:06:03.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:06:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:06:03.637+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:06:03.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:06:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:06:03.638+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:06:04.660 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:06:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:06:04.291+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:06:04.660 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:06:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:06:04.291+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:06:04.660 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:06:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:06:04.291+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:06:04.661 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:06:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:06:04.381+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:06:04.661 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:06:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:06:04.381+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:06:04.661 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:06:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:06:04.381+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:06:04.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:06:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:06:04.660+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:06:04.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:06:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:06:04.660+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:06:04.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:06:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:06:04.660+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:06:04.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:06:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:06:04.660+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:06:05.613 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:06:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:06:05.280+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:06:05.613 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:06:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:06:05.280+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:06:05.613 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:06:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:06:05.280+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:06:05.613 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:06:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:06:05.381+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:06:05.614 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:06:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:06:05.381+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:06:05.614 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:06:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:06:05.381+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:06:05.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:06:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:06:05.612+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:06:05.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:06:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:06:05.612+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:06:05.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:06:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:06:05.613+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:06:05.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:06:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:06:05.613+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:06:06.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:06:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:06:06.419+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:06:06.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:06:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:06:06.419+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:06:06.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:06:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:06:06.419+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:06:06.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:06:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:06:06.238+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:06:06.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:06:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:06:06.238+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:06:06.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:06:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:06:06.238+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:06:06.761 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:06:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:06:06.569+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:06:06.761 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:06:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:06:06.569+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:06:06.761 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:06:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:06:06.569+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:06:06.761 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:06:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:06:06.569+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:06:07.241 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:06:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:06:06.760+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 4752 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:06:07.588 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:06:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:06:07.379+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:06:07.589 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:06:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:06:07.379+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:06:07.589 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:06:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:06:07.379+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:06:07.589 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:06:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:06:07.267+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:06:07.589 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:06:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:06:07.267+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:06:07.589 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:06:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:06:07.267+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:06:07.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:06:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:06:07.588+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:06:07.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:06:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:06:07.588+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:06:07.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:06:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:06:07.588+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:06:07.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:06:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:06:07.588+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:06:08.606 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:06:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:06:08.310+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:06:08.606 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:06:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:06:08.310+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:06:08.606 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:06:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:06:08.310+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:06:08.606 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:06:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:06:08.344+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:06:08.607 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:06:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:06:08.344+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:06:08.607 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:06:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:06:08.344+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:06:08.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:06:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:06:08.605+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:06:08.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:06:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:06:08.605+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:06:08.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:06:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:06:08.605+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:06:08.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:06:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:06:08.605+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:06:09.576 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:06:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:06:09.307+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:06:09.576 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:06:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:06:09.307+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:06:09.576 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:06:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:06:09.307+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:06:09.577 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:06:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:06:09.296+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:06:09.577 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:06:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:06:09.296+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:06:09.577 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:06:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:06:09.296+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:06:09.885 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:06:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:06:09.576+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:06:09.885 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:06:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:06:09.576+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:06:09.885 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:06:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:06:09.576+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:06:09.885 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:06:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:06:09.576+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:06:10.609 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:06:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:06:10.308+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:06:10.609 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:06:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:06:10.308+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:06:10.609 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:06:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:06:10.308+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:06:10.610 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:06:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:06:10.264+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:06:10.610 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:06:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:06:10.264+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:06:10.610 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:06:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:06:10.264+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:06:10.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:06:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:06:10.609+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:06:10.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:06:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:06:10.609+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:06:10.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:06:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:06:10.609+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:06:10.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:06:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:06:10.609+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:06:11.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:06:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:06:11.239+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:06:11.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:06:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:06:11.239+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:06:11.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:06:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:06:11.239+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:06:11.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:06:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:06:11.349+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:06:11.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:06:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:06:11.349+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:06:11.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:06:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:06:11.349+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:06:11.888 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:06:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:06:11.762+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 4858 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:06:11.888 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:06:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:06:11.590+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:06:11.888 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:06:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:06:11.590+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:06:11.888 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:06:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:06:11.590+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:06:11.888 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:06:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:06:11.590+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:06:12.562 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:06:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:06:12.324+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:06:12.562 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:06:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:06:12.324+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:06:12.562 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:06:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:06:12.324+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:06:12.562 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:06:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:06:12.250+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:06:12.563 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:06:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:06:12.250+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:06:12.563 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:06:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:06:12.250+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:06:12.983 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:06:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:06:12.562+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:06:12.983 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:06:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:06:12.562+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:06:12.983 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:06:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:06:12.562+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:06:12.983 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:06:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:06:12.562+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:06:13.549 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:06:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:06:13.244+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:06:13.549 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:06:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:06:13.244+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:06:13.550 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:06:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:06:13.244+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:06:13.550 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:06:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:06:13.326+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:06:13.550 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:06:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:06:13.326+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:06:13.550 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:06:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:06:13.326+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:06:13.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:06:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:06:13.612+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:06:13.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:06:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:06:13.612+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:06:13.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:06:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:06:13.612+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:06:13.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:06:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:06:13.612+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:06:14.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:06:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:06:14.198+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:06:14.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:06:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:06:14.198+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:06:14.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:06:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:06:14.198+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:06:14.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:06:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:06:14.339+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:06:14.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:06:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:06:14.339+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:06:14.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:06:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:06:14.339+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:06:14.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:06:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:06:14.604+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:06:14.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:06:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:06:14.604+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:06:14.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:06:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:06:14.604+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:06:14.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:06:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:06:14.604+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:06:15.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:06:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:06:15.208+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:06:15.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:06:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:06:15.208+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:06:15.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:06:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:06:15.208+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:06:15.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:06:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:06:15.367+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:06:15.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:06:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:06:15.367+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:06:15.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:06:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:06:15.367+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:06:15.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:06:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:06:15.583+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:06:15.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:06:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:06:15.583+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:06:15.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:06:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:06:15.583+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:06:15.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:06:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:06:15.583+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:06:16.346 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:06:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:06:16.218+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:06:16.347 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:06:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:06:16.218+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:06:16.347 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:06:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:06:16.218+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:06:16.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:06:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:06:16.346+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:06:16.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:06:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:06:16.346+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:06:16.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:06:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:06:16.346+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:06:16.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:06:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:06:16.557+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:06:16.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:06:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:06:16.557+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:06:16.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:06:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:06:16.557+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:06:16.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:06:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:06:16.557+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:06:17.231 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:06:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:06:16.763+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 4957 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:06:17.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:06:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:06:17.228+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:06:17.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:06:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:06:17.228+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:06:17.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:06:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:06:17.228+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:06:17.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:06:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:06:17.396+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:06:17.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:06:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:06:17.396+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:06:17.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:06:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:06:17.396+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:06:17.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:06:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:06:17.524+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:06:17.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:06:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:06:17.524+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:06:17.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:06:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:06:17.524+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:06:17.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:06:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:06:17.524+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:06:18.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:06:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:06:18.224+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:06:18.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:06:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:06:18.224+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:06:18.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:06:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:06:18.224+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:06:18.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:06:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:06:18.417+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:06:18.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:06:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:06:18.417+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:06:18.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:06:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:06:18.417+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:06:18.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:06:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:06:18.561+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:06:18.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:06:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:06:18.561+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:06:18.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:06:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:06:18.561+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:06:18.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:06:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:06:18.561+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:06:19.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:06:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:06:19.197+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:06:19.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:06:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:06:19.197+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:06:19.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:06:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:06:19.197+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:06:19.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:06:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:06:19.391+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:06:19.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:06:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:06:19.391+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:06:19.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:06:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:06:19.391+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:06:19.885 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:06:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:06:19.575+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:06:19.885 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:06:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:06:19.575+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:06:19.885 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:06:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:06:19.575+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:06:19.886 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:06:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:06:19.575+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:06:20.567 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:06:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:06:20.247+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:06:20.567 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:06:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:06:20.247+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:06:20.567 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:06:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:06:20.247+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:06:20.568 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:06:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:06:20.419+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:06:20.568 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:06:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:06:20.419+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:06:20.568 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:06:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:06:20.419+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:06:20.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:06:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:06:20.567+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:06:20.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:06:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:06:20.567+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:06:20.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:06:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:06:20.567+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:06:20.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:06:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:06:20.567+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:06:21.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:06:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:06:21.231+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:06:21.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:06:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:06:21.231+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:06:21.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:06:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:06:21.231+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:06:21.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:06:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:06:21.389+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:06:21.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:06:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:06:21.389+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:06:21.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:06:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:06:21.389+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:06:21.887 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:06:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:06:21.764+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 5039 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:06:21.887 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:06:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:06:21.585+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:06:21.887 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:06:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:06:21.585+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:06:21.887 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:06:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:06:21.585+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:06:21.887 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:06:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:06:21.585+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:06:22.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:06:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:06:22.186+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:06:22.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:06:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:06:22.186+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:06:22.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:06:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:06:22.186+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:06:22.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:06:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:06:22.435+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:06:22.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:06:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:06:22.435+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:06:22.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:06:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:06:22.435+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:06:22.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:06:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:06:22.590+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:06:22.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:06:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:06:22.590+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:06:22.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:06:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:06:22.590+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:06:22.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:06:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:06:22.590+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:06:23.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:06:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:06:23.211+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:06:23.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:06:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:06:23.211+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:06:23.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:06:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:06:23.211+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:06:23.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:06:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:06:23.432+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:06:23.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:06:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:06:23.432+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:06:23.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:06:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:06:23.432+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:06:23.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:06:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:06:23.598+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:06:23.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:06:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:06:23.598+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:06:23.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:06:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:06:23.598+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:06:23.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:06:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:06:23.598+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:06:24.648 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:06:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:06:24.250+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:06:24.648 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:06:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:06:24.250+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:06:24.648 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:06:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:06:24.250+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:06:24.649 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:06:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:06:24.458+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:06:24.649 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:06:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:06:24.458+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:06:24.649 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:06:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:06:24.458+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:06:24.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:06:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:06:24.648+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:06:24.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:06:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:06:24.648+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:06:24.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:06:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:06:24.648+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:06:24.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:06:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:06:24.648+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:06:25.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:06:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:06:25.437+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:06:25.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:06:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:06:25.437+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:06:25.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:06:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:06:25.437+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:06:25.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:06:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:06:25.209+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:06:25.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:06:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:06:25.209+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:06:25.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:06:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:06:25.209+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:06:25.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:06:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:06:25.598+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:06:25.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:06:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:06:25.598+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:06:25.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:06:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:06:25.598+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:06:25.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:06:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:06:25.599+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:06:26.595 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:06:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:06:26.420+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:06:26.595 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:06:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:06:26.420+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:06:26.595 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:06:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:06:26.420+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:06:26.595 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:06:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:06:26.242+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:06:26.595 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:06:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:06:26.242+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:06:26.595 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:06:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:06:26.242+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:06:26.889 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:06:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:06:26.764+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 5121 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:06:26.889 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:06:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:06:26.594+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:06:26.889 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:06:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:06:26.594+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:06:26.889 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:06:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:06:26.594+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:06:26.889 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:06:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:06:26.594+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:06:27.613 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:06:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:06:27.265+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:06:27.613 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:06:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:06:27.265+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:06:27.613 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:06:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:06:27.265+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:06:27.613 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:06:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:06:27.458+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:06:27.613 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:06:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:06:27.458+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:06:27.613 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:06:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:06:27.458+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:06:27.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:06:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:06:27.611+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:06:27.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:06:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:06:27.611+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:06:27.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:06:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:06:27.611+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:06:27.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:06:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:06:27.612+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:06:28.592 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:06:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:06:28.280+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:06:28.592 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:06:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:06:28.280+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:06:28.592 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:06:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:06:28.280+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:06:28.592 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:06:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:06:28.432+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:06:28.593 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:06:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:06:28.432+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:06:28.593 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:06:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:06:28.432+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:06:28.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:06:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:06:28.591+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:06:28.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:06:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:06:28.591+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:06:28.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:06:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:06:28.591+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:06:28.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:06:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:06:28.591+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:06:29.542 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:06:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:06:29.253+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:06:29.542 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:06:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:06:29.253+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:06:29.542 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:06:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:06:29.254+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:06:29.543 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:06:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:06:29.454+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:06:29.543 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:06:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:06:29.454+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:06:29.543 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:06:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:06:29.454+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:06:29.884 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:06:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:06:29.541+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:06:29.885 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:06:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:06:29.541+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:06:29.885 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:06:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:06:29.541+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:06:29.885 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:06:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:06:29.541+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:06:30.587 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:06:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:06:30.254+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:06:30.587 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:06:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:06:30.254+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:06:30.587 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:06:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:06:30.254+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:06:30.587 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:06:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:06:30.422+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:06:30.588 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:06:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:06:30.422+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:06:30.588 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:06:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:06:30.422+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:06:30.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:06:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:06:30.586+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:06:30.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:06:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:06:30.586+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:06:30.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:06:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:06:30.586+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:06:30.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:06:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:06:30.586+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:06:31.544 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:06:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:06:31.266+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:06:31.545 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:06:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:06:31.266+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:06:31.545 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:06:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:06:31.266+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:06:31.545 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:06:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:06:31.395+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:06:31.545 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:06:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:06:31.395+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:06:31.545 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:06:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:06:31.395+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:06:31.892 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:06:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:06:31.766+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 5204 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:06:31.892 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:06:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:06:31.543+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:06:31.893 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:06:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:06:31.543+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:06:31.893 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:06:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:06:31.543+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:06:31.893 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:06:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:06:31.543+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:06:32.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:06:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:06:32.312+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:06:32.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:06:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:06:32.312+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:06:32.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:06:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:06:32.312+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:06:32.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:06:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:06:32.423+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:06:32.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:06:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:06:32.423+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:06:32.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:06:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:06:32.423+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:06:32.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:06:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:06:32.557+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:06:32.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:06:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:06:32.557+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:06:32.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:06:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:06:32.557+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:06:32.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:06:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:06:32.557+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:06:33.586 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:06:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:06:33.310+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:06:33.586 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:06:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:06:33.310+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:06:33.586 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:06:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:06:33.310+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:06:33.587 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:06:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:06:33.381+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:06:33.587 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:06:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:06:33.381+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:06:33.587 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:06:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:06:33.381+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:06:33.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:06:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:06:33.585+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:06:33.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:06:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:06:33.585+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:06:33.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:06:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:06:33.585+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:06:33.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:06:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:06:33.585+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:06:34.635 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:06:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:06:34.383+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:06:34.636 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:06:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:06:34.383+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:06:34.636 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:06:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:06:34.383+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:06:34.636 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:06:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:06:34.263+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:06:34.636 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:06:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:06:34.263+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:06:34.636 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:06:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:06:34.263+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:06:34.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:06:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:06:34.635+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:06:34.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:06:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:06:34.635+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:06:34.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:06:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:06:34.635+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:06:34.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:06:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:06:34.635+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:06:35.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:06:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:06:35.218+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:06:35.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:06:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:06:35.218+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:06:35.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:06:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:06:35.218+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:06:35.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:06:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:06:35.360+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:06:35.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:06:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:06:35.360+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:06:35.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:06:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:06:35.360+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:06:35.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:06:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:06:35.678+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:06:35.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:06:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:06:35.678+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:06:35.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:06:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:06:35.678+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:06:35.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:06:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:06:35.678+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:06:36.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:06:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:06:36.199+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:06:36.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:06:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:06:36.199+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:06:36.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:06:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:06:36.199+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:06:36.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:06:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:06:36.405+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:06:36.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:06:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:06:36.405+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:06:36.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:06:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:06:36.405+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:06:36.768 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:06:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:06:36.665+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:06:36.768 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:06:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:06:36.665+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:06:36.768 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:06:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:06:36.665+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:06:36.769 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:06:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:06:36.665+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:06:37.170 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:06:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:06:36.767+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 5287 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:06:37.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:06:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:06:37.407+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:06:37.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:06:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:06:37.407+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:06:37.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:06:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:06:37.407+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:06:37.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:06:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:06:37.169+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:06:37.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:06:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:06:37.169+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:06:37.492 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:06:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:06:37.169+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:06:37.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:06:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:06:37.635+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:06:37.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:06:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:06:37.635+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:06:37.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:06:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:06:37.635+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:06:37.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:06:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:06:37.635+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:06:38.434 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:06:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:06:38.143+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:06:38.434 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:06:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:06:38.143+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:06:38.435 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:06:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:06:38.143+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:06:38.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:06:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:06:38.433+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:06:38.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:06:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:06:38.433+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:06:38.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:06:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:06:38.433+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:06:38.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:06:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:06:38.615+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:06:38.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:06:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:06:38.615+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:06:38.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:06:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:06:38.615+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:06:38.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:06:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:06:38.615+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:06:39.414 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:06:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:06:39.157+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:06:39.414 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:06:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:06:39.157+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:06:39.414 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:06:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:06:39.157+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:06:39.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:06:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:06:39.413+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:06:39.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:06:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:06:39.413+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:06:39.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:06:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:06:39.413+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:06:39.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:06:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:06:39.636+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:06:39.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:06:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:06:39.636+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:06:39.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:06:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:06:39.636+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:06:39.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:06:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:06:39.636+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:06:40.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:06:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:06:40.438+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:06:40.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:06:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:06:40.438+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:06:40.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:06:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:06:40.438+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:06:40.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:06:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:06:40.201+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:06:40.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:06:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:06:40.201+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:06:40.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:06:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:06:40.201+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:06:40.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:06:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:06:40.617+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:06:40.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:06:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:06:40.617+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:06:40.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:06:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:06:40.617+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:06:40.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:06:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:06:40.618+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:06:41.618 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:06:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:06:41.251+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:06:41.619 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:06:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:06:41.251+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:06:41.619 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:06:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:06:41.251+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:06:41.619 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:06:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:06:41.413+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:06:41.619 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:06:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:06:41.413+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:06:41.619 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:06:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:06:41.413+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:06:41.893 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:06:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:06:41.768+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 5369 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:06:41.893 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:06:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:06:41.618+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:06:41.893 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:06:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:06:41.618+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:06:41.893 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:06:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:06:41.618+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:06:41.894 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:06:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:06:41.618+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:06:42.667 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:06:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:06:42.285+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:06:42.667 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:06:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:06:42.285+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:06:42.667 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:06:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:06:42.285+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:06:42.667 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:06:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:06:42.375+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:06:42.667 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:06:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:06:42.375+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:06:42.667 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:06:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:06:42.375+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:06:42.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:06:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:06:42.666+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:06:42.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:06:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:06:42.666+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:06:42.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:06:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:06:42.666+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:06:42.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:06:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:06:42.666+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:06:43.664 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:06:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:06:43.320+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:06:43.664 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:06:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:06:43.320+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:06:43.664 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:06:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:06:43.320+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:06:43.664 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:06:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:06:43.416+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:06:43.664 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:06:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:06:43.416+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:06:43.664 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:06:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:06:43.416+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:06:43.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:06:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:06:43.662+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:06:43.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:06:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:06:43.662+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:06:43.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:06:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:06:43.662+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:06:43.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:06:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:06:43.662+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:06:44.660 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:06:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:06:44.327+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:06:44.660 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:06:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:06:44.327+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:06:44.660 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:06:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:06:44.327+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:06:44.660 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:06:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:06:44.409+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:06:44.660 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:06:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:06:44.409+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:06:44.660 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:06:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:06:44.409+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:06:44.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:06:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:06:44.658+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:06:44.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:06:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:06:44.658+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:06:44.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:06:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:06:44.658+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:06:44.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:06:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:06:44.658+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:06:45.627 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:06:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:06:45.300+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:06:45.627 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:06:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:06:45.300+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:06:45.627 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:06:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:06:45.300+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:06:45.628 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:06:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:06:45.449+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:06:45.628 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:06:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:06:45.449+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:06:45.628 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:06:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:06:45.449+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:06:45.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:06:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:06:45.626+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:06:45.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:06:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:06:45.626+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:06:45.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:06:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:06:45.626+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:06:45.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:06:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:06:45.626+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:06:46.608 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:06:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:06:46.343+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:06:46.608 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:06:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:06:46.343+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:06:46.608 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:06:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:06:46.343+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:06:46.608 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:06:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:06:46.407+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:06:46.608 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:06:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:06:46.407+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:06:46.608 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:06:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:06:46.407+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:06:46.898 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:06:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:06:46.770+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 5451 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:06:46.898 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:06:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:06:46.606+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:06:46.898 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:06:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:06:46.606+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:06:46.898 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:06:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:06:46.606+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:06:46.898 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:06:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:06:46.606+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:06:47.645 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:06:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:06:47.424+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:06:47.645 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:06:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:06:47.424+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:06:47.645 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:06:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:06:47.424+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:06:47.645 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:06:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:06:47.298+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:06:47.645 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:06:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:06:47.298+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:06:47.645 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:06:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:06:47.298+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:06:47.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:06:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:06:47.644+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:06:47.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:06:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:06:47.644+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:06:47.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:06:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:06:47.644+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:06:47.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:06:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:06:47.644+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:06:48.637 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:06:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:06:48.284+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:06:48.637 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:06:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:06:48.284+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:06:48.637 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:06:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:06:48.284+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:06:48.638 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:06:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:06:48.404+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:06:48.638 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:06:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:06:48.404+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:06:48.638 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:06:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:06:48.404+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:06:48.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:06:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:06:48.637+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:06:48.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:06:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:06:48.637+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:06:48.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:06:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:06:48.637+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:06:48.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:06:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:06:48.637+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:06:49.597 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:06:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:06:49.291+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:06:49.597 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:06:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:06:49.291+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:06:49.598 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:06:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:06:49.291+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:06:49.598 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:06:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:06:49.442+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:06:49.598 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:06:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:06:49.442+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:06:49.598 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:06:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:06:49.443+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:06:49.848 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:06:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:06:49.596+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:06:49.848 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:06:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:06:49.596+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:06:49.848 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:06:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:06:49.596+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:06:49.848 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:06:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:06:49.596+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:06:50.549 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:06:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:06:50.291+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:06:50.549 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:06:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:06:50.291+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:06:50.549 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:06:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:06:50.291+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:06:50.549 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:06:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:06:50.478+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:06:50.550 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:06:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:06:50.478+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:06:50.550 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:06:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:06:50.478+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:06:50.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:06:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:06:50.548+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:06:50.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:06:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:06:50.548+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:06:50.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:06:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:06:50.548+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:06:50.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:06:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:06:50.548+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:06:51.660 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:06:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:06:51.499+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:06:51.660 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:06:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:06:51.499+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:06:51.660 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:06:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:06:51.499+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:06:51.661 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:06:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:06:51.323+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:06:51.661 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:06:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:06:51.323+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:06:51.661 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:06:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:06:51.323+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:06:51.661 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:06:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:06:51.514+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:06:51.661 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:06:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:06:51.514+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:06:51.661 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:06:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:06:51.514+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:06:51.661 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:06:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:06:51.514+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:06:51.991 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:06:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:06:51.771+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 5534 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:06:52.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:06:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:06:52.472+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:06:52.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:06:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:06:52.472+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:06:52.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:06:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:06:52.472+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:06:52.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:06:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:06:52.340+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:06:52.742 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:06:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:06:52.340+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:06:52.742 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:06:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:06:52.340+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:06:52.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:06:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:06:52.514+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:06:52.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:06:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:06:52.514+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:06:52.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:06:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:06:52.514+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:06:52.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:06:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:06:52.514+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:06:53.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:06:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:06:53.440+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:06:53.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:06:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:06:53.440+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:06:53.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:06:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:06:53.440+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:06:53.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:06:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:06:53.369+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:06:53.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:06:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:06:53.369+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:06:53.742 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:06:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:06:53.369+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:06:53.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:06:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:06:53.498+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:06:53.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:06:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:06:53.498+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:06:53.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:06:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:06:53.498+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:06:53.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:06:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:06:53.498+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:06:54.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:06:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:06:54.386+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:06:54.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:06:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:06:54.386+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:06:54.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:06:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:06:54.386+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:06:54.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:06:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:06:54.480+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:06:54.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:06:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:06:54.480+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:06:54.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:06:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:06:54.480+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:06:54.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:06:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:06:54.482+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:06:54.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:06:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:06:54.482+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:06:54.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:06:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:06:54.482+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:06:54.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:06:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:06:54.482+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:06:55.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:06:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:06:55.375+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:06:55.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:06:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:06:55.375+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:06:55.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:06:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:06:55.375+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:06:55.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:06:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:06:55.483+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:06:55.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:06:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:06:55.483+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:06:55.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:06:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:06:55.483+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:06:55.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:06:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:06:55.447+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:06:55.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:06:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:06:55.447+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:06:55.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:06:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:06:55.447+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:06:55.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:06:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:06:55.447+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:06:56.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:06:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:06:56.487+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:06:56.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:06:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:06:56.487+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:06:56.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:06:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:06:56.487+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:06:56.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:06:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:06:56.401+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:06:56.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:06:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:06:56.401+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:06:56.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:06:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:06:56.401+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:06:56.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:06:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:06:56.496+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:06:56.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:06:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:06:56.496+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:06:56.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:06:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:06:56.496+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:06:56.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:06:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:06:56.496+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:06:57.241 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:06:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:06:56.772+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 5617 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:06:57.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:06:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:06:57.513+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:06:57.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:06:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:06:57.513+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:06:57.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:06:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:06:57.513+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:06:57.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:06:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:06:57.489+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:06:57.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:06:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:06:57.489+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:06:57.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:06:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:06:57.489+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:06:57.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:06:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:06:57.489+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:06:57.742 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:06:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:06:57.367+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:06:57.742 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:06:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:06:57.367+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:06:57.742 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:06:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:06:57.367+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:06:58.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:06:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:06:58.559+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:06:58.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:06:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:06:58.559+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:06:58.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:06:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:06:58.559+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:06:58.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:06:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:06:58.401+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:06:58.742 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:06:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:06:58.401+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:06:58.742 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:06:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:06:58.401+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:06:58.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:06:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:06:58.511+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:06:58.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:06:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:06:58.511+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:06:58.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:06:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:06:58.511+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:06:58.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:06:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:06:58.511+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:06:59.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:06:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:06:59.538+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:06:59.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:06:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:06:59.538+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:06:59.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:06:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:06:59.538+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:06:59.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:06:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:06:59.499+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:06:59.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:06:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:06:59.499+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:06:59.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:06:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:06:59.499+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:06:59.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:06:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:06:59.499+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:06:59.742 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:06:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:06:59.425+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:06:59.742 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:06:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:06:59.425+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:06:59.742 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:06:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:06:59.425+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:07:00.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:07:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:07:00.515+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:07:00.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:07:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:07:00.515+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:07:00.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:07:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:07:00.515+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:07:00.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:07:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:07:00.406+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:07:00.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:07:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:07:00.406+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:07:00.742 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:07:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:07:00.406+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:07:00.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:07:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:07:00.502+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:07:00.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:07:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:07:00.502+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:07:00.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:07:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:07:00.502+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:07:00.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:07:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:07:00.502+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:07:01.660 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:07:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:07:01.515+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:07:01.660 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:07:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:07:01.515+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:07:01.660 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:07:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:07:01.515+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:07:01.661 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:07:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:07:01.366+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:07:01.661 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:07:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:07:01.366+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:07:01.661 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:07:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:07:01.366+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:07:01.661 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:07:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:07:01.537+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:07:01.661 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:07:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:07:01.537+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:07:01.661 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:07:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:07:01.537+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:07:01.661 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:07:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:07:01.537+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:07:01.991 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:07:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:07:01.775+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 5699 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:07:02.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:07:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:07:02.556+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:07:02.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:07:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:07:02.556+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:07:02.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:07:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:07:02.556+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:07:02.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:07:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:07:02.351+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:07:02.742 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:07:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:07:02.351+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:07:02.742 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:07:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:07:02.351+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:07:02.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:07:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:07:02.531+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:07:02.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:07:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:07:02.531+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:07:02.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:07:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:07:02.531+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:07:02.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:07:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:07:02.532+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:07:03.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:07:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:07:03.356+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:07:03.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:07:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:07:03.356+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:07:03.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:07:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:07:03.356+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:07:03.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:07:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:07:03.591+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:07:03.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:07:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:07:03.591+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:07:03.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:07:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:07:03.591+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:07:03.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:07:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:07:03.482+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:07:03.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:07:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:07:03.482+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:07:03.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:07:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:07:03.482+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:07:03.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:07:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:07:03.482+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:07:04.435 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T08:07:04.433+0000 7fa27dd3d700 0 monclient(hunting): authenticate timed out after 300 2024-02-20T08:07:04.435 INFO:teuthology.orchestra.run.smithi133.stderr:[errno 110] RADOS timed out (error connecting to the cluster) 2024-02-20T08:07:04.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:07:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:07:04.388+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:07:04.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:07:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:07:04.388+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:07:04.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:07:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:07:04.388+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:07:04.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:07:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:07:04.616+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:07:04.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:07:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:07:04.616+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:07:04.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:07:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:07:04.616+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:07:04.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:07:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:07:04.502+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:07:04.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:07:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:07:04.502+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:07:04.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:07:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:07:04.502+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:07:04.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:07:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:07:04.502+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:07:05.178 DEBUG:teuthology.orchestra.run.smithi133:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4537327c-cfc4-11ee-95bb-87774f69a715 -e sha1=91ff50e903dcc99ee3ecf9523050878eb452da5c -- bash -c 'ceph versions | jq -e '"'"'.mgr | keys'"'"' | grep $sha1' 2024-02-20T08:07:05.443 INFO:teuthology.orchestra.run.smithi133.stderr:Inferring config /var/lib/ceph/4537327c-cfc4-11ee-95bb-87774f69a715/mon.smithi133/config 2024-02-20T08:07:05.697 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:07:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:07:05.568+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:07:05.697 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:07:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:07:05.568+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:07:05.697 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:07:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:07:05.568+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:07:05.697 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:07:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:07:05.413+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:07:05.697 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:07:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:07:05.413+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:07:05.697 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:07:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:07:05.413+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:07:05.698 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:07:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:07:05.542+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:07:05.698 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:07:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:07:05.542+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:07:05.698 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:07:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:07:05.542+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:07:05.698 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:07:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:07:05.542+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:07:06.555 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:07:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:07:06.525+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:07:06.555 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:07:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:07:06.525+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:07:06.555 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:07:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:07:06.525+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:07:06.556 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:07:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:07:06.389+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:43.250915+0000 front 2024-02-20T07:56:43.251003+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:07:06.556 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:07:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:07:06.389+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:43.251032+0000 front 2024-02-20T07:56:43.251149+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:07:06.556 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:07:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:07:06.389+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:43.251063+0000 front 2024-02-20T07:56:43.251171+0000 (oldest deadline 2024-02-20T07:57:05.550836+0000) 2024-02-20T08:07:06.556 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:07:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:07:06.517+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:07:06.556 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:07:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:07:06.517+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:07:06.556 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:07:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:07:06.517+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:07:06.556 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:07:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:07:06.517+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:07:06.898 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:07:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:07:06.775+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 5781 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:07:07.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:07:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:07:07.491+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:07:07.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:07:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:07:07.491+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:07:07.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:07:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:07:07.491+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:07:07.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:07:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:07:07.477+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:07:07.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:07:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:07:07.477+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:07:07.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:07:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:07:07.477+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:07:07.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:07:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:07:07.477+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:07:08.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:07:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:07:08.501+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:07:08.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:07:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:07:08.501+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:07:08.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:07:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:07:08.501+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:07:08.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:07:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:07:08.434+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:07:08.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:07:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:07:08.434+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:07:08.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:07:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:07:08.434+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:07:08.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:07:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:07:08.434+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:07:09.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:07:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:07:09.482+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:07:09.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:07:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:07:09.482+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:07:09.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:07:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:07:09.482+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:07:09.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:07:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:07:09.460+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:07:09.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:07:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:07:09.460+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:07:09.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:07:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:07:09.460+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:07:09.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:07:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:07:09.460+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:07:10.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:07:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:07:10.504+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:07:10.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:07:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:07:10.504+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:07:10.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:07:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:07:10.504+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:07:10.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:07:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:07:10.459+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:07:10.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:07:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:07:10.459+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:07:10.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:07:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:07:10.459+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:07:10.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:07:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:07:10.459+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:07:11.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:07:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:07:11.480+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:07:11.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:07:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:07:11.480+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:07:11.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:07:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:07:11.480+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:07:11.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:07:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:07:11.502+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:07:11.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:07:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:07:11.502+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:07:11.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:07:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:07:11.502+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:07:11.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:07:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:07:11.503+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:07:12.241 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:07:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:07:11.777+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 5864 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:07:12.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:07:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:07:12.495+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:07:12.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:07:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:07:12.495+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:07:12.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:07:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:07:12.495+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:07:12.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:07:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:07:12.473+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:07:12.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:07:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:07:12.473+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:07:12.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:07:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:07:12.473+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:07:12.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:07:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:07:12.473+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:07:13.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:07:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:07:13.486+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:07:13.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:07:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:07:13.486+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:07:13.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:07:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:07:13.486+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:07:13.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:07:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:07:13.486+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:07:13.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:07:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:07:13.477+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:07:13.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:07:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:07:13.477+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:07:13.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:07:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:07:13.477+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:07:14.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:07:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:07:14.428+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:07:14.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:07:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:07:14.428+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:07:14.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:07:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:07:14.428+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:07:14.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:07:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:07:14.479+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:07:14.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:07:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:07:14.479+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:07:14.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:07:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:07:14.479+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:07:14.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:07:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:07:14.479+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:07:15.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:07:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:07:15.493+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:07:15.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:07:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:07:15.493+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:07:15.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:07:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:07:15.493+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:07:15.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:07:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:07:15.493+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:07:15.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:07:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:07:15.417+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:07:15.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:07:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:07:15.417+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:07:15.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:07:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:07:15.417+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:07:16.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:07:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:07:16.502+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:07:16.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:07:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:07:16.502+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:07:16.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:07:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:07:16.502+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:07:16.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:07:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:07:16.502+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:07:16.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:07:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:07:16.393+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:07:16.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:07:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:07:16.393+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:07:16.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:07:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:07:16.393+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:07:17.241 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:07:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:07:16.778+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 5946 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:07:17.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:07:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:07:17.360+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:07:17.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:07:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:07:17.360+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:07:17.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:07:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:07:17.360+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:07:17.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:07:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:07:17.545+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:07:17.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:07:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:07:17.545+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:07:17.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:07:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:07:17.545+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:07:17.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:07:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:07:17.545+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:07:18.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:07:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:07:18.401+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:40.782314+0000 front 2024-02-20T07:56:40.782261+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:07:18.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:07:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:07:18.401+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:40.782557+0000 front 2024-02-20T07:56:40.782379+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:07:18.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:07:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:07:18.401+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:40.782350+0000 front 2024-02-20T07:56:40.782580+0000 (oldest deadline 2024-02-20T07:57:06.082160+0000) 2024-02-20T08:07:18.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:07:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:07:18.516+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:07:18.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:07:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:07:18.516+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:07:18.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:07:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:07:18.516+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:07:18.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:07:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:07:18.516+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:07:19.885 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:07:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:07:19.532+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:07:19.885 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:07:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:07:19.532+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:07:19.885 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:07:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:07:19.532+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:07:19.885 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:07:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:07:19.532+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:07:20.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:07:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:07:20.487+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:07:20.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:07:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:07:20.487+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:07:20.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:07:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:07:20.487+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:07:20.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:07:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:07:20.487+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:07:21.780 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:07:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:07:21.506+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:07:21.780 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:07:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:07:21.506+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:07:21.780 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:07:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:07:21.506+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:07:21.780 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:07:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:07:21.506+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:07:22.241 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:07:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:07:21.779+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 6029 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:07:22.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:07:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:07:22.552+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:07:22.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:07:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:07:22.552+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:07:22.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:07:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:07:22.552+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:07:22.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:07:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:07:22.552+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:07:23.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:07:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:07:23.530+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:07:23.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:07:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:07:23.530+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:07:23.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:07:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:07:23.530+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:07:23.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:07:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:07:23.530+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:07:24.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:07:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:07:24.496+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:07:24.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:07:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:07:24.496+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:07:24.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:07:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:07:24.496+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:07:24.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:07:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:07:24.496+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:07:25.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:07:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:07:25.506+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:07:25.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:07:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:07:25.506+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:07:25.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:07:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:07:25.506+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:07:25.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:07:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:07:25.506+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:07:26.781 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:07:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:07:26.490+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:07:26.782 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:07:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:07:26.490+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:07:26.782 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:07:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:07:26.490+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:07:26.782 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:07:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:07:26.490+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:07:27.241 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:07:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:07:26.780+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 6110 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:07:27.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:07:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:07:27.514+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:07:27.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:07:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:07:27.514+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:07:27.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:07:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:07:27.514+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:07:27.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:07:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:07:27.514+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:07:28.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:07:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:07:28.500+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:07:28.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:07:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:07:28.500+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:07:28.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:07:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:07:28.500+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:07:28.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:07:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:07:28.500+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:07:29.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:07:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:07:29.605+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:07:29.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:07:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:07:29.605+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:07:29.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:07:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:07:29.605+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:07:29.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:07:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:07:29.480+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 since back 2024-02-20T07:56:44.074489+0000 front 2024-02-20T07:56:44.074509+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:07:29.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:07:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:07:29.480+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 since back 2024-02-20T07:56:44.074640+0000 front 2024-02-20T07:56:44.074596+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:07:29.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:07:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:07:29.480+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 since back 2024-02-20T07:56:44.074627+0000 front 2024-02-20T07:56:44.074736+0000 (oldest deadline 2024-02-20T07:57:09.974480+0000) 2024-02-20T08:07:29.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:07:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:07:29.480+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:07:30.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:07:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:07:30.499+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:07:30.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:07:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:07:30.633+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:07:30.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:07:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:07:30.633+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:07:30.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:07:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:07:30.633+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:07:31.783 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:07:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:07:31.498+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:07:31.783 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:07:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:07:31.667+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:07:31.783 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:07:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:07:31.667+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:07:31.783 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:07:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:07:31.667+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:07:32.241 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:07:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:07:31.782+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 6194 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:07:32.710 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:07:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:07:32.451+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:07:32.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:07:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:07:32.709+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:07:32.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:07:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:07:32.709+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:07:32.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:07:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:07:32.709+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:07:33.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:07:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:07:33.501+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:07:33.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:07:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:07:33.719+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:07:33.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:07:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:07:33.719+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:07:33.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:07:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:07:33.719+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:07:34.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:07:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:07:34.521+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:07:34.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:07:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:07:34.685+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:07:34.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:07:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:07:34.685+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:07:34.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:07:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:07:34.685+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:07:35.927 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:07:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:07:35.550+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:07:35.928 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:07:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:07:35.668+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:07:35.928 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:07:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:07:35.668+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:07:35.928 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:07:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:07:35.668+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:07:36.908 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:07:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:07:36.783+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 6276 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:07:36.908 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:07:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:07:36.576+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:07:36.908 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:07:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:07:36.633+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:07:36.908 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:07:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:07:36.633+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:07:36.908 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:07:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:07:36.633+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:07:37.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:07:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:07:37.604+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:07:37.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:07:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:07:37.677+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:07:37.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:07:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:07:37.677+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:07:37.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:07:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:07:37.677+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:07:38.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:07:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:07:38.589+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:07:38.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:07:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:07:38.727+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:07:38.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:07:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:07:38.727+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:07:38.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:07:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:07:38.727+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:07:39.884 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:07:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:07:39.596+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:07:39.884 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:07:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:07:39.767+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:07:39.885 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:07:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:07:39.767+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:07:39.885 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:07:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:07:39.767+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:07:39.885 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:07:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:07:39.671+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:07:39.885 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:07:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:07:39.671+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:07:39.885 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:07:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:07:39.671+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:07:40.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:07:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:07:40.567+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:07:40.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:07:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:07:40.787+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:07:40.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:07:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:07:40.787+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:07:40.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:07:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:07:40.787+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:07:40.992 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:07:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:07:40.633+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:07:40.992 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:07:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:07:40.633+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:07:40.992 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:07:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:07:40.633+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:07:41.913 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:07:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:07:41.784+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 6359 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:07:41.913 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:07:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:07:41.615+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:07:41.913 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:07:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:07:41.767+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:07:41.914 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:07:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:07:41.767+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:07:41.914 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:07:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:07:41.767+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:07:41.914 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:07:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:07:41.598+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:07:41.914 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:07:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:07:41.598+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:07:41.914 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:07:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:07:41.598+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:07:42.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:07:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:07:42.763+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:07:42.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:07:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:07:42.763+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:07:42.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:07:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:07:42.763+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:07:42.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:07:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:07:42.624+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:07:42.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:07:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:07:42.586+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:07:42.992 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:07:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:07:42.586+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:07:42.992 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:07:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:07:42.586+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:07:43.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:07:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:07:43.653+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:07:43.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:07:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:07:43.742+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:07:43.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:07:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:07:43.742+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:07:43.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:07:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:07:43.742+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:07:43.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:07:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:07:43.598+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:07:43.992 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:07:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:07:43.598+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:07:43.992 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:07:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:07:43.598+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:07:44.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:07:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:07:44.724+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:07:44.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:07:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:07:44.724+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:07:44.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:07:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:07:44.724+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:07:44.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:07:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:07:44.630+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:07:44.992 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:07:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:07:44.590+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:07:44.992 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:07:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:07:44.590+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:07:44.992 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:07:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:07:44.590+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:07:45.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:07:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:07:45.744+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:07:45.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:07:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:07:45.744+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:07:45.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:07:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:07:45.744+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:07:45.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:07:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:07:45.592+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:07:45.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:07:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:07:45.546+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:07:45.992 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:07:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:07:45.546+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:07:45.992 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:07:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:07:45.546+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:07:46.787 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:07:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:07:46.558+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:07:46.787 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:07:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:07:46.523+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:07:46.787 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:07:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:07:46.523+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:07:46.787 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:07:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:07:46.523+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:07:46.787 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:07:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:07:46.699+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:07:46.787 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:07:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:07:46.699+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:07:46.788 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:07:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:07:46.699+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:07:47.241 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:07:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:07:46.785+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 6440 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:07:47.740 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:07:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:07:47.526+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:07:47.740 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:07:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:07:47.479+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:07:47.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:07:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:07:47.479+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:07:47.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:07:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:07:47.479+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:07:47.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:07:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:07:47.739+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:07:47.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:07:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:07:47.739+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:07:47.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:07:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:07:47.739+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:07:48.781 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:07:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:07:48.538+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:07:48.782 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:07:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:07:48.496+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:07:48.782 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:07:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:07:48.496+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:07:48.782 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:07:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:07:48.496+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:07:49.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:07:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:07:48.780+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:07:49.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:07:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:07:48.780+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:07:49.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:07:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:07:48.780+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:07:49.790 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:07:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:07:49.568+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:07:49.790 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:07:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:07:49.492+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:07:49.790 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:07:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:07:49.492+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:07:49.790 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:07:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:07:49.492+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:07:50.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:07:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:07:49.790+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:07:50.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:07:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:07:49.790+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:07:50.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:07:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:07:49.790+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:07:50.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:07:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:07:50.568+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:07:50.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:07:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:07:50.745+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:07:50.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:07:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:07:50.745+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:07:50.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:07:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:07:50.745+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:07:50.992 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:07:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:07:50.522+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:07:50.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:07:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:07:50.522+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:07:50.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:07:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:07:50.522+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:07:51.810 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:07:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:07:51.788+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 6524 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:07:51.810 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:07:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:07:51.611+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:07:51.811 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:07:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:07:51.776+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:07:51.811 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:07:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:07:51.776+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:07:51.811 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:07:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:07:51.776+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:07:51.811 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:07:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:07:51.551+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:07:51.811 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:07:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:07:51.551+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:07:51.811 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:07:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:07:51.551+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:07:52.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:07:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:07:52.750+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:07:52.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:07:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:07:52.750+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:07:52.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:07:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:07:52.750+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:07:52.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:07:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:07:52.624+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:07:52.992 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:07:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:07:52.551+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:07:52.992 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:07:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:07:52.551+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:07:52.992 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:07:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:07:52.551+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:07:53.790 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:07:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:07:53.519+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:07:53.790 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:07:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:07:53.519+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:07:53.790 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:07:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:07:53.519+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:07:53.790 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:07:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:07:53.615+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:07:53.790 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:07:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:07:53.615+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:07:53.790 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:07:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:07:53.615+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:07:53.790 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:07:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:07:53.615+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:07:54.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:07:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:07:53.789+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:07:54.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:07:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:07:53.789+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:07:54.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:07:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:07:53.789+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:07:54.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:07:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:07:54.572+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:07:54.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:07:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:07:54.572+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:07:54.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:07:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:07:54.572+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:07:54.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:07:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:07:54.572+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:07:54.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:07:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:07:54.490+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:07:54.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:07:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:07:54.490+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:07:54.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:07:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:07:54.490+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:07:55.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:07:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:07:54.753+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:07:55.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:07:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:07:54.753+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:07:55.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:07:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:07:54.753+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:07:55.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:07:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:07:55.716+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:07:55.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:07:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:07:55.716+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:07:55.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:07:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:07:55.716+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:07:55.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:07:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:07:55.597+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:07:55.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:07:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:07:55.597+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:07:55.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:07:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:07:55.597+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:07:55.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:07:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:07:55.597+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:07:55.992 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:07:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:07:55.534+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:07:55.992 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:07:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:07:55.534+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:07:55.992 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:07:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:07:55.534+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:07:56.803 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:07:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:07:56.789+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 6606 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:07:56.803 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:07:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:07:56.706+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:07:56.803 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:07:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:07:56.706+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:07:56.803 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:07:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:07:56.706+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:07:56.804 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:07:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:07:56.551+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:07:56.804 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:07:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:07:56.551+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:07:56.804 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:07:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:07:56.551+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:07:56.804 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:07:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:07:56.553+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:07:56.804 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:07:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:07:56.553+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:07:56.804 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:07:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:07:56.553+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:07:56.804 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:07:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:07:56.554+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:07:57.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:07:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:07:57.559+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:07:57.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:07:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:07:57.559+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:07:57.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:07:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:07:57.559+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:07:57.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:07:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:07:57.726+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:07:57.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:07:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:07:57.726+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:07:57.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:07:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:07:57.726+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:07:57.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:07:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:07:57.590+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:07:57.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:07:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:07:57.590+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:07:57.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:07:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:07:57.590+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:07:57.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:07:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:07:57.590+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:07:58.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:07:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:07:58.548+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:07:58.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:07:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:07:58.548+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:07:58.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:07:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:07:58.548+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:07:58.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:07:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:07:58.776+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:07:58.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:07:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:07:58.776+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:07:58.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:07:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:07:58.776+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:07:58.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:07:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:07:58.559+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:07:58.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:07:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:07:58.559+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:07:58.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:07:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:07:58.559+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:07:58.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:07:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:07:58.559+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:07:59.885 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:07:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:07:59.753+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:07:59.885 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:07:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:07:59.753+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:07:59.885 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:07:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:07:59.753+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:07:59.885 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:07:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:07:59.558+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:07:59.885 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:07:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:07:59.558+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:07:59.885 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:07:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:07:59.558+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:07:59.885 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:07:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:07:59.531+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:07:59.885 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:07:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:07:59.531+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:07:59.886 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:07:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:07:59.531+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:07:59.886 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:07:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:07:59.531+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:08:00.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:08:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:08:00.726+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:08:00.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:08:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:08:00.726+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:08:00.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:08:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:08:00.726+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:08:00.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:08:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:08:00.541+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:08:00.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:08:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:08:00.541+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:08:00.992 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:08:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:08:00.541+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:08:00.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:08:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:08:00.565+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:08:00.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:08:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:08:00.565+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:08:00.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:08:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:08:00.565+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:08:00.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:08:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:08:00.565+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:08:01.818 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:08:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:08:01.790+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 6689 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:08:01.818 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:08:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:08:01.712+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:08:01.818 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:08:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:08:01.712+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:08:01.818 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:08:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:08:01.712+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:08:01.819 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:08:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:08:01.575+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:08:01.819 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:08:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:08:01.575+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:08:01.819 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:08:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:08:01.575+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:08:01.819 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:08:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:08:01.560+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:08:01.819 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:08:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:08:01.560+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:08:01.819 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:08:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:08:01.560+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:08:01.819 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:08:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:08:01.560+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:08:02.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:08:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:08:02.605+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:08:02.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:08:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:08:02.605+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:08:02.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:08:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:08:02.605+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:08:02.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:08:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:08:02.606+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:08:02.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:08:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:08:02.697+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:08:02.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:08:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:08:02.697+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:08:02.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:08:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:08:02.697+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:08:02.992 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:08:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:08:02.589+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:08:02.992 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:08:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:08:02.589+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:08:02.992 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:08:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:08:02.589+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:08:03.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:08:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:08:03.610+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:08:03.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:08:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:08:03.610+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:08:03.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:08:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:08:03.610+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:08:03.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:08:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:08:03.723+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:08:03.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:08:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:08:03.723+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:08:03.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:08:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:08:03.723+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:08:03.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:08:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:08:03.596+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:08:03.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:08:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:08:03.596+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:08:03.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:08:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:08:03.596+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:08:03.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:08:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:08:03.596+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:08:04.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:08:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:08:04.655+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:08:04.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:08:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:08:04.655+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:08:04.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:08:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:08:04.655+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:08:04.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:08:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:08:04.739+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:08:04.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:08:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:08:04.739+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:08:04.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:08:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:08:04.739+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:08:04.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:08:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:08:04.554+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:08:04.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:08:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:08:04.554+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:08:04.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:08:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:08:04.554+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:08:04.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:08:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:08:04.554+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:08:05.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:08:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:08:05.680+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:08:05.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:08:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:08:05.680+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:08:05.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:08:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:08:05.680+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:08:05.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:08:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:08:05.732+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:08:05.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:08:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:08:05.732+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:08:05.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:08:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:08:05.732+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:08:05.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:08:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:08:05.549+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:08:05.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:08:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:08:05.549+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:08:05.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:08:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:08:05.549+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:08:05.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:08:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:08:05.549+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:08:06.892 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:08:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:08:06.792+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 6770 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:08:06.892 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:08:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:08:06.576+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:08:06.893 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:08:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:08:06.576+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:08:06.893 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:08:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:08:06.576+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:08:06.893 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:08:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:08:06.577+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:08:06.893 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:08:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:08:06.716+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:08:06.893 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:08:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:08:06.716+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:08:06.893 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:08:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:08:06.716+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:08:06.893 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:08:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:08:06.680+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:08:06.893 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:08:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:08:06.680+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:08:06.893 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:08:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:08:06.680+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:08:07.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:08:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:08:07.595+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:08:07.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:08:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:08:07.595+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:08:07.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:08:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:08:07.595+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:08:07.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:08:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:08:07.595+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:08:07.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:08:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:08:07.728+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:08:07.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:08:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:08:07.729+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:08:07.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:08:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:08:07.729+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:08:07.992 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:08:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:08:07.653+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:08:07.992 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:08:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:08:07.653+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:08:07.992 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:08:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:08:07.653+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:08:08.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:08:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:08:08.698+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:08:08.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:08:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:08:08.698+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:08:08.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:08:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:08:08.698+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:08:08.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:08:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:08:08.684+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:08:08.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:08:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:08:08.684+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:08:08.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:08:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:08:08.684+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:08:08.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:08:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:08:08.635+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:08:08.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:08:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:08:08.635+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:08:08.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:08:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:08:08.635+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:08:08.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:08:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:08:08.635+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:08:09.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:08:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:08:09.726+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:08:09.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:08:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:08:09.726+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:08:09.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:08:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:08:09.726+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:08:09.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:08:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:08:09.682+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:08:09.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:08:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:08:09.682+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:08:09.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:08:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:08:09.682+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:08:09.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:08:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:08:09.671+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:08:09.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:08:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:08:09.671+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:08:09.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:08:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:08:09.671+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:08:09.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:08:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:08:09.671+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:08:10.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:08:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:08:10.702+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:08:10.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:08:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:08:10.702+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:08:10.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:08:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:08:10.702+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:08:10.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:08:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:08:10.706+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:08:10.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:08:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:08:10.706+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:08:10.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:08:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:08:10.706+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:08:10.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:08:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:08:10.683+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:08:10.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:08:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:08:10.683+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:08:10.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:08:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:08:10.683+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:08:10.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:08:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:08:10.683+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:08:11.927 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:08:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:08:11.794+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 6854 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:08:11.928 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:08:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:08:11.745+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:08:11.928 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:08:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:08:11.745+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:08:11.928 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:08:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:08:11.745+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:08:11.928 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:08:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:08:11.700+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:08:11.928 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:08:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:08:11.700+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:08:11.928 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:08:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:08:11.700+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:08:11.929 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:08:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:08:11.634+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:08:11.929 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:08:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:08:11.634+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:08:11.929 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:08:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:08:11.634+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:08:11.929 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:08:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:08:11.634+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:08:12.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:08:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:08:12.714+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:08:12.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:08:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:08:12.714+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:08:12.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:08:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:08:12.714+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:08:12.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:08:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:08:12.685+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:08:12.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:08:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:08:12.685+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:08:12.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:08:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:08:12.685+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:08:12.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:08:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:08:12.588+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:08:12.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:08:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:08:12.588+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:08:12.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:08:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:08:12.588+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:08:12.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:08:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:08:12.588+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:08:13.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:08:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:08:13.673+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:08:13.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:08:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:08:13.673+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:08:13.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:08:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:08:13.673+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:08:13.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:08:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:08:13.674+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:08:13.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:08:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:08:13.674+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:08:13.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:08:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:08:13.674+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:08:13.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:08:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:08:13.564+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:08:13.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:08:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:08:13.564+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:08:13.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:08:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:08:13.564+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:08:13.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:08:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:08:13.564+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:08:14.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:08:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:08:14.680+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:08:14.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:08:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:08:14.680+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:08:14.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:08:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:08:14.680+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:08:14.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:08:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:08:14.657+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:08:14.992 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:08:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:08:14.657+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:08:14.992 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:08:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:08:14.657+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:08:14.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:08:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:08:14.575+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:08:14.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:08:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:08:14.575+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:08:14.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:08:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:08:14.575+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:08:14.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:08:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:08:14.575+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:08:15.863 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:08:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:08:15.573+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:08:15.863 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:08:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:08:15.573+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:08:15.863 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:08:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:08:15.573+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:08:15.863 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:08:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:08:15.574+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:08:15.864 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:08:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:08:15.685+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:08:15.864 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:08:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:08:15.685+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:08:15.864 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:08:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:08:15.685+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:08:15.864 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:08:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:08:15.719+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:08:15.864 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:08:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:08:15.719+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:08:15.864 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:08:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:08:15.719+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:08:16.819 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:08:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:08:16.794+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 6936 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:08:16.820 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:08:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:08:16.722+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:08:16.820 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:08:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:08:16.722+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:08:16.820 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:08:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:08:16.722+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:08:16.820 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:08:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:08:16.710+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:08:16.821 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:08:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:08:16.710+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:08:16.821 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:08:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:08:16.710+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:08:16.821 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:08:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:08:16.560+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:08:16.821 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:08:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:08:16.560+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:08:16.821 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:08:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:08:16.560+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:08:16.821 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:08:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:08:16.560+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:08:17.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:08:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:08:17.740+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:08:17.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:08:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:08:17.740+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:08:17.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:08:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:08:17.740+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:08:17.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:08:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:08:17.556+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:08:17.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:08:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:08:17.556+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:08:17.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:08:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:08:17.556+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:08:17.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:08:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:08:17.556+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:08:17.992 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:08:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:08:17.671+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:08:17.992 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:08:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:08:17.671+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:08:17.992 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:08:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:08:17.671+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:08:18.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:08:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:08:18.716+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:08:18.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:08:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:08:18.716+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:08:18.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:08:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:08:18.716+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:08:18.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:08:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:08:18.587+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:08:18.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:08:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:08:18.587+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:08:18.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:08:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:08:18.587+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:08:18.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:08:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:08:18.587+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:08:18.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:08:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:08:18.624+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:08:18.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:08:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:08:18.624+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:08:18.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:08:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:08:18.624+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:08:19.884 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:08:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:08:19.620+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:08:19.884 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:08:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:08:19.620+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:08:19.884 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:08:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:08:19.620+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:08:19.884 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:08:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:08:19.682+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:08:19.885 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:08:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:08:19.682+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:08:19.885 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:08:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:08:19.682+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:08:19.885 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:08:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:08:19.631+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:08:19.885 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:08:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:08:19.631+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:08:19.885 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:08:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:08:19.631+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:08:19.885 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:08:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:08:19.631+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:08:20.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:08:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:08:20.651+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:08:20.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:08:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:08:20.651+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:08:20.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:08:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:08:20.651+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:08:20.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:08:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:08:20.665+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:08:20.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:08:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:08:20.665+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:08:20.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:08:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:08:20.665+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:08:20.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:08:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:08:20.601+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:08:20.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:08:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:08:20.601+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:08:20.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:08:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:08:20.601+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:08:20.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:08:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:08:20.601+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:08:21.920 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:08:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:08:21.796+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 7019 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:08:21.920 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:08:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:08:21.673+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:08:21.920 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:08:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:08:21.673+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:08:21.920 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:08:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:08:21.673+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:08:21.921 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:08:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:08:21.623+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:08:21.921 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:08:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:08:21.623+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:08:21.921 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:08:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:08:21.623+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:08:21.921 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:08:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:08:21.623+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:08:21.921 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:08:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:08:21.690+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:08:21.921 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:08:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:08:21.690+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:08:21.921 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:08:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:08:21.690+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:08:22.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:08:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:08:22.634+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:08:22.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:08:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:08:22.634+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:08:22.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:08:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:08:22.634+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:08:22.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:08:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:08:22.655+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:08:22.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:08:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:08:22.655+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:08:22.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:08:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:08:22.655+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:08:22.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:08:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:08:22.655+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:08:22.992 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:08:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:08:22.661+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:08:22.992 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:08:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:08:22.661+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:08:22.992 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:08:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:08:22.661+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:08:23.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:08:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:08:23.611+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:08:23.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:08:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:08:23.611+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:08:23.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:08:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:08:23.611+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:08:23.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:08:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:08:23.644+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:08:23.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:08:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:08:23.644+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:08:23.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:08:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:08:23.644+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:08:23.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:08:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:08:23.644+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:08:23.992 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:08:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:08:23.655+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:08:23.992 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:08:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:08:23.655+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:08:23.992 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:08:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:08:23.655+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:08:24.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:08:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:08:24.601+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:08:24.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:08:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:08:24.601+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:08:24.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:08:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:08:24.601+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:08:24.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:08:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:08:24.598+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:08:24.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:08:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:08:24.598+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:08:24.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:08:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:08:24.598+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:08:24.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:08:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:08:24.598+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:08:24.992 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:08:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:08:24.691+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:08:24.992 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:08:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:08:24.691+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:08:24.992 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:08:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:08:24.691+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:08:25.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:08:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:08:25.582+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:08:25.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:08:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:08:25.582+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:08:25.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:08:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:08:25.582+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:08:25.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:08:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:08:25.683+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:08:25.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:08:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:08:25.683+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:08:25.992 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:08:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:08:25.683+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:08:25.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:08:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:08:25.645+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:08:25.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:08:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:08:25.645+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:08:25.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:08:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:08:25.645+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:08:25.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:08:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:08:25.645+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:08:26.809 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:08:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:08:26.797+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 7100 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:08:26.809 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:08:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:08:26.557+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:08:26.810 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:08:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:08:26.557+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:08:26.810 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:08:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:08:26.557+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:08:26.810 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:08:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:08:26.625+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:08:26.810 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:08:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:08:26.625+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:08:26.810 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:08:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:08:26.625+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:08:26.810 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:08:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:08:26.625+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:08:26.810 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:08:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:08:26.637+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:08:26.810 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:08:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:08:26.637+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:08:26.810 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:08:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:08:26.637+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:08:27.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:08:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:08:27.559+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:08:27.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:08:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:08:27.559+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:08:27.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:08:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:08:27.559+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:08:27.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:08:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:08:27.621+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:08:27.992 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:08:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:08:27.621+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:08:27.992 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:08:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:08:27.621+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:08:27.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:08:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:08:27.594+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:08:27.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:08:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:08:27.594+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:08:27.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:08:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:08:27.594+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:08:27.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:08:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:08:27.594+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:08:28.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:08:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:08:28.521+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:08:28.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:08:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:08:28.521+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:08:28.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:08:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:08:28.521+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:08:28.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:08:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:08:28.614+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:08:28.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:08:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:08:28.614+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:08:28.992 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:08:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:08:28.614+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:08:28.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:08:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:08:28.642+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:08:28.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:08:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:08:28.642+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:08:28.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:08:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:08:28.642+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:08:28.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:08:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:08:28.642+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:08:29.885 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:08:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:08:29.549+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:08:29.885 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:08:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:08:29.549+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:08:29.885 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:08:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:08:29.549+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:08:29.885 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:08:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:08:29.622+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:08:29.886 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:08:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:08:29.622+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:08:29.886 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:08:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:08:29.622+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:08:29.886 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:08:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:08:29.622+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:08:29.886 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:08:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:08:29.605+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:08:29.886 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:08:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:08:29.605+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:08:29.886 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:08:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:08:29.605+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:08:30.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:08:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:08:30.618+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:08:30.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:08:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:08:30.618+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:08:30.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:08:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:08:30.618+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:08:30.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:08:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:08:30.588+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:08:30.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:08:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:08:30.588+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:08:30.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:08:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:08:30.588+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:08:30.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:08:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:08:30.602+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:08:30.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:08:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:08:30.602+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:08:30.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:08:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:08:30.602+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:08:30.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:08:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:08:30.602+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:08:31.800 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:08:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:08:31.589+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:08:31.800 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:08:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:08:31.589+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:08:31.800 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:08:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:08:31.589+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:08:31.800 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:08:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:08:31.589+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:08:31.801 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:08:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:08:31.605+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:08:31.801 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:08:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:08:31.605+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:08:31.801 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:08:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:08:31.605+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:08:31.801 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:08:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:08:31.615+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:08:31.801 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:08:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:08:31.615+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:08:31.801 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:08:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:08:31.615+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:08:32.241 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:08:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:08:31.799+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 7184 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:08:32.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:08:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:08:32.630+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:08:32.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:08:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:08:32.630+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:08:32.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:08:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:08:32.630+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:08:32.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:08:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:08:32.614+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:08:32.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:08:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:08:32.614+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:08:32.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:08:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:08:32.614+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:08:32.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:08:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:08:32.614+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:08:32.992 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:08:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:08:32.593+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:08:32.992 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:08:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:08:32.593+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:08:32.992 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:08:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:08:32.593+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:08:33.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:08:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:08:33.642+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:08:33.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:08:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:08:33.642+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:08:33.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:08:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:08:33.642+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:08:33.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:08:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:08:33.625+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:08:33.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:08:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:08:33.625+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:08:33.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:08:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:08:33.625+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:08:33.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:08:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:08:33.625+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:08:33.992 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:08:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:08:33.545+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:08:33.992 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:08:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:08:33.545+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:08:33.992 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:08:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:08:33.545+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:08:34.602 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:08:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:08:34.522+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:08:34.602 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:08:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:08:34.522+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:08:34.602 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:08:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:08:34.522+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:08:34.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:08:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:08:34.655+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:08:34.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:08:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:08:34.655+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:08:34.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:08:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:08:34.655+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:08:34.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:08:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:08:34.602+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:08:34.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:08:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:08:34.602+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:08:34.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:08:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:08:34.602+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:08:34.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:08:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:08:34.602+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:08:35.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:08:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:08:35.665+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:08:35.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:08:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:08:35.665+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:08:35.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:08:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:08:35.665+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:08:35.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:08:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:08:35.534+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:08:35.992 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:08:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:08:35.534+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:08:35.992 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:08:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:08:35.534+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:08:35.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:08:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:08:35.584+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:08:35.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:08:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:08:35.584+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:08:35.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:08:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:08:35.584+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:08:35.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:08:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:08:35.584+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:08:36.802 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:08:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:08:36.574+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:08:36.802 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:08:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:08:36.574+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:08:36.802 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:08:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:08:36.574+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:08:36.802 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:08:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:08:36.574+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:08:36.802 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:08:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:08:36.551+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:08:36.802 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:08:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:08:36.551+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:08:36.802 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:08:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:08:36.551+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:08:36.803 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:08:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:08:36.641+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:08:36.803 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:08:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:08:36.641+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:08:36.803 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:08:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:08:36.641+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:08:37.241 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:08:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:08:36.800+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 7266 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:08:37.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:08:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:08:37.596+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:08:37.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:08:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:08:37.596+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:08:37.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:08:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:08:37.596+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:08:37.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:08:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:08:37.544+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:08:37.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:08:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:08:37.544+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:08:37.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:08:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:08:37.544+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:08:37.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:08:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:08:37.549+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:08:37.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:08:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:08:37.549+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:08:37.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:08:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:08:37.549+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:08:37.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:08:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:08:37.549+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:08:38.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:08:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:08:38.627+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:08:38.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:08:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:08:38.627+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:08:38.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:08:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:08:38.627+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:08:38.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:08:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:08:38.542+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:08:38.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:08:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:08:38.542+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:08:38.992 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:08:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:08:38.542+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:08:38.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:08:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:08:38.553+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:08:38.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:08:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:08:38.553+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:08:38.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:08:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:08:38.553+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:08:38.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:08:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:08:38.554+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:08:39.884 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:08:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:08:39.630+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:08:39.884 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:08:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:08:39.630+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:08:39.885 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:08:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:08:39.630+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:08:39.885 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:08:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:08:39.558+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:08:39.885 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:08:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:08:39.558+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:08:39.885 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:08:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:08:39.558+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:08:39.885 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:08:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:08:39.516+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:08:39.886 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:08:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:08:39.516+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:08:39.886 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:08:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:08:39.516+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:08:39.886 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:08:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:08:39.516+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:08:40.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:08:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:08:40.533+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:08:40.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:08:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:08:40.533+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:08:40.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:08:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:08:40.533+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:08:40.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:08:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:08:40.533+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:08:40.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:08:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:08:40.662+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:08:40.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:08:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:08:40.662+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:08:40.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:08:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:08:40.662+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:08:40.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:08:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:08:40.605+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:08:40.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:08:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:08:40.605+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:08:40.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:08:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:08:40.605+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:08:41.803 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:08:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:08:41.625+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:08:41.803 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:08:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:08:41.625+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:08:41.803 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:08:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:08:41.625+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:08:41.804 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:08:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:08:41.561+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:08:41.804 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:08:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:08:41.561+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:08:41.804 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:08:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:08:41.561+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:08:41.804 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:08:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:08:41.497+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:08:41.804 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:08:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:08:41.497+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:08:41.804 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:08:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:08:41.497+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:08:41.804 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:08:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:08:41.497+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:08:42.241 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:08:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:08:41.802+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 7349 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:08:42.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:08:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:08:42.603+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:08:42.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:08:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:08:42.603+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:08:42.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:08:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:08:42.603+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:08:42.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:08:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:08:42.497+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:08:42.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:08:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:08:42.497+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:08:42.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:08:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:08:42.497+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:08:42.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:08:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:08:42.497+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:08:42.992 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:08:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:08:42.546+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:08:42.992 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:08:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:08:42.546+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:08:42.992 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:08:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:08:42.546+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:08:43.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:08:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:08:43.647+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:08:43.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:08:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:08:43.647+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:08:43.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:08:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:08:43.647+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:08:43.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:08:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:08:43.450+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:08:43.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:08:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:08:43.450+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:08:43.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:08:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:08:43.450+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:08:43.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:08:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:08:43.450+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:08:43.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:08:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:08:43.511+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:08:43.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:08:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:08:43.511+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:08:43.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:08:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:08:43.511+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:08:44.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:08:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:08:44.602+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:08:44.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:08:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:08:44.602+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:08:44.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:08:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:08:44.602+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:08:44.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:08:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:08:44.463+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:08:44.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:08:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:08:44.463+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:08:44.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:08:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:08:44.463+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:08:44.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:08:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:08:44.463+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:08:44.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:08:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:08:44.538+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:08:44.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:08:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:08:44.538+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:08:44.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:08:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:08:44.538+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:08:45.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:08:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:08:45.607+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:08:45.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:08:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:08:45.607+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:08:45.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:08:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:08:45.607+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:08:45.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:08:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:08:45.467+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:08:45.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:08:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:08:45.467+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:08:45.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:08:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:08:45.467+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:08:45.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:08:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:08:45.467+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:08:45.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:08:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:08:45.490+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:08:45.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:08:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:08:45.490+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:08:45.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:08:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:08:45.490+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:08:46.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:08:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:08:46.639+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:08:46.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:08:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:08:46.639+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:08:46.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:08:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:08:46.639+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:08:46.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:08:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:08:46.456+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:08:46.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:08:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:08:46.456+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:08:46.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:08:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:08:46.456+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:08:46.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:08:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:08:46.497+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:08:46.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:08:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:08:46.497+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:08:46.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:08:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:08:46.497+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:08:46.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:08:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:08:46.497+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:08:47.241 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:08:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:08:46.804+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 7430 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:08:47.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:08:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:08:47.636+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:08:47.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:08:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:08:47.636+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:08:47.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:08:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:08:47.636+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:08:47.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:08:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:08:47.537+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:08:47.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:08:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:08:47.537+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:08:47.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:08:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:08:47.537+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:08:47.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:08:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:08:47.537+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:08:47.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:08:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:08:47.452+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:08:47.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:08:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:08:47.452+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:08:47.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:08:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:08:47.452+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:08:48.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:08:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:08:48.624+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:08:48.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:08:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:08:48.624+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:08:48.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:08:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:08:48.624+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:08:48.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:08:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:08:48.587+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:08:48.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:08:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:08:48.587+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:08:48.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:08:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:08:48.587+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:08:48.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:08:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:08:48.587+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:08:48.992 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:08:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:08:48.497+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:08:48.992 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:08:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:08:48.498+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:08:48.992 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:08:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:08:48.498+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:08:49.884 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:08:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:08:49.507+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:08:49.885 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:08:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:08:49.507+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:08:49.885 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:08:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:08:49.507+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:08:49.885 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:08:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:08:49.616+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:08:49.885 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:08:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:08:49.616+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:08:49.885 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:08:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:08:49.616+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:08:49.885 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:08:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:08:49.553+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:08:49.885 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:08:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:08:49.553+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:08:49.886 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:08:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:08:49.553+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:08:49.886 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:08:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:08:49.553+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:08:50.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:08:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:08:50.577+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:08:50.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:08:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:08:50.577+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:08:50.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:08:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:08:50.577+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:08:50.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:08:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:08:50.516+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:08:50.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:08:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:08:50.516+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:08:50.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:08:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:08:50.516+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:08:50.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:08:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:08:50.516+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:08:50.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:08:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:08:50.461+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:08:50.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:08:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:08:50.461+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:08:50.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:08:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:08:50.461+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:08:51.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:08:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:08:51.482+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:08:51.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:08:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:08:51.482+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:08:51.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:08:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:08:51.482+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:08:51.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:08:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:08:51.540+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:08:51.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:08:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:08:51.540+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:08:51.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:08:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:08:51.540+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:08:51.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:08:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:08:51.540+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:08:51.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:08:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:08:51.554+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:08:51.742 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:08:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:08:51.554+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:08:51.742 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:08:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:08:51.554+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:08:52.240 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:08:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:08:51.806+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 7514 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:08:52.727 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:08:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:08:52.581+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:08:52.727 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:08:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:08:52.581+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:08:52.727 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:08:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:08:52.581+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:08:52.728 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:08:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:08:52.519+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:08:52.728 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:08:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:08:52.519+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:08:52.728 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:08:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:08:52.519+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:08:52.728 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:08:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:08:52.519+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:08:52.728 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:08:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:08:52.470+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:08:52.728 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:08:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:08:52.470+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:08:52.728 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:08:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:08:52.470+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:08:53.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:08:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:08:53.522+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:08:53.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:08:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:08:53.523+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:08:53.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:08:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:08:53.523+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:08:53.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:08:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:08:53.523+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:08:53.742 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:08:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:08:53.618+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:08:53.742 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:08:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:08:53.618+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:08:53.742 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:08:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:08:53.618+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:08:53.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:08:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:08:53.447+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:08:53.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:08:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:08:53.447+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:08:53.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:08:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:08:53.447+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:08:54.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:08:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:08:54.588+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:08:54.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:08:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:08:54.588+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:08:54.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:08:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:08:54.588+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:08:54.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:08:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:08:54.490+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:08:54.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:08:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:08:54.490+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:08:54.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:08:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:08:54.490+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:08:54.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:08:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:08:54.503+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:08:54.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:08:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:08:54.503+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:08:54.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:08:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:08:54.503+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:08:54.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:08:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:08:54.503+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:08:55.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:08:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:08:55.559+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:08:55.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:08:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:08:55.559+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:08:55.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:08:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:08:55.559+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:08:55.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:08:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:08:55.441+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:08:55.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:08:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:08:55.441+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:08:55.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:08:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:08:55.441+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:08:55.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:08:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:08:55.539+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:08:55.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:08:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:08:55.539+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:08:55.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:08:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:08:55.539+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:08:55.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:08:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:08:55.539+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:08:56.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:08:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:08:56.524+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:08:56.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:08:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:08:56.524+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:08:56.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:08:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:08:56.524+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:08:56.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:08:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:08:56.573+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:08:56.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:08:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:08:56.573+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:08:56.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:08:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:08:56.573+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:08:56.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:08:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:08:56.573+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:08:56.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:08:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:08:56.488+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:08:56.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:08:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:08:56.488+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:08:56.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:08:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:08:56.488+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:08:57.241 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:08:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:08:56.807+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 7596 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:08:57.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:08:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:08:57.536+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:08:57.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:08:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:08:57.536+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:08:57.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:08:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:08:57.536+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:08:57.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:08:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:08:57.561+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:08:57.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:08:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:08:57.561+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:08:57.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:08:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:08:57.561+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:08:57.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:08:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:08:57.561+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:08:57.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:08:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:08:57.476+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:08:57.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:08:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:08:57.476+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:08:57.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:08:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:08:57.476+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:08:58.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:08:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:08:58.494+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:08:58.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:08:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:08:58.494+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:08:58.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:08:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:08:58.494+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:08:58.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:08:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:08:58.452+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:08:58.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:08:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:08:58.452+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:08:58.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:08:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:08:58.452+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:08:58.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:08:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:08:58.608+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:08:58.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:08:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:08:58.608+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:08:58.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:08:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:08:58.608+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:08:58.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:08:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:08:58.608+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:08:59.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:08:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:08:59.454+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:08:59.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:08:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:08:59.454+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:08:59.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:08:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:08:59.454+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:08:59.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:08:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:08:59.408+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:08:59.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:08:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:08:59.408+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:08:59.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:08:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:08:59.408+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:08:59.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:08:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:08:59.622+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:08:59.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:08:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:08:59.622+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:08:59.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:08:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:08:59.622+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:08:59.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:08:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:08:59.622+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:09:00.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:09:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:09:00.495+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:09:00.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:09:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:09:00.495+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:09:00.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:09:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:09:00.495+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:09:00.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:09:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:09:00.632+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:09:00.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:09:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:09:00.632+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:09:00.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:09:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:09:00.632+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:09:00.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:09:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:09:00.632+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:09:00.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:09:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:09:00.432+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:09:00.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:09:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:09:00.432+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:09:00.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:09:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:09:00.432+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:09:01.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:09:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:09:01.498+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:09:01.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:09:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:09:01.498+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:09:01.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:09:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:09:01.498+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:09:01.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:09:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:09:01.652+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:09:01.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:09:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:09:01.652+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:09:01.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:09:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:09:01.652+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:09:01.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:09:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:09:01.652+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:09:01.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:09:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:09:01.454+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:09:01.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:09:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:09:01.454+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:09:01.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:09:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:09:01.454+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:09:02.241 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:09:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:09:01.809+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 7679 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:09:02.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:09:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:09:02.482+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:09:02.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:09:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:09:02.482+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:09:02.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:09:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:09:02.482+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:09:02.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:09:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:09:02.676+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:09:02.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:09:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:09:02.676+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:09:02.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:09:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:09:02.676+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:09:02.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:09:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:09:02.676+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:09:02.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:09:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:09:02.474+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:09:02.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:09:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:09:02.474+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:09:02.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:09:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:09:02.474+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:09:03.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:09:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:09:03.470+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:09:03.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:09:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:09:03.470+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:09:03.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:09:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:09:03.470+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:09:03.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:09:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:09:03.684+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:09:03.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:09:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:09:03.684+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:09:03.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:09:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:09:03.684+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:09:03.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:09:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:09:03.685+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:09:03.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:09:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:09:03.495+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:09:03.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:09:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:09:03.495+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:09:03.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:09:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:09:03.495+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:09:04.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:09:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:09:04.501+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:09:04.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:09:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:09:04.501+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:09:04.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:09:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:09:04.501+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:09:04.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:09:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:09:04.691+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:09:04.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:09:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:09:04.691+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:09:04.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:09:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:09:04.691+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:09:04.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:09:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:09:04.691+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:09:04.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:09:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:09:04.479+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:09:04.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:09:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:09:04.479+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:09:04.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:09:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:09:04.479+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:09:05.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:09:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:09:05.488+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:09:05.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:09:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:09:05.488+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:09:05.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:09:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:09:05.488+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:09:05.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:09:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:09:05.704+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:09:05.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:09:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:09:05.704+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:09:05.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:09:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:09:05.704+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:09:05.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:09:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:09:05.704+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:09:05.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:09:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:09:05.509+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:09:05.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:09:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:09:05.509+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:09:05.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:09:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:09:05.509+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:09:06.810 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:09:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:09:06.705+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:09:06.810 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:09:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:09:06.705+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:09:06.810 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:09:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:09:06.705+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:09:06.810 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:09:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:09:06.705+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:09:06.811 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:09:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:09:06.505+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:09:06.811 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:09:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:09:06.505+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:09:06.811 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:09:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:09:06.505+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:09:06.811 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:09:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:09:06.546+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:09:06.811 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:09:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:09:06.546+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:09:06.811 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:09:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:09:06.546+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:09:07.241 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:09:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:09:06.809+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 7757 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:09:07.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:09:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:09:07.541+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:09:07.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:09:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:09:07.541+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:09:07.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:09:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:09:07.541+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:09:07.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:09:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:09:07.554+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:09:07.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:09:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:09:07.554+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:09:07.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:09:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:09:07.554+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:09:07.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:09:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:09:07.727+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:09:07.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:09:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:09:07.727+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:09:07.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:09:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:09:07.727+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:09:07.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:09:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:09:07.728+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:09:08.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:09:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:09:08.512+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:09:08.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:09:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:09:08.512+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:09:08.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:09:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:09:08.512+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:09:08.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:09:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:09:08.568+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:09:08.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:09:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:09:08.568+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:09:08.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:09:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:09:08.568+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:09:08.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:09:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:09:08.678+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:09:08.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:09:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:09:08.678+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:09:08.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:09:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:09:08.678+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:09:08.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:09:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:09:08.678+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:09:09.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:09:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:09:09.571+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:09:09.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:09:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:09:09.571+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:09:09.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:09:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:09:09.571+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:09:09.744 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:09:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:09:09.470+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:09:09.744 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:09:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:09:09.470+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:09:09.744 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:09:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:09:09.470+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:09:09.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:09:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:09:09.666+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:09:09.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:09:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:09:09.666+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:09:09.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:09:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:09:09.666+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:09:09.745 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:09:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:09:09.666+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:09:10.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:09:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:09:10.611+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:09:10.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:09:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:09:10.611+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:09:10.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:09:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:09:10.611+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:09:10.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:09:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:09:10.421+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:09:10.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:09:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:09:10.421+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:09:10.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:09:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:09:10.421+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:09:10.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:09:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:09:10.670+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:09:10.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:09:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:09:10.670+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:09:10.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:09:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:09:10.670+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:09:10.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:09:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:09:10.670+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:09:11.669 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:09:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:09:11.569+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:09:11.669 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:09:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:09:11.569+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:09:11.669 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:09:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:09:11.569+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:09:11.669 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:09:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:09:11.407+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:09:11.670 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:09:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:09:11.407+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:09:11.670 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:09:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:09:11.407+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:09:11.670 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:09:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:09:11.643+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:09:11.670 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:09:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:09:11.643+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:09:11.670 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:09:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:09:11.643+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:09:11.670 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:09:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:09:11.643+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:09:11.936 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:09:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:09:11.811+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 7838 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:09:12.664 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:09:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:09:12.388+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:09:12.664 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:09:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:09:12.388+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:09:12.664 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:09:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:09:12.388+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:09:12.664 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:09:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:09:12.526+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:09:12.664 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:09:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:09:12.526+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:09:12.664 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:09:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:09:12.526+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:09:12.990 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:09:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:09:12.663+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:09:12.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:09:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:09:12.663+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:09:12.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:09:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:09:12.663+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:09:12.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:09:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:09:12.663+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:09:13.697 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:09:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:09:13.366+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:09:13.697 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:09:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:09:13.366+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:09:13.697 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:09:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:09:13.366+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:09:13.697 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:09:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:09:13.574+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:09:13.697 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:09:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:09:13.574+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:09:13.697 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:09:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:09:13.574+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:09:13.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:09:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:09:13.695+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:09:13.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:09:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:09:13.695+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:09:13.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:09:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:09:13.695+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:09:13.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:09:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:09:13.695+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:09:14.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:09:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:09:14.368+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:09:14.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:09:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:09:14.368+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:09:14.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:09:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:09:14.368+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:09:14.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:09:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:09:14.534+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:09:14.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:09:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:09:14.534+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:09:14.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:09:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:09:14.534+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:09:14.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:09:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:09:14.727+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:09:14.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:09:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:09:14.727+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:09:14.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:09:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:09:14.727+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:09:14.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:09:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:09:14.727+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:09:15.685 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:09:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:09:15.377+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:09:15.685 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:09:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:09:15.377+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:09:15.685 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:09:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:09:15.377+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:09:15.685 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:09:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:09:15.494+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:09:15.685 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:09:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:09:15.494+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:09:15.686 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:09:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:09:15.494+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:09:15.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:09:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:09:15.684+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:09:15.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:09:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:09:15.684+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:09:15.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:09:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:09:15.684+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:09:15.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:09:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:09:15.684+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:09:16.723 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:09:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:09:16.524+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:09:16.723 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:09:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:09:16.524+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:09:16.723 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:09:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:09:16.524+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:09:16.723 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:09:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:09:16.393+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:09:16.723 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:09:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:09:16.393+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:09:16.723 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:09:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:09:16.393+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:09:16.991 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:09:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:09:16.813+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 7919 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:09:16.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:09:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:09:16.721+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:09:16.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:09:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:09:16.721+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:09:16.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:09:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:09:16.721+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:09:16.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:09:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:09:16.722+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:09:17.720 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:09:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:09:17.440+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:09:17.720 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:09:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:09:17.440+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:09:17.720 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:09:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:09:17.440+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:09:17.720 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:09:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:09:17.478+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:09:17.721 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:09:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:09:17.478+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:09:17.721 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:09:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:09:17.478+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:09:17.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:09:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:09:17.718+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:09:17.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:09:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:09:17.718+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:09:17.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:09:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:09:17.718+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:09:17.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:09:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:09:17.719+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:09:18.682 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:09:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:09:18.404+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:09:18.682 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:09:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:09:18.404+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:09:18.682 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:09:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:09:18.404+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:09:18.683 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:09:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:09:18.516+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:09:18.683 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:09:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:09:18.516+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:09:18.683 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:09:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:09:18.516+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:09:18.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:09:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:09:18.681+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:09:18.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:09:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:09:18.681+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:09:18.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:09:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:09:18.681+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:09:18.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:09:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:09:18.681+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:09:19.704 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:09:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:09:19.517+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:09:19.704 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:09:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:09:19.517+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:09:19.704 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:09:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:09:19.517+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:09:19.704 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:09:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:09:19.401+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:09:19.704 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:09:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:09:19.401+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:09:19.704 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:09:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:09:19.401+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:09:19.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:09:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:09:19.702+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:09:19.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:09:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:09:19.702+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:09:19.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:09:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:09:19.702+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:09:19.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:09:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:09:19.702+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:09:20.688 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:09:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:09:20.380+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:09:20.688 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:09:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:09:20.380+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:09:20.688 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:09:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:09:20.381+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:09:20.688 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:09:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:09:20.490+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:09:20.689 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:09:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:09:20.490+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:09:20.689 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:09:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:09:20.490+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:09:20.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:09:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:09:20.687+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:09:20.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:09:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:09:20.687+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:09:20.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:09:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:09:20.687+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:09:20.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:09:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:09:20.687+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:09:21.670 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:09:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:09:21.482+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:09:21.670 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:09:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:09:21.482+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:09:21.670 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:09:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:09:21.482+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:09:21.670 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:09:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:09:21.418+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:09:21.670 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:09:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:09:21.418+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:09:21.670 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:09:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:09:21.418+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:09:21.939 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:09:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:09:21.814+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 7996 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:09:21.940 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:09:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:09:21.699+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:09:21.940 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:09:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:09:21.699+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:09:21.940 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:09:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:09:21.699+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:09:21.940 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:09:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:09:21.700+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:09:22.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:09:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:09:22.459+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:09:22.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:09:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:09:22.459+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:09:22.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:09:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:09:22.459+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:09:22.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:09:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:09:22.664+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:09:22.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:09:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:09:22.664+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:09:22.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:09:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:09:22.664+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:09:22.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:09:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:09:22.664+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:09:22.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:09:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:09:22.458+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:09:22.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:09:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:09:22.458+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:09:22.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:09:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:09:22.458+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:09:23.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:09:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:09:23.427+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:09:23.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:09:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:09:23.427+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:09:23.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:09:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:09:23.427+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:09:23.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:09:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:09:23.647+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:09:23.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:09:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:09:23.647+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:09:23.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:09:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:09:23.647+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:09:23.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:09:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:09:23.647+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:09:23.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:09:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:09:23.494+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:09:23.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:09:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:09:23.494+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:09:23.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:09:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:09:23.494+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:09:24.696 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:09:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:09:24.488+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:09:24.696 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:09:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:09:24.488+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:09:24.696 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:09:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:09:24.488+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:09:24.697 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:09:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:09:24.402+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:09:24.697 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:09:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:09:24.402+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:09:24.697 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:09:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:09:24.402+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:09:24.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:09:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:09:24.695+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:09:24.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:09:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:09:24.695+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:09:24.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:09:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:09:24.695+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:09:24.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:09:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:09:24.695+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:09:25.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:09:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:09:25.450+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:09:25.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:09:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:09:25.450+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:09:25.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:09:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:09:25.450+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:09:25.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:09:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:09:25.502+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:09:25.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:09:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:09:25.502+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:09:25.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:09:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:09:25.502+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:09:25.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:09:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:09:25.697+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:09:25.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:09:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:09:25.697+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:09:25.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:09:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:09:25.697+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:09:25.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:09:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:09:25.697+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:09:26.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:09:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:09:26.472+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:09:26.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:09:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:09:26.472+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:09:26.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:09:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:09:26.472+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:09:26.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:09:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:09:26.668+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:09:26.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:09:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:09:26.668+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:09:26.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:09:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:09:26.668+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:09:26.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:09:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:09:26.668+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:09:26.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:09:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:09:26.483+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:09:26.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:09:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:09:26.483+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:09:26.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:09:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:09:26.483+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:09:27.240 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:09:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:09:26.816+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 8071 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:09:27.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:09:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:09:27.425+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:09:27.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:09:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:09:27.425+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:09:27.742 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:09:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:09:27.425+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:09:27.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:09:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:09:27.659+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:09:27.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:09:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:09:27.659+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:09:27.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:09:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:09:27.659+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:09:27.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:09:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:09:27.660+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:09:27.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:09:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:09:27.495+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:09:27.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:09:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:09:27.495+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:09:27.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:09:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:09:27.495+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:09:28.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:09:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:09:28.468+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:09:28.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:09:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:09:28.468+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:09:28.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:09:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:09:28.468+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:09:28.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:09:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:09:28.537+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:09:28.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:09:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:09:28.537+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:09:28.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:09:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:09:28.537+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:09:28.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:09:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:09:28.631+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:09:28.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:09:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:09:28.631+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:09:28.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:09:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:09:28.631+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:09:28.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:09:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:09:28.631+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:09:29.885 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:09:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:09:29.546+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:09:29.885 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:09:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:09:29.546+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:09:29.885 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:09:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:09:29.546+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:09:29.885 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:09:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:09:29.504+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:09:29.886 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:09:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:09:29.504+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:09:29.886 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:09:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:09:29.504+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:09:29.886 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:09:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:09:29.650+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:09:29.886 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:09:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:09:29.650+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:09:29.886 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:09:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:09:29.650+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:09:29.886 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:09:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:09:29.650+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:09:30.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:09:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:09:30.496+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:09:30.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:09:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:09:30.496+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:09:30.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:09:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:09:30.496+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:09:30.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:09:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:09:30.520+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:09:30.992 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:09:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:09:30.520+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:09:30.992 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:09:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:09:30.520+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:09:30.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:09:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:09:30.655+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:09:30.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:09:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:09:30.655+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:09:30.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:09:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:09:30.655+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:09:30.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:09:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:09:30.655+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:09:31.819 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:09:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:09:31.523+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:09:31.819 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:09:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:09:31.523+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:09:31.819 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:09:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:09:31.523+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:09:31.820 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:09:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:09:31.663+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:09:31.820 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:09:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:09:31.663+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:09:31.820 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:09:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:09:31.663+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:09:31.820 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:09:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:09:31.663+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:09:31.820 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:09:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:09:31.528+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:09:31.820 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:09:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:09:31.528+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:09:31.820 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:09:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:09:31.528+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:09:32.241 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:09:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:09:31.818+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 8149 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:09:32.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:09:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:09:32.554+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:09:32.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:09:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:09:32.554+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:09:32.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:09:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:09:32.554+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:09:32.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:09:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:09:32.538+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:09:32.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:09:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:09:32.538+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:09:32.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:09:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:09:32.538+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:09:32.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:09:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:09:32.702+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:09:32.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:09:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:09:32.702+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:09:32.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:09:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:09:32.702+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:09:32.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:09:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:09:32.702+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:09:33.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:09:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:09:33.584+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:09:33.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:09:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:09:33.584+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:09:33.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:09:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:09:33.584+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:09:33.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:09:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:09:33.549+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:09:33.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:09:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:09:33.549+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:09:33.992 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:09:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:09:33.549+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:09:33.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:09:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:09:33.655+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:09:33.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:09:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:09:33.655+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:09:33.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:09:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:09:33.655+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:09:33.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:09:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:09:33.655+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:09:34.990 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:09:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:09:34.540+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:09:34.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:09:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:09:34.540+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:09:34.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:09:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:09:34.540+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:09:34.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:09:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:09:34.665+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:09:34.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:09:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:09:34.665+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:09:34.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:09:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:09:34.665+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:09:34.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:09:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:09:34.665+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:09:34.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:09:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:09:34.593+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:09:34.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:09:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:09:34.593+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:09:34.992 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:09:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:09:34.593+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:09:35.990 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:09:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:09:35.530+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:09:35.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:09:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:09:35.530+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:09:35.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:09:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:09:35.530+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:09:35.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:09:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:09:35.624+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:09:35.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:09:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:09:35.624+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:09:35.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:09:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:09:35.624+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:09:35.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:09:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:09:35.624+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:09:35.992 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:09:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:09:35.624+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:09:35.992 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:09:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:09:35.624+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:09:35.992 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:09:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:09:35.624+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:09:36.821 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:09:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:09:36.515+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:09:36.821 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:09:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:09:36.515+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:09:36.821 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:09:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:09:36.515+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:09:36.821 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:09:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:09:36.651+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:09:36.821 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:09:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:09:36.651+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:09:36.821 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:09:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:09:36.651+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:09:36.822 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:09:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:09:36.651+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:09:36.822 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:09:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:09:36.623+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:09:36.822 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:09:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:09:36.623+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:09:36.822 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:09:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:09:36.623+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:09:37.241 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:09:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:09:36.820+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 8225 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:09:37.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:09:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:09:37.502+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:09:37.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:09:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:09:37.502+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:09:37.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:09:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:09:37.502+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:09:37.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:09:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:09:37.608+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:09:37.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:09:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:09:37.608+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:09:37.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:09:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:09:37.608+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:09:37.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:09:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:09:37.638+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:09:37.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:09:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:09:37.638+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:09:37.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:09:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:09:37.638+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:09:37.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:09:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:09:37.638+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:09:38.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:09:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:09:38.551+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:09:38.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:09:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:09:38.551+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:09:38.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:09:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:09:38.551+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:09:38.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:09:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:09:38.624+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:09:38.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:09:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:09:38.624+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:09:38.992 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:09:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:09:38.624+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:09:38.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:09:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:09:38.603+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:09:38.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:09:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:09:38.603+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:09:38.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:09:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:09:38.603+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:09:38.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:09:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:09:38.603+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:09:39.885 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:09:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:09:39.587+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:09:39.885 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:09:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:09:39.587+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:09:39.885 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:09:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:09:39.587+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:09:39.885 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:09:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:09:39.610+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:09:39.885 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:09:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:09:39.610+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:09:39.886 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:09:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:09:39.610+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:09:39.886 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:09:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:09:39.626+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:09:39.886 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:09:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:09:39.626+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:09:39.886 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:09:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:09:39.626+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:09:39.886 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:09:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:09:39.626+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:09:40.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:09:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:09:40.592+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:09:40.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:09:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:09:40.592+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:09:40.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:09:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:09:40.592+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:09:40.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:09:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:09:40.567+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:09:40.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:09:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:09:40.567+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:09:40.992 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:09:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:09:40.567+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:09:40.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:09:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:09:40.641+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:09:40.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:09:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:09:40.641+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:09:40.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:09:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:09:40.641+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:09:40.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:09:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:09:40.641+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:09:41.822 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:09:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:09:41.583+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:09:41.822 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:09:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:09:41.583+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:09:41.822 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:09:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:09:41.583+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:09:41.823 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:09:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:09:41.678+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:09:41.823 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:09:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:09:41.678+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:09:41.823 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:09:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:09:41.678+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:09:41.823 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:09:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:09:41.678+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:09:41.823 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:09:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:09:41.562+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:09:41.823 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:09:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:09:41.562+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:09:41.823 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:09:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:09:41.562+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:09:42.241 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:09:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:09:41.822+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 8305 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:09:42.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:09:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:09:42.608+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:09:42.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:09:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:09:42.608+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:09:42.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:09:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:09:42.608+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:09:42.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:09:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:09:42.514+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:09:42.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:09:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:09:42.514+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:09:42.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:09:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:09:42.514+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:09:42.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:09:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:09:42.652+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:09:42.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:09:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:09:42.652+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:09:42.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:09:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:09:42.652+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:09:42.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:09:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:09:42.652+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:09:43.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:09:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:09:43.640+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:09:43.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:09:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:09:43.640+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:09:43.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:09:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:09:43.640+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:09:43.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:09:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:09:43.539+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:09:43.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:09:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:09:43.539+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:09:43.992 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:09:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:09:43.539+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:09:43.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:09:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:09:43.641+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:09:43.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:09:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:09:43.641+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:09:43.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:09:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:09:43.641+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:09:43.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:09:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:09:43.641+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:09:44.951 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:09:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:09:44.664+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:09:44.951 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:09:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:09:44.664+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:09:44.951 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:09:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:09:44.664+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:09:44.951 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:09:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:09:44.664+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:09:44.952 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:09:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:09:44.648+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:09:44.952 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:09:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:09:44.648+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:09:44.952 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:09:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:09:44.648+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:09:44.952 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:09:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:09:44.562+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:09:44.952 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:09:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:09:44.562+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:09:44.952 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:09:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:09:44.562+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:09:45.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:09:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:09:45.676+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:09:45.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:09:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:09:45.676+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:09:45.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:09:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:09:45.676+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:09:45.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:09:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:09:45.571+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:09:45.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:09:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:09:45.571+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:09:45.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:09:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:09:45.571+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:09:45.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:09:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:09:45.672+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:09:45.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:09:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:09:45.672+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:09:45.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:09:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:09:45.672+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:09:45.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:09:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:09:45.672+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:09:46.845 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:09:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:09:46.823+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 8383 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:09:46.845 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:09:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:09:46.698+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:09:46.845 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:09:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:09:46.698+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:09:46.845 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:09:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:09:46.698+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:09:46.846 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:09:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:09:46.589+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:09:46.846 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:09:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:09:46.589+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:09:46.846 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:09:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:09:46.589+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:09:46.846 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:09:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:09:46.691+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:09:46.846 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:09:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:09:46.691+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:09:46.846 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:09:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:09:46.691+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:09:46.846 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:09:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:09:46.691+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:09:47.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:09:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:09:47.721+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:09:47.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:09:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:09:47.721+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:09:47.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:09:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:09:47.721+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:09:47.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:09:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:09:47.692+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:09:47.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:09:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:09:47.692+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:09:47.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:09:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:09:47.692+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:09:47.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:09:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:09:47.692+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:09:47.992 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:09:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:09:47.591+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:09:47.992 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:09:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:09:47.591+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:09:47.992 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:09:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:09:47.591+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:09:48.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:09:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:09:48.676+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:09:48.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:09:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:09:48.676+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:09:48.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:09:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:09:48.676+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:09:48.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:09:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:09:48.567+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:09:48.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:09:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:09:48.567+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:09:48.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:09:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:09:48.567+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:09:48.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:09:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:09:48.739+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:09:48.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:09:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:09:48.739+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:09:48.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:09:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:09:48.739+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:09:48.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:09:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:09:48.739+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:09:49.885 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:09:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:09:49.705+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:09:49.885 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:09:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:09:49.705+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:09:49.885 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:09:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:09:49.705+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:09:49.886 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:09:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:09:49.594+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:09:49.886 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:09:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:09:49.594+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:09:49.886 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:09:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:09:49.594+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:09:49.886 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:09:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:09:49.737+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:09:49.886 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:09:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:09:49.737+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:09:49.886 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:09:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:09:49.737+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:09:49.886 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:09:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:09:49.737+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:09:50.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:09:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:09:50.670+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:09:50.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:09:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:09:50.670+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:09:50.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:09:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:09:50.670+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:09:50.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:09:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:09:50.572+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:09:50.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:09:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:09:50.572+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:09:50.992 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:09:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:09:50.572+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:09:50.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:09:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:09:50.751+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:09:50.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:09:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:09:50.751+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:09:50.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:09:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:09:50.751+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:09:50.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:09:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:09:50.751+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:09:51.825 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:09:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:09:51.753+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:09:51.825 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:09:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:09:51.753+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:09:51.825 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:09:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:09:51.753+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:09:51.826 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:09:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:09:51.753+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:09:51.826 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:09:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:09:51.697+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:09:51.826 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:09:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:09:51.697+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:09:51.826 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:09:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:09:51.697+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:09:51.826 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:09:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:09:51.562+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:09:51.826 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:09:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:09:51.562+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:09:51.826 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:09:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:09:51.562+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:09:52.241 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:09:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:09:51.825+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 8464 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:09:52.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:09:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:09:52.523+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:09:52.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:09:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:09:52.523+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:09:52.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:09:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:09:52.523+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:09:52.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:09:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:09:52.666+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:09:52.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:09:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:09:52.666+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:09:52.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:09:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:09:52.666+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:09:52.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:09:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:09:52.731+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:09:52.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:09:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:09:52.731+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:09:52.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:09:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:09:52.731+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:09:52.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:09:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:09:52.732+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:09:53.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:09:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:09:53.634+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:09:53.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:09:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:09:53.634+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:09:53.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:09:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:09:53.634+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:09:53.992 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:09:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:09:53.517+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:09:53.992 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:09:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:09:53.517+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:09:53.992 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:09:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:09:53.517+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:09:53.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:09:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:09:53.725+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:09:53.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:09:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:09:53.725+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:09:53.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:09:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:09:53.725+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:09:53.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:09:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:09:53.725+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:09:54.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:09:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:09:54.489+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:09:54.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:09:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:09:54.489+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:09:54.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:09:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:09:54.489+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:09:54.742 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:09:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:09:54.631+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:09:54.742 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:09:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:09:54.631+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:09:54.742 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:09:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:09:54.631+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:09:54.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:09:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:09:54.733+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:09:54.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:09:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:09:54.733+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:09:54.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:09:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:09:54.733+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:09:54.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:09:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:09:54.733+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:09:55.774 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:09:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:09:55.618+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:09:55.774 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:09:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:09:55.618+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:09:55.774 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:09:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:09:55.618+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:09:55.774 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:09:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:09:55.499+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:09:55.775 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:09:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:09:55.499+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:09:55.775 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:09:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:09:55.499+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:09:56.241 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:09:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:09:55.773+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:09:56.241 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:09:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:09:55.773+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:09:56.241 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:09:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:09:55.773+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:09:56.241 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:09:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:09:55.773+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:09:56.827 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:09:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:09:56.754+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:09:56.827 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:09:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:09:56.754+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:09:56.827 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:09:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:09:56.754+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:09:56.827 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:09:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:09:56.754+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:09:56.827 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:09:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:09:56.521+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:09:56.827 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:09:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:09:56.521+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:09:56.827 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:09:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:09:56.521+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:09:56.828 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:09:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:09:56.662+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:09:56.828 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:09:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:09:56.662+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:09:56.828 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:09:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:09:56.662+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:09:57.241 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:09:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:09:56.825+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 8546 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:09:57.801 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:09:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:09:57.496+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:09:57.802 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:09:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:09:57.496+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:09:57.802 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:09:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:09:57.496+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:09:57.802 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:09:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:09:57.704+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:09:57.802 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:09:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:09:57.704+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:09:57.802 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:09:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:09:57.704+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:09:58.241 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:09:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:09:57.800+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:09:58.241 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:09:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:09:57.800+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:09:58.241 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:09:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:09:57.800+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:09:58.241 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:09:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:09:57.800+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:09:58.832 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:09:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:09:58.662+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:09:58.832 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:09:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:09:58.662+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:09:58.832 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:09:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:09:58.662+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:09:58.833 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:09:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:09:58.541+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:09:58.833 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:09:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:09:58.541+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:09:58.833 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:09:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:09:58.541+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:09:59.241 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:09:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:09:58.831+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:09:59.241 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:09:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:09:58.831+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:09:59.241 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:09:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:09:58.831+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:09:59.241 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:09:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:09:58.831+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:09:59.855 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:09:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:09:59.656+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:09:59.856 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:09:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:09:59.656+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:09:59.856 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:09:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:09:59.656+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:09:59.856 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:09:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:09:59.505+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:09:59.856 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:09:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:09:59.505+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:09:59.856 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:09:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:09:59.505+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:10:00.241 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:09:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:09:59.855+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:10:00.241 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:09:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:09:59.855+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:10:00.241 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:09:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:09:59.855+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:10:00.241 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:09:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:09:59.855+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:10:00.610 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:10:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:10:00.458+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:10:00.610 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:10:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:10:00.458+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:10:00.610 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:10:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:10:00.458+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:10:00.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:10:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:10:00.815+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:10:00.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:10:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:10:00.815+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:10:00.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:10:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:10:00.815+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:10:00.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:10:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:10:00.816+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:10:00.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:10:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:10:00.609+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:10:00.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:10:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:10:00.609+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:10:00.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:10:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:10:00.609+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:10:01.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:10:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:10:01.442+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:10:01.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:10:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:10:01.442+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:10:01.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:10:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:10:01.442+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:10:01.742 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:10:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:10:01.561+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:10:01.742 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:10:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:10:01.561+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:10:01.742 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:10:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:10:01.561+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:10:02.241 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:10:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:10:01.827+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 8629 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:10:02.241 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:10:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:10:01.766+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:10:02.241 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:10:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:10:01.766+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:10:02.241 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:10:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:10:01.766+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:10:02.241 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:10:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:10:01.767+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:10:02.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:10:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:10:02.609+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:10:02.742 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:10:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:10:02.609+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:10:02.742 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:10:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:10:02.609+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:10:02.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:10:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:10:02.435+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:10:02.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:10:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:10:02.435+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:10:02.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:10:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:10:02.435+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:10:03.241 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:10:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:10:02.783+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:10:03.241 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:10:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:10:02.783+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:10:03.241 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:10:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:10:02.783+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:10:03.241 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:10:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:10:02.783+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:10:03.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:10:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:10:03.577+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:10:03.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:10:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:10:03.577+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:10:03.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:10:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:10:03.577+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:10:03.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:10:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:10:03.396+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:10:03.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:10:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:10:03.396+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:10:03.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:10:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:10:03.396+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:10:04.241 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:10:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:10:03.768+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:10:04.241 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:10:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:10:03.768+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:10:04.241 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:10:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:10:03.768+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:10:04.241 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:10:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:10:03.768+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:10:04.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:10:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:10:04.369+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:10:04.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:10:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:10:04.369+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:10:04.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:10:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:10:04.369+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:10:04.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:10:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:10:04.597+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:10:04.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:10:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:10:04.597+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:10:04.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:10:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:10:04.597+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:10:05.241 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:10:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:10:04.789+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:10:05.241 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:10:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:10:04.789+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:10:05.241 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:10:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:10:04.789+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:10:05.241 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:10:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:10:04.789+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:10:05.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:10:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:10:05.557+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:10:05.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:10:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:10:05.557+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:10:05.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:10:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:10:05.557+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:10:05.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:10:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:10:05.393+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:10:05.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:10:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:10:05.393+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:10:05.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:10:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:10:05.393+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:10:06.241 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:10:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:10:05.778+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:10:06.241 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:10:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:10:05.778+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:10:06.241 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:10:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:10:05.778+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:10:06.241 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:10:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:10:05.778+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:10:06.511 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:10:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:10:06.398+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:10:06.511 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:10:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:10:06.398+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:10:06.511 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:10:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:10:06.398+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:10:06.772 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:10:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:10:06.511+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:10:06.772 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:10:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:10:06.511+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:10:06.772 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:10:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:10:06.511+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:10:07.241 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:10:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:10:06.829+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 8710 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:10:07.241 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:10:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:10:06.771+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:10:07.241 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:10:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:10:06.771+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:10:07.241 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:10:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:10:06.771+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:10:07.241 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:10:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:10:06.771+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:10:07.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:10:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:10:07.544+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:10:07.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:10:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:10:07.544+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:10:07.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:10:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:10:07.544+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:10:07.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:10:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:10:07.386+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:10:07.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:10:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:10:07.386+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:10:07.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:10:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:10:07.386+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:10:08.241 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:10:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:10:07.741+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:10:08.241 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:10:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:10:07.741+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:10:08.241 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:10:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:10:07.741+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:10:08.241 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:10:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:10:07.741+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:10:08.710 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:10:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:10:08.552+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:10:08.710 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:10:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:10:08.552+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:10:08.710 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:10:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:10:08.552+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:10:08.711 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:10:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:10:08.397+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:10:08.711 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:10:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:10:08.397+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:10:08.711 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:10:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:10:08.397+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:10:08.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:10:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:10:08.709+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:10:08.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:10:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:10:08.709+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:10:08.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:10:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:10:08.709+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:10:08.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:10:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:10:08.709+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:10:09.702 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:10:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:10:09.582+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:10:09.702 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:10:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:10:09.582+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:10:09.702 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:10:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:10:09.582+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:10:09.702 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:10:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:10:09.370+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:10:09.702 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:10:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:10:09.370+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:10:09.702 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:10:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:10:09.370+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:10:09.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:10:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:10:09.701+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:10:09.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:10:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:10:09.701+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:10:09.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:10:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:10:09.701+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:10:09.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:10:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:10:09.701+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:10:10.662 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:10:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:10:10.615+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:10:10.662 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:10:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:10:10.615+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:10:10.662 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:10:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:10:10.615+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:10:10.662 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:10:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:10:10.372+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:10:10.662 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:10:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:10:10.372+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:10:10.663 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:10:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:10:10.372+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:10:10.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:10:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:10:10.661+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:10:10.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:10:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:10:10.661+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:10:10.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:10:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:10:10.661+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:10:10.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:10:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:10:10.661+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:10:11.674 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:10:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:10:11.386+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:10:11.674 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:10:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:10:11.386+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:10:11.674 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:10:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:10:11.386+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:10:11.675 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:10:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:10:11.608+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:10:11.675 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:10:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:10:11.608+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:10:11.675 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:10:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:10:11.608+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:10:11.675 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:10:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:10:11.621+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:10:11.675 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:10:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:10:11.621+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:10:11.675 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:10:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:10:11.621+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:10:11.675 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:10:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:10:11.621+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:10:11.956 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:10:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:10:11.830+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 8794 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:10:12.599 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:10:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:10:12.372+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:10:12.599 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:10:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:10:12.372+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:10:12.600 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:10:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:10:12.372+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:10:12.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:10:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:10:12.641+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:10:12.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:10:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:10:12.641+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:10:12.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:10:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:10:12.641+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:10:12.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:10:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:10:12.623+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:10:12.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:10:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:10:12.623+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:10:12.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:10:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:10:12.623+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:10:12.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:10:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:10:12.623+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:10:13.684 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:10:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:10:13.392+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:10:13.685 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:10:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:10:13.392+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:10:13.685 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:10:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:10:13.392+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:10:13.685 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:10:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:10:13.623+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:10:13.685 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:10:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:10:13.623+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:10:13.685 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:10:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:10:13.623+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:10:13.685 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:10:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:10:13.623+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:10:13.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:10:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:10:13.683+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:10:13.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:10:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:10:13.683+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:10:13.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:10:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:10:13.683+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:10:14.667 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:10:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:10:14.367+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:10:14.667 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:10:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:10:14.368+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:10:14.667 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:10:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:10:14.368+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:10:14.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:10:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:10:14.687+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:10:14.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:10:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:10:14.687+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:10:14.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:10:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:10:14.687+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:10:14.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:10:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:10:14.666+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:10:14.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:10:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:10:14.666+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:10:14.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:10:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:10:14.666+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:10:14.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:10:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:10:14.666+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:10:15.671 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:10:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:10:15.325+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:10:15.671 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:10:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:10:15.325+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:10:15.671 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:10:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:10:15.325+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:10:15.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:10:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:10:15.702+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:10:15.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:10:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:10:15.702+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:10:15.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:10:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:10:15.702+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:10:15.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:10:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:10:15.682+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:10:15.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:10:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:10:15.682+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:10:15.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:10:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:10:15.682+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:10:15.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:10:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:10:15.682+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:10:16.634 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:10:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:10:16.329+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:10:16.634 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:10:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:10:16.329+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:10:16.634 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:10:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:10:16.329+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:10:16.958 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:10:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:10:16.832+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 8872 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:10:16.958 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:10:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:10:16.744+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:10:16.958 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:10:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:10:16.744+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:10:16.958 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:10:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:10:16.744+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:10:16.959 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:10:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:10:16.634+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:10:16.959 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:10:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:10:16.634+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:10:16.959 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:10:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:10:16.634+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:10:16.959 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:10:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:10:16.634+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:10:17.682 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:10:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:10:17.360+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:10:17.682 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:10:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:10:17.360+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:10:17.682 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:10:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:10:17.360+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:10:17.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:10:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:10:17.719+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:10:17.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:10:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:10:17.719+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:10:17.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:10:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:10:17.719+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:10:17.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:10:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:10:17.681+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:10:17.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:10:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:10:17.681+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:10:17.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:10:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:10:17.681+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:10:17.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:10:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:10:17.681+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:10:18.657 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:10:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:10:18.377+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:10:18.657 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:10:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:10:18.377+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:10:18.657 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:10:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:10:18.377+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:10:18.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:10:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:10:18.706+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:10:18.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:10:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:10:18.706+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:10:18.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:10:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:10:18.706+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:10:18.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:10:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:10:18.656+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:10:18.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:10:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:10:18.656+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:10:18.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:10:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:10:18.656+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:10:18.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:10:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:10:18.656+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:10:19.696 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:10:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:10:19.334+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:10:19.696 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:10:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:10:19.334+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:10:19.696 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:10:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:10:19.334+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:10:19.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:10:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:10:19.695+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:10:19.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:10:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:10:19.695+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:10:19.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:10:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:10:19.695+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:10:19.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:10:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:10:19.695+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:10:19.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:10:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:10:19.698+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:10:19.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:10:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:10:19.698+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:10:19.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:10:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:10:19.698+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:10:20.657 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:10:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:10:20.307+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:10:20.657 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:10:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:10:20.307+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:10:20.657 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:10:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:10:20.307+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:10:20.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:10:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:10:20.694+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:10:20.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:10:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:10:20.694+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:10:20.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:10:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:10:20.694+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:10:20.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:10:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:10:20.657+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:10:20.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:10:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:10:20.657+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:10:20.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:10:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:10:20.657+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:10:20.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:10:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:10:20.657+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:10:21.675 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:10:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:10:21.277+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:10:21.675 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:10:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:10:21.277+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:10:21.675 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:10:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:10:21.277+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:10:21.960 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:10:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:10:21.834+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 8954 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:10:21.961 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:10:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:10:21.693+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:10:21.961 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:10:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:10:21.693+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:10:21.961 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:10:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:10:21.693+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:10:21.961 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:10:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:10:21.674+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:10:21.961 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:10:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:10:21.674+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:10:21.961 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:10:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:10:21.674+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:10:21.961 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:10:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:10:21.674+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:10:22.639 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:10:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:10:22.265+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:10:22.639 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:10:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:10:22.265+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:10:22.639 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:10:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:10:22.265+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:10:22.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:10:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:10:22.666+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:10:22.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:10:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:10:22.666+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:10:22.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:10:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:10:22.666+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:10:22.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:10:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:10:22.638+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:10:22.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:10:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:10:22.638+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:10:22.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:10:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:10:22.638+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:10:22.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:10:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:10:22.638+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:10:23.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:10:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:10:23.236+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:10:23.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:10:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:10:23.236+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:10:23.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:10:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:10:23.236+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:10:23.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:10:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:10:23.645+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:10:23.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:10:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:10:23.645+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:10:23.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:10:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:10:23.645+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:10:23.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:10:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:10:23.677+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:10:23.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:10:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:10:23.677+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:10:23.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:10:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:10:23.677+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:10:23.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:10:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:10:23.677+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:10:24.640 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:10:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:10:24.261+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:10:24.640 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:10:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:10:24.261+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:10:24.640 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:10:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:10:24.261+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:10:24.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:10:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:10:24.639+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:10:24.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:10:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:10:24.639+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:10:24.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:10:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:10:24.639+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:10:24.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:10:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:10:24.682+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:10:24.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:10:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:10:24.682+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:10:24.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:10:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:10:24.682+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:10:24.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:10:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:10:24.682+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:10:25.673 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:10:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:10:25.245+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:10:25.673 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:10:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:10:25.245+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:10:25.673 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:10:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:10:25.245+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:10:25.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:10:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:10:25.673+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:10:25.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:10:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:10:25.673+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:10:25.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:10:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:10:25.673+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:10:25.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:10:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:10:25.698+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:10:25.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:10:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:10:25.698+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:10:25.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:10:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:10:25.698+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:10:25.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:10:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:10:25.698+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:10:26.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:10:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:10:26.212+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:10:26.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:10:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:10:26.212+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:10:26.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:10:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:10:26.212+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:10:26.959 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:10:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:10:26.835+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 9036 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:10:26.960 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:10:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:10:26.658+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:10:26.960 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:10:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:10:26.658+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:10:26.960 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:10:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:10:26.658+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:10:26.960 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:10:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:10:26.693+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:10:26.960 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:10:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:10:26.693+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:10:26.960 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:10:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:10:26.693+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:10:26.960 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:10:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:10:26.693+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:10:27.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:10:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:10:27.165+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:10:27.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:10:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:10:27.165+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:10:27.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:10:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:10:27.165+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:10:27.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:10:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:10:27.654+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:10:27.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:10:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:10:27.654+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:10:27.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:10:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:10:27.654+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:10:27.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:10:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:10:27.671+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:10:27.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:10:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:10:27.671+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:10:27.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:10:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:10:27.671+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:10:27.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:10:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:10:27.671+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:10:28.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:10:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:10:28.130+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:10:28.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:10:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:10:28.130+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:10:28.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:10:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:10:28.130+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:10:28.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:10:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:10:28.644+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:10:28.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:10:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:10:28.644+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:10:28.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:10:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:10:28.644+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:10:28.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:10:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:10:28.644+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:10:28.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:10:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:10:28.644+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:10:28.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:10:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:10:28.644+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:10:28.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:10:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:10:28.644+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:10:29.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:10:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:10:29.083+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:10:29.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:10:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:10:29.083+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:10:29.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:10:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:10:29.083+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:10:29.884 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:10:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:10:29.617+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:10:29.884 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:10:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:10:29.617+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:10:29.884 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:10:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:10:29.617+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:10:29.884 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:10:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:10:29.637+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:10:29.885 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:10:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:10:29.637+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:10:29.885 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:10:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:10:29.637+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:10:29.885 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:10:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:10:29.637+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:10:30.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:10:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:10:30.038+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:10:30.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:10:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:10:30.038+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:10:30.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:10:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:10:30.038+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:10:30.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:10:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:10:30.596+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:10:30.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:10:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:10:30.596+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:10:30.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:10:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:10:30.596+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:10:30.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:10:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:10:30.593+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:10:30.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:10:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:10:30.593+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:10:30.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:10:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:10:30.593+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:10:30.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:10:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:10:30.593+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:10:31.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:10:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:10:31.080+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:10:31.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:10:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:10:31.080+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:10:31.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:10:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:10:31.080+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:10:31.838 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:10:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:10:31.610+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:10:31.838 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:10:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:10:31.610+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:10:31.838 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:10:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:10:31.610+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:10:31.838 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:10:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:10:31.547+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:10:31.839 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:10:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:10:31.547+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:10:31.839 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:10:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:10:31.547+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:10:31.839 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:10:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:10:31.547+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:10:32.241 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:10:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:10:31.838+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 9119 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:10:32.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:10:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:10:32.077+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:10:32.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:10:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:10:32.077+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:10:32.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:10:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:10:32.077+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:10:32.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:10:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:10:32.579+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:10:32.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:10:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:10:32.579+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:10:32.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:10:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:10:32.579+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:10:32.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:10:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:10:32.572+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:10:32.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:10:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:10:32.572+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:10:32.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:10:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:10:32.572+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:10:32.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:10:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:10:32.573+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:10:33.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:10:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:10:33.073+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:10:33.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:10:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:10:33.073+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:10:33.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:10:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:10:33.073+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:10:33.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:10:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:10:33.626+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:10:33.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:10:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:10:33.626+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:10:33.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:10:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:10:33.626+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:10:33.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:10:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:10:33.598+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:10:33.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:10:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:10:33.598+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:10:33.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:10:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:10:33.598+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:10:33.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:10:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:10:33.598+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:10:34.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:10:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:10:34.043+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:10:34.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:10:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:10:34.043+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:10:34.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:10:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:10:34.043+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:10:34.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:10:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:10:34.620+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:10:34.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:10:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:10:34.620+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:10:34.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:10:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:10:34.620+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:10:34.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:10:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:10:34.612+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:10:34.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:10:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:10:34.612+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:10:34.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:10:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:10:34.612+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:10:34.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:10:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:10:34.612+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:10:35.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:10:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:10:35.031+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:10:35.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:10:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:10:35.031+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:10:35.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:10:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:10:35.031+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:10:35.984 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:10:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:10:35.641+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:10:35.985 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:10:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:10:35.641+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:10:35.985 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:10:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:10:35.641+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:10:35.985 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:10:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:10:35.622+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:10:35.985 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:10:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:10:35.622+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:10:35.985 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:10:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:10:35.622+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:10:35.985 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:10:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:10:35.622+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:10:36.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:10:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:10:35.984+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:10:36.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:10:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:10:35.984+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:10:36.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:10:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:10:35.984+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:10:36.965 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:10:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:10:36.839+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 9202 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:10:36.965 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:10:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:10:36.660+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:10:36.965 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:10:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:10:36.660+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:10:36.965 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:10:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:10:36.660+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:10:36.965 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:10:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:10:36.663+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:10:36.965 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:10:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:10:36.663+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:10:36.965 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:10:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:10:36.663+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:10:36.966 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:10:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:10:36.664+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:10:37.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:10:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:10:36.982+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:10:37.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:10:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:10:36.982+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:10:37.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:10:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:10:36.982+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:10:37.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:10:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:10:37.675+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:10:37.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:10:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:10:37.675+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:10:37.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:10:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:10:37.675+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:10:37.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:10:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:10:37.675+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:10:37.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:10:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:10:37.675+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:10:37.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:10:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:10:37.675+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:10:37.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:10:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:10:37.675+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:10:38.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:10:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:10:37.994+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:10:38.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:10:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:10:37.994+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:10:38.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:10:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:10:37.994+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:10:38.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:10:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:10:38.675+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:10:38.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:10:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:10:38.675+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:10:38.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:10:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:10:38.675+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:10:38.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:10:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:10:38.680+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:10:38.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:10:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:10:38.680+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:10:38.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:10:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:10:38.680+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:10:38.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:10:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:10:38.680+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:10:39.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:10:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:10:39.018+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:10:39.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:10:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:10:39.018+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:10:39.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:10:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:10:39.018+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:10:39.885 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:10:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:10:39.681+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:10:39.885 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:10:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:10:39.681+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:10:39.885 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:10:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:10:39.681+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:10:39.885 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:10:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:10:39.633+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:10:39.885 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:10:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:10:39.633+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:10:39.885 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:10:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:10:39.633+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:10:39.885 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:10:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:10:39.633+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:10:40.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:10:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:10:40.037+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:10:40.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:10:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:10:40.037+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:10:40.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:10:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:10:40.037+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:10:40.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:10:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:10:40.731+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:10:40.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:10:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:10:40.731+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:10:40.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:10:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:10:40.731+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:10:40.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:10:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:10:40.618+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:10:40.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:10:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:10:40.618+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:10:40.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:10:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:10:40.618+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:10:40.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:10:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:10:40.618+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:10:41.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:10:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:10:40.999+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:10:41.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:10:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:10:40.999+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:10:41.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:10:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:10:40.999+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:10:41.842 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:10:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:10:41.726+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:10:41.842 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:10:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:10:41.726+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:10:41.842 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:10:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:10:41.726+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:10:41.842 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:10:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:10:41.571+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:10:41.842 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:10:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:10:41.571+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:10:41.842 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:10:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:10:41.571+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:10:41.842 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:10:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:10:41.571+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:10:42.241 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:10:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:10:41.841+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 9284 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:10:42.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:10:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:10:42.040+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:10:42.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:10:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:10:42.040+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:10:42.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:10:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:10:42.040+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:10:42.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:10:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:10:42.685+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:10:42.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:10:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:10:42.685+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:10:42.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:10:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:10:42.685+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:10:42.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:10:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:10:42.554+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:10:42.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:10:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:10:42.554+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:10:42.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:10:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:10:42.554+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:10:42.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:10:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:10:42.554+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:10:43.319 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:10:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:10:43.038+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:10:43.320 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:10:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:10:43.038+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:10:43.320 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:10:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:10:43.038+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:10:43.570 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:10:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:10:43.538+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:10:43.570 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:10:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:10:43.538+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:10:43.570 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:10:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:10:43.538+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:10:43.571 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:10:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:10:43.538+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:10:43.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:10:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:10:43.672+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:10:43.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:10:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:10:43.672+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:10:43.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:10:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:10:43.672+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:10:44.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:10:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:10:44.055+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:10:44.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:10:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:10:44.055+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:10:44.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:10:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:10:44.055+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:10:44.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:10:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:10:44.532+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:10:44.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:10:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:10:44.532+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:10:44.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:10:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:10:44.532+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:10:44.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:10:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:10:44.533+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:10:44.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:10:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:10:44.638+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:10:44.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:10:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:10:44.638+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:10:44.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:10:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:10:44.638+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:10:45.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:10:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:10:45.090+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:10:45.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:10:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:10:45.090+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:10:45.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:10:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:10:45.090+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:10:45.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:10:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:10:45.536+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:10:45.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:10:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:10:45.536+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:10:45.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:10:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:10:45.536+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:10:45.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:10:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:10:45.537+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:10:45.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:10:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:10:45.674+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:10:45.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:10:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:10:45.674+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:10:45.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:10:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:10:45.674+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:10:46.391 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:10:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:10:46.078+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:10:46.391 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:10:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:10:46.078+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:10:46.391 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:10:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:10:46.078+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:10:46.648 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:10:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:10:46.495+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:10:46.648 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:10:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:10:46.495+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:10:46.648 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:10:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:10:46.495+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:10:46.648 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:10:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:10:46.495+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:10:46.969 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:10:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:10:46.843+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 9366 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:10:46.969 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:10:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:10:46.648+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:10:46.969 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:10:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:10:46.648+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:10:46.969 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:10:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:10:46.648+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:10:47.240 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:10:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:10:47.039+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:10:47.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:10:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:10:47.039+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:10:47.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:10:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:10:47.039+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:10:47.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:10:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:10:47.688+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:10:47.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:10:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:10:47.688+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:10:47.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:10:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:10:47.688+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:10:47.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:10:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:10:47.447+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:10:47.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:10:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:10:47.447+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:10:47.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:10:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:10:47.447+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:10:47.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:10:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:10:47.447+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:10:48.460 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:10:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:10:48.076+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:10:48.461 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:10:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:10:48.076+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:10:48.461 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:10:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:10:48.076+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:10:48.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:10:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:10:48.676+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:10:48.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:10:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:10:48.676+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:10:48.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:10:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:10:48.676+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:10:48.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:10:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:10:48.459+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:10:48.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:10:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:10:48.459+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:10:48.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:10:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:10:48.459+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:10:48.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:10:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:10:48.459+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:10:49.482 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:10:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:10:49.047+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:10:49.482 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:10:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:10:49.047+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:10:49.482 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:10:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:10:49.047+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:10:49.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:10:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:10:49.712+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:10:49.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:10:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:10:49.712+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:10:49.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:10:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:10:49.712+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:10:49.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:10:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:10:49.480+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:10:49.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:10:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:10:49.480+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:10:49.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:10:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:10:49.480+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:10:49.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:10:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:10:49.480+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:10:50.143 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:10:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:10:50.078+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:10:50.143 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:10:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:10:50.078+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:10:50.143 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:10:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:10:50.078+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:10:50.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:10:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:10:50.473+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:10:50.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:10:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:10:50.473+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:10:50.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:10:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:10:50.473+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:10:50.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:10:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:10:50.473+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:10:51.057 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:10:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:10:50.750+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:10:51.057 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:10:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:10:50.750+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:10:51.057 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:10:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:10:50.750+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:10:51.458 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:10:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:10:51.056+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:10:51.459 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:10:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:10:51.056+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:10:51.459 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:10:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:10:51.056+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:10:51.728 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:10:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:10:51.457+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:10:51.728 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:10:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:10:51.457+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:10:51.728 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:10:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:10:51.457+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:10:51.728 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:10:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:10:51.457+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:10:51.991 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:10:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:10:51.844+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 9449 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:10:51.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:10:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:10:51.726+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:10:51.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:10:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:10:51.726+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:10:51.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:10:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:10:51.726+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:10:52.427 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:10:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:10:52.016+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:10:52.427 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:10:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:10:52.016+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:10:52.427 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:10:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:10:52.016+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:10:52.738 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:10:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:10:52.426+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:10:52.738 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:10:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:10:52.426+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:10:52.738 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:10:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:10:52.426+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:10:52.738 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:10:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:10:52.426+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:10:52.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:10:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:10:52.737+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:10:52.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:10:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:10:52.737+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:10:52.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:10:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:10:52.737+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:10:53.475 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:10:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:10:53.048+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:10:53.475 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:10:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:10:53.048+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:10:53.475 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:10:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:10:53.048+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:10:53.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:10:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:10:53.473+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:10:53.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:10:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:10:53.473+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:10:53.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:10:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:10:53.473+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:10:53.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:10:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:10:53.474+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:10:53.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:10:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:10:53.697+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:10:53.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:10:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:10:53.697+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:10:53.742 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:10:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:10:53.697+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:10:54.468 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:10:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:10:54.066+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:10:54.468 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:10:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:10:54.066+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:10:54.468 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:10:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:10:54.066+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:10:54.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:10:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:10:54.467+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:10:54.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:10:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:10:54.467+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:10:54.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:10:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:10:54.467+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:10:54.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:10:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:10:54.467+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:10:54.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:10:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:10:54.682+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:10:54.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:10:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:10:54.682+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:10:54.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:10:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:10:54.682+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:10:55.476 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:10:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:10:55.033+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:10:55.476 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:10:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:10:55.033+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:10:55.476 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:10:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:10:55.033+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:10:55.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:10:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:10:55.475+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:10:55.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:10:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:10:55.475+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:10:55.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:10:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:10:55.475+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:10:55.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:10:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:10:55.475+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:10:55.742 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:10:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:10:55.677+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:10:55.742 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:10:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:10:55.677+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:10:55.742 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:10:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:10:55.677+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:10:56.472 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:10:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:10:56.037+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:10:56.472 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:10:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:10:56.037+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:10:56.472 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:10:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:10:56.037+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:10:56.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:10:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:10:56.471+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:10:56.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:10:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:10:56.471+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:10:56.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:10:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:10:56.471+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:10:56.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:10:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:10:56.471+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:10:56.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:10:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:10:56.654+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:10:56.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:10:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:10:56.654+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:10:56.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:10:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:10:56.654+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:10:57.241 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:10:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:10:56.846+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 9532 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:10:57.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:10:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:10:56.999+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:10:57.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:10:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:10:56.999+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:10:57.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:10:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:10:56.999+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:10:57.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:10:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:10:57.471+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:10:57.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:10:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:10:57.471+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:10:57.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:10:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:10:57.471+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:10:57.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:10:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:10:57.471+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:10:57.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:10:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:10:57.617+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:10:57.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:10:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:10:57.617+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:10:57.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:10:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:10:57.617+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:10:58.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:10:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:10:57.951+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:10:58.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:10:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:10:57.951+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:10:58.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:10:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:10:57.951+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:10:58.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:10:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:10:58.433+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:10:58.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:10:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:10:58.433+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:10:58.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:10:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:10:58.433+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:10:58.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:10:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:10:58.433+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:10:58.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:10:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:10:58.595+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:10:58.742 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:10:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:10:58.595+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:10:58.742 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:10:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:10:58.595+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:10:59.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:10:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:10:58.915+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:10:59.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:10:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:10:58.915+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:10:59.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:10:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:10:58.915+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:10:59.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:10:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:10:59.483+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:10:59.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:10:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:10:59.483+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:10:59.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:10:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:10:59.483+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:10:59.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:10:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:10:59.483+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:10:59.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:10:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:10:59.586+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:10:59.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:10:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:10:59.586+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:10:59.742 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:10:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:10:59.586+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:11:00.143 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:10:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:10:59.912+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:11:00.144 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:10:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:10:59.912+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:11:00.144 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:10:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:10:59.912+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:11:00.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:11:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:11:00.628+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:11:00.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:11:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:11:00.628+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:11:00.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:11:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:11:00.628+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:11:00.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:11:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:11:00.477+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:11:00.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:11:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:11:00.477+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:11:00.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:11:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:11:00.477+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:11:00.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:11:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:11:00.477+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:11:01.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:11:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:11:00.937+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:11:01.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:11:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:11:00.937+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:11:01.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:11:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:11:00.937+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:11:01.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:11:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:11:01.677+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:11:01.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:11:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:11:01.677+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:11:01.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:11:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:11:01.677+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:11:01.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:11:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:11:01.488+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:11:01.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:11:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:11:01.488+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:11:01.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:11:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:11:01.488+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:11:01.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:11:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:11:01.488+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:11:02.241 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:11:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:11:01.847+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 9614 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:11:02.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:11:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:11:01.909+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:11:02.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:11:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:11:01.909+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:11:02.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:11:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:11:01.909+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:11:02.906 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:11:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:11:02.668+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:11:02.907 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:11:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:11:02.668+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:11:02.907 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:11:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:11:02.668+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:11:02.907 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:11:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:11:02.500+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:11:02.907 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:11:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:11:02.500+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:11:02.907 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:11:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:11:02.500+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:11:02.907 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:11:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:11:02.500+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:11:03.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:11:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:11:02.905+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:11:03.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:11:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:11:02.905+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:11:03.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:11:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:11:02.905+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:11:03.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:11:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:11:03.648+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:11:03.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:11:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:11:03.648+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:11:03.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:11:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:11:03.648+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:11:03.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:11:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:11:03.483+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:11:03.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:11:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:11:03.483+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:11:03.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:11:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:11:03.483+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:11:03.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:11:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:11:03.483+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:11:04.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:11:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:11:03.872+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:11:04.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:11:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:11:03.872+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:11:04.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:11:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:11:03.872+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:11:04.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:11:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:11:04.615+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:11:04.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:11:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:11:04.615+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:11:04.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:11:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:11:04.615+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:11:04.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:11:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:11:04.440+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:11:04.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:11:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:11:04.440+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:11:04.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:11:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:11:04.440+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:11:04.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:11:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:11:04.440+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:11:05.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:11:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:11:04.849+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:11:05.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:11:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:11:04.849+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:11:05.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:11:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:11:04.849+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:11:05.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:11:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:11:05.651+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:11:05.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:11:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:11:05.651+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:11:05.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:11:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:11:05.651+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:11:05.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:11:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:11:05.450+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:11:05.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:11:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:11:05.450+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:11:05.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:11:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:11:05.450+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:11:05.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:11:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:11:05.450+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:11:06.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:11:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:11:05.866+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:11:06.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:11:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:11:05.866+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:11:06.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:11:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:11:05.866+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:11:06.676 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:11:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:11:06.440+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:11:06.676 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:11:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:11:06.440+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:11:06.676 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:11:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:11:06.440+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:11:06.676 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:11:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:11:06.440+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:11:06.974 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:11:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:11:06.849+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 9696 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:11:06.975 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:11:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:11:06.674+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:11:06.975 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:11:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:11:06.674+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:11:06.975 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:11:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:11:06.674+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:11:06.975 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:11:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:11:06.844+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:11:06.975 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:11:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:11:06.844+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:11:06.975 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:11:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:11:06.844+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:11:07.721 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:11:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:11:07.447+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:11:07.721 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:11:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:11:07.447+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:11:07.721 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:11:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:11:07.447+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:11:07.722 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:11:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:11:07.448+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:11:07.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:11:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:11:07.721+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:11:07.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:11:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:11:07.721+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:11:07.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:11:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:11:07.721+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:11:07.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:11:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:11:07.847+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:11:07.992 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:11:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:11:07.847+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:11:07.992 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:11:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:11:07.847+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:11:08.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:11:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:11:08.676+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:11:08.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:11:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:11:08.676+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:11:08.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:11:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:11:08.676+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:11:08.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:11:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:11:08.474+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:11:08.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:11:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:11:08.474+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:11:08.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:11:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:11:08.474+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:11:08.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:11:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:11:08.474+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:11:09.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:11:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:11:08.834+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:11:09.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:11:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:11:08.834+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:11:09.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:11:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:11:08.834+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:11:09.704 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:11:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:11:09.425+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:11:09.704 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:11:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:11:09.425+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:11:09.704 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:11:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:11:09.425+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:11:09.704 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:11:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:11:09.425+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:11:09.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:11:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:11:09.703+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:11:09.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:11:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:11:09.703+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:11:09.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:11:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:11:09.703+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:11:09.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:11:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:11:09.800+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:11:09.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:11:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:11:09.800+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:11:09.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:11:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:11:09.800+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:11:10.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:11:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:11:10.688+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:11:10.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:11:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:11:10.688+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:11:10.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:11:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:11:10.688+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:11:10.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:11:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:11:10.455+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:11:10.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:11:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:11:10.455+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:11:10.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:11:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:11:10.455+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:11:10.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:11:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:11:10.455+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:11:11.240 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:11:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:11:10.795+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:11:11.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:11:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:11:10.795+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:11:11.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:11:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:11:10.795+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:11:11.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:11:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:11:11.459+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:11:11.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:11:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:11:11.459+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:11:11.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:11:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:11:11.459+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:11:11.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:11:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:11:11.460+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:11:11.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:11:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:11:11.702+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:11:11.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:11:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:11:11.702+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:11:11.742 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:11:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:11:11.702+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:11:12.241 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:11:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:11:11.851+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 9779 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:11:12.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:11:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:11:11.801+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:11:12.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:11:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:11:11.801+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:11:12.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:11:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:11:11.801+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:11:12.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:11:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:11:12.451+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:11:12.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:11:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:11:12.451+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:11:12.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:11:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:11:12.451+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:11:12.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:11:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:11:12.451+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:11:12.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:11:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:11:12.658+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:11:12.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:11:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:11:12.658+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:11:12.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:11:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:11:12.658+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:11:13.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:11:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:11:12.830+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:11:13.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:11:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:11:12.830+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:11:13.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:11:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:11:12.830+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:11:13.806 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:11:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:11:13.491+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:11:13.806 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:11:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:11:13.491+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:11:13.806 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:11:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:11:13.491+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:11:13.806 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:11:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:11:13.491+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:11:13.806 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:11:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:11:13.641+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:11:13.807 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:11:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:11:13.641+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:11:13.807 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:11:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:11:13.641+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:11:14.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:11:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:11:13.805+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:11:14.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:11:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:11:13.805+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:11:14.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:11:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:11:13.805+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:11:14.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:11:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:11:14.592+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:11:14.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:11:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:11:14.592+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:11:14.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:11:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:11:14.592+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:11:14.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:11:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:11:14.758+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:11:14.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:11:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:11:14.758+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:11:14.992 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:11:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:11:14.758+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:11:14.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:11:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:11:14.511+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:11:14.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:11:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:11:14.511+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:11:14.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:11:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:11:14.511+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:11:14.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:11:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:11:14.511+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:11:15.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:11:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:11:15.622+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:11:15.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:11:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:11:15.622+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:11:15.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:11:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:11:15.622+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:11:15.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:11:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:11:15.775+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:11:15.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:11:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:11:15.775+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:11:15.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:11:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:11:15.775+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:11:15.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:11:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:11:15.540+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:11:15.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:11:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:11:15.540+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:11:15.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:11:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:11:15.540+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:11:15.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:11:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:11:15.540+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:11:16.854 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:11:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:11:16.614+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:11:16.854 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:11:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:11:16.614+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:11:16.854 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:11:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:11:16.614+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:11:16.854 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:11:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:11:16.743+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:11:16.855 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:11:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:11:16.743+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:11:16.855 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:11:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:11:16.743+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:11:16.855 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:11:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:11:16.527+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:11:16.855 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:11:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:11:16.527+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:11:16.855 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:11:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:11:16.527+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:11:16.855 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:11:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:11:16.527+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:11:17.240 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:11:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:11:16.853+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 9862 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:11:17.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:11:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:11:17.600+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:11:17.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:11:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:11:17.600+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:11:17.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:11:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:11:17.600+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:11:17.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:11:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:11:17.705+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:11:17.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:11:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:11:17.705+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:11:17.992 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:11:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:11:17.705+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:11:17.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:11:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:11:17.496+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:11:17.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:11:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:11:17.496+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:11:17.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:11:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:11:17.496+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:11:17.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:11:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:11:17.496+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:11:18.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:11:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:11:18.614+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:11:18.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:11:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:11:18.614+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:11:18.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:11:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:11:18.614+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:11:18.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:11:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:11:18.702+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:11:18.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:11:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:11:18.702+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:11:18.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:11:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:11:18.702+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:11:18.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:11:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:11:18.485+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:11:18.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:11:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:11:18.485+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:11:18.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:11:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:11:18.485+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:11:18.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:11:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:11:18.485+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:11:19.884 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:11:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:11:19.741+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:11:19.884 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:11:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:11:19.741+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:11:19.884 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:11:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:11:19.741+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:11:19.884 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:11:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:11:19.660+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:11:19.885 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:11:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:11:19.660+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:11:19.885 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:11:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:11:19.660+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:11:19.885 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:11:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:11:19.515+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:11:19.885 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:11:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:11:19.515+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:11:19.885 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:11:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:11:19.515+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:11:19.885 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:11:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:11:19.516+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:11:20.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:11:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:11:20.679+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:11:20.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:11:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:11:20.679+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:11:20.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:11:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:11:20.679+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:11:20.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:11:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:11:20.696+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:11:20.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:11:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:11:20.696+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:11:20.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:11:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:11:20.696+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:11:20.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:11:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:11:20.483+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:11:20.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:11:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:11:20.483+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:11:20.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:11:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:11:20.483+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:11:20.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:11:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:11:20.484+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:11:21.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:11:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:11:21.690+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:11:21.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:11:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:11:21.690+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:11:21.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:11:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:11:21.690+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:11:21.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:11:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:11:21.457+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:11:21.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:11:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:11:21.457+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:11:21.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:11:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:11:21.457+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:11:21.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:11:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:11:21.457+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:11:21.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:11:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:11:21.663+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:11:21.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:11:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:11:21.663+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:11:21.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:11:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:11:21.663+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:11:22.241 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:11:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:11:21.855+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 9944 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:11:22.692 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:11:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:11:22.413+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:11:22.693 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:11:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:11:22.413+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:11:22.693 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:11:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:11:22.413+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:11:22.693 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:11:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:11:22.413+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:11:22.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:11:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:11:22.738+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:11:22.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:11:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:11:22.738+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:11:22.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:11:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:11:22.738+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:11:22.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:11:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:11:22.692+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:11:22.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:11:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:11:22.692+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:11:22.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:11:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:11:22.692+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:11:23.572 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:11:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:11:23.399+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:11:23.573 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:11:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:11:23.399+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:11:23.573 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:11:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:11:23.399+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:11:23.573 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:11:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:11:23.399+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:11:23.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:11:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:11:23.775+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:11:23.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:11:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:11:23.775+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:11:23.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:11:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:11:23.775+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:11:23.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:11:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:11:23.669+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:11:23.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:11:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:11:23.669+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:11:23.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:11:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:11:23.669+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:11:24.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:11:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:11:24.403+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:11:24.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:11:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:11:24.403+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:11:24.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:11:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:11:24.403+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:11:24.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:11:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:11:24.403+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:11:24.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:11:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:11:24.645+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:11:24.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:11:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:11:24.645+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:11:24.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:11:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:11:24.645+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:11:25.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:11:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:11:24.783+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:11:25.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:11:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:11:24.783+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:11:25.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:11:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:11:24.783+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:11:25.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:11:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:11:25.356+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:11:25.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:11:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:11:25.356+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:11:25.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:11:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:11:25.356+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:11:25.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:11:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:11:25.356+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:11:25.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:11:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:11:25.599+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:11:25.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:11:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:11:25.599+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:11:25.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:11:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:11:25.599+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:11:26.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:11:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:11:25.769+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:11:26.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:11:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:11:25.769+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:11:26.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:11:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:11:25.769+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:11:26.640 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:11:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:11:26.326+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:11:26.640 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:11:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:11:26.326+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:11:26.640 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:11:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:11:26.326+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:11:26.640 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:11:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:11:26.326+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:11:26.983 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:11:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:11:26.857+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 10026 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:11:26.983 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:11:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:11:26.754+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:11:26.983 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:11:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:11:26.754+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:11:26.983 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:11:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:11:26.754+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:11:26.983 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:11:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:11:26.640+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:11:26.984 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:11:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:11:26.640+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:11:26.984 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:11:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:11:26.640+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:11:27.617 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:11:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:11:27.290+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:11:27.617 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:11:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:11:27.290+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:11:27.617 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:11:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:11:27.290+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:11:27.618 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:11:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:11:27.290+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:11:27.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:11:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:11:27.723+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:11:27.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:11:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:11:27.723+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:11:27.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:11:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:11:27.723+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:11:27.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:11:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:11:27.617+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:11:27.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:11:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:11:27.617+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:11:27.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:11:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:11:27.617+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:11:28.654 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:11:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:11:28.291+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:11:28.655 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:11:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:11:28.291+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:11:28.655 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:11:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:11:28.291+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:11:28.655 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:11:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:11:28.292+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:11:28.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:11:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:11:28.770+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:11:28.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:11:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:11:28.770+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:11:28.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:11:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:11:28.770+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:11:28.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:11:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:11:28.654+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:11:28.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:11:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:11:28.654+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:11:28.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:11:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:11:28.654+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:11:29.674 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:11:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:11:29.337+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:11:29.674 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:11:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:11:29.337+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:11:29.674 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:11:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:11:29.337+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:11:29.674 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:11:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:11:29.337+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:11:29.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:11:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:11:29.791+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:11:29.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:11:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:11:29.791+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:11:29.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:11:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:11:29.791+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:11:29.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:11:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:11:29.674+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:11:29.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:11:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:11:29.674+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:11:29.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:11:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:11:29.674+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:11:30.491 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:11:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:11:30.306+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:11:30.491 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:11:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:11:30.306+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:11:30.491 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:11:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:11:30.306+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:11:30.491 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:11:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:11:30.306+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:11:30.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:11:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:11:30.799+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:11:30.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:11:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:11:30.800+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:11:30.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:11:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:11:30.800+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:11:30.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:11:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:11:30.696+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:11:30.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:11:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:11:30.696+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:11:30.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:11:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:11:30.696+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:11:31.680 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:11:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:11:31.293+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:11:31.680 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:11:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:11:31.293+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:11:31.681 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:11:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:11:31.293+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:11:31.681 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:11:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:11:31.293+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:11:31.985 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:11:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:11:31.859+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 10109 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:11:31.986 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:11:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:11:31.846+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:11:31.986 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:11:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:11:31.846+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:11:31.986 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:11:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:11:31.846+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:11:31.986 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:11:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:11:31.693+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:11:31.986 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:11:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:11:31.693+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:11:31.986 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:11:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:11:31.693+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:11:32.676 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:11:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:11:32.323+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:11:32.676 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:11:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:11:32.323+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:11:32.676 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:11:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:11:32.323+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:11:32.677 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:11:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:11:32.323+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:11:32.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:11:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:11:32.889+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:11:32.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:11:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:11:32.889+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:11:32.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:11:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:11:32.889+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:11:32.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:11:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:11:32.676+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:11:32.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:11:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:11:32.676+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:11:32.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:11:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:11:32.676+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:11:33.573 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:11:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:11:33.308+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:11:33.574 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:11:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:11:33.308+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:11:33.574 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:11:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:11:33.308+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:11:33.574 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:11:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:11:33.308+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:11:33.878 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:11:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:11:33.687+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:11:33.878 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:11:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:11:33.687+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:11:33.878 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:11:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:11:33.687+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:11:34.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:11:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:11:33.878+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:11:34.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:11:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:11:33.878+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:11:34.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:11:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:11:33.878+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:11:34.672 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:11:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:11:34.265+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:11:34.673 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:11:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:11:34.265+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:11:34.673 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:11:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:11:34.265+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:11:34.673 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:11:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:11:34.265+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:11:34.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:11:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:11:34.918+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:11:34.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:11:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:11:34.918+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:11:34.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:11:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:11:34.918+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:11:34.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:11:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:11:34.672+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:11:34.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:11:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:11:34.672+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:11:34.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:11:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:11:34.672+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:11:35.635 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:11:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:11:35.250+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:11:35.635 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:11:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:11:35.250+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:11:35.635 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:11:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:11:35.250+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:11:35.635 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:11:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:11:35.250+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:11:35.909 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:11:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:11:35.635+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:11:35.909 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:11:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:11:35.635+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:11:35.909 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:11:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:11:35.635+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:11:36.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:11:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:11:35.909+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:11:36.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:11:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:11:35.909+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:11:36.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:11:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:11:35.909+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:11:36.625 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:11:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:11:36.242+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:11:36.625 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:11:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:11:36.242+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:11:36.625 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:11:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:11:36.242+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:11:36.626 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:11:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:11:36.242+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:11:36.884 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:11:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:11:36.861+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 10192 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:11:36.885 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:11:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:11:36.625+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:11:36.885 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:11:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:11:36.625+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:11:36.885 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:11:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:11:36.625+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:11:36.885 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:11:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:11:36.873+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:11:36.885 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:11:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:11:36.873+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:11:36.885 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:11:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:11:36.873+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:11:37.583 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:11:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:11:37.269+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:11:37.584 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:11:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:11:37.269+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:11:37.584 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:11:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:11:37.269+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:11:37.584 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:11:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:11:37.269+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:11:37.880 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:11:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:11:37.584+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:11:37.881 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:11:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:11:37.584+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:11:37.881 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:11:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:11:37.584+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:11:38.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:11:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:11:37.881+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:11:38.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:11:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:11:37.881+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:11:38.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:11:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:11:37.881+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:11:38.615 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:11:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:11:38.542+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:11:38.615 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:11:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:11:38.542+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:11:38.615 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:11:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:11:38.542+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:11:38.616 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:11:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:11:38.310+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:11:38.616 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:11:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:11:38.310+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:11:38.616 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:11:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:11:38.310+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:11:38.616 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:11:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:11:38.310+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:11:39.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:11:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:11:38.875+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:11:39.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:11:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:11:38.875+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:11:39.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:11:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:11:38.875+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:11:39.546 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:11:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:11:39.295+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:11:39.546 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:11:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:11:39.295+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:11:39.546 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:11:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:11:39.295+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:11:39.547 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:11:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:11:39.295+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:11:39.884 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:11:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:11:39.546+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:11:39.884 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:11:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:11:39.546+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:11:39.884 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:11:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:11:39.546+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:11:40.146 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:11:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:11:39.884+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:11:40.146 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:11:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:11:39.884+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:11:40.147 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:11:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:11:39.884+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:11:40.491 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:11:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:11:40.296+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:11:40.491 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:11:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:11:40.296+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:11:40.491 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:11:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:11:40.296+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:11:40.491 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:11:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:11:40.296+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:11:40.842 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:11:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:11:40.509+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:11:40.842 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:11:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:11:40.509+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:11:40.842 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:11:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:11:40.509+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:11:41.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:11:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:11:40.841+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:11:41.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:11:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:11:40.841+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:11:41.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:11:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:11:40.841+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:11:41.681 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:11:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:11:41.495+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:11:41.681 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:11:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:11:41.495+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:11:41.681 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:11:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:11:41.495+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:11:41.682 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:11:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:11:41.264+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:11:41.682 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:11:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:11:41.264+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:11:41.682 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:11:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:11:41.264+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:11:41.682 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:11:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:11:41.264+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:11:41.989 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:11:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:11:41.862+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 10274 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:11:41.989 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:11:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:11:41.885+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:11:41.989 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:11:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:11:41.885+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:11:41.989 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:11:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:11:41.885+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:11:42.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:11:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:11:42.475+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:11:42.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:11:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:11:42.475+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:11:42.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:11:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:11:42.475+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:11:42.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:11:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:11:42.273+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:11:42.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:11:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:11:42.273+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:11:42.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:11:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:11:42.273+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:11:42.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:11:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:11:42.273+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:11:43.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:11:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:11:42.890+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:11:43.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:11:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:11:42.890+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:11:43.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:11:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:11:42.890+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:11:43.574 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:11:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:11:43.466+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:11:43.574 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:11:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:11:43.466+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:11:43.574 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:11:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:11:43.466+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:11:43.574 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:11:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:11:43.278+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:11:43.574 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:11:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:11:43.278+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:11:43.574 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:11:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:11:43.278+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:11:43.575 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:11:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:11:43.278+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:11:44.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:11:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:11:43.846+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:11:44.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:11:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:11:43.846+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:11:44.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:11:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:11:43.846+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:11:44.688 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:11:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:11:44.510+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:11:44.688 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:11:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:11:44.510+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:11:44.688 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:11:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:11:44.510+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:11:44.688 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:11:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:11:44.261+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:11:44.688 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:11:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:11:44.261+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:11:44.689 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:11:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:11:44.261+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:11:44.689 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:11:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:11:44.261+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:11:44.990 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:11:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:11:44.844+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:11:44.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:11:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:11:44.844+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:11:44.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:11:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:11:44.844+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:11:45.383 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:11:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:11:45.215+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:11:45.384 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:11:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:11:45.215+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:11:45.384 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:11:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:11:45.215+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:11:45.384 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:11:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:11:45.215+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:11:45.720 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:11:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:11:45.487+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:11:45.720 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:11:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:11:45.487+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:11:45.720 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:11:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:11:45.487+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:11:45.977 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:11:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:11:45.826+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:11:45.977 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:11:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:11:45.826+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:11:45.977 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:11:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:11:45.826+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:11:46.241 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:11:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:11:46.187+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:11:46.241 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:11:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:11:46.187+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:11:46.241 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:11:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:11:46.187+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:11:46.241 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:11:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:11:46.187+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:11:46.740 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:11:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:11:46.473+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:11:46.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:11:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:11:46.473+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:11:46.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:11:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:11:46.473+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:11:47.145 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:11:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:11:46.863+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 10356 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:11:47.145 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:11:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:11:46.823+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:11:47.145 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:11:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:11:46.823+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:11:47.145 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:11:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:11:46.823+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:11:47.435 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:11:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:11:47.144+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:11:47.435 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:11:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:11:47.144+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:11:47.435 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:11:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:11:47.144+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:11:47.435 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:11:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:11:47.144+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:11:47.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:11:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:11:47.434+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:11:47.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:11:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:11:47.434+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:11:47.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:11:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:11:47.434+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:11:48.129 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:11:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:11:47.808+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:11:48.129 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:11:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:11:47.808+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:11:48.129 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:11:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:11:47.808+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:11:48.451 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:11:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:11:48.128+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:11:48.451 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:11:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:11:48.128+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:11:48.451 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:11:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:11:48.128+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:11:48.451 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:11:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:11:48.128+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:11:48.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:11:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:11:48.450+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:11:48.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:11:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:11:48.450+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:11:48.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:11:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:11:48.450+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:11:49.164 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:11:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:11:48.814+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:11:49.164 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:11:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:11:48.814+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:11:49.164 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:11:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:11:48.814+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:11:49.481 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:11:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:11:49.163+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:11:49.481 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:11:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:11:49.163+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:11:49.481 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:11:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:11:49.163+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:11:49.481 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:11:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:11:49.163+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:11:49.740 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:11:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:11:49.480+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:11:49.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:11:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:11:49.480+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:11:49.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:11:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:11:49.480+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:11:50.147 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:11:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:11:49.849+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:11:50.147 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:11:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:11:49.849+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:11:50.147 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:11:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:11:49.849+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:11:50.431 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:11:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:11:50.160+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:11:50.431 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:11:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:11:50.160+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:11:50.432 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:11:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:11:50.160+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:11:50.432 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:11:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:11:50.160+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:11:50.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:11:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:11:50.430+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:11:50.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:11:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:11:50.430+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:11:50.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:11:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:11:50.430+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:11:51.158 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:11:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:11:50.808+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:11:51.159 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:11:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:11:50.808+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:11:51.159 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:11:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:11:50.808+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:11:51.422 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:11:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:11:51.157+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:11:51.422 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:11:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:11:51.157+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:11:51.422 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:11:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:11:51.157+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:11:51.422 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:11:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:11:51.157+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:11:51.682 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:11:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:11:51.421+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:11:51.683 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:11:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:11:51.421+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:11:51.683 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:11:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:11:51.421+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:11:51.988 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:11:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:11:51.802+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:11:51.989 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:11:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:11:51.802+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:11:51.989 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:11:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:11:51.802+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:11:51.989 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:11:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:11:51.865+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 10439 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:11:52.241 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:11:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:11:52.133+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:11:52.241 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:11:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:11:52.133+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:11:52.241 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:11:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:11:52.133+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:11:52.241 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:11:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:11:52.133+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:11:52.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:11:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:11:52.468+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:11:52.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:11:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:11:52.468+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:11:52.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:11:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:11:52.468+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:11:53.084 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:11:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:11:52.829+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:11:53.085 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:11:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:11:52.829+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:11:53.085 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:11:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:11:52.829+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:11:53.464 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:11:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:11:53.084+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:11:53.464 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:11:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:11:53.084+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:11:53.464 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:11:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:11:53.084+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:11:53.464 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:11:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:11:53.084+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:11:53.817 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:11:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:11:53.498+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:11:53.817 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:11:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:11:53.498+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:11:53.817 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:11:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:11:53.498+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:11:54.112 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:11:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:11:53.816+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:11:54.112 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:11:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:11:53.816+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:11:54.112 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:11:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:11:53.816+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:11:54.484 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:11:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:11:54.111+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:11:54.484 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:11:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:11:54.111+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:11:54.484 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:11:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:11:54.111+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:11:54.484 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:11:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:11:54.111+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:11:54.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:11:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:11:54.483+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:11:54.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:11:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:11:54.483+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:11:54.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:11:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:11:54.483+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:11:55.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:11:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:11:54.858+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:11:55.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:11:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:11:54.858+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:11:55.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:11:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:11:54.858+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:11:55.241 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:11:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:11:55.094+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:11:55.241 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:11:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:11:55.094+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:11:55.241 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:11:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:11:55.094+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:11:55.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:11:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:11:55.094+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:11:55.880 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:11:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:11:55.512+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:11:55.880 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:11:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:11:55.512+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:11:55.880 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:11:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:11:55.512+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:11:56.136 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:11:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:11:55.878+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:11:56.136 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:11:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:11:55.878+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:11:56.137 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:11:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:11:55.878+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:11:56.388 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:11:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:11:56.135+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:11:56.388 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:11:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:11:56.135+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:11:56.388 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:11:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:11:56.135+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:11:56.388 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:11:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:11:56.135+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:11:56.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:11:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:11:56.462+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:11:56.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:11:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:11:56.462+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:11:56.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:11:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:11:56.462+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:11:57.145 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:11:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:11:56.867+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 10522 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:11:57.145 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:11:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:11:56.911+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:11:57.145 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:11:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:11:56.911+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:11:57.145 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:11:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:11:56.911+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:11:57.458 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:11:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:11:57.144+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:11:57.458 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:11:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:11:57.144+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:11:57.458 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:11:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:11:57.144+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:11:57.458 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:11:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:11:57.144+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:11:57.740 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:11:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:11:57.457+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:11:57.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:11:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:11:57.457+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:11:57.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:11:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:11:57.457+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:11:58.162 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:11:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:11:57.870+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:11:58.162 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:11:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:11:57.870+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:11:58.162 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:11:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:11:57.870+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:11:58.445 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:11:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:11:58.161+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:11:58.445 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:11:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:11:58.161+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:11:58.445 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:11:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:11:58.161+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:11:58.445 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:11:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:11:58.161+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:11:58.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:11:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:11:58.444+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:11:58.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:11:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:11:58.444+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:11:58.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:11:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:11:58.444+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:11:59.136 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:11:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:11:58.831+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:11:59.136 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:11:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:11:58.831+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:11:59.136 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:11:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:11:58.831+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:11:59.457 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:11:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:11:59.135+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:11:59.457 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:11:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:11:59.135+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:11:59.457 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:11:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:11:59.135+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:11:59.457 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:11:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:11:59.135+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:11:59.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:11:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:11:59.456+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:11:59.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:11:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:11:59.456+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:11:59.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:11:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:11:59.456+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:12:00.148 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:11:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:11:59.834+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:12:00.148 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:11:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:11:59.834+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:12:00.148 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:11:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:11:59.834+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:12:00.491 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:12:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:12:00.177+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:12:00.491 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:12:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:12:00.177+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:12:00.491 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:12:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:12:00.177+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:12:00.491 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:12:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:12:00.177+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:12:00.790 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:12:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:12:00.501+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:12:00.790 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:12:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:12:00.501+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:12:00.790 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:12:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:12:00.501+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:12:01.185 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:12:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:12:00.788+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:12:01.185 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:12:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:12:00.788+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:12:01.185 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:12:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:12:00.788+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:12:01.462 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:12:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:12:01.184+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:12:01.462 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:12:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:12:01.184+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:12:01.462 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:12:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:12:01.184+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:12:01.462 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:12:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:12:01.184+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:12:01.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:12:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:12:01.462+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:12:01.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:12:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:12:01.462+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:12:01.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:12:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:12:01.462+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:12:02.151 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:12:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:12:01.869+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 10604 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:12:02.151 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:12:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:12:01.759+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:12:02.151 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:12:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:12:01.759+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:12:02.151 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:12:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:12:01.759+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:12:02.429 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:12:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:12:02.150+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:12:02.429 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:12:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:12:02.150+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:12:02.429 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:12:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:12:02.150+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:12:02.429 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:12:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:12:02.150+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:12:02.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:12:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:12:02.427+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:12:02.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:12:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:12:02.427+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:12:02.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:12:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:12:02.427+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:12:03.139 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:12:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:12:02.764+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:12:03.139 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:12:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:12:02.764+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:12:03.139 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:12:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:12:02.764+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:12:03.463 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:12:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:12:03.137+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:12:03.463 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:12:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:12:03.137+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:12:03.463 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:12:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:12:03.137+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:12:03.463 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:12:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:12:03.138+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:12:03.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:12:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:12:03.462+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:12:03.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:12:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:12:03.462+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:12:03.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:12:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:12:03.462+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:12:04.119 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:12:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:12:03.796+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:12:04.119 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:12:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:12:03.796+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:12:04.119 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:12:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:12:03.796+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:12:04.491 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:12:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:12:04.118+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:12:04.491 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:12:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:12:04.118+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:12:04.491 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:12:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:12:04.118+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:12:04.491 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:12:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:12:04.118+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:12:04.792 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:12:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:12:04.491+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:12:04.792 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:12:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:12:04.491+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:12:04.792 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:12:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:12:04.491+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:12:05.120 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:12:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:12:04.791+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:12:05.120 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:12:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:12:04.791+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:12:05.120 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:12:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:12:04.791+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:12:05.472 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:12:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:12:05.119+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:12:05.472 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:12:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:12:05.119+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:12:05.473 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:12:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:12:05.119+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:12:05.473 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:12:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:12:05.119+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:12:05.740 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:12:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:12:05.472+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:12:05.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:12:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:12:05.472+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:12:05.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:12:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:12:05.472+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:12:06.104 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:12:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:12:05.825+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:12:06.104 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:12:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:12:05.825+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:12:06.105 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:12:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:12:05.825+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:12:06.489 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:12:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:12:06.104+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:12:06.489 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:12:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:12:06.104+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:12:06.489 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:12:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:12:06.104+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:12:06.489 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:12:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:12:06.104+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:12:06.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:12:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:12:06.489+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:12:06.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:12:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:12:06.489+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:12:06.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:12:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:12:06.489+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:12:07.092 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:12:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:12:06.871+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 10686 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:12:07.092 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:12:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:12:06.794+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:12:07.092 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:12:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:12:06.794+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:12:07.093 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:12:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:12:06.794+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:12:07.114 INFO:teuthology.orchestra.run.smithi133.stderr:2024-02-20T08:12:07.111+0000 7f5a33776700 0 monclient(hunting): authenticate timed out after 300 2024-02-20T08:12:07.114 INFO:teuthology.orchestra.run.smithi133.stderr:[errno 110] RADOS timed out (error connecting to the cluster) 2024-02-20T08:12:07.417 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:12:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:12:07.091+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:12:07.417 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:12:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:12:07.091+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:12:07.417 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:12:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:12:07.091+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:12:07.417 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:12:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:12:07.091+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:12:07.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:12:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:12:07.528+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:12:07.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:12:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:12:07.528+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:12:07.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:12:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:12:07.528+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:12:07.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-20T08:12:07.832 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/task/parallel.py", line 56, in task p.spawn(_run_spawned, ctx, confg, taskname) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/task/parallel.py", line 64, in _run_spawned mgr = run_tasks.run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/task/sequential.py", line 47, in task mgr = run_tasks.run_one_task(taskname, ctx=ctx, config=confg) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/github.com_batrick_ceph_3d72bcfd7affbb4a923c191d212814b4934cfeaa/qa/tasks/cephadm.py", line 1364, in shell _shell(ctx, cluster_name, remote, File "/home/teuthworker/src/github.com_batrick_ceph_3d72bcfd7affbb4a923c191d212814b4934cfeaa/qa/tasks/cephadm.py", line 39, in _shell return remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi133 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4537327c-cfc4-11ee-95bb-87774f69a715 -e sha1=91ff50e903dcc99ee3ecf9523050878eb452da5c -- bash -c \'ceph versions | jq -e \'"\'"\'.mgr | keys\'"\'"\' | grep $sha1\'' 2024-02-20T08:12:08.046 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=3743dfb8b3974a0ea422048339080efc Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/task/parallel.py", line 56, in task p.spawn(_run_spawned, ctx, confg, taskname) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/task/parallel.py", line 64, in _run_spawned mgr = run_tasks.run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/task/sequential.py", line 47, in task mgr = run_tasks.run_one_task(taskname, ctx=ctx, config=confg) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/github.com_batrick_ceph_3d72bcfd7affbb4a923c191d212814b4934cfeaa/qa/tasks/cephadm.py", line 1364, in shell _shell(ctx, cluster_name, remote, File "/home/teuthworker/src/github.com_batrick_ceph_3d72bcfd7affbb4a923c191d212814b4934cfeaa/qa/tasks/cephadm.py", line 39, in _shell return remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi133 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4537327c-cfc4-11ee-95bb-87774f69a715 -e sha1=91ff50e903dcc99ee3ecf9523050878eb452da5c -- bash -c \'ceph versions | jq -e \'"\'"\'.mgr | keys\'"\'"\' | grep $sha1\'' 2024-02-20T08:12:08.050 DEBUG:teuthology.run_tasks:Unwinding manager kclient 2024-02-20T08:12:08.062 INFO:tasks.kclient:Unmounting kernel clients... 2024-02-20T08:12:08.063 DEBUG:teuthology.orchestra.run.smithi133:> set -ex 2024-02-20T08:12:08.063 DEBUG:teuthology.orchestra.run.smithi133:> dd if=/proc/self/mounts of=/dev/stdout 2024-02-20T08:12:08.089 DEBUG:teuthology.orchestra.run.smithi133:> set -ex 2024-02-20T08:12:08.089 DEBUG:teuthology.orchestra.run.smithi133:> dd if=/proc/self/mounts of=/dev/stdout 2024-02-20T08:12:08.110 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:12:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:12:07.825+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:12:08.110 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:12:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:12:07.825+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:12:08.110 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:12:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:12:07.825+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:12:08.129 DEBUG:teuthology.orchestra.run.smithi133:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-20T08:12:08.491 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:12:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:12:08.108+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:12:08.491 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:12:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:12:08.108+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:12:08.491 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:12:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:12:08.108+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:12:08.491 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:12:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:12:08.108+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:12:08.853 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:12:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:12:08.574+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:12:08.853 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:12:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:12:08.574+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:12:08.853 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:12:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:12:08.574+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:12:09.130 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:12:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:12:08.853+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:12:09.130 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:12:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:12:08.853+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:12:09.130 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:12:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:12:08.853+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:12:09.491 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:12:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:12:09.130+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:12:09.491 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:12:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:12:09.130+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:12:09.491 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:12:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:12:09.130+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:12:09.491 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:12:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:12:09.130+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:12:09.817 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:12:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:12:09.526+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:12:09.817 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:12:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:12:09.526+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:12:09.817 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:12:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:12:09.526+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:12:10.104 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:12:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:12:09.817+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:12:10.104 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:12:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:12:09.817+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:12:10.104 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:12:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:12:09.817+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:12:10.491 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:12:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:12:10.103+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:12:10.491 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:12:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:12:10.103+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:12:10.491 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:12:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:12:10.103+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:12:10.491 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:12:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:12:10.103+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:12:10.810 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:12:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:12:10.534+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:12:10.811 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:12:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:12:10.534+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:12:10.811 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:12:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:12:10.534+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:12:11.072 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:12:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:12:10.810+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:12:11.072 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:12:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:12:10.810+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:12:11.072 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:12:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:12:10.810+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:12:11.491 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:12:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:12:11.071+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:12:11.491 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:12:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:12:11.071+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:12:11.491 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:12:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:12:11.071+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:12:11.491 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:12:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:12:11.071+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:12:11.825 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:12:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:12:11.491+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:12:11.826 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:12:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:12:11.491+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:12:11.826 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:12:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:12:11.491+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:12:12.104 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:12:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:12:11.873+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 10769 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:12:12.104 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:12:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:12:11.825+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:12:12.105 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:12:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:12:11.825+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:12:12.105 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:12:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:12:11.825+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:12:12.407 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:12:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:12:12.104+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:12:12.407 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:12:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:12:12.104+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:12:12.408 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:12:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:12:12.104+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:12:12.408 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:12:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:12:12.104+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:12:12.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:12:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:12:12.447+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:12:12.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:12:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:12:12.447+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:12:12.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:12:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:12:12.447+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:12:13.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:12:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:12:12.865+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:12:13.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:12:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:12:12.865+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:12:13.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:12:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:12:12.865+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:12:13.241 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:12:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:12:13.098+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:12:13.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:12:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:12:13.098+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:12:13.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:12:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:12:13.098+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:12:13.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:12:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:12:13.098+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:12:13.740 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:12:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:12:13.475+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:12:13.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:12:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:12:13.475+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:12:13.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:12:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:12:13.475+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:12:14.119 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:12:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:12:13.821+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:12:14.120 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:12:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:12:13.821+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:12:14.120 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:12:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:12:13.821+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:12:14.428 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:12:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:12:14.119+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:12:14.428 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:12:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:12:14.119+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:12:14.428 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:12:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:12:14.119+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:12:14.428 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:12:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:12:14.120+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:12:14.740 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:12:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:12:14.427+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:12:14.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:12:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:12:14.427+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:12:14.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:12:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:12:14.427+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:12:15.135 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:12:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:12:14.851+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:12:15.136 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:12:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:12:14.851+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:12:15.136 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:12:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:12:14.851+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:12:15.479 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:12:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:12:15.384+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:12:15.479 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:12:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:12:15.384+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:12:15.480 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:12:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:12:15.384+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:12:15.480 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:12:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:12:15.135+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:12:15.480 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:12:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:12:15.135+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:12:15.480 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:12:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:12:15.135+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:12:15.481 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:12:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:12:15.136+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:12:16.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:12:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:12:15.877+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:12:16.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:12:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:12:15.877+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:12:16.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:12:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:12:15.877+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:12:16.241 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:12:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:12:16.109+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:12:16.241 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:12:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:12:16.109+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:12:16.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:12:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:12:16.109+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:12:16.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:12:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:12:16.109+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:12:16.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:12:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:12:16.400+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:12:16.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:12:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:12:16.400+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:12:16.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:12:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:12:16.400+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:12:16.997 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:12:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:12:16.875+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 10853 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:12:16.997 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:12:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:12:16.912+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:12:16.997 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:12:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:12:16.912+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:12:16.998 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:12:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:12:16.912+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:12:17.386 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:12:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:12:17.086+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:12:17.386 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:12:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:12:17.086+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:12:17.386 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:12:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:12:17.086+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:12:17.386 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:12:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:12:17.086+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:12:17.740 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:12:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:12:17.385+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:12:17.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:12:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:12:17.385+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:12:17.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:12:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:12:17.385+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:12:18.018 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:12:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:12:17.944+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:12:18.019 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:12:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:12:17.944+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:12:18.019 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:12:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:12:17.944+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:12:18.372 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:12:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:12:18.066+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:12:18.372 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:12:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:12:18.066+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:12:18.372 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:12:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:12:18.066+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:12:18.372 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:12:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:12:18.066+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:12:18.740 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:12:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:12:18.371+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:12:18.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:12:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:12:18.371+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:12:18.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:12:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:12:18.371+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:12:19.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:12:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:12:18.947+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:12:19.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:12:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:12:18.947+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:12:19.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:12:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:12:18.947+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:12:19.241 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:12:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:12:19.079+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:12:19.241 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:12:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:12:19.079+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:12:19.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:12:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:12:19.079+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:12:19.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:12:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:12:19.079+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:12:19.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:12:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:12:19.347+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:12:19.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:12:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:12:19.347+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:12:19.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:12:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:12:19.347+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:12:20.150 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:12:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:12:19.988+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:12:20.150 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:12:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:12:19.988+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:12:20.151 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:12:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:12:19.988+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:12:20.151 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:12:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:12:20.075+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:12:20.151 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:12:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:12:20.075+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:12:20.151 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:12:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:12:20.075+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:12:20.151 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:12:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:12:20.075+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:12:20.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:12:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:12:20.322+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:12:20.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:12:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:12:20.322+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:12:20.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:12:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:12:20.322+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:12:21.320 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:12:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:12:21.008+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:12:21.321 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:12:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:12:21.008+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:12:21.321 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:12:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:12:21.008+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:12:21.321 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:12:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:12:21.099+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:12:21.321 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:12:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:12:21.099+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:12:21.321 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:12:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:12:21.099+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:12:21.321 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:12:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:12:21.099+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:12:21.623 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:12:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:12:21.320+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:12:21.623 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:12:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:12:21.320+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:12:21.623 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:12:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:12:21.320+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:12:22.241 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:12:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:12:21.877+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 10935 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:12:22.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:12:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:12:22.003+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:12:22.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:12:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:12:22.003+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:12:22.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:12:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:12:22.003+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:12:22.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:12:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:12:22.094+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:12:22.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:12:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:12:22.094+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:12:22.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:12:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:12:22.094+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:12:22.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:12:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:12:22.094+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:12:22.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:12:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:12:22.310+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:12:22.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:12:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:12:22.310+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:12:22.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:12:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:12:22.310+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:12:23.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:12:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:12:23.268+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:12:23.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:12:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:12:23.268+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:12:23.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:12:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:12:23.268+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:12:23.492 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:12:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:12:23.037+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:12:23.492 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:12:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:12:23.038+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:12:23.492 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:12:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:12:23.038+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:12:23.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:12:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:12:23.114+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:12:23.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:12:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:12:23.114+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:12:23.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:12:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:12:23.114+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:12:23.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:12:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:12:23.114+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:12:24.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:12:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:12:24.240+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:12:24.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:12:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:12:24.240+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:12:24.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:12:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:12:24.240+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:12:24.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:12:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:12:24.058+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:12:24.492 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:12:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:12:24.058+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:12:24.492 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:12:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:12:24.058+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:12:24.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:12:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:12:24.091+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:12:24.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:12:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:12:24.091+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:12:24.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:12:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:12:24.091+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:12:24.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:12:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:12:24.091+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:12:25.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:12:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:12:25.250+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:12:25.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:12:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:12:25.250+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:12:25.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:12:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:12:25.250+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:12:25.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:12:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:12:25.047+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:12:25.492 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:12:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:12:25.047+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:12:25.492 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:12:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:12:25.047+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:12:25.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:12:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:12:25.072+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:12:25.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:12:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:12:25.072+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:12:25.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:12:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:12:25.072+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:12:25.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:12:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:12:25.072+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:12:26.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:12:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:12:26.256+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:12:26.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:12:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:12:26.256+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:12:26.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:12:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:12:26.256+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:12:26.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:12:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:12:26.009+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:12:26.492 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:12:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:12:26.009+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:12:26.492 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:12:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:12:26.009+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:12:26.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:12:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:12:26.090+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:12:26.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:12:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:12:26.090+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:12:26.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:12:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:12:26.090+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:12:26.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:12:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:12:26.090+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:12:27.241 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:12:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:12:26.879+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 11017 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:12:27.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:12:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:12:26.976+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:12:27.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:12:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:12:26.976+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:12:27.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:12:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:12:26.976+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:12:27.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:12:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:12:27.086+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:12:27.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:12:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:12:27.086+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:12:27.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:12:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:12:27.086+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:12:27.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:12:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:12:27.086+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:12:27.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:12:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:12:27.284+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:12:27.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:12:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:12:27.284+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:12:27.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:12:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:12:27.284+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:12:28.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:12:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:12:27.962+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:12:28.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:12:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:12:27.962+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:12:28.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:12:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:12:27.962+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:12:28.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:12:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:12:28.094+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:12:28.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:12:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:12:28.094+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:12:28.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:12:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:12:28.094+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:12:28.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:12:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:12:28.094+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:12:28.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:12:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:12:28.254+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:12:28.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:12:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:12:28.254+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:12:28.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:12:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:12:28.254+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:12:29.215 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:12:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:12:28.923+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:12:29.215 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:12:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:12:28.923+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:12:29.215 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:12:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:12:28.923+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:12:29.216 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:12:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:12:29.134+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:12:29.216 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:12:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:12:29.134+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:12:29.216 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:12:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:12:29.134+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:12:29.216 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:12:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:12:29.134+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:12:29.490 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:12:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:12:29.215+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:12:29.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:12:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:12:29.215+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:12:29.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:12:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:12:29.215+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:12:30.151 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:12:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:12:29.913+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:12:30.152 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:12:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:12:29.913+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:12:30.152 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:12:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:12:29.913+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:12:30.152 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:12:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:12:30.101+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:12:30.152 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:12:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:12:30.101+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:12:30.152 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:12:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:12:30.101+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:12:30.152 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:12:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:12:30.101+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:12:30.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:12:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:12:30.169+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:12:30.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:12:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:12:30.169+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:12:30.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:12:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:12:30.169+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:12:31.112 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:12:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:12:30.943+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:12:31.113 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:12:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:12:30.943+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:12:31.113 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:12:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:12:30.943+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:12:31.491 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:12:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:12:31.113+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:12:31.491 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:12:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:12:31.113+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:12:31.491 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:12:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:12:31.113+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:12:31.491 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:12:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:12:31.113+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:12:31.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:12:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:12:31.201+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:12:31.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:12:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:12:31.201+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:12:31.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:12:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:12:31.201+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:12:31.951 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:12:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:12:31.881+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 11100 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:12:32.241 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:12:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:12:32.121+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:12:32.241 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:12:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:12:32.121+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:12:32.241 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:12:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:12:32.121+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:12:32.241 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:12:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:12:32.121+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:12:32.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:12:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:12:31.949+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:12:32.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:12:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:12:31.949+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:12:32.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:12:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:12:31.949+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:12:32.740 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:12:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:12:32.247+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:12:32.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:12:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:12:32.247+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:12:32.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:12:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:12:32.247+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:12:33.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:12:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:12:32.945+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:12:33.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:12:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:12:32.945+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:12:33.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:12:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:12:32.945+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:12:33.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:12:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:12:33.160+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:12:33.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:12:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:12:33.160+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:12:33.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:12:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:12:33.160+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:12:33.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:12:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:12:33.160+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:12:33.577 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:12:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:12:33.253+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:12:33.577 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:12:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:12:33.253+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:12:33.578 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:12:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:12:33.253+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:12:34.206 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:12:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:12:33.907+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:12:34.206 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:12:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:12:33.907+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:12:34.206 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:12:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:12:33.907+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:12:34.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:12:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:12:34.221+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:12:34.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:12:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:12:34.221+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:12:34.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:12:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:12:34.221+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:12:34.491 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:12:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:12:34.205+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:12:34.491 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:12:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:12:34.205+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:12:34.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:12:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:12:34.205+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:12:34.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:12:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:12:34.205+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:12:35.225 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:12:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:12:34.897+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:12:35.226 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:12:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:12:34.897+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:12:35.226 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:12:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:12:34.897+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:12:35.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:12:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:12:35.241+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:12:35.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:12:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:12:35.241+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:12:35.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:12:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:12:35.241+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:12:35.491 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:12:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:12:35.225+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:12:35.491 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:12:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:12:35.225+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:12:35.491 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:12:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:12:35.225+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:12:35.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:12:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:12:35.225+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:12:36.204 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:12:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:12:35.941+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:12:36.204 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:12:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:12:35.941+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:12:36.205 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:12:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:12:35.941+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:12:36.490 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:12:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:12:36.247+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:12:36.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:12:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:12:36.247+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:12:36.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:12:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:12:36.247+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:12:36.491 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:12:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:12:36.203+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:12:36.491 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:12:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:12:36.203+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:12:36.491 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:12:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:12:36.203+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:12:36.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:12:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:12:36.203+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:12:37.188 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:12:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:12:36.881+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 11183 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:12:37.188 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:12:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:12:36.983+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:12:37.188 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:12:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:12:36.983+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:12:37.188 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:12:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:12:36.983+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:12:37.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:12:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:12:37.205+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:12:37.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:12:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:12:37.205+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:12:37.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:12:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:12:37.205+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:12:37.491 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:12:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:12:37.186+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:12:37.491 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:12:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:12:37.186+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:12:37.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:12:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:12:37.186+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:12:37.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:12:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:12:37.187+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:12:38.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:12:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:12:38.021+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:12:38.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:12:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:12:38.021+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:12:38.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:12:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:12:38.021+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:12:38.491 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:12:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:12:38.177+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:12:38.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:12:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:12:38.177+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:12:38.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:12:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:12:38.177+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:12:38.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:12:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:12:38.177+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:12:38.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:12:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:12:38.224+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:12:38.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:12:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:12:38.224+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:12:38.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:12:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:12:38.224+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:12:39.231 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:12:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:12:38.974+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:12:39.232 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:12:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:12:38.974+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:12:39.232 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:12:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:12:38.974+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:12:39.232 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:12:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:12:39.179+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:12:39.232 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:12:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:12:39.179+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:12:39.232 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:12:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:12:39.179+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:12:39.232 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:12:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:12:39.179+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:12:39.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:12:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:12:39.230+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:12:39.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:12:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:12:39.230+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:12:39.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:12:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:12:39.230+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:12:40.152 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:12:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:12:39.957+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:12:40.152 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:12:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:12:39.957+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:12:40.153 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:12:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:12:39.957+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:12:40.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:12:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:12:40.241+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:12:40.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:12:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:12:40.241+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:12:40.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:12:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:12:40.241+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:12:40.491 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:12:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:12:40.208+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:12:40.491 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:12:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:12:40.208+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:12:40.491 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:12:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:12:40.208+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:12:40.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:12:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:12:40.208+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:12:41.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:12:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:12:40.924+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:12:41.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:12:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:12:40.924+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:12:41.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:12:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:12:40.924+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:12:41.686 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:12:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:12:41.279+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:12:41.687 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:12:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:12:41.279+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:12:41.687 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:12:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:12:41.279+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:12:41.687 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:12:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:12:41.253+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:12:41.687 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:12:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:12:41.253+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:12:41.687 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:12:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:12:41.253+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:12:41.687 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:12:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:12:41.253+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:12:41.991 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:12:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:12:41.883+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 11265 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:12:41.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:12:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:12:41.909+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:12:41.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:12:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:12:41.909+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:12:41.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:12:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:12:41.909+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:12:42.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:12:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:12:42.318+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:12:42.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:12:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:12:42.318+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:12:42.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:12:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:12:42.318+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:12:42.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:12:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:12:42.279+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:12:42.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:12:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:12:42.279+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:12:42.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:12:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:12:42.279+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:12:42.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:12:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:12:42.279+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:12:43.236 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:12:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:12:42.907+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:12:43.236 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:12:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:12:42.907+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:12:43.236 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:12:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:12:42.907+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:12:43.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:12:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:12:43.353+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:12:43.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:12:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:12:43.353+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:12:43.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:12:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:12:43.353+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:12:43.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:12:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:12:43.235+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:12:43.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:12:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:12:43.235+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:12:43.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:12:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:12:43.235+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:12:43.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:12:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:12:43.235+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:12:44.192 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:12:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:12:43.857+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:12:44.192 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:12:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:12:43.857+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:12:44.192 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:12:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:12:43.857+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:12:44.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:12:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:12:44.335+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:12:44.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:12:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:12:44.335+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:12:44.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:12:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:12:44.335+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:12:44.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:12:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:12:44.189+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:12:44.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:12:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:12:44.189+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:12:44.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:12:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:12:44.189+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:12:44.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:12:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:12:44.189+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:12:45.226 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:12:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:12:44.829+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:12:45.226 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:12:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:12:44.829+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:12:45.227 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:12:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:12:44.829+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:12:45.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:12:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:12:45.329+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:12:45.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:12:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:12:45.329+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:12:45.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:12:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:12:45.329+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:12:45.491 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:12:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:12:45.225+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:12:45.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:12:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:12:45.225+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:12:45.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:12:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:12:45.225+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:12:45.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:12:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:12:45.225+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:12:46.199 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:12:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:12:45.806+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:12:46.199 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:12:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:12:45.806+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:12:46.199 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:12:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:12:45.806+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:12:46.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:12:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:12:46.290+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:12:46.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:12:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:12:46.290+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:12:46.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:12:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:12:46.290+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:12:46.491 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:12:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:12:46.235+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:12:46.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:12:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:12:46.235+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:12:46.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:12:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:12:46.235+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:12:46.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:12:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:12:46.235+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:12:47.241 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:12:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:12:46.885+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 11347 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:12:47.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:12:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:12:46.794+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:12:47.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:12:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:12:46.794+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:12:47.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:12:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:12:46.794+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:12:47.736 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:12:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:12:47.269+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:12:47.736 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:12:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:12:47.269+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:12:47.736 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:12:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:12:47.269+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:12:47.736 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:12:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:12:47.269+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:12:47.737 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:12:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:12:47.288+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:12:47.737 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:12:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:12:47.288+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:12:47.737 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:12:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:12:47.288+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:12:48.240 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:12:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:12:47.802+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:12:48.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:12:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:12:47.802+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:12:48.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:12:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:12:47.802+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:12:48.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:12:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:12:48.249+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:12:48.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:12:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:12:48.249+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:12:48.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:12:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:12:48.249+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:12:48.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:12:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:12:48.249+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:12:48.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:12:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:12:48.291+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:12:48.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:12:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:12:48.291+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:12:48.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:12:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:12:48.291+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:12:49.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:12:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:12:48.831+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:12:49.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:12:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:12:48.831+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:12:49.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:12:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:12:48.831+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:12:49.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:12:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:12:49.240+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:12:49.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:12:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:12:49.240+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:12:49.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:12:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:12:49.240+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:12:49.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:12:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:12:49.240+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:12:49.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:12:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:12:49.283+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:12:49.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:12:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:12:49.283+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:12:49.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:12:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:12:49.283+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:12:50.153 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:12:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:12:49.821+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:12:50.153 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:12:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:12:49.821+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:12:50.153 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:12:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:12:49.821+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:12:50.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:12:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:12:50.290+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:12:50.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:12:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:12:50.290+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:12:50.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:12:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:12:50.290+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:12:50.491 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:12:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:12:50.212+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:12:50.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:12:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:12:50.212+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:12:50.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:12:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:12:50.212+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:12:50.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:12:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:12:50.212+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:12:51.213 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:12:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:12:50.835+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:12:51.214 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:12:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:12:50.835+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:12:51.214 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:12:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:12:50.835+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:12:51.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:12:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:12:51.337+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:12:51.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:12:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:12:51.337+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:12:51.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:12:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:12:51.337+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:12:51.491 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:12:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:12:51.213+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:12:51.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:12:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:12:51.213+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:12:51.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:12:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:12:51.213+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:12:51.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:12:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:12:51.213+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:12:51.991 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:12:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:12:51.887+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 11430 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:12:51.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:12:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:12:51.812+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:12:51.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:12:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:12:51.812+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:12:51.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:12:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:12:51.812+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:12:52.318 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:12:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:12:52.237+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:12:52.319 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:12:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:12:52.237+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:12:52.319 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:12:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:12:52.237+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:12:52.319 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:12:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:12:52.237+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:12:52.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:12:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:12:52.318+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:12:52.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:12:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:12:52.318+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:12:52.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:12:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:12:52.318+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:12:53.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:12:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:12:52.797+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:12:53.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:12:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:12:52.797+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:12:53.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:12:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:12:52.797+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:12:53.579 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:12:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:12:53.241+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:12:53.579 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:12:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:12:53.241+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:12:53.579 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:12:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:12:53.241+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:12:53.579 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:12:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:12:53.241+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:12:53.580 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:12:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:12:53.329+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:12:53.580 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:12:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:12:53.329+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:12:53.580 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:12:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:12:53.329+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:12:54.223 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:12:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:12:53.841+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:12:54.223 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:12:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:12:53.841+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:12:54.223 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:12:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:12:53.841+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:12:54.491 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:12:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:12:54.222+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:12:54.491 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:12:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:12:54.222+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:12:54.491 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:12:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:12:54.222+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:12:54.491 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:12:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:12:54.222+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:12:54.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:12:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:12:54.344+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:12:54.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:12:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:12:54.344+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:12:54.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:12:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:12:54.344+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:12:55.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:12:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:12:54.871+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:12:55.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:12:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:12:54.871+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:12:55.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:12:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:12:54.871+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:12:55.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:12:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:12:55.247+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:12:55.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:12:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:12:55.247+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:12:55.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:12:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:12:55.247+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:12:55.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:12:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:12:55.247+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:12:55.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:12:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:12:55.342+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:12:55.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:12:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:12:55.342+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:12:55.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:12:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:12:55.342+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:12:56.237 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:12:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:12:55.898+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:12:56.237 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:12:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:12:55.898+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:12:56.237 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:12:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:12:55.898+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:12:56.491 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:12:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:12:56.236+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:12:56.491 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:12:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:12:56.236+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:12:56.491 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:12:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:12:56.236+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:12:56.491 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:12:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:12:56.236+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:12:56.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:12:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:12:56.358+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:12:56.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:12:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:12:56.358+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:12:56.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:12:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:12:56.358+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:12:57.241 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:12:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:12:56.889+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 11513 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:12:57.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:12:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:12:56.873+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:12:57.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:12:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:12:56.873+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:12:57.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:12:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:12:56.873+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:12:57.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:12:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:12:57.255+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:12:57.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:12:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:12:57.255+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:12:57.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:12:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:12:57.255+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:12:57.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:12:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:12:57.255+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:12:57.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:12:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:12:57.332+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:12:57.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:12:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:12:57.332+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:12:57.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:12:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:12:57.332+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:12:58.240 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:12:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:12:57.851+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:12:58.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:12:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:12:57.851+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:12:58.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:12:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:12:57.851+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:12:58.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:12:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:12:58.289+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:12:58.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:12:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:12:58.289+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:12:58.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:12:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:12:58.289+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:12:58.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:12:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:12:58.283+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:12:58.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:12:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:12:58.283+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:12:58.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:12:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:12:58.283+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:12:58.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:12:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:12:58.283+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:12:59.240 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:12:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:12:58.864+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:12:59.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:12:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:12:58.864+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:12:59.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:12:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:12:58.864+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:12:59.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:12:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:12:59.298+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:12:59.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:12:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:12:59.298+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:12:59.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:12:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:12:59.298+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:12:59.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:12:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:12:59.245+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:12:59.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:12:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:12:59.245+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:12:59.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:12:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:12:59.245+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:12:59.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:12:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:12:59.245+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:13:00.154 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:12:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:12:59.868+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:13:00.155 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:12:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:12:59.868+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:13:00.155 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:12:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:12:59.868+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:13:00.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:13:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:13:00.268+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:13:00.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:13:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:13:00.268+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:13:00.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:13:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:13:00.268+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:13:00.491 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:13:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:13:00.277+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:13:00.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:13:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:13:00.277+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:13:00.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:13:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:13:00.277+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:13:00.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:13:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:13:00.277+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:13:01.240 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:13:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:13:00.899+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:13:01.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:13:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:13:00.899+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:13:01.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:13:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:13:00.899+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:13:01.559 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:13:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:13:01.320+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:13:01.560 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:13:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:13:01.320+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:13:01.560 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:13:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:13:01.320+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:13:01.560 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:13:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:13:01.320+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:13:01.560 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:13:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:13:01.261+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:13:01.560 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:13:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:13:01.261+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:13:01.560 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:13:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:13:01.261+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:13:02.241 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:13:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:13:01.891+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 11595 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:13:02.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:13:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:13:01.875+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:13:02.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:13:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:13:01.875+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:13:02.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:13:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:13:01.875+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:13:02.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:13:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:13:02.344+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:13:02.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:13:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:13:02.344+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:13:02.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:13:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:13:02.344+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:13:02.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:13:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:13:02.344+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:13:02.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:13:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:13:02.298+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:13:02.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:13:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:13:02.298+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:13:02.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:13:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:13:02.298+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:13:03.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:13:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:13:02.857+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:13:03.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:13:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:13:02.857+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:13:03.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:13:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:13:02.857+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:13:03.580 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:13:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:13:03.287+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:13:03.580 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:13:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:13:03.287+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:13:03.581 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:13:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:13:03.287+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:13:03.581 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:13:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:13:03.344+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:13:03.581 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:13:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:13:03.344+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:13:03.581 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:13:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:13:03.344+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:13:03.581 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:13:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:13:03.344+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:13:03.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:13:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:13:03.824+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:13:03.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:13:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:13:03.824+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:13:03.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:13:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:13:03.824+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:13:04.631 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:13:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:13:04.274+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:13:04.631 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:13:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:13:04.274+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:13:04.631 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:13:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:13:04.274+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:13:04.632 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:13:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:13:04.326+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:13:04.632 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:13:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:13:04.326+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:13:04.632 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:13:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:13:04.326+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:13:04.632 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:13:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:13:04.326+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:13:04.990 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:13:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:13:04.823+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:13:04.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:13:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:13:04.823+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:13:04.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:13:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:13:04.823+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:13:05.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:13:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:13:05.232+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:13:05.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:13:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:13:05.232+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:13:05.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:13:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:13:05.232+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:13:05.491 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:13:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:13:05.290+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:13:05.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:13:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:13:05.290+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:13:05.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:13:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:13:05.290+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:13:05.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:13:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:13:05.291+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:13:06.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:13:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:13:05.789+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:13:06.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:13:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:13:05.789+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:13:06.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:13:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:13:05.789+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:13:06.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:13:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:13:06.275+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:13:06.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:13:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:13:06.275+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:13:06.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:13:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:13:06.275+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:13:06.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:13:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:13:06.242+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:13:06.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:13:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:13:06.242+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:13:06.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:13:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:13:06.242+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:13:06.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:13:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:13:06.242+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:13:07.017 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:13:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:13:06.893+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 11677 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:13:07.018 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:13:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:13:06.745+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:13:07.018 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:13:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:13:06.745+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:13:07.018 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:13:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:13:06.745+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:13:07.283 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:13:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:13:07.243+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:13:07.283 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:13:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:13:07.243+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:13:07.283 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:13:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:13:07.243+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:13:07.703 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:13:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:13:07.283+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:13:07.704 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:13:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:13:07.283+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:13:07.704 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:13:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:13:07.283+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:13:07.704 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:13:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:13:07.283+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:13:07.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:13:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:13:07.713+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:13:07.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:13:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:13:07.713+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:13:07.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:13:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:13:07.713+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:13:08.690 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:13:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:13:08.242+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:13:08.691 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:13:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:13:08.242+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:13:08.691 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:13:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:13:08.242+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:13:08.691 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:13:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:13:08.252+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:13:08.691 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:13:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:13:08.252+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:13:08.691 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:13:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:13:08.252+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:13:08.691 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:13:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:13:08.252+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:13:08.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:13:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:13:08.690+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:13:08.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:13:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:13:08.690+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:13:08.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:13:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:13:08.690+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:13:09.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:13:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:13:09.196+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:13:09.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:13:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:13:09.196+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:13:09.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:13:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:13:09.196+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:13:09.491 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:13:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:13:09.289+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:13:09.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:13:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:13:09.289+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:13:09.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:13:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:13:09.289+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:13:09.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:13:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:13:09.289+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:13:09.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:13:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:13:09.696+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:13:09.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:13:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:13:09.696+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:13:09.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:13:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:13:09.696+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:13:10.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:13:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:13:10.147+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:13:10.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:13:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:13:10.147+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:13:10.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:13:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:13:10.147+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:13:10.491 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:13:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:13:10.253+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:13:10.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:13:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:13:10.253+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:13:10.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:13:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:13:10.253+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:13:10.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:13:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:13:10.253+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:13:10.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:13:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:13:10.687+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:13:10.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:13:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:13:10.687+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:13:10.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:13:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:13:10.687+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:13:11.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:13:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:13:11.161+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:13:11.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:13:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:13:11.161+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:13:11.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:13:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:13:11.161+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:13:11.491 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:13:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:13:11.242+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:13:11.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:13:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:13:11.242+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:13:11.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:13:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:13:11.242+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:13:11.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:13:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:13:11.242+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:13:11.920 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:13:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:13:11.895+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 11760 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:13:11.920 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:13:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:13:11.669+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:13:11.920 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:13:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:13:11.669+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:13:11.920 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:13:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:13:11.669+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:13:12.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:13:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:13:12.205+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:13:12.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:13:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:13:12.205+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:13:12.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:13:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:13:12.205+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:13:12.491 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:13:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:13:12.195+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:13:12.491 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:13:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:13:12.195+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:13:12.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:13:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:13:12.195+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:13:12.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:13:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:13:12.196+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:13:12.990 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:13:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:13:12.700+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:13:12.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:13:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:13:12.700+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:13:12.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:13:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:13:12.700+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:13:13.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:13:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:13:13.197+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:13:13.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:13:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:13:13.197+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:13:13.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:13:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:13:13.197+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:13:13.491 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:13:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:13:13.221+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:13:13.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:13:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:13:13.221+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:13:13.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:13:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:13:13.221+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:13:13.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:13:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:13:13.221+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:13:13.847 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:13:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:13:13.661+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:13:13.847 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:13:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:13:13.661+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:13:13.847 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:13:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:13:13.661+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:13:14.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:13:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:13:14.172+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:13:14.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:13:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:13:14.172+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:13:14.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:13:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:13:14.172+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:13:14.491 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:13:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:13:14.217+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:13:14.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:13:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:13:14.217+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:13:14.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:13:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:13:14.217+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:13:14.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:13:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:13:14.217+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:13:14.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:13:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:13:14.631+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:13:14.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:13:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:13:14.631+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:13:14.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:13:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:13:14.631+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:13:15.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:13:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:13:15.176+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:13:15.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:13:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:13:15.176+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:13:15.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:13:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:13:15.176+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:13:15.491 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:13:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:13:15.216+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:13:15.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:13:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:13:15.216+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:13:15.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:13:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:13:15.216+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:13:15.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:13:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:13:15.216+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:13:15.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:13:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:13:15.645+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:13:15.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:13:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:13:15.645+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:13:15.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:13:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:13:15.645+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:13:16.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:13:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:13:16.167+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:13:16.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:13:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:13:16.167+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:13:16.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:13:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:13:16.167+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:13:16.491 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:13:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:13:16.192+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:13:16.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:13:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:13:16.192+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:13:16.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:13:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:13:16.192+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:13:16.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:13:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:13:16.192+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:13:16.991 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:13:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:13:16.897+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 11842 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:13:16.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:13:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:13:16.691+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:13:16.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:13:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:13:16.691+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:13:16.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:13:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:13:16.691+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:13:17.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:13:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:13:17.131+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:13:17.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:13:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:13:17.131+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:13:17.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:13:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:13:17.131+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:13:17.491 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:13:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:13:17.192+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:13:17.491 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:13:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:13:17.192+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:13:17.491 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:13:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:13:17.192+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:13:17.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:13:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:13:17.192+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:13:17.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:13:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:13:17.688+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:13:17.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:13:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:13:17.688+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:13:17.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:13:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:13:17.688+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:13:18.456 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:13:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:13:18.173+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:13:18.456 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:13:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:13:18.173+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:13:18.456 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:13:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:13:18.173+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:13:18.456 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:13:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:13:18.223+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:13:18.457 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:13:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:13:18.223+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:13:18.457 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:13:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:13:18.223+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:13:18.457 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:13:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:13:18.223+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:13:18.740 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:13:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:13:18.653+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:13:18.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:13:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:13:18.654+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:13:18.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:13:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:13:18.654+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:13:19.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:13:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:13:19.135+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:13:19.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:13:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:13:19.135+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:13:19.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:13:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:13:19.135+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:13:19.491 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:13:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:13:19.192+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:13:19.491 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:13:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:13:19.192+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:13:19.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:13:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:13:19.192+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:13:19.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:13:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:13:19.192+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:13:19.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:13:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:13:19.647+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:13:19.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:13:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:13:19.647+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:13:19.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:13:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:13:19.647+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:13:20.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:13:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:13:20.184+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:13:20.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:13:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:13:20.184+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:13:20.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:13:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:13:20.184+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:13:20.491 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:13:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:13:20.176+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:13:20.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:13:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:13:20.176+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:13:20.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:13:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:13:20.176+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:13:20.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:13:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:13:20.176+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:13:20.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:13:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:13:20.621+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:13:20.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:13:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:13:20.621+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:13:20.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:13:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:13:20.621+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:13:21.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:13:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:13:21.159+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:13:21.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:13:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:13:21.159+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:13:21.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:13:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:13:21.159+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:13:21.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:13:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:13:21.225+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:13:21.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:13:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:13:21.225+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:13:21.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:13:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:13:21.225+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:13:21.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:13:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:13:21.225+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:13:21.923 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:13:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:13:21.899+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 11925 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:13:21.923 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:13:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:13:21.665+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:13:21.923 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:13:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:13:21.665+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:13:21.924 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:13:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:13:21.665+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:13:22.206 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:13:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:13:22.155+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:13:22.206 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:13:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:13:22.155+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:13:22.206 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:13:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:13:22.155+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:13:22.491 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:13:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:13:22.206+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:13:22.491 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:13:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:13:22.206+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:13:22.491 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:13:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:13:22.206+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:13:22.491 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:13:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:13:22.206+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:13:22.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:13:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:13:22.624+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:13:22.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:13:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:13:22.624+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:13:22.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:13:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:13:22.624+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:13:23.490 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:13:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:13:23.222+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:13:23.491 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:13:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:13:23.222+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:13:23.491 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:13:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:13:23.222+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:13:23.491 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:13:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:13:23.222+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:13:23.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:13:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:13:23.166+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:13:23.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:13:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:13:23.166+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:13:23.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:13:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:13:23.166+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:13:23.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:13:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:13:23.668+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:13:23.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:13:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:13:23.668+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:13:23.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:13:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:13:23.668+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:13:24.491 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:13:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:13:24.197+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:13:24.491 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:13:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:13:24.197+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:13:24.491 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:13:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:13:24.197+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:13:24.491 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:13:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:13:24.197+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:13:24.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:13:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:13:24.187+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:13:24.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:13:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:13:24.187+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:13:24.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:13:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:13:24.187+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:13:24.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:13:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:13:24.641+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:13:24.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:13:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:13:24.641+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:13:24.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:13:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:13:24.641+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:13:25.491 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:13:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:13:25.223+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:13:25.491 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:13:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:13:25.223+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:13:25.491 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:13:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:13:25.223+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:13:25.491 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:13:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:13:25.223+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:13:25.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:13:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:13:25.145+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:13:25.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:13:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:13:25.145+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:13:25.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:13:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:13:25.145+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:13:25.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:13:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:13:25.644+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:13:25.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:13:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:13:25.644+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:13:25.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:13:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:13:25.644+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:13:26.490 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:13:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:13:26.257+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:13:26.491 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:13:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:13:26.257+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:13:26.491 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:13:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:13:26.257+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:13:26.491 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:13:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:13:26.257+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:13:26.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:13:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:13:26.116+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:13:26.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:13:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:13:26.116+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:13:26.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:13:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:13:26.116+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:13:26.901 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:13:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:13:26.614+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:13:26.901 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:13:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:13:26.614+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:13:26.901 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:13:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:13:26.614+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:13:27.155 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:13:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:13:26.900+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 12006 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:13:27.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:13:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:13:27.154+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:13:27.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:13:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:13:27.154+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:13:27.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:13:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:13:27.154+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:13:27.491 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:13:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:13:27.209+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:13:27.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:13:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:13:27.209+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:13:27.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:13:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:13:27.209+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:13:27.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:13:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:13:27.209+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:13:27.990 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:13:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:13:27.637+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:13:27.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:13:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:13:27.637+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:13:27.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:13:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:13:27.637+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:13:28.491 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:13:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:13:28.239+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:13:28.491 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:13:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:13:28.239+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:13:28.491 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:13:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:13:28.239+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:13:28.491 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:13:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:13:28.239+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:13:28.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:13:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:13:28.137+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:13:28.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:13:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:13:28.137+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:13:28.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:13:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:13:28.137+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:13:28.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:13:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:13:28.590+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:13:28.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:13:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:13:28.590+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:13:28.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:13:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:13:28.590+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:13:29.491 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:13:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:13:29.289+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:13:29.491 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:13:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:13:29.289+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:13:29.491 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:13:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:13:29.289+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:13:29.491 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:13:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:13:29.289+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:13:29.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:13:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:13:29.116+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:13:29.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:13:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:13:29.116+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:13:29.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:13:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:13:29.116+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:13:29.884 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:13:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:13:29.568+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:13:29.885 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:13:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:13:29.568+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:13:29.885 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:13:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:13:29.568+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:13:30.156 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:13:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:13:30.123+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:13:30.156 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:13:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:13:30.123+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:13:30.156 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:13:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:13:30.123+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:13:30.491 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:13:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:13:30.262+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:13:30.491 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:13:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:13:30.262+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:13:30.491 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:13:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:13:30.262+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:13:30.491 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:13:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:13:30.262+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:13:30.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:13:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:13:30.559+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:13:30.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:13:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:13:30.559+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:13:30.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:13:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:13:30.559+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:13:31.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:13:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:13:31.100+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:13:31.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:13:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:13:31.100+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:13:31.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:13:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:13:31.100+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:13:31.491 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:13:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:13:31.219+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:13:31.491 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:13:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:13:31.219+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:13:31.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:13:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:13:31.219+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:13:31.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:13:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:13:31.219+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:13:31.903 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:13:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:13:31.535+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:13:31.903 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:13:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:13:31.535+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:13:31.903 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:13:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:13:31.535+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:13:32.182 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:13:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:13:31.902+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 12090 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:13:32.182 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:13:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:13:32.054+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:13:32.182 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:13:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:13:32.054+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:13:32.182 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:13:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:13:32.054+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:13:32.491 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:13:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:13:32.181+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:13:32.491 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:13:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:13:32.181+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:13:32.491 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:13:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:13:32.181+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:13:32.491 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:13:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:13:32.181+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:13:32.990 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:13:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:13:32.498+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:13:32.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:13:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:13:32.498+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:13:32.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:13:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:13:32.498+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:13:33.481 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:13:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:13:33.069+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:13:33.481 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:13:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:13:33.069+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:13:33.481 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:13:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:13:33.069+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:13:33.482 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:13:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:13:33.205+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:13:33.482 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:13:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:13:33.205+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:13:33.482 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:13:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:13:33.205+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:13:33.482 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:13:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:13:33.205+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:13:33.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:13:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:13:33.480+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:13:33.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:13:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:13:33.480+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:13:33.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:13:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:13:33.480+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:13:34.463 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:13:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:13:34.101+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:13:34.463 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:13:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:13:34.101+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:13:34.463 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:13:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:13:34.101+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:13:34.464 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:13:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:13:34.254+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:13:34.464 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:13:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:13:34.254+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:13:34.464 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:13:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:13:34.254+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:13:34.464 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:13:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:13:34.254+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:13:34.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:13:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:13:34.462+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:13:34.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:13:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:13:34.462+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:13:34.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:13:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:13:34.462+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:13:35.351 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:13:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:13:35.056+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:13:35.351 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:13:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:13:35.056+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:13:35.352 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:13:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:13:35.056+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:13:35.352 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:13:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:13:35.230+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:13:35.352 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:13:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:13:35.230+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:13:35.352 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:13:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:13:35.230+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:13:35.352 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:13:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:13:35.230+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:13:35.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:13:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:13:35.473+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:13:35.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:13:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:13:35.473+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:13:35.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:13:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:13:35.473+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:13:36.489 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:13:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:13:36.024+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:13:36.490 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:13:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:13:36.024+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:13:36.490 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:13:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:13:36.024+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:13:36.490 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:13:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:13:36.258+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:13:36.490 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:13:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:13:36.258+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:13:36.490 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:13:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:13:36.258+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:13:36.490 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:13:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:13:36.258+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:13:36.740 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:13:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:13:36.488+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:13:36.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:13:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:13:36.488+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:13:36.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:13:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:13:36.488+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:13:37.241 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:13:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:13:36.904+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 12172 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:13:37.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:13:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:13:37.028+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:13:37.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:13:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:13:37.028+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:13:37.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:13:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:13:37.028+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:13:37.538 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:13:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:13:37.246+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:13:37.538 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:13:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:13:37.246+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:13:37.538 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:13:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:13:37.246+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:13:37.538 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:13:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:13:37.246+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:13:37.990 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:13:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:13:37.536+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:13:37.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:13:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:13:37.536+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:13:37.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:13:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:13:37.536+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:13:38.424 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:13:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:13:38.078+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:13:38.424 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:13:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:13:38.078+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:13:38.424 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:13:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:13:38.078+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:13:38.424 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:13:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:13:38.205+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:13:38.425 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:13:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:13:38.205+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:13:38.425 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:13:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:13:38.205+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:13:38.425 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:13:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:13:38.205+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:13:38.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:13:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:13:38.556+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:13:38.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:13:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:13:38.556+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:13:38.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:13:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:13:38.556+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:13:39.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:13:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:13:39.054+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:13:39.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:13:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:13:39.054+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:13:39.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:13:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:13:39.054+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:13:39.491 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:13:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:13:39.188+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:13:39.491 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:13:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:13:39.188+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:13:39.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:13:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:13:39.188+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:13:39.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:13:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:13:39.188+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:13:39.885 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:13:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:13:39.523+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:13:39.885 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:13:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:13:39.523+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:13:39.885 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:13:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:13:39.523+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:13:40.157 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:13:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:13:40.074+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:13:40.157 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:13:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:13:40.074+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:13:40.158 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:13:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:13:40.074+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:13:40.475 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:13:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:13:40.170+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:13:40.475 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:13:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:13:40.170+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:13:40.476 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:13:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:13:40.170+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:13:40.476 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:13:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:13:40.170+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:13:40.740 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:13:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:13:40.474+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:13:40.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:13:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:13:40.474+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:13:40.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:13:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:13:40.474+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:13:41.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:13:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:13:41.053+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:13:41.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:13:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:13:41.053+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:13:41.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:13:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:13:41.053+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:13:41.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:13:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:13:41.152+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:13:41.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:13:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:13:41.152+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:13:41.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:13:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:13:41.152+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:13:41.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:13:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:13:41.152+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:13:41.907 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:13:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:13:41.515+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:13:41.907 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:13:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:13:41.515+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:13:41.907 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:13:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:13:41.515+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:13:42.241 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:13:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:13:41.906+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 12255 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:13:42.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:13:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:13:42.094+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:13:42.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:13:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:13:42.094+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:13:42.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:13:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:13:42.094+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:13:42.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:13:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:13:42.118+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:13:42.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:13:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:13:42.118+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:13:42.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:13:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:13:42.118+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:13:42.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:13:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:13:42.118+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:13:42.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:13:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:13:42.489+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:13:42.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:13:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:13:42.489+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:13:42.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:13:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:13:42.489+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:13:43.453 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:13:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:13:43.114+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:13:43.453 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:13:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:13:43.114+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:13:43.453 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:13:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:13:43.114+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:13:43.453 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:13:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:13:43.083+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:13:43.453 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:13:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:13:43.083+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:13:43.454 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:13:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:13:43.083+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:13:43.454 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:13:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:13:43.083+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:13:43.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:13:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:13:43.452+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:13:43.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:13:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:13:43.452+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:13:43.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:13:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:13:43.452+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:13:44.465 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:13:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:13:44.070+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:13:44.465 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:13:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:13:44.070+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:13:44.465 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:13:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:13:44.070+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:13:44.465 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:13:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:13:44.070+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:13:44.466 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:13:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:13:44.161+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:13:44.466 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:13:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:13:44.161+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:13:44.466 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:13:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:13:44.161+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:13:44.740 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:13:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:13:44.464+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:13:44.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:13:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:13:44.464+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:13:44.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:13:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:13:44.464+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:13:45.429 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:13:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:13:45.104+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:13:45.429 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:13:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:13:45.104+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:13:45.429 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:13:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:13:45.104+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:13:45.429 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:13:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:13:45.104+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:13:45.429 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:13:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:13:45.129+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:13:45.430 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:13:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:13:45.129+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:13:45.430 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:13:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:13:45.129+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:13:45.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:13:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:13:45.428+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:13:45.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:13:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:13:45.428+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:13:45.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:13:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:13:45.428+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:13:46.384 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:13:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:13:46.096+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:13:46.384 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:13:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:13:46.096+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:13:46.384 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:13:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:13:46.096+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:13:46.384 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:13:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:13:46.096+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:13:46.384 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:13:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:13:46.178+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:13:46.385 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:13:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:13:46.178+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:13:46.385 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:13:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:13:46.178+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:13:46.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:13:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:13:46.382+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:13:46.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:13:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:13:46.382+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:13:46.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:13:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:13:46.382+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:13:47.220 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:13:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:13:46.908+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 12330 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:13:47.220 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:13:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:13:47.056+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:13:47.221 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:13:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:13:47.056+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:13:47.221 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:13:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:13:47.056+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:13:47.221 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:13:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:13:47.056+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:13:47.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:13:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:13:47.408+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:13:47.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:13:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:13:47.408+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:13:47.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:13:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:13:47.408+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:13:47.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:13:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:13:47.220+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:13:47.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:13:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:13:47.220+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:13:47.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:13:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:13:47.220+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:13:48.413 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:13:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:13:48.240+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:13:48.414 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:13:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:13:48.240+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:13:48.414 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:13:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:13:48.240+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:13:48.414 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:13:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:13:48.037+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:13:48.414 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:13:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:13:48.037+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:13:48.414 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:13:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:13:48.037+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:13:48.415 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:13:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:13:48.037+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:13:48.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:13:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:13:48.412+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:13:48.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:13:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:13:48.412+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:13:48.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:13:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:13:48.412+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:13:49.400 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:13:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:13:49.252+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:13:49.400 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:13:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:13:49.252+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:13:49.400 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:13:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:13:49.252+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:13:49.401 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:13:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:13:49.066+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:13:49.401 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:13:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:13:49.066+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:13:49.401 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:13:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:13:49.066+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:13:49.401 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:13:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:13:49.066+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:13:49.688 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:13:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:13:49.399+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:13:49.688 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:13:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:13:49.399+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:13:49.688 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:13:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:13:49.399+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:13:50.398 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:13:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:13:50.291+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:13:50.399 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:13:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:13:50.291+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:13:50.399 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:13:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:13:50.291+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:13:50.399 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:13:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:13:50.086+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:13:50.399 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:13:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:13:50.086+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:13:50.399 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:13:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:13:50.086+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:13:50.399 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:13:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:13:50.086+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:13:50.711 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:13:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:13:50.398+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:13:50.711 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:13:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:13:50.398+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:13:50.711 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:13:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:13:50.398+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:13:51.382 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:13:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:13:51.278+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:13:51.382 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:13:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:13:51.278+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:13:51.382 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:13:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:13:51.278+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:13:51.383 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:13:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:13:51.071+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:13:51.383 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:13:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:13:51.071+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:13:51.383 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:13:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:13:51.071+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:13:51.383 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:13:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:13:51.071+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:13:51.690 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:13:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:13:51.382+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:13:51.690 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:13:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:13:51.382+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:13:51.690 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:13:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:13:51.382+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:13:51.991 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:13:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:13:51.909+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 12401 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:13:52.298 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:13:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:13:52.086+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:13:52.298 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:13:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:13:52.086+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:13:52.298 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:13:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:13:52.086+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:13:52.298 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:13:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:13:52.086+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:13:52.740 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:13:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:13:52.382+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:13:52.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:13:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:13:52.382+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:13:52.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:13:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:13:52.382+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:13:52.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:13:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:13:52.296+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:13:52.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:13:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:13:52.296+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:13:52.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:13:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:13:52.296+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:13:53.361 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:13:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:13:53.257+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:13:53.361 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:13:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:13:53.257+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:13:53.361 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:13:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:13:53.257+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:13:53.361 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:13:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:13:53.047+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:13:53.361 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:13:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:13:53.047+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:13:53.361 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:13:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:13:53.047+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:13:53.362 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:13:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:13:53.047+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:13:53.740 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:13:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:13:53.360+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:13:53.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:13:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:13:53.360+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:13:53.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:13:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:13:53.360+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:13:54.238 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:13:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:13:54.014+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:13:54.238 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:13:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:13:54.014+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:13:54.239 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:13:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:13:54.014+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:13:54.239 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:13:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:13:54.014+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:13:54.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:13:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:13:54.363+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:13:54.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:13:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:13:54.363+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:13:54.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:13:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:13:54.363+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:13:54.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:13:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:13:54.238+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:13:54.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:13:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:13:54.238+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:13:54.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:13:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:13:54.238+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:13:55.328 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:13:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:13:55.238+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:13:55.329 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:13:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:13:55.238+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:13:55.329 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:13:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:13:55.238+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:13:55.329 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:13:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:13:55.018+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:13:55.329 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:13:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:13:55.018+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:13:55.329 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:13:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:13:55.018+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:13:55.329 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:13:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:13:55.018+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:13:55.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:13:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:13:55.328+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:13:55.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:13:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:13:55.328+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:13:55.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:13:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:13:55.328+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:13:56.372 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:13:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:13:56.042+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:13:56.372 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:13:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:13:56.042+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:13:56.372 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:13:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:13:56.042+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:13:56.372 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:13:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:13:56.043+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:13:56.373 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:13:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:13:56.238+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:13:56.373 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:13:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:13:56.238+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:13:56.373 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:13:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:13:56.238+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:13:56.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:13:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:13:56.372+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:13:56.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:13:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:13:56.372+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:13:56.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:13:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:13:56.372+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:13:57.241 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:13:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:13:56.912+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 12471 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:13:57.241 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:13:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:13:57.046+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:13:57.241 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:13:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:13:57.046+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:13:57.241 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:13:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:13:57.046+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:13:57.241 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:13:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:13:57.046+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:13:57.740 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:13:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:13:57.252+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:13:57.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:13:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:13:57.252+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:13:57.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:13:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:13:57.252+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:13:57.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:13:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:13:57.343+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:13:57.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:13:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:13:57.343+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:13:57.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:13:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:13:57.343+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:13:58.279 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:13:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:13:58.025+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:13:58.279 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:13:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:13:58.025+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:13:58.279 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:13:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:13:58.025+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:13:58.279 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:13:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:13:58.025+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:13:58.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:13:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:13:58.279+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:13:58.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:13:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:13:58.279+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:13:58.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:13:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:13:58.279+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:13:58.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:13:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:13:58.350+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:13:58.742 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:13:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:13:58.350+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:13:58.742 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:13:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:13:58.350+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:13:59.307 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:13:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:13:58.995+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:13:59.308 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:13:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:13:58.995+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:13:59.308 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:13:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:13:58.995+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:13:59.308 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:13:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:13:58.995+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:13:59.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:13:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:13:59.316+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:13:59.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:13:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:13:59.316+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:13:59.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:13:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:13:59.316+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:13:59.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:13:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:13:59.307+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:13:59.742 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:13:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:13:59.307+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:13:59.742 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:13:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:13:59.307+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:14:00.158 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:13:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:13:59.981+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:14:00.158 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:13:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:13:59.981+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:14:00.159 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:13:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:13:59.981+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:14:00.159 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:13:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:13:59.981+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:14:00.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:14:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:14:00.336+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:14:00.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:14:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:14:00.336+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:14:00.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:14:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:14:00.336+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:14:00.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:14:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:14:00.360+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:14:00.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:14:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:14:00.360+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:14:00.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:14:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:14:00.360+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:14:01.241 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:14:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:14:00.977+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:14:01.241 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:14:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:14:00.977+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:14:01.241 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:14:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:14:00.977+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:14:01.241 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:14:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:14:00.977+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:14:01.691 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:14:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:14:01.332+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:14:01.691 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:14:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:14:01.332+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:14:01.691 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:14:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:14:01.332+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:14:01.691 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:14:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:14:01.365+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:14:01.692 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:14:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:14:01.365+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:14:01.692 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:14:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:14:01.365+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:14:01.991 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:14:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:14:01.914+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 12541 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:14:02.340 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:14:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:14:02.009+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:14:02.340 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:14:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:14:02.009+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:14:02.340 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:14:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:14:02.009+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:14:02.340 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:14:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:14:02.009+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:14:02.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:14:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:14:02.360+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:14:02.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:14:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:14:02.360+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:14:02.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:14:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:14:02.360+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:14:02.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:14:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:14:02.340+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:14:02.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:14:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:14:02.340+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:14:02.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:14:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:14:02.340+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:14:03.366 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:14:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:14:03.015+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:14:03.366 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:14:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:14:03.015+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:14:03.366 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:14:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:14:03.015+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:14:03.366 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:14:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:14:03.015+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:14:03.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:14:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:14:03.366+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:14:03.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:14:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:14:03.366+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:14:03.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:14:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:14:03.366+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:14:03.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:14:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:14:03.389+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:14:03.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:14:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:14:03.389+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:14:03.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:14:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:14:03.389+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:14:04.341 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:14:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:14:04.029+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:14:04.341 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:14:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:14:04.029+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:14:04.341 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:14:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:14:04.029+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:14:04.341 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:14:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:14:04.030+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:14:04.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:14:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:14:04.346+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:14:04.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:14:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:14:04.346+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:14:04.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:14:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:14:04.346+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:14:04.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:14:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:14:04.341+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:14:04.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:14:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:14:04.341+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:14:04.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:14:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:14:04.341+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:14:05.313 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:14:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:14:05.039+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:14:05.313 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:14:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:14:05.039+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:14:05.313 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:14:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:14:05.039+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:14:05.313 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:14:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:14:05.039+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:14:05.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:14:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:14:05.313+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:14:05.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:14:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:14:05.313+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:14:05.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:14:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:14:05.313+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:14:05.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:14:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:14:05.361+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:14:05.742 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:14:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:14:05.361+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:14:05.742 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:14:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:14:05.361+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:14:06.344 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:14:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:14:06.078+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:14:06.344 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:14:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:14:06.078+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:14:06.344 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:14:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:14:06.078+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:14:06.344 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:14:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:14:06.078+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:14:06.740 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:14:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:14:06.395+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:14:06.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:14:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:14:06.395+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:14:06.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:14:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:14:06.395+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:14:06.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:14:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:14:06.344+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:14:06.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:14:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:14:06.344+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:14:06.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:14:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:14:06.344+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:14:07.240 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:14:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:14:06.916+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 12610 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:14:07.241 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:14:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:14:07.109+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:14:07.241 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:14:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:14:07.109+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:14:07.241 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:14:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:14:07.109+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:14:07.241 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:14:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:14:07.110+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:14:07.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:14:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:14:07.360+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:14:07.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:14:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:14:07.360+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:14:07.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:14:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:14:07.360+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:14:07.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:14:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:14:07.342+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:14:07.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:14:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:14:07.342+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:14:07.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:14:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:14:07.342+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:14:08.208 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-02-20T08:14:08.209 WARNING:tasks.kclient:Ordinary umount failed, forcing... 2024-02-20T08:14:08.209 DEBUG:teuthology.orchestra.run.smithi133:> set -ex 2024-02-20T08:14:08.210 DEBUG:teuthology.orchestra.run.smithi133:> dd if=/proc/self/mounts of=/dev/stdout 2024-02-20T08:14:08.231 DEBUG:teuthology.orchestra.run.smithi133:> set -ex 2024-02-20T08:14:08.231 DEBUG:teuthology.orchestra.run.smithi133:> dd if=/proc/self/mounts of=/dev/stdout 2024-02-20T08:14:08.290 DEBUG:teuthology.orchestra.run.smithi133:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-20T08:14:08.351 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:14:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:14:08.083+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:14:08.352 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:14:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:14:08.083+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:14:08.352 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:14:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:14:08.083+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:14:08.352 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:14:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:14:08.083+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:14:08.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:14:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:14:08.351+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:14:08.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:14:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:14:08.351+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:14:08.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:14:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:14:08.351+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:14:08.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:14:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:14:08.387+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:14:08.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:14:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:14:08.387+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:14:08.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:14:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:14:08.387+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:14:09.437 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:14:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:14:09.315+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:14:09.437 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:14:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:14:09.315+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:14:09.437 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:14:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:14:09.315+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:14:09.438 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:14:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:14:09.131+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:14:09.438 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:14:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:14:09.131+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:14:09.438 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:14:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:14:09.131+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:14:09.438 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:14:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:14:09.131+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:14:09.740 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:14:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:14:09.437+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:14:09.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:14:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:14:09.437+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:14:09.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:14:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:14:09.437+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:14:10.413 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:14:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:14:10.323+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:14:10.413 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:14:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:14:10.323+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:14:10.413 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:14:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:14:10.323+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:14:10.414 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:14:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:14:10.157+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:14:10.414 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:14:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:14:10.157+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:14:10.414 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:14:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:14:10.157+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:14:10.414 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:14:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:14:10.157+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:14:10.740 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:14:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:14:10.413+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:14:10.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:14:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:14:10.413+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:14:10.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:14:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:14:10.413+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:14:11.418 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:14:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:14:11.275+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:14:11.418 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:14:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:14:11.275+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:14:11.418 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:14:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:14:11.275+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:14:11.418 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:14:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:14:11.148+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:14:11.419 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:14:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:14:11.148+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:14:11.419 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:14:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:14:11.148+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:14:11.419 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:14:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:14:11.149+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:14:11.691 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:14:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:14:11.418+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:14:11.691 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:14:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:14:11.418+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:14:11.692 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:14:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:14:11.418+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:14:11.990 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:14:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:14:11.918+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 12682 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:14:12.430 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:14:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:14:12.171+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:14:12.430 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:14:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:14:12.171+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:14:12.431 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:14:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:14:12.171+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:14:12.431 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:14:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:14:12.171+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:14:12.431 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:14:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:14:12.266+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:14:12.431 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:14:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:14:12.266+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:14:12.431 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:14:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:14:12.266+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:14:12.740 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:14:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:14:12.430+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:14:12.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:14:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:14:12.430+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:14:12.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:14:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:14:12.430+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:14:13.386 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:14:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:14:13.129+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:14:13.386 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:14:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:14:13.129+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:14:13.386 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:14:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:14:13.129+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:14:13.387 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:14:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:14:13.129+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:14:13.387 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:14:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:14:13.228+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:14:13.387 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:14:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:14:13.228+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:14:13.387 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:14:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:14:13.228+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:14:13.740 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:14:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:14:13.386+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:14:13.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:14:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:14:13.386+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:14:13.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:14:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:14:13.386+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:14:14.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:14:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:14:14.392+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:14:14.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:14:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:14:14.392+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:14:14.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:14:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:14:14.392+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:14:14.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:14:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:14:14.156+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:14:14.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:14:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:14:14.156+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:14:14.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:14:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:14:14.156+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:14:14.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:14:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:14:14.156+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:14:14.492 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:14:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:14:14.222+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:14:14.492 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:14:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:14:14.222+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:14:14.492 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:14:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:14:14.222+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:14:15.424 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:14:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:14:15.266+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:14:15.425 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:14:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:14:15.266+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:14:15.425 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:14:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:14:15.266+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:14:15.425 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:14:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:14:15.140+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:14:15.425 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:14:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:14:15.140+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:14:15.425 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:14:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:14:15.140+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:14:15.426 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:14:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:14:15.140+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:14:15.740 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:14:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:14:15.425+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:14:15.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:14:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:14:15.425+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:14:15.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:14:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:14:15.425+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:14:16.405 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:14:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:14:16.106+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:14:16.406 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:14:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:14:16.106+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:14:16.406 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:14:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:14:16.106+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:14:16.406 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:14:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:14:16.106+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:14:16.406 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:14:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:14:16.298+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:14:16.406 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:14:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:14:16.298+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:14:16.407 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:14:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:14:16.298+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:14:16.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:14:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:14:16.404+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:14:16.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:14:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:14:16.404+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:14:16.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:14:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:14:16.404+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:14:17.241 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:14:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:14:16.919+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 12752 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:14:17.241 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:14:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:14:17.098+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:14:17.241 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:14:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:14:17.098+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:14:17.241 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:14:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:14:17.098+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:14:17.241 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:14:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:14:17.098+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:14:17.740 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:14:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:14:17.290+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:14:17.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:14:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:14:17.290+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:14:17.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:14:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:14:17.290+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:14:17.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:14:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:14:17.369+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:14:17.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:14:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:14:17.369+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:14:17.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:14:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:14:17.369+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:14:18.402 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:14:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:14:18.255+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:14:18.402 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:14:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:14:18.255+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:14:18.403 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:14:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:14:18.255+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:14:18.403 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:14:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:14:18.116+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:14:18.403 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:14:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:14:18.117+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:14:18.403 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:14:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:14:18.117+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:14:18.403 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:14:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:14:18.117+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:14:18.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:14:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:14:18.401+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:14:18.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:14:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:14:18.401+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:14:18.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:14:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:14:18.401+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:14:19.413 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:14:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:14:19.211+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:14:19.413 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:14:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:14:19.211+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:14:19.413 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:14:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:14:19.211+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:14:19.414 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:14:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:14:19.123+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:14:19.414 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:14:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:14:19.123+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:14:19.414 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:14:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:14:19.123+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:14:19.414 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:14:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:14:19.123+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:14:19.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:14:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:14:19.412+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:14:19.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:14:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:14:19.412+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:14:19.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:14:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:14:19.412+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:14:20.160 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:14:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:14:20.120+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:14:20.160 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:14:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:14:20.120+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:14:20.160 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:14:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:14:20.120+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:14:20.160 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:14:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:14:20.120+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:14:20.435 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:14:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:14:20.255+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:14:20.435 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:14:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:14:20.255+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:14:20.435 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:14:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:14:20.255+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:14:20.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:14:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:14:20.434+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:14:20.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:14:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:14:20.434+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:14:20.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:14:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:14:20.434+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:14:21.207 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:14:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:14:21.106+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:14:21.207 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:14:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:14:21.106+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:14:21.207 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:14:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:14:21.106+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:14:21.207 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:14:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:14:21.106+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:14:21.490 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:14:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:14:21.455+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:14:21.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:14:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:14:21.455+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:14:21.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:14:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:14:21.455+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:14:21.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:14:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:14:21.206+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:14:21.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:14:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:14:21.206+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:14:21.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:14:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:14:21.206+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:14:21.943 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:14:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:14:21.921+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 12824 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:14:22.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:14:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:14:22.163+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:14:22.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:14:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:14:22.163+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:14:22.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:14:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:14:22.163+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:14:22.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:14:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:14:22.118+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:14:22.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:14:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:14:22.118+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:14:22.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:14:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:14:22.118+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:14:22.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:14:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:14:22.118+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:14:22.740 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:14:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:14:22.420+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:14:22.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:14:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:14:22.420+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:14:22.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:14:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:14:22.420+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:14:23.411 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:14:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:14:23.212+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:14:23.412 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:14:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:14:23.212+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:14:23.412 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:14:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:14:23.212+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:14:23.412 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:14:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:14:23.086+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:14:23.412 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:14:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:14:23.086+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:14:23.412 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:14:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:14:23.086+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:14:23.412 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:14:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:14:23.086+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:14:23.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:14:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:14:23.411+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:14:23.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:14:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:14:23.411+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:14:23.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:14:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:14:23.411+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:14:24.380 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:14:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:14:24.181+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:14:24.380 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:14:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:14:24.181+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:14:24.380 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:14:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:14:24.181+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:14:24.380 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:14:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:14:24.114+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:14:24.381 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:14:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:14:24.114+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:14:24.381 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:14:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:14:24.114+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:14:24.381 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:14:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:14:24.114+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:14:24.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:14:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:14:24.379+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:14:24.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:14:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:14:24.379+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:14:24.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:14:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:14:24.379+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:14:25.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:14:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:14:25.211+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:14:25.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:14:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:14:25.211+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:14:25.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:14:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:14:25.211+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:14:25.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:14:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:14:25.334+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:14:25.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:14:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:14:25.334+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:14:25.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:14:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:14:25.334+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:14:25.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:14:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:14:25.116+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:14:25.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:14:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:14:25.116+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:14:25.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:14:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:14:25.116+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:14:25.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:14:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:14:25.116+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:14:26.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:14:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:14:26.252+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:14:26.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:14:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:14:26.252+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:14:26.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:14:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:14:26.252+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:14:26.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:14:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:14:26.354+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:14:26.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:14:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:14:26.354+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:14:26.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:14:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:14:26.354+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:14:26.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:14:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:14:26.154+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:14:26.495 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:14:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:14:26.154+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:14:26.495 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:14:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:14:26.155+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:14:26.495 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:14:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:14:26.155+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:14:27.199 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:14:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:14:26.923+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 12894 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:14:27.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:14:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:14:27.291+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:14:27.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:14:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:14:27.291+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:14:27.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:14:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:14:27.291+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:14:27.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:14:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:14:27.324+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:14:27.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:14:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:14:27.324+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:14:27.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:14:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:14:27.324+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:14:27.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:14:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:14:27.198+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:14:27.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:14:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:14:27.198+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:14:27.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:14:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:14:27.198+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:14:27.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:14:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:14:27.198+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:14:28.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:14:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:14:28.246+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:14:28.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:14:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:14:28.246+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:14:28.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:14:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:14:28.246+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:14:28.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:14:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:14:28.302+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:14:28.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:14:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:14:28.302+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:14:28.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:14:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:14:28.302+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:14:28.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:14:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:14:28.179+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:14:28.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:14:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:14:28.179+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:14:28.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:14:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:14:28.179+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:14:28.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:14:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:14:28.179+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:14:29.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:14:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:14:29.270+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:14:29.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:14:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:14:29.270+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:14:29.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:14:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:14:29.270+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:14:29.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:14:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:14:29.256+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:14:29.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:14:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:14:29.256+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:14:29.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:14:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:14:29.256+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:14:29.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:14:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:14:29.212+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:14:29.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:14:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:14:29.212+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:14:29.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:14:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:14:29.212+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:14:29.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:14:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:14:29.212+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:14:30.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:14:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:14:30.221+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:14:30.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:14:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:14:30.221+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:14:30.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:14:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:14:30.221+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:14:30.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:14:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:14:30.237+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:14:30.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:14:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:14:30.237+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:14:30.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:14:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:14:30.237+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:14:30.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:14:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:14:30.217+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:14:30.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:14:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:14:30.217+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:14:30.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:14:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:14:30.217+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:14:30.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:14:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:14:30.217+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:14:31.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:14:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:14:31.200+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:14:31.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:14:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:14:31.200+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:14:31.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:14:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:14:31.200+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:14:31.491 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:14:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:14:31.227+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:14:31.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:14:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:14:31.227+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:14:31.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:14:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:14:31.227+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:14:31.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:14:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:14:31.228+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:14:31.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:14:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:14:31.242+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:14:31.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:14:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:14:31.242+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:14:31.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:14:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:14:31.242+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:14:31.948 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:14:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:14:31.925+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 12966 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:14:32.222 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:14:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:14:32.160+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:14:32.222 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:14:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:14:32.160+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:14:32.222 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:14:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:14:32.160+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:14:32.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:14:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:14:32.239+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:14:32.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:14:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:14:32.239+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:14:32.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:14:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:14:32.239+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:14:32.491 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:14:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:14:32.220+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:14:32.491 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:14:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:14:32.220+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:14:32.491 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:14:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:14:32.220+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:14:32.491 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:14:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:14:32.221+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:14:33.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:14:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:14:33.211+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:14:33.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:14:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:14:33.211+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:14:33.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:14:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:14:33.211+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:14:33.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:14:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:14:33.192+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:14:33.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:14:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:14:33.192+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:14:33.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:14:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:14:33.192+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:14:33.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:14:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:14:33.233+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:14:33.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:14:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:14:33.233+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:14:33.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:14:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:14:33.233+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:14:33.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:14:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:14:33.233+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:14:34.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:14:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:14:34.234+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:14:34.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:14:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:14:34.234+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:14:34.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:14:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:14:34.234+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:14:34.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:14:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:14:34.173+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:14:34.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:14:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:14:34.173+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:14:34.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:14:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:14:34.173+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:14:34.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:14:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:14:34.227+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:14:34.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:14:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:14:34.227+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:14:34.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:14:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:14:34.227+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:14:34.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:14:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:14:34.227+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:14:35.490 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:14:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:14:35.274+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:14:35.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:14:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:14:35.274+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:14:35.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:14:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:14:35.274+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:14:35.491 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:14:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:14:35.227+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:14:35.491 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:14:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:14:35.227+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:14:35.491 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:14:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:14:35.227+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:14:35.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:14:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:14:35.227+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:14:35.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:14:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:14:35.192+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:14:35.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:14:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:14:35.192+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:14:35.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:14:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:14:35.192+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:14:36.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:14:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:14:36.300+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:14:36.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:14:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:14:36.300+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:14:36.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:14:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:14:36.300+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:14:36.491 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:14:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:14:36.206+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:14:36.491 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:14:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:14:36.206+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:14:36.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:14:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:14:36.206+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:14:36.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:14:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:14:36.206+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:14:36.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:14:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:14:36.166+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:14:36.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:14:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:14:36.166+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:14:36.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:14:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:14:36.166+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:14:37.052 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:14:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:14:36.927+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 13035 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:14:37.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:14:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:14:37.138+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:14:37.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:14:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:14:37.138+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:14:37.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:14:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:14:37.138+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:14:37.491 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:14:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:14:37.245+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:14:37.491 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:14:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:14:37.245+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:14:37.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:14:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:14:37.245+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:14:37.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:14:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:14:37.245+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:14:37.492 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:14:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:14:37.298+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:14:37.492 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:14:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:14:37.298+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:14:37.492 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:14:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:14:37.298+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:14:38.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:14:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:14:38.326+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:14:38.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:14:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:14:38.326+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:14:38.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:14:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:14:38.326+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:14:38.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:14:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:14:38.240+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:14:38.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:14:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:14:38.240+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:14:38.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:14:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:14:38.240+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:14:38.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:14:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:14:38.240+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:14:38.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:14:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:14:38.143+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:14:38.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:14:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:14:38.143+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:14:38.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:14:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:14:38.143+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:14:39.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:14:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:14:39.184+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:14:39.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:14:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:14:39.184+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:14:39.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:14:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:14:39.184+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:14:39.492 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:14:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:14:39.283+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:14:39.492 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:14:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:14:39.283+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:14:39.492 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:14:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:14:39.283+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:14:39.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:14:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:14:39.190+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:14:39.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:14:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:14:39.190+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:14:39.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:14:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:14:39.190+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:14:39.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:14:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:14:39.191+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:14:40.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:14:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:14:40.309+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:14:40.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:14:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:14:40.309+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:14:40.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:14:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:14:40.309+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:14:40.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:14:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:14:40.222+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:14:40.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:14:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:14:40.222+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:14:40.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:14:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:14:40.222+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:14:40.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:14:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:14:40.157+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:14:40.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:14:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:14:40.157+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:14:40.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:14:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:14:40.157+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:14:40.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:14:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:14:40.157+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:14:41.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:14:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:14:41.341+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:14:41.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:14:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:14:41.341+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:14:41.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:14:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:14:41.341+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:14:41.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:14:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:14:41.212+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:14:41.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:14:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:14:41.212+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:14:41.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:14:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:14:41.212+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:14:41.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:14:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:14:41.154+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:14:41.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:14:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:14:41.154+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:14:41.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:14:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:14:41.154+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:14:41.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:14:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:14:41.154+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:14:41.951 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:14:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:14:41.930+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 13106 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:14:42.241 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:14:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:14:42.177+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:14:42.241 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:14:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:14:42.177+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:14:42.241 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:14:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:14:42.177+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:14:42.241 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:14:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:14:42.177+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:14:42.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:14:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:14:42.254+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:14:42.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:14:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:14:42.254+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:14:42.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:14:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:14:42.254+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:14:42.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:14:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:14:42.332+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:14:42.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:14:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:14:42.332+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:14:42.742 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:14:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:14:42.332+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:14:43.241 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:14:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:14:43.168+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:14:43.241 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:14:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:14:43.168+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:14:43.241 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:14:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:14:43.168+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:14:43.241 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:14:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:14:43.168+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:14:43.587 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:14:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:14:43.295+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:14:43.587 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:14:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:14:43.295+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:14:43.587 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:14:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:14:43.295+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:14:43.587 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:14:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:14:43.335+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:14:43.587 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:14:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:14:43.335+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:14:43.587 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:14:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:14:43.335+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:14:44.379 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:14:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:14:44.277+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:14:44.379 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:14:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:14:44.277+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:14:44.379 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:14:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:14:44.277+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:14:44.379 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:14:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:14:44.120+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:14:44.380 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:14:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:14:44.120+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:14:44.380 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:14:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:14:44.120+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:14:44.380 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:14:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:14:44.120+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:14:44.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:14:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:14:44.378+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:14:44.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:14:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:14:44.378+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:14:44.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:14:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:14:44.378+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:14:45.490 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:14:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:14:45.378+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:14:45.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:14:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:14:45.378+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:14:45.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:14:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:14:45.378+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:14:45.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:14:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:14:45.247+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:14:45.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:14:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:14:45.247+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:14:45.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:14:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:14:45.247+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:14:45.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:14:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:14:45.158+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:14:45.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:14:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:14:45.158+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:14:45.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:14:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:14:45.158+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:14:45.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:14:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:14:45.158+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:14:46.411 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:14:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:14:46.204+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:14:46.411 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:14:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:14:46.204+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:14:46.412 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:14:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:14:46.204+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:14:46.412 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:14:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:14:46.126+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:14:46.412 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:14:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:14:46.126+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:14:46.412 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:14:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:14:46.126+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:14:46.413 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:14:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:14:46.126+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:14:46.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:14:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:14:46.411+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:14:46.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:14:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:14:46.411+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:14:46.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:14:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:14:46.411+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:14:47.241 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:14:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:14:46.932+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 13175 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:14:47.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:14:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:14:47.170+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:14:47.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:14:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:14:47.170+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:14:47.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:14:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:14:47.170+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:14:47.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:14:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:14:47.110+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:14:47.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:14:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:14:47.110+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:14:47.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:14:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:14:47.110+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:14:47.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:14:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:14:47.110+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:14:47.740 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:14:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:14:47.414+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:14:47.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:14:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:14:47.414+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:14:47.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:14:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:14:47.414+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:14:48.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:14:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:14:48.397+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:14:48.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:14:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:14:48.397+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:14:48.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:14:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:14:48.397+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:14:48.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:14:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:14:48.153+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:14:48.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:14:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:14:48.153+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:14:48.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:14:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:14:48.153+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:14:48.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:14:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:14:48.159+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:14:48.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:14:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:14:48.159+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:14:48.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:14:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:14:48.159+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:14:48.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:14:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:14:48.159+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:14:49.385 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:14:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:14:49.112+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:14:49.385 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:14:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:14:49.112+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:14:49.385 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:14:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:14:49.112+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:14:49.385 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:14:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:14:49.157+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:14:49.385 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:14:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:14:49.157+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:14:49.385 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:14:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:14:49.157+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:14:49.386 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:14:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:14:49.157+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:14:49.740 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:14:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:14:49.384+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:14:49.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:14:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:14:49.384+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:14:49.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:14:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:14:49.384+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:14:50.148 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:14:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:14:50.071+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:14:50.149 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:14:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:14:50.071+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:14:50.149 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:14:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:14:50.071+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:14:50.491 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:14:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:14:50.148+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:14:50.491 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:14:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:14:50.148+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:14:50.491 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:14:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:14:50.148+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:14:50.491 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:14:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:14:50.149+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:14:50.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:14:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:14:50.371+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:14:50.492 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:14:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:14:50.371+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:14:50.492 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:14:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:14:50.371+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:14:51.364 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:14:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:14:51.110+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:14:51.364 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:14:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:14:51.110+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:14:51.364 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:14:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:14:51.110+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:14:51.364 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:14:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:14:51.111+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:14:51.364 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:14:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:14:51.084+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:14:51.365 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:14:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:14:51.084+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:14:51.365 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:14:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:14:51.084+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:14:51.693 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:14:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:14:51.364+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:14:51.693 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:14:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:14:51.364+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:14:51.694 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:14:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:14:51.364+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:14:51.945 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:14:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:14:51.935+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 13246 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:14:52.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:14:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:14:52.042+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:14:52.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:14:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:14:52.042+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:14:52.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:14:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:14:52.042+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:14:52.241 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:14:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:14:52.127+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:14:52.241 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:14:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:14:52.127+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:14:52.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:14:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:14:52.127+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:14:52.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:14:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:14:52.127+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:14:52.664 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:14:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:14:52.349+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:14:52.664 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:14:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:14:52.349+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:14:52.664 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:14:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:14:52.349+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:14:53.329 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:14:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:14:53.002+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:14:53.330 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:14:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:14:53.002+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:14:53.330 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:14:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:14:53.002+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:14:53.330 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:14:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:14:53.160+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:14:53.330 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:14:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:14:53.160+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:14:53.330 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:14:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:14:53.160+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:14:53.330 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:14:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:14:53.160+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:14:53.587 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:14:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:14:53.329+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:14:53.587 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:14:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:14:53.329+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:14:53.587 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:14:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:14:53.329+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:14:54.294 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:14:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:14:53.993+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:14:54.294 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:14:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:14:53.993+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:14:54.294 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:14:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:14:53.993+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:14:54.294 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:14:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:14:54.139+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:14:54.295 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:14:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:14:54.140+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:14:54.295 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:14:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:14:54.140+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:14:54.295 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:14:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:14:54.140+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:14:54.740 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:14:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:14:54.294+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:14:54.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:14:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:14:54.294+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:14:54.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:14:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:14:54.294+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:14:55.308 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:14:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:14:55.029+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:14:55.308 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:14:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:14:55.029+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:14:55.308 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:14:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:14:55.029+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:14:55.309 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:14:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:14:55.156+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:14:55.309 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:14:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:14:55.156+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:14:55.309 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:14:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:14:55.156+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:14:55.309 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:14:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:14:55.156+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:14:55.740 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:14:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:14:55.308+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:14:55.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:14:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:14:55.308+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:14:55.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:14:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:14:55.308+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:14:56.262 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:14:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:14:56.000+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:14:56.262 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:14:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:14:56.000+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:14:56.262 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:14:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:14:56.000+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:14:56.263 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:14:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:14:56.187+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:14:56.263 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:14:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:14:56.187+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:14:56.263 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:14:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:14:56.187+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:14:56.263 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:14:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:14:56.187+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:14:56.740 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:14:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:14:56.262+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:14:56.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:14:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:14:56.262+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:14:56.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:14:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:14:56.262+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:14:57.241 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:14:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:14:56.937+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 13315 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:14:57.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:14:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:14:57.023+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:14:57.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:14:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:14:57.023+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:14:57.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:14:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:14:57.023+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:14:57.241 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:14:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:14:57.178+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:14:57.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:14:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:14:57.178+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:14:57.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:14:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:14:57.178+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:14:57.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:14:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:14:57.178+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:14:57.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:14:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:14:57.287+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:14:57.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:14:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:14:57.287+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:14:57.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:14:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:14:57.287+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:14:58.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:14:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:14:58.265+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:14:58.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:14:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:14:58.265+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:14:58.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:14:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:14:58.265+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:14:58.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:14:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:14:58.047+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:14:58.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:14:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:14:58.047+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:14:58.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:14:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:14:58.047+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:14:58.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:14:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:14:58.134+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:14:58.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:14:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:14:58.134+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:14:58.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:14:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:14:58.134+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:14:58.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:14:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:14:58.134+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:14:59.305 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:14:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:14:59.019+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:14:59.305 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:14:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:14:59.019+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:14:59.305 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:14:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:14:59.019+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:14:59.306 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:14:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:14:59.103+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:14:59.306 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:14:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:14:59.103+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:14:59.306 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:14:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:14:59.103+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:14:59.306 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:14:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:14:59.104+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:14:59.740 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:14:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:14:59.305+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:14:59.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:14:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:14:59.305+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:14:59.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:14:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:14:59.305+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:15:00.167 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:15:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:15:00.015+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:15:00.167 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:15:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:15:00.015+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:15:00.167 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:15:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:15:00.015+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:15:00.167 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:15:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:15:00.128+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:15:00.168 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:15:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:15:00.128+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:15:00.168 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:15:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:15:00.128+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:15:00.168 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:15:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:15:00.128+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:15:00.490 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:15:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:15:00.260+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:15:00.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:15:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:15:00.260+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:15:00.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:15:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:15:00.260+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:15:01.263 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:15:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:15:01.170+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:15:01.263 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:15:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:15:01.170+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:15:01.263 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:15:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:15:01.170+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:15:01.263 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:15:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:15:01.170+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:15:01.264 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:15:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:15:01.010+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:15:01.264 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:15:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:15:01.010+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:15:01.264 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:15:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:15:01.010+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:15:01.694 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:15:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:15:01.263+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:15:01.694 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:15:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:15:01.263+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:15:01.694 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:15:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:15:01.263+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:15:01.948 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:15:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:15:01.939+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 13386 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:15:02.203 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:15:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:15:02.011+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:15:02.203 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:15:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:15:02.011+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:15:02.203 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:15:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:15:02.011+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:15:02.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:15:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:15:02.222+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:15:02.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:15:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:15:02.222+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:15:02.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:15:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:15:02.222+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:15:02.491 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:15:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:15:02.203+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:15:02.491 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:15:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:15:02.203+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:15:02.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:15:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:15:02.203+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:15:02.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:15:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:15:02.203+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:15:03.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:15:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:15:02.993+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:15:03.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:15:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:15:02.993+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:15:03.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:15:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:15:02.993+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:15:03.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:15:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:15:03.229+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:15:03.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:15:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:15:03.229+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:15:03.492 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:15:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:15:03.229+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:15:03.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:15:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:15:03.205+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:15:03.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:15:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:15:03.205+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:15:03.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:15:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:15:03.205+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:15:03.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:15:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:15:03.205+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:15:04.439 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:15:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:15:04.225+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:15:04.439 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:15:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:15:04.225+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:15:04.440 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:15:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:15:04.225+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:15:04.440 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:15:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:15:04.033+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:15:04.440 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:15:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:15:04.033+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:15:04.440 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:15:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:15:04.033+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:15:04.440 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:15:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:15:04.229+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:15:04.441 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:15:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:15:04.229+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:15:04.441 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:15:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:15:04.229+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:15:04.441 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:15:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:15:04.229+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:15:05.271 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:15:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:15:05.007+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:15:05.272 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:15:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:15:05.007+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:15:05.272 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:15:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:15:05.007+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:15:05.272 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:15:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:15:05.226+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:15:05.272 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:15:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:15:05.226+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:15:05.272 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:15:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:15:05.226+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:15:05.272 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:15:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:15:05.226+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:15:05.740 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:15:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:15:05.271+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:15:05.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:15:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:15:05.271+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:15:05.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:15:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:15:05.271+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:15:06.258 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:15:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:15:06.005+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:15:06.258 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:15:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:15:06.005+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:15:06.259 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:15:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:15:06.005+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:15:06.259 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:15:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:15:06.214+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:15:06.259 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:15:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:15:06.214+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:15:06.259 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:15:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:15:06.214+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:15:06.259 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:15:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:15:06.214+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:15:06.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:15:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:15:06.257+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:15:06.742 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:15:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:15:06.257+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:15:06.742 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:15:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:15:06.258+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:15:07.201 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:15:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:15:06.941+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 13455 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:15:07.202 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:15:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:15:06.957+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:15:07.202 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:15:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:15:06.957+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:15:07.202 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:15:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:15:06.957+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:15:07.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:15:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:15:07.244+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:15:07.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:15:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:15:07.244+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:15:07.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:15:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:15:07.244+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:15:07.491 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:15:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:15:07.202+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:15:07.491 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:15:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:15:07.202+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:15:07.491 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:15:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:15:07.202+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:15:07.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:15:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:15:07.202+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:15:08.237 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:15:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:15:07.973+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:15:08.237 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:15:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:15:07.973+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:15:08.237 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:15:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:15:07.973+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:15:08.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:15:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:15:08.240+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:15:08.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:15:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:15:08.241+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:15:08.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:15:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:15:08.241+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:15:08.491 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:15:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:15:08.236+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:15:08.491 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:15:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:15:08.236+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:15:08.491 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:15:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:15:08.236+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:15:08.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:15:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:15:08.236+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:15:09.284 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:15:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:15:09.012+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:15:09.284 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:15:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:15:09.012+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:15:09.284 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:15:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:15:09.012+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:15:09.284 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:15:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:15:09.236+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:15:09.285 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:15:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:15:09.236+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:15:09.285 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:15:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:15:09.236+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:15:09.285 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:15:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:15:09.236+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:15:09.740 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:15:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:15:09.283+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:15:09.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:15:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:15:09.283+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:15:09.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:15:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:15:09.283+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:15:10.167 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:15:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:15:10.034+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:15:10.168 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:15:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:15:10.034+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:15:10.168 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:15:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:15:10.034+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:15:10.490 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:15:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:15:10.257+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:15:10.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:15:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:15:10.257+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:15:10.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:15:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:15:10.257+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:15:10.491 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:15:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:15:10.196+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:15:10.491 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:15:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:15:10.196+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:15:10.491 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:15:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:15:10.196+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:15:10.491 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:15:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:15:10.196+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:15:11.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:15:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:15:11.208+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:15:11.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:15:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:15:11.208+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:15:11.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:15:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:15:11.208+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:15:11.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:15:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:15:10.999+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:15:11.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:15:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:15:10.999+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:15:11.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:15:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:15:10.999+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:15:11.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:15:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:15:11.210+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:15:11.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:15:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:15:11.210+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:15:11.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:15:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:15:11.210+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:15:11.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:15:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:15:11.211+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:15:11.955 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:15:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:15:11.942+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 13526 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:15:12.237 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:15:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:15:12.032+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:15:12.237 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:15:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:15:12.032+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:15:12.238 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:15:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:15:12.032+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:15:12.238 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:15:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:15:12.178+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:15:12.238 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:15:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:15:12.178+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:15:12.238 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:15:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:15:12.178+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:15:12.238 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:15:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:15:12.178+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:15:12.490 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:15:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:15:12.237+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:15:12.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:15:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:15:12.237+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:15:12.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:15:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:15:12.237+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:15:13.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:15:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:15:13.279+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:15:13.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:15:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:15:13.279+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:15:13.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:15:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:15:13.279+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:15:13.491 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:15:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:15:13.198+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:15:13.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:15:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:15:13.198+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:15:13.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:15:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:15:13.198+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:15:13.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:15:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:15:13.198+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:15:13.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:15:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:15:13.073+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:15:13.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:15:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:15:13.074+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:15:13.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:15:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:15:13.074+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:15:14.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:15:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:15:14.275+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:15:14.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:15:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:15:14.275+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:15:14.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:15:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:15:14.275+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:15:14.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:15:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:15:14.113+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:15:14.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:15:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:15:14.113+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:15:14.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:15:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:15:14.113+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:15:14.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:15:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:15:14.224+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:15:14.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:15:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:15:14.224+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:15:14.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:15:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:15:14.224+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:15:14.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:15:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:15:14.224+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:15:15.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:15:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:15:15.239+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:15:15.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:15:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:15:15.239+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:15:15.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:15:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:15:15.239+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:15:15.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:15:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:15:15.249+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:15:15.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:15:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:15:15.249+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:15:15.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:15:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:15:15.249+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:15:15.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:15:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:15:15.249+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:15:15.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:15:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:15:15.114+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:15:15.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:15:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:15:15.114+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:15:15.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:15:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:15:15.114+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:15:16.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:15:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:15:16.279+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:15:16.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:15:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:15:16.279+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:15:16.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:15:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:15:16.279+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:15:16.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:15:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:15:16.128+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:15:16.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:15:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:15:16.128+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:15:16.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:15:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:15:16.128+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:15:16.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:15:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:15:16.274+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:15:16.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:15:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:15:16.274+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:15:16.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:15:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:15:16.274+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:15:16.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:15:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:15:16.274+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:15:17.241 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:15:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:15:16.944+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 13595 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:15:17.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:15:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:15:17.152+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:15:17.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:15:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:15:17.152+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:15:17.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:15:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:15:17.152+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:15:17.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:15:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:15:17.305+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:15:17.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:15:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:15:17.305+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:15:17.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:15:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:15:17.305+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:15:17.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:15:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:15:17.322+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:15:17.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:15:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:15:17.322+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:15:17.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:15:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:15:17.322+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:15:17.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:15:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:15:17.322+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:15:18.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:15:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:15:18.276+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:15:18.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:15:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:15:18.276+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:15:18.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:15:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:15:18.276+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:15:18.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:15:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:15:18.150+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:15:18.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:15:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:15:18.151+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:15:18.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:15:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:15:18.151+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:15:18.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:15:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:15:18.318+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:15:18.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:15:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:15:18.318+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:15:18.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:15:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:15:18.318+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:15:18.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:15:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:15:18.318+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:15:19.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:15:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:15:19.259+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:15:19.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:15:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:15:19.259+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:15:19.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:15:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:15:19.259+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:15:19.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:15:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:15:19.117+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:15:19.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:15:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:15:19.117+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:15:19.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:15:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:15:19.117+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:15:19.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:15:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:15:19.352+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:15:19.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:15:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:15:19.352+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:15:19.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:15:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:15:19.352+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:15:19.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:15:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:15:19.352+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:15:20.169 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:15:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:15:20.111+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:15:20.169 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:15:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:15:20.111+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:15:20.169 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:15:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:15:20.111+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:15:20.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:15:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:15:20.304+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:15:20.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:15:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:15:20.304+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:15:20.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:15:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:15:20.304+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:15:20.491 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:15:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:15:20.365+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:15:20.491 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:15:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:15:20.365+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:15:20.491 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:15:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:15:20.365+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:15:20.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:15:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:15:20.365+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:15:21.337 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:15:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:15:21.075+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:15:21.338 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:15:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:15:21.075+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:15:21.338 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:15:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:15:21.075+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:15:21.696 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:15:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:15:21.337+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:15:21.696 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:15:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:15:21.337+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:15:21.696 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:15:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:15:21.337+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:15:21.697 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:15:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:15:21.347+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:15:21.697 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:15:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:15:21.347+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:15:21.697 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:15:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:15:21.347+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:15:21.697 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:15:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:15:21.348+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:15:22.240 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:15:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:15:21.946+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 13666 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:15:22.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:15:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:15:22.034+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:15:22.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:15:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:15:22.034+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:15:22.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:15:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:15:22.034+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:15:22.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:15:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:15:22.294+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:15:22.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:15:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:15:22.294+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:15:22.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:15:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:15:22.294+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:15:22.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:15:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:15:22.315+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:15:22.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:15:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:15:22.315+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:15:22.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:15:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:15:22.315+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:15:22.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:15:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:15:22.315+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:15:23.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:15:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:15:23.059+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:15:23.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:15:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:15:23.059+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:15:23.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:15:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:15:23.059+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:15:23.589 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:15:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:15:23.275+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:15:23.589 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:15:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:15:23.275+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:15:23.589 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:15:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:15:23.275+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:15:23.589 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:15:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:15:23.316+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:15:23.590 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:15:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:15:23.316+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:15:23.590 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:15:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:15:23.316+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:15:23.590 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:15:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:15:23.316+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:15:24.491 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:15:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:15:24.286+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:15:24.491 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:15:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:15:24.286+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:15:24.491 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:15:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:15:24.286+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:15:24.491 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:15:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:15:24.286+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:15:24.492 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:15:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:15:24.245+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:15:24.492 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:15:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:15:24.245+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:15:24.492 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:15:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:15:24.245+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:15:24.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:15:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:15:24.081+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:15:24.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:15:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:15:24.081+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:15:24.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:15:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:15:24.081+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:15:25.491 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:15:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:15:25.300+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:15:25.491 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:15:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:15:25.300+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:15:25.491 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:15:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:15:25.300+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:15:25.491 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:15:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:15:25.300+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:15:25.492 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:15:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:15:25.279+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:15:25.492 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:15:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:15:25.279+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:15:25.492 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:15:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:15:25.279+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:15:25.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:15:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:15:25.110+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:15:25.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:15:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:15:25.110+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:15:25.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:15:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:15:25.110+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:15:26.240 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:15:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:15:26.157+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:15:26.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:15:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:15:26.157+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:15:26.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:15:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:15:26.157+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:15:26.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:15:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:15:26.263+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:15:26.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:15:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:15:26.263+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:15:26.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:15:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:15:26.263+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:15:26.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:15:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:15:26.342+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:15:26.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:15:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:15:26.342+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:15:26.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:15:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:15:26.342+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:15:26.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:15:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:15:26.342+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:15:27.241 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:15:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:15:26.948+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 13735 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:15:27.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:15:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:15:27.173+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:15:27.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:15:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:15:27.173+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:15:27.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:15:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:15:27.173+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:15:27.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:15:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:15:27.292+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:15:27.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:15:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:15:27.292+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:15:27.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:15:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:15:27.292+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:15:27.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:15:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:15:27.369+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:15:27.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:15:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:15:27.369+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:15:27.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:15:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:15:27.369+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:15:27.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:15:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:15:27.369+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:15:28.491 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:15:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:15:28.380+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:15:28.491 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:15:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:15:28.380+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:15:28.491 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:15:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:15:28.380+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:15:28.491 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:15:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:15:28.380+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:15:28.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:15:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:15:28.306+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:15:28.492 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:15:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:15:28.306+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:15:28.492 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:15:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:15:28.306+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:15:28.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:15:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:15:28.169+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:15:28.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:15:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:15:28.169+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:15:28.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:15:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:15:28.169+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:15:29.414 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:15:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:15:29.258+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:15:29.414 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:15:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:15:29.258+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:15:29.415 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:15:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:15:29.258+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:15:29.415 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:15:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:15:29.197+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:15:29.415 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:15:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:15:29.197+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:15:29.415 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:15:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:15:29.197+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:15:29.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:15:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:15:29.414+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:15:29.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:15:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:15:29.414+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:15:29.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:15:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:15:29.414+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:15:29.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:15:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:15:29.414+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:15:30.447 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:15:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:15:30.276+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:15:30.447 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:15:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:15:30.276+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:15:30.447 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:15:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:15:30.276+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:15:30.448 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:15:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:15:30.214+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:15:30.448 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:15:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:15:30.214+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:15:30.448 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:15:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:15:30.214+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:15:30.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:15:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:15:30.438+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:15:30.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:15:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:15:30.438+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:15:30.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:15:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:15:30.438+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:15:30.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:15:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:15:30.438+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:15:31.480 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:15:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:15:31.236+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:15:31.480 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:15:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:15:31.236+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:15:31.480 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:15:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:15:31.236+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:15:31.481 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:15:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:15:31.182+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:15:31.481 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:15:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:15:31.182+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:15:31.481 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:15:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:15:31.182+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:15:31.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:15:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:15:31.480+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:15:31.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:15:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:15:31.480+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:15:31.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:15:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:15:31.480+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:15:31.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:15:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:15:31.480+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:15:32.230 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:15:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:15:31.951+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 13806 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:15:32.231 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:15:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:15:32.187+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:15:32.231 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:15:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:15:32.187+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:15:32.231 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:15:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:15:32.187+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:15:32.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:15:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:15:32.230+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:15:32.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:15:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:15:32.230+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:15:32.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:15:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:15:32.230+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:15:32.491 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:15:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:15:32.455+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:15:32.491 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:15:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:15:32.455+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:15:32.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:15:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:15:32.455+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:15:32.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:15:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:15:32.455+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:15:33.428 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:15:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:15:33.219+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:15:33.428 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:15:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:15:33.219+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:15:33.429 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:15:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:15:33.219+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:15:33.429 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:15:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:15:33.168+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:15:33.429 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:15:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:15:33.168+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:15:33.429 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:15:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:15:33.168+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:15:33.740 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:15:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:15:33.428+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:15:33.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:15:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:15:33.428+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:15:33.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:15:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:15:33.428+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:15:33.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:15:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:15:33.428+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:15:34.443 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:15:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:15:34.121+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:15:34.443 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:15:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:15:34.121+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:15:34.443 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:15:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:15:34.121+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:15:34.443 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:15:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:15:34.211+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:15:34.443 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:15:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:15:34.211+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:15:34.443 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:15:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:15:34.211+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:15:34.740 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:15:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:15:34.442+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:15:34.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:15:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:15:34.442+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:15:34.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:15:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:15:34.442+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:15:34.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:15:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:15:34.442+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:15:35.473 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:15:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:15:35.182+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:15:35.474 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:15:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:15:35.182+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:15:35.474 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:15:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:15:35.182+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:15:35.474 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:15:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:15:35.101+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:15:35.474 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:15:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:15:35.101+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:15:35.474 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:15:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:15:35.101+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:15:35.740 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:15:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:15:35.472+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:15:35.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:15:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:15:35.472+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:15:35.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:15:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:15:35.472+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:15:35.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:15:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:15:35.472+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:15:36.425 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:15:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:15:36.150+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:15:36.426 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:15:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:15:36.150+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:15:36.426 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:15:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:15:36.150+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:15:36.426 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:15:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:15:36.106+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:15:36.426 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:15:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:15:36.106+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:15:36.426 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:15:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:15:36.106+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:15:36.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:15:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:15:36.425+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:15:36.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:15:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:15:36.425+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:15:36.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:15:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:15:36.425+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:15:36.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:15:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:15:36.425+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:15:37.241 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:15:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:15:36.953+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 13875 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:15:37.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:15:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:15:37.148+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:15:37.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:15:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:15:37.148+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:15:37.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:15:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:15:37.148+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:15:37.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:15:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:15:37.123+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:15:37.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:15:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:15:37.123+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:15:37.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:15:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:15:37.123+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:15:37.740 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:15:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:15:37.470+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:15:37.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:15:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:15:37.470+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:15:37.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:15:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:15:37.470+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:15:37.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:15:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:15:37.470+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:15:38.455 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:15:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:15:38.118+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:15:38.456 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:15:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:15:38.118+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:15:38.456 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:15:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:15:38.118+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:15:38.456 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:15:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:15:38.079+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:15:38.456 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:15:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:15:38.079+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:15:38.457 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:15:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:15:38.079+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:15:38.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:15:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:15:38.455+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:15:38.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:15:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:15:38.455+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:15:38.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:15:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:15:38.455+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:15:38.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:15:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:15:38.455+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:15:39.408 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:15:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:15:39.069+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:15:39.409 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:15:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:15:39.069+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:15:39.409 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:15:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:15:39.069+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:15:39.409 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:15:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:15:39.085+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:15:39.409 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:15:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:15:39.085+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:15:39.409 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:15:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:15:39.085+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:15:39.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:15:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:15:39.408+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:15:39.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:15:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:15:39.408+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:15:39.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:15:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:15:39.408+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:15:39.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:15:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:15:39.409+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:15:40.170 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:15:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:15:40.069+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:15:40.170 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:15:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:15:40.069+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:15:40.170 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:15:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:15:40.069+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:15:40.171 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:15:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:15:40.067+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:15:40.171 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:15:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:15:40.067+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:15:40.171 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:15:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:15:40.067+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:15:40.491 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:15:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:15:40.417+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:15:40.491 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:15:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:15:40.417+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:15:40.491 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:15:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:15:40.417+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:15:40.491 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:15:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:15:40.417+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:15:41.303 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:15:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:15:41.028+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:15:41.303 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:15:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:15:41.028+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:15:41.303 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:15:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:15:41.028+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:15:41.304 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:15:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:15:41.035+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:15:41.304 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:15:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:15:41.035+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:15:41.304 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:15:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:15:41.035+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:15:41.697 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:15:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:15:41.402+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:15:41.698 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:15:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:15:41.402+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:15:41.698 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:15:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:15:41.402+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:15:41.698 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:15:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:15:41.403+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:15:42.241 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:15:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:15:41.956+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 13946 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:15:42.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:15:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:15:42.006+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:15:42.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:15:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:15:42.006+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:15:42.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:15:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:15:42.006+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:15:42.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:15:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:15:42.036+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:15:42.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:15:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:15:42.036+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:15:42.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:15:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:15:42.036+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:15:42.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:15:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:15:42.413+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:15:42.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:15:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:15:42.413+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:15:42.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:15:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:15:42.413+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:15:42.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:15:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:15:42.413+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:15:43.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:15:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:15:42.987+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:15:43.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:15:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:15:42.987+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:15:43.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:15:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:15:42.987+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:15:43.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:15:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:15:43.058+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:15:43.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:15:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:15:43.058+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:15:43.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:15:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:15:43.058+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:15:43.740 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:15:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:15:43.458+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:15:43.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:15:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:15:43.458+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:15:43.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:15:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:15:43.458+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:15:43.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:15:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:15:43.458+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:15:44.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:15:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:15:44.092+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:15:44.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:15:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:15:44.092+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:15:44.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:15:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:15:44.092+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:15:44.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:15:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:15:43.984+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:15:44.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:15:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:15:43.984+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:15:44.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:15:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:15:43.985+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:15:44.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:15:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:15:44.449+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:15:44.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:15:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:15:44.449+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:15:44.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:15:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:15:44.449+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:15:44.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:15:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:15:44.449+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:15:45.426 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:15:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:15:45.012+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:15:45.427 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:15:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:15:45.012+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:15:45.427 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:15:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:15:45.012+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:15:45.427 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:15:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:15:45.075+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:15:45.427 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:15:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:15:45.075+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:15:45.427 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:15:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:15:45.075+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:15:45.740 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:15:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:15:45.426+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:15:45.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:15:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:15:45.426+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:15:45.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:15:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:15:45.426+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:15:45.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:15:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:15:45.426+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:15:46.437 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:15:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:15:46.007+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:15:46.437 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:15:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:15:46.007+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:15:46.437 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:15:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:15:46.007+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:15:46.437 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:15:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:15:46.073+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:15:46.437 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:15:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:15:46.073+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:15:46.437 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:15:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:15:46.073+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:15:46.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:15:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:15:46.436+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:15:46.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:15:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:15:46.436+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:15:46.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:15:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:15:46.436+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:15:46.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:15:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:15:46.437+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:15:47.240 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:15:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:15:46.958+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 14016 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:15:47.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:15:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:15:47.053+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:15:47.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:15:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:15:47.053+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:15:47.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:15:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:15:47.053+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:15:47.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:15:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:15:47.034+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:15:47.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:15:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:15:47.034+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:15:47.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:15:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:15:47.034+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:15:47.740 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:15:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:15:47.483+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:15:47.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:15:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:15:47.483+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:15:47.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:15:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:15:47.483+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:15:47.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:15:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:15:47.483+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:15:48.468 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:15:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:15:48.004+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:15:48.468 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:15:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:15:48.004+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:15:48.469 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:15:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:15:48.004+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:15:48.469 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:15:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:15:48.082+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:15:48.469 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:15:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:15:48.082+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:15:48.469 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:15:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:15:48.082+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:15:48.740 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:15:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:15:48.465+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:15:48.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:15:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:15:48.465+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:15:48.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:15:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:15:48.465+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:15:48.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:15:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:15:48.465+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:15:49.455 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:15:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:15:49.034+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:15:49.455 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:15:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:15:49.034+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:15:49.455 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:15:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:15:49.034+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:15:49.456 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:15:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:15:49.129+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:15:49.456 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:15:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:15:49.129+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:15:49.456 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:15:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:15:49.129+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:15:49.740 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:15:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:15:49.455+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:15:49.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:15:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:15:49.455+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:15:49.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:15:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:15:49.455+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:15:49.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:15:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:15:49.455+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:15:50.135 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:15:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:15:50.076+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:15:50.135 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:15:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:15:50.076+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:15:50.135 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:15:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:15:50.076+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:15:50.136 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:15:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:15:50.134+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:15:50.136 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:15:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:15:50.134+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:15:50.136 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:15:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:15:50.134+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:15:50.740 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:15:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:15:50.463+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:15:50.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:15:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:15:50.463+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:15:50.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:15:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:15:50.463+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:15:50.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:15:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:15:50.463+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:15:51.416 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:15:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:15:51.080+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:15:51.416 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:15:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:15:51.080+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:15:51.416 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:15:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:15:51.080+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:15:51.417 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:15:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:15:51.175+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:15:51.417 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:15:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:15:51.175+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:15:51.417 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:15:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:15:51.175+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:15:51.698 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:15:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:15:51.416+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:15:51.698 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:15:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:15:51.416+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:15:51.699 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:15:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:15:51.416+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:15:51.699 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:15:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:15:51.416+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:15:52.241 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:15:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:15:51.960+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 14088 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:15:52.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:15:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:15:52.184+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:15:52.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:15:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:15:52.184+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:15:52.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:15:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:15:52.184+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:15:52.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:15:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:15:52.059+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:15:52.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:15:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:15:52.059+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:15:52.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:15:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:15:52.059+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:15:52.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:15:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:15:52.436+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:15:52.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:15:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:15:52.436+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:15:52.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:15:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:15:52.436+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:15:52.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:15:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:15:52.436+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:15:53.458 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:15:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:15:53.048+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:15:53.459 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:15:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:15:53.048+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:15:53.459 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:15:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:15:53.048+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:15:53.459 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:15:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:15:53.188+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:15:53.459 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:15:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:15:53.188+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:15:53.459 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:15:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:15:53.188+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:15:53.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:15:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:15:53.458+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:15:53.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:15:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:15:53.458+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:15:53.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:15:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:15:53.458+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:15:53.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:15:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:15:53.459+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:15:54.477 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:15:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:15:54.051+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:15:54.478 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:15:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:15:54.051+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:15:54.478 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:15:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:15:54.051+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:15:54.478 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:15:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:15:54.179+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:15:54.478 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:15:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:15:54.179+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:15:54.478 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:15:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:15:54.179+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:15:54.740 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:15:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:15:54.477+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:15:54.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:15:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:15:54.477+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:15:54.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:15:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:15:54.477+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:15:54.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:15:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:15:54.477+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:15:55.488 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:15:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:15:55.029+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:15:55.488 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:15:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:15:55.029+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:15:55.488 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:15:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:15:55.029+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:15:55.489 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:15:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:15:55.213+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:15:55.489 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:15:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:15:55.213+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:15:55.489 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:15:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:15:55.213+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:15:55.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:15:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:15:55.488+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:15:55.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:15:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:15:55.488+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:15:55.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:15:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:15:55.488+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:15:55.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:15:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:15:55.488+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:15:56.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:15:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:15:55.989+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:15:56.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:15:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:15:55.989+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:15:56.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:15:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:15:55.989+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:15:56.663 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:15:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:15:56.449+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:15:56.664 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:15:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:15:56.449+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:15:56.664 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:15:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:15:56.449+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:15:56.664 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:15:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:15:56.449+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:15:56.664 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:15:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:15:56.261+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:15:56.664 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:15:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:15:56.261+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:15:56.664 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:15:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:15:56.261+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:15:57.241 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:15:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:15:56.963+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 14158 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:15:57.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:15:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:15:56.954+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:15:57.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:15:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:15:56.954+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:15:57.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:15:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:15:56.954+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:15:57.740 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:15:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:15:57.256+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:15:57.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:15:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:15:57.256+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:15:57.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:15:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:15:57.256+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:15:57.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:15:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:15:57.484+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:15:57.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:15:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:15:57.484+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:15:57.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:15:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:15:57.484+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:15:57.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:15:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:15:57.485+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:15:58.240 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:15:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:15:57.953+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:15:58.240 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:15:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:15:57.953+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:15:58.240 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:15:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:15:57.953+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:15:58.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:15:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:15:58.240+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:15:58.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:15:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:15:58.240+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:15:58.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:15:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:15:58.240+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:15:58.939 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:15:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:15:58.517+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:15:58.939 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:15:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:15:58.517+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:15:58.939 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:15:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:15:58.517+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:15:58.940 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:15:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:15:58.517+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:15:59.237 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:15:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:15:58.939+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:15:59.238 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:15:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:15:58.939+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:15:59.238 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:15:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:15:58.939+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:15:59.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:15:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:15:59.238+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:15:59.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:15:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:15:59.238+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:15:59.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:15:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:15:59.238+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:15:59.884 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:15:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:15:59.509+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:15:59.885 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:15:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:15:59.509+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:15:59.885 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:15:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:15:59.509+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:15:59.885 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:15:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:15:59.509+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:16:00.172 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:15:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:15:59.898+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:16:00.172 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:15:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:15:59.898+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:16:00.172 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:15:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:15:59.898+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:16:00.490 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:16:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:16:00.236+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:16:00.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:16:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:16:00.236+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:16:00.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:16:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:16:00.236+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:16:00.876 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:16:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:16:00.535+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:16:00.876 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:16:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:16:00.535+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:16:00.877 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:16:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:16:00.535+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:16:00.877 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:16:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:16:00.535+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:16:01.187 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:16:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:16:00.875+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:16:01.188 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:16:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:16:00.875+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:16:01.188 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:16:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:16:00.875+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:16:01.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:16:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:16:01.187+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:16:01.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:16:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:16:01.187+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:16:01.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:16:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:16:01.187+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:16:01.869 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:16:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:16:01.530+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:16:01.870 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:16:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:16:01.530+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:16:01.870 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:16:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:16:01.530+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:16:01.870 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:16:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:16:01.530+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:16:02.182 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:16:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:16:01.964+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 14230 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:16:02.182 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:16:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:16:01.869+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:16:02.182 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:16:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:16:01.869+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:16:02.183 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:16:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:16:01.869+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:16:02.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:16:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:16:02.181+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:16:02.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:16:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:16:02.181+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:16:02.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:16:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:16:02.181+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:16:02.867 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:16:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:16:02.570+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:16:02.867 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:16:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:16:02.570+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:16:02.868 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:16:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:16:02.570+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:16:02.868 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:16:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:16:02.570+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:16:03.156 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:16:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:16:02.866+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:16:03.157 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:16:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:16:02.866+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:16:03.157 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:16:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:16:02.866+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:16:03.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:16:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:16:03.155+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:16:03.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:16:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:16:03.155+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:16:03.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:16:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:16:03.155+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:16:03.870 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:16:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:16:03.610+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:16:03.870 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:16:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:16:03.610+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:16:03.871 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:16:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:16:03.610+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:16:03.871 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:16:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:16:03.610+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:16:04.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:16:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:16:04.111+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:16:04.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:16:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:16:04.111+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:16:04.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:16:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:16:04.111+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:16:04.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:16:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:16:03.869+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:16:04.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:16:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:16:03.869+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:16:04.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:16:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:16:03.869+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:16:04.824 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:16:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:16:04.567+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:16:04.824 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:16:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:16:04.567+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:16:04.825 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:16:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:16:04.567+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:16:04.825 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:16:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:16:04.567+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:16:05.106 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:16:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:16:04.823+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:16:05.107 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:16:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:16:04.823+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:16:05.107 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:16:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:16:04.823+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:16:05.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:16:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:16:05.105+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:16:05.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:16:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:16:05.105+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:16:05.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:16:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:16:05.105+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:16:05.990 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:16:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:16:05.842+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:16:05.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:16:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:16:05.842+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:16:05.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:16:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:16:05.842+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:16:05.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:16:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:16:05.607+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:16:05.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:16:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:16:05.607+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:16:05.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:16:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:16:05.607+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:16:05.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:16:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:16:05.607+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:16:06.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:16:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:16:06.068+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:16:06.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:16:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:16:06.068+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:16:06.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:16:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:16:06.068+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:16:06.864 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:16:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:16:06.608+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:16:06.864 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:16:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:16:06.608+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:16:06.864 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:16:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:16:06.608+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:16:06.864 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:16:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:16:06.608+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:16:07.241 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:16:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:16:06.966+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 14300 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:16:07.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:16:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:16:06.863+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:16:07.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:16:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:16:06.863+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:16:07.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:16:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:16:06.863+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:16:07.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:16:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:16:07.046+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:16:07.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:16:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:16:07.046+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:16:07.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:16:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:16:07.046+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:16:07.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:16:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:16:07.835+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:16:07.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:16:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:16:07.835+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:16:07.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:16:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:16:07.835+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:16:07.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:16:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:16:07.645+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:16:07.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:16:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:16:07.645+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:16:07.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:16:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:16:07.645+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:16:07.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:16:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:16:07.645+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:16:08.368 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:16:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:16:08.009+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:16:08.368 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:16:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:16:08.009+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:16:08.368 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:16:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:16:08.009+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:16:08.369 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-02-20T08:16:08.370 DEBUG:tasks.cephfs.mount:Force/lazy unmounting on client.0 2024-02-20T08:16:08.371 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-20T08:16:08.371 DEBUG:teuthology.orchestra.run.smithi133:> sudo umount --lazy --force /home/ubuntu/cephtest/mnt.0 2024-02-20T08:16:08.403 INFO:tasks.workunit.client.0.smithi133.stderr:mkdir: cannot create directory ‘/home/ubuntu/cephtest/mnt.0/client.0’: Input/output error 2024-02-20T08:16:08.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-20T08:16:08.405 INFO:tasks.workunit:Stopping ['suites/fsstress.sh'] on client.0... 2024-02-20T08:16:08.405 DEBUG:teuthology.orchestra.run.smithi133:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2024-02-20T08:16:08.406 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi133.front.sepia.ceph.com 2024-02-20T08:16:08.406 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-20T08:16:08.406 DEBUG:teuthology.orchestra.run.smithi133:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-02-20T08:16:08.473 DEBUG:teuthology.orchestra.run.smithi183:> set -ex 2024-02-20T08:16:08.473 DEBUG:teuthology.orchestra.run.smithi183:> dd if=/proc/self/mounts of=/dev/stdout 2024-02-20T08:16:08.952 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:16:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:16:08.805+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:16:08.952 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:16:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:16:08.805+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:16:08.953 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:16:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:16:08.805+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:16:08.953 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:16:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:16:08.646+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:16:08.954 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:16:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:16:08.646+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:16:08.954 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:16:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:16:08.646+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:16:08.955 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:16:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:16:08.646+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:16:09.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:16:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:16:09.009+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:16:09.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:16:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:16:09.009+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:16:09.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:16:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:16:09.009+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:16:09.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:16:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:16:09.791+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:16:09.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:16:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:16:09.791+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:16:09.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:16:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:16:09.791+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:16:09.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:16:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:16:09.669+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:16:09.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:16:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:16:09.669+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:16:09.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:16:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:16:09.669+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:16:09.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:16:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:16:09.669+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:16:10.490 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:16:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:16:10.052+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:16:10.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:16:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:16:10.052+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:16:10.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:16:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:16:10.052+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:16:10.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:16:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:16:10.771+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:16:10.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:16:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:16:10.771+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:16:10.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:16:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:16:10.771+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:16:10.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:16:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:16:10.650+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:16:10.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:16:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:16:10.650+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:16:10.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:16:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:16:10.650+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:16:10.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:16:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:16:10.650+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:16:11.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:16:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:16:11.042+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:16:11.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:16:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:16:11.042+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:16:11.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:16:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:16:11.042+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:16:11.969 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:16:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:16:11.644+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:16:11.969 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:16:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:16:11.644+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:16:11.970 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:16:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:16:11.644+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:16:11.970 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:16:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:16:11.644+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:16:11.970 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:16:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:16:11.723+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:16:11.970 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:16:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:16:11.723+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:16:11.970 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:16:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:16:11.723+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:16:12.246 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:16:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:16:11.968+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 14372 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:16:12.247 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:16:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:16:12.086+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:16:12.247 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:16:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:16:12.086+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:16:12.247 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:16:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:16:12.086+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:16:12.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:16:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:16:12.742+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:16:12.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:16:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:16:12.742+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:16:12.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:16:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:16:12.742+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:16:12.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:16:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:16:12.652+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:16:12.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:16:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:16:12.652+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:16:12.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:16:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:16:12.652+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:16:12.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:16:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:16:12.652+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:16:13.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:16:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:16:13.104+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:16:13.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:16:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:16:13.104+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:16:13.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:16:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:16:13.104+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:16:13.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:16:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:16:13.768+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:16:13.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:16:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:16:13.768+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:16:13.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:16:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:16:13.768+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:16:13.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:16:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:16:13.602+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:16:13.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:16:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:16:13.602+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:16:13.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:16:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:16:13.602+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:16:13.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:16:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:16:13.602+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:16:14.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:16:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:16:14.088+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:16:14.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:16:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:16:14.088+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:16:14.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:16:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:16:14.088+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:16:14.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:16:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:16:14.633+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:16:14.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:16:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:16:14.633+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:16:14.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:16:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:16:14.633+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:16:14.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:16:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:16:14.633+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:16:14.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:16:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:16:14.734+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:16:14.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:16:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:16:14.734+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:16:14.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:16:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:16:14.734+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:16:15.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:16:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:16:15.089+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:16:15.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:16:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:16:15.089+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:16:15.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:16:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:16:15.089+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:16:15.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:16:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:16:15.734+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:16:15.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:16:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:16:15.735+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:16:15.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:16:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:16:15.735+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:16:15.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:16:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:16:15.600+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:16:15.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:16:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:16:15.600+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:16:15.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:16:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:16:15.600+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:16:15.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:16:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:16:15.600+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:16:16.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:16:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:16:16.103+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:16:16.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:16:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:16:16.103+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:16:16.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:16:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:16:16.103+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:16:16.971 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:16:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:16:16.752+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:16:16.972 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:16:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:16:16.752+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:16:16.972 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:16:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:16:16.752+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:16:16.972 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:16:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:16:16.613+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:16:16.972 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:16:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:16:16.613+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:16:16.972 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:16:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:16:16.613+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:16:16.973 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:16:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:16:16.614+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:16:17.240 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:16:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:16:16.970+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 14442 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:16:17.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:16:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:16:17.075+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:16:17.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:16:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:16:17.075+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:16:17.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:16:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:16:17.075+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:16:17.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:16:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:16:17.738+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:16:17.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:16:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:16:17.738+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:16:17.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:16:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:16:17.738+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:16:17.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:16:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:16:17.656+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:16:17.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:16:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:16:17.656+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:16:17.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:16:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:16:17.656+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:16:17.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:16:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:16:17.656+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:16:18.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:16:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:16:18.091+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:16:18.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:16:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:16:18.091+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:16:18.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:16:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:16:18.091+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:16:18.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:16:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:16:18.733+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:16:18.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:16:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:16:18.733+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:16:18.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:16:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:16:18.733+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:16:18.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:16:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:16:18.671+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:16:18.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:16:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:16:18.671+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:16:18.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:16:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:16:18.671+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:16:18.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:16:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:16:18.671+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:16:19.490 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:16:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:16:19.073+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:16:19.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:16:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:16:19.073+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:16:19.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:16:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:16:19.073+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:16:19.884 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:16:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:16:19.748+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:16:19.885 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:16:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:16:19.748+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:16:19.885 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:16:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:16:19.748+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:16:19.885 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:16:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:16:19.625+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:16:19.886 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:16:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:16:19.625+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:16:19.886 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:16:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:16:19.625+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:16:19.886 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:16:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:16:19.625+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:16:20.173 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:16:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:16:20.097+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:16:20.174 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:16:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:16:20.097+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:16:20.174 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:16:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:16:20.097+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:16:20.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:16:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:16:20.762+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:16:20.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:16:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:16:20.762+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:16:20.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:16:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:16:20.762+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:16:20.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:16:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:16:20.624+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:16:20.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:16:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:16:20.624+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:16:20.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:16:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:16:20.624+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:16:20.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:16:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:16:20.624+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:16:21.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:16:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:16:21.058+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:16:21.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:16:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:16:21.058+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:16:21.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:16:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:16:21.058+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:16:21.974 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:16:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:16:21.746+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:16:21.974 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:16:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:16:21.746+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:16:21.974 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:16:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:16:21.746+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:16:21.975 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:16:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:16:21.672+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:16:21.975 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:16:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:16:21.672+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:16:21.975 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:16:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:16:21.672+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:16:21.975 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:16:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:16:21.672+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:16:22.241 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:16:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:16:21.973+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 14513 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:16:22.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:16:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:16:22.062+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:16:22.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:16:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:16:22.062+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:16:22.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:16:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:16:22.062+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:16:22.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:16:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:16:22.718+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:16:22.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:16:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:16:22.718+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:16:22.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:16:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:16:22.718+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:16:22.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:16:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:16:22.673+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:16:22.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:16:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:16:22.673+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:16:22.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:16:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:16:22.673+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:16:22.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:16:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:16:22.673+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:16:23.490 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:16:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:16:23.084+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:16:23.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:16:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:16:23.084+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:16:23.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:16:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:16:23.084+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:16:23.990 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:16:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:16:23.760+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:16:23.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:16:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:16:23.760+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:16:23.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:16:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:16:23.760+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:16:23.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:16:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:16:23.670+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:16:23.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:16:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:16:23.670+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:16:23.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:16:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:16:23.670+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:16:23.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:16:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:16:23.670+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:16:24.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:16:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:16:24.096+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:16:24.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:16:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:16:24.096+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:16:24.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:16:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:16:24.096+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:16:24.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:16:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:16:24.729+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:16:24.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:16:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:16:24.729+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:16:24.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:16:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:16:24.729+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:16:24.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:16:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:16:24.718+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:16:24.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:16:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:16:24.719+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:16:24.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:16:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:16:24.719+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:16:24.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:16:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:16:24.719+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:16:25.490 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:16:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:16:25.136+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:16:25.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:16:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:16:25.136+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:16:25.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:16:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:16:25.136+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:16:25.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:16:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:16:25.718+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:16:25.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:16:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:16:25.718+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:16:25.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:16:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:16:25.718+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:16:25.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:16:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:16:25.756+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:16:25.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:16:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:16:25.756+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:16:25.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:16:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:16:25.756+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:16:25.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:16:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:16:25.756+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:16:26.490 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:16:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:16:26.158+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:16:26.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:16:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:16:26.158+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:16:26.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:16:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:16:26.158+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:16:26.976 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:16:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:16:26.774+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:16:26.977 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:16:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:16:26.774+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:16:26.977 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:16:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:16:26.774+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:16:26.977 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:16:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:16:26.774+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:16:26.977 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:16:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:16:26.668+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:16:26.977 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:16:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:16:26.668+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:16:26.977 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:16:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:16:26.668+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:16:27.241 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:16:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:16:26.975+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 14584 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:16:27.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:16:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:16:27.118+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:16:27.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:16:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:16:27.118+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:16:27.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:16:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:16:27.118+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:16:27.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:16:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:16:27.628+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:16:27.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:16:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:16:27.628+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:16:27.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:16:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:16:27.628+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:16:27.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:16:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:16:27.778+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:16:27.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:16:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:16:27.778+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:16:27.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:16:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:16:27.778+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:16:27.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:16:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:16:27.778+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:16:28.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:16:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:16:28.093+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:16:28.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:16:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:16:28.093+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:16:28.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:16:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:16:28.093+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:16:28.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:16:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:16:28.664+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:16:28.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:16:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:16:28.664+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:16:28.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:16:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:16:28.664+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:16:28.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:16:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:16:28.762+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:16:28.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:16:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:16:28.762+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:16:28.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:16:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:16:28.762+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:16:28.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:16:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:16:28.762+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:16:29.490 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:16:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:16:29.096+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:16:29.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:16:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:16:29.096+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:16:29.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:16:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:16:29.096+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:16:29.990 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:16:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:16:29.650+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:16:29.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:16:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:16:29.650+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:16:29.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:16:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:16:29.650+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:16:29.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:16:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:16:29.773+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:16:29.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:16:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:16:29.773+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:16:29.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:16:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:16:29.773+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:16:29.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:16:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:16:29.773+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:16:30.455 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:16:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:16:30.073+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:16:30.455 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:16:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:16:30.073+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:16:30.455 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:16:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:16:30.073+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:16:30.740 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:16:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:16:30.605+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:16:30.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:16:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:16:30.605+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:16:30.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:16:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:16:30.605+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:16:31.025 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:16:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:16:30.749+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:16:31.025 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:16:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:16:30.749+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:16:31.025 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:16:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:16:30.749+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:16:31.025 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:16:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:16:30.749+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:16:31.490 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:16:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:16:31.025+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:16:31.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:16:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:16:31.025+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:16:31.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:16:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:16:31.025+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:16:31.979 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:16:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:16:31.766+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:16:31.979 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:16:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:16:31.766+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:16:31.979 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:16:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:16:31.766+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:16:31.979 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:16:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:16:31.766+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:16:31.980 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:16:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:16:31.652+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:16:31.980 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:16:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:16:31.652+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:16:31.980 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:16:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:16:31.652+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:16:32.241 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:16:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:16:31.978+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 14653 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:16:32.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:16:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:16:32.015+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:16:32.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:16:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:16:32.015+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:16:32.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:16:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:16:32.016+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:16:32.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:16:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:16:32.816+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:16:32.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:16:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:16:32.816+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:16:32.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:16:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:16:32.816+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:16:32.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:16:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:16:32.816+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:16:32.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:16:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:16:32.620+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:16:32.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:16:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:16:32.620+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:16:32.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:16:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:16:32.620+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:16:33.490 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:16:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:16:33.043+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:16:33.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:16:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:16:33.043+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:16:33.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:16:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:16:33.043+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:16:33.815 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:16:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:16:33.631+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:16:33.815 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:16:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:16:33.631+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:16:33.815 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:16:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:16:33.631+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:16:34.241 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:16:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:16:33.815+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:16:34.241 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:16:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:16:33.815+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:16:34.241 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:16:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:16:33.815+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:16:34.241 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:16:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:16:33.815+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:16:34.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:16:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:16:34.039+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:16:34.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:16:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:16:34.039+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:16:34.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:16:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:16:34.039+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:16:34.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:16:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:16:34.809+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:16:34.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:16:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:16:34.809+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:16:34.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:16:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:16:34.809+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:16:34.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:16:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:16:34.809+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:16:34.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:16:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:16:34.675+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:16:34.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:16:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:16:34.675+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:16:34.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:16:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:16:34.675+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:16:35.490 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:16:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:16:35.020+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:16:35.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:16:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:16:35.020+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:16:35.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:16:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:16:35.020+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:16:35.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:16:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:16:35.777+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:16:35.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:16:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:16:35.777+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:16:35.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:16:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:16:35.777+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:16:35.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:16:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:16:35.777+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:16:35.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:16:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:16:35.697+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:16:35.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:16:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:16:35.697+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:16:35.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:16:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:16:35.697+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:16:36.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:16:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:16:36.010+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:16:36.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:16:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:16:36.010+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:16:36.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:16:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:16:36.010+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:16:36.781 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:16:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:16:36.662+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:16:36.781 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:16:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:16:36.662+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:16:36.781 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:16:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:16:36.662+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:16:37.105 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:16:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:16:36.981+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 14723 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:16:37.105 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:16:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:16:36.991+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:16:37.105 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:16:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:16:36.991+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:16:37.105 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:16:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:16:36.991+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:16:37.106 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:16:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:16:36.780+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:16:37.106 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:16:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:16:36.780+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:16:37.106 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:16:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:16:36.780+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:16:37.106 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:16:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:16:36.781+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:16:37.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:16:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:16:37.825+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:16:37.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:16:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:16:37.825+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:16:37.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:16:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:16:37.825+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:16:37.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:16:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:16:37.825+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:16:37.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:16:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:16:37.696+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:16:37.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:16:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:16:37.696+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:16:37.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:16:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:16:37.696+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:16:38.490 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:16:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:16:38.018+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:16:38.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:16:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:16:38.018+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:16:38.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:16:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:16:38.018+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:16:38.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:16:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:16:38.782+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:16:38.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:16:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:16:38.782+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:16:38.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:16:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:16:38.782+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:16:38.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:16:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:16:38.782+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:16:38.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:16:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:16:38.730+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:16:38.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:16:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:16:38.730+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:16:38.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:16:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:16:38.730+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:16:39.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:16:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:16:39.043+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:16:39.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:16:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:16:39.043+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:16:39.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:16:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:16:39.043+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:16:39.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:16:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:16:39.792+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:16:39.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:16:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:16:39.792+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:16:39.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:16:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:16:39.792+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:16:39.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:16:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:16:39.793+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:16:39.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:16:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:16:39.719+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:16:39.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:16:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:16:39.719+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:16:39.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:16:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:16:39.719+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:16:40.490 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:16:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:16:40.082+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:16:40.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:16:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:16:40.082+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:16:40.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:16:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:16:40.082+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:16:40.990 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:16:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:16:40.820+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:16:40.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:16:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:16:40.820+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:16:40.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:16:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:16:40.820+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:16:40.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:16:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:16:40.820+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:16:40.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:16:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:16:40.696+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:16:40.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:16:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:16:40.696+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:16:40.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:16:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:16:40.696+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:16:41.490 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:16:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:16:41.053+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:16:41.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:16:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:16:41.053+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:16:41.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:16:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:16:41.053+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:16:41.983 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:16:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:16:41.843+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:16:41.984 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:16:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:16:41.843+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:16:41.984 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:16:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:16:41.843+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:16:41.984 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:16:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:16:41.843+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:16:41.984 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:16:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:16:41.733+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:16:41.984 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:16:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:16:41.733+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:16:41.985 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:16:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:16:41.733+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:16:42.240 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:16:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:16:41.983+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 14793 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:16:42.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:16:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:16:42.051+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:16:42.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:16:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:16:42.051+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:16:42.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:16:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:16:42.051+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:16:42.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:16:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:16:42.798+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:16:42.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:16:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:16:42.798+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:16:42.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:16:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:16:42.798+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:16:42.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:16:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:16:42.798+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:16:42.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:16:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:16:42.689+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:16:42.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:16:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:16:42.689+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:16:42.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:16:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:16:42.689+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:16:43.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:16:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:16:43.078+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:16:43.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:16:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:16:43.078+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:16:43.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:16:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:16:43.078+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:16:43.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:16:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:16:43.802+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:16:43.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:16:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:16:43.802+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:16:43.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:16:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:16:43.802+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:16:43.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:16:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:16:43.802+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:16:43.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:16:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:16:43.659+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:16:43.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:16:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:16:43.659+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:16:43.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:16:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:16:43.659+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:16:44.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:16:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:16:44.058+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:16:44.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:16:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:16:44.058+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:16:44.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:16:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:16:44.058+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:16:44.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:16:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:16:44.654+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:16:44.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:16:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:16:44.654+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:16:44.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:16:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:16:44.654+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:16:44.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:16:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:16:44.829+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:16:44.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:16:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:16:44.829+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:16:44.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:16:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:16:44.829+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:16:44.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:16:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:16:44.829+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:16:45.427 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:16:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:16:45.053+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:16:45.427 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:16:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:16:45.053+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:16:45.427 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:16:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:16:45.053+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:16:45.740 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:16:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:16:45.658+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:16:45.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:16:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:16:45.658+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:16:45.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:16:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:16:45.658+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:16:46.240 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:16:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:16:46.018+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:16:46.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:16:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:16:46.018+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:16:46.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:16:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:16:46.018+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:16:46.241 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:16:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:16:45.841+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:16:46.241 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:16:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:16:45.841+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:16:46.241 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:16:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:16:45.841+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:16:46.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:16:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:16:45.841+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:16:46.985 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:16:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:16:46.828+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:16:46.986 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:16:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:16:46.828+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:16:46.986 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:16:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:16:46.828+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:16:46.986 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:16:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:16:46.828+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:16:46.986 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:16:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:16:46.675+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:16:46.986 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:16:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:16:46.675+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:16:46.987 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:16:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:16:46.675+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:16:47.008 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:16:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:16:46.985+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 14864 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:16:47.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:16:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:16:47.022+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:16:47.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:16:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:16:47.022+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:16:47.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:16:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:16:47.022+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:16:47.986 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:16:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:16:47.695+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:16:47.986 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:16:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:16:47.695+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:16:47.986 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:16:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:16:47.695+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:16:47.987 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:16:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:16:47.842+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:16:47.987 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:16:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:16:47.842+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:16:47.987 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:16:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:16:47.842+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:16:47.987 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:16:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:16:47.842+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:16:48.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:16:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:16:47.986+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:16:48.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:16:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:16:47.986+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:16:48.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:16:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:16:47.986+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:16:48.943 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:16:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:16:48.688+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:16:48.943 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:16:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:16:48.688+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:16:48.943 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:16:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:16:48.688+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:16:48.944 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:16:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:16:48.808+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:16:48.944 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:16:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:16:48.808+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:16:48.944 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:16:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:16:48.808+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:16:48.944 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:16:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:16:48.808+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:16:49.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:16:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:16:48.943+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:16:49.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:16:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:16:48.943+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:16:49.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:16:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:16:48.943+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:16:49.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:16:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:16:49.675+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:16:49.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:16:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:16:49.675+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:16:49.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:16:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:16:49.675+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:16:49.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:16:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:16:49.905+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:16:49.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:16:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:16:49.905+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:16:49.992 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:16:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:16:49.905+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:16:49.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:16:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:16:49.773+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:16:49.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:16:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:16:49.773+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:16:49.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:16:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:16:49.773+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:16:49.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:16:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:16:49.773+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:16:50.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:16:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:16:50.719+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:16:50.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:16:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:16:50.719+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:16:50.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:16:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:16:50.719+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:16:50.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:16:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:16:50.922+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:16:50.992 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:16:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:16:50.922+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:16:50.992 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:16:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:16:50.922+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:16:50.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:16:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:16:50.756+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:16:50.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:16:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:16:50.756+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:16:50.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:16:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:16:50.756+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:16:50.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:16:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:16:50.756+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:16:51.959 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:16:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:16:51.701+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:16:51.959 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:16:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:16:51.701+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:16:51.959 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:16:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:16:51.701+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:16:51.960 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:16:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:16:51.886+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:16:51.960 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:16:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:16:51.886+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:16:51.960 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:16:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:16:51.886+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:16:51.960 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:16:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:16:51.750+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:16:51.960 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:16:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:16:51.750+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:16:51.960 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:16:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:16:51.750+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:16:51.961 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:16:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:16:51.750+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:16:52.241 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:16:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:16:51.987+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 14935 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:16:53.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:16:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:16:52.744+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:16:53.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:16:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:16:52.744+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:16:53.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:16:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:16:52.744+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:16:53.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:16:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:16:52.913+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:16:53.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:16:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:16:52.913+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:16:53.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:16:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:16:52.913+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:16:53.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:16:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:16:52.782+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:16:53.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:16:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:16:52.782+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:16:53.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:16:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:16:52.782+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:16:53.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:16:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:16:52.782+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:16:54.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:16:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:16:53.785+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:16:54.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:16:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:16:53.785+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:16:54.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:16:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:16:53.785+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:16:54.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:16:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:16:53.889+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:16:54.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:16:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:16:53.889+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:16:54.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:16:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:16:53.889+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:16:54.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:16:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:16:53.783+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:16:54.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:16:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:16:53.783+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:16:54.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:16:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:16:53.783+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:16:54.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:16:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:16:53.784+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:16:55.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:16:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:16:54.818+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:16:55.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:16:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:16:54.818+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:16:55.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:16:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:16:54.818+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:16:55.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:16:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:16:54.913+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:16:55.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:16:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:16:54.913+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:16:55.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:16:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:16:54.913+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:16:55.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:16:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:16:54.796+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:16:55.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:16:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:16:54.796+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:16:55.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:16:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:16:54.796+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:16:55.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:16:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:16:54.796+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:16:56.026 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:16:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:16:55.821+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:16:56.026 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:16:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:16:55.821+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:16:56.027 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:16:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:16:55.821+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:16:56.027 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:16:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:16:55.887+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:16:56.027 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:16:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:16:55.887+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:16:56.027 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:16:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:16:55.887+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:16:56.028 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:16:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:16:55.801+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:16:56.028 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:16:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:16:55.801+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:16:56.028 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:16:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:16:55.801+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:16:56.028 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:16:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:16:55.801+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:16:56.841 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:16:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:16:56.758+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:16:56.841 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:16:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:16:56.758+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:16:56.842 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:16:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:16:56.758+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:16:56.842 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:16:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:16:56.758+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:16:57.110 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:16:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:16:56.988+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 15006 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:16:57.111 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:16:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:16:56.840+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:16:57.111 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:16:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:16:56.840+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:16:57.111 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:16:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:16:56.840+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:16:57.111 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:16:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:16:56.887+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:16:57.111 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:16:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:16:56.887+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:16:57.112 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:16:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:16:56.887+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:16:57.990 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:16:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:16:57.731+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:16:57.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:16:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:16:57.731+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:16:57.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:16:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:16:57.731+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:16:57.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:16:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:16:57.731+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:16:57.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:16:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:16:57.800+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:16:57.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:16:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:16:57.800+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:16:57.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:16:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:16:57.800+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:16:57.992 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:16:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:16:57.907+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:16:57.992 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:16:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:16:57.907+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:16:57.992 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:16:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:16:57.907+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:16:58.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:16:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:16:58.782+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:16:58.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:16:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:16:58.782+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:16:58.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:16:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:16:58.782+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:16:58.992 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:16:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:16:58.895+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:16:58.992 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:16:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:16:58.895+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:16:58.992 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:16:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:16:58.895+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:16:58.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:16:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:16:58.715+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:16:58.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:16:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:16:58.715+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:16:58.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:16:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:16:58.715+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:16:58.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:16:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:16:58.715+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:16:59.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:16:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:16:59.734+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:16:59.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:16:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:16:59.734+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:16:59.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:16:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:16:59.734+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:16:59.992 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:16:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:16:59.883+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:16:59.992 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:16:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:16:59.883+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:16:59.992 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:16:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:16:59.883+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:16:59.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:16:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:16:59.708+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:16:59.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:16:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:16:59.708+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:16:59.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:16:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:16:59.708+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:16:59.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:16:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:16:59.708+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:17:00.990 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:17:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:17:00.758+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:17:00.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:17:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:17:00.758+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:17:00.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:17:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:17:00.758+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:17:00.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:17:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:17:00.907+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:17:00.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:17:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:17:00.907+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:17:00.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:17:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:17:00.907+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:17:00.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:17:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:17:00.659+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:17:00.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:17:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:17:00.659+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:17:00.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:17:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:17:00.659+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:17:00.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:17:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:17:00.659+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:17:01.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:17:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:17:01.902+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:17:01.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:17:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:17:01.902+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:17:01.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:17:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:17:01.902+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:17:01.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:17:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:17:01.656+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:17:01.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:17:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:17:01.656+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:17:01.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:17:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:17:01.656+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:17:01.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:17:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:17:01.656+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:17:01.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:17:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:17:01.714+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:17:01.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:17:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:17:01.714+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:17:01.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:17:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:17:01.714+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:17:02.490 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:17:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:17:01.991+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 15077 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:17:02.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:17:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:17:02.887+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:17:02.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:17:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:17:02.887+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:17:02.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:17:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:17:02.887+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:17:02.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:17:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:17:02.762+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:17:02.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:17:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:17:02.762+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:17:02.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:17:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:17:02.762+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:17:02.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:17:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:17:02.660+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:17:02.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:17:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:17:02.660+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:17:02.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:17:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:17:02.660+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:17:02.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:17:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:17:02.660+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:17:03.907 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:17:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:17:03.797+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:17:03.908 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:17:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:17:03.797+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:17:03.908 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:17:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:17:03.797+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:17:03.908 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:17:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:17:03.623+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:17:03.908 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:17:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:17:03.623+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:17:03.909 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:17:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:17:03.623+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:17:03.909 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:17:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:17:03.623+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:17:04.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:17:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:17:03.906+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:17:04.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:17:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:17:03.906+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:17:04.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:17:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:17:03.906+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:17:04.911 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:17:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:17:04.804+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:17:04.911 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:17:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:17:04.804+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:17:04.911 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:17:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:17:04.804+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:17:04.912 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:17:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:17:04.614+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:17:04.912 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:17:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:17:04.614+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:17:04.912 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:17:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:17:04.614+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:17:04.912 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:17:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:17:04.614+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:17:05.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:17:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:17:04.910+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:17:05.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:17:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:17:04.910+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:17:05.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:17:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:17:04.910+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:17:05.936 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:17:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:17:05.840+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:17:05.937 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:17:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:17:05.840+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:17:05.937 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:17:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:17:05.840+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:17:05.937 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:17:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:17:05.652+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:17:05.937 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:17:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:17:05.653+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:17:05.937 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:17:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:17:05.653+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:17:05.937 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:17:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:17:05.653+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:17:06.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:17:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:17:05.936+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:17:06.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:17:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:17:05.936+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:17:06.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:17:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:17:05.936+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:17:06.942 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:17:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:17:06.652+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:17:06.942 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:17:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:17:06.652+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:17:06.942 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:17:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:17:06.652+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:17:06.942 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:17:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:17:06.652+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:17:06.943 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:17:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:17:06.817+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:17:06.943 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:17:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:17:06.817+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:17:06.943 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:17:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:17:06.817+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:17:07.240 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:17:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:17:06.993+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 15147 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:17:07.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:17:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:17:06.941+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:17:07.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:17:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:17:06.941+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:17:07.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:17:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:17:06.941+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:17:07.767 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:17:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:17:07.661+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:17:07.767 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:17:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:17:07.661+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:17:07.767 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:17:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:17:07.661+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:17:07.767 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:17:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:17:07.661+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:17:08.079 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:17:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:17:07.956+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:17:08.079 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:17:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:17:07.956+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:17:08.079 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:17:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:17:07.956+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:17:08.079 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:17:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:17:07.804+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:17:08.080 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:17:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:17:07.804+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:17:08.080 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:17:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:17:07.804+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:17:08.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:17:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:17:08.680+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:17:08.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:17:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:17:08.680+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:17:08.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:17:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:17:08.680+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:17:08.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:17:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:17:08.680+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:17:08.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:17:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:17:08.778+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:17:08.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:17:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:17:08.778+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:17:08.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:17:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:17:08.778+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:17:09.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:17:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:17:09.000+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:17:09.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:17:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:17:09.000+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:17:09.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:17:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:17:09.000+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:17:09.953 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:17:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:17:09.649+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:17:09.953 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:17:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:17:09.649+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:17:09.953 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:17:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:17:09.649+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:17:09.954 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:17:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:17:09.650+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:17:09.954 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:17:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:17:09.784+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:17:09.954 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:17:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:17:09.784+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:17:09.954 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:17:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:17:09.784+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:17:10.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:17:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:17:09.952+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:17:10.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:17:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:17:09.952+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:17:10.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:17:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:17:09.952+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:17:10.740 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:17:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:17:10.629+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:17:10.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:17:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:17:10.629+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:17:10.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:17:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:17:10.629+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:17:10.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:17:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:17:10.629+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:17:11.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:17:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:17:10.928+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:17:11.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:17:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:17:10.928+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:17:11.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:17:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:17:10.928+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:17:11.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:17:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:17:10.800+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:17:11.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:17:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:17:10.800+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:17:11.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:17:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:17:10.800+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:17:11.976 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:17:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:17:11.679+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:17:11.976 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:17:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:17:11.679+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:17:11.976 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:17:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:17:11.679+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:17:11.976 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:17:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:17:11.679+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:17:11.976 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:17:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:17:11.825+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:17:11.977 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:17:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:17:11.825+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:17:11.977 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:17:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:17:11.825+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:17:12.241 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:17:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:17:11.996+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 15217 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:17:12.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:17:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:17:11.975+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:17:12.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:17:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:17:11.975+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:17:12.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:17:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:17:11.975+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:17:12.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:17:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:17:12.840+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:17:12.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:17:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:17:12.840+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:17:12.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:17:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:17:12.840+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:17:12.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:17:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:17:12.728+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:17:12.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:17:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:17:12.728+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:17:12.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:17:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:17:12.728+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:17:12.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:17:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:17:12.728+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:17:13.463 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:17:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:17:13.016+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:17:13.463 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:17:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:17:13.016+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:17:13.463 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:17:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:17:13.016+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:17:13.740 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:17:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:17:13.678+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:17:13.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:17:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:17:13.678+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:17:13.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:17:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:17:13.678+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:17:13.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:17:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:17:13.678+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:17:14.240 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:17:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:17:13.852+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:17:14.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:17:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:17:13.852+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:17:14.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:17:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:17:13.852+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:17:14.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:17:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:17:14.064+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:17:14.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:17:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:17:14.064+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:17:14.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:17:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:17:14.064+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:17:14.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:17:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:17:14.836+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:17:14.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:17:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:17:14.836+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:17:14.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:17:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:17:14.836+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:17:14.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:17:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:17:14.692+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:17:14.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:17:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:17:14.692+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:17:14.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:17:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:17:14.692+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:17:14.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:17:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:17:14.692+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:17:15.490 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:17:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:17:15.070+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:17:15.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:17:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:17:15.070+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:17:15.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:17:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:17:15.070+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:17:15.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:17:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:17:15.795+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:17:15.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:17:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:17:15.795+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:17:15.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:17:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:17:15.795+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:17:15.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:17:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:17:15.647+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:17:15.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:17:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:17:15.647+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:17:15.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:17:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:17:15.647+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:17:15.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:17:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:17:15.647+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:17:16.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:17:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:17:16.034+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:17:16.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:17:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:17:16.034+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:17:16.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:17:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:17:16.034+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:17:16.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:17:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:17:16.824+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:17:16.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:17:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:17:16.824+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:17:16.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:17:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:17:16.824+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:17:16.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:17:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:17:16.629+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:17:16.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:17:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:17:16.629+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:17:16.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:17:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:17:16.629+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:17:16.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:17:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:17:16.629+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:17:17.491 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:17:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:17:16.998+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 15287 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:17:17.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:17:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:17:17.002+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:17:17.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:17:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:17:17.002+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:17:17.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:17:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:17:17.002+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:17:17.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:17:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:17:17.830+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:17:17.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:17:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:17:17.830+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:17:17.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:17:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:17:17.830+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:17:17.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:17:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:17:17.675+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:17:17.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:17:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:17:17.675+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:17:17.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:17:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:17:17.675+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:17:17.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:17:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:17:17.675+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:17:18.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:17:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:17:17.998+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:17:18.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:17:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:17:17.998+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:17:18.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:17:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:17:17.998+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:17:18.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:17:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:17:18.852+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:17:18.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:17:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:17:18.852+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:17:18.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:17:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:17:18.852+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:17:18.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:17:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:17:18.708+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:17:18.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:17:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:17:18.708+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:17:18.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:17:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:17:18.708+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:17:18.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:17:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:17:18.708+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:17:19.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:17:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:17:19.006+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:17:19.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:17:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:17:19.006+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:17:19.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:17:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:17:19.006+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:17:19.964 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:17:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:17:19.870+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:17:19.964 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:17:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:17:19.870+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:17:19.965 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:17:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:17:19.870+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:17:19.965 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:17:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:17:19.708+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:17:19.965 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:17:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:17:19.708+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:17:19.965 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:17:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:17:19.708+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:17:19.965 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:17:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:17:19.708+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:17:20.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:17:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:17:19.964+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:17:20.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:17:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:17:19.964+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:17:20.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:17:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:17:19.964+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:17:20.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:17:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:17:20.883+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:17:20.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:17:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:17:20.883+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:17:20.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:17:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:17:20.883+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:17:20.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:17:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:17:20.715+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:17:20.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:17:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:17:20.715+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:17:20.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:17:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:17:20.715+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:17:20.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:17:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:17:20.715+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:17:21.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:17:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:17:21.012+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:17:21.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:17:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:17:21.012+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:17:21.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:17:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:17:21.012+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:17:21.968 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:17:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:17:21.871+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:17:21.968 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:17:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:17:21.871+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:17:21.968 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:17:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:17:21.871+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:17:21.969 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:17:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:17:21.752+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:17:21.969 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:17:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:17:21.752+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:17:21.969 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:17:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:17:21.752+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:17:21.969 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:17:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:17:21.752+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:17:22.240 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:17:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:17:22.000+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 15357 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:17:22.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:17:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:17:21.967+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:17:22.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:17:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:17:21.967+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:17:22.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:17:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:17:21.967+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:17:22.975 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:17:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:17:22.842+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:17:22.975 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:17:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:17:22.842+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:17:22.975 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:17:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:17:22.842+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:17:22.975 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:17:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:17:22.743+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:17:22.976 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:17:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:17:22.743+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:17:22.976 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:17:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:17:22.743+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:17:22.976 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:17:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:17:22.743+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:17:23.240 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:17:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:17:22.975+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:17:23.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:17:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:17:22.975+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:17:23.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:17:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:17:22.975+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:17:24.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:17:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:17:23.980+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:17:24.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:17:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:17:23.980+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:17:24.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:17:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:17:23.980+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:17:24.245 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:17:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:17:23.796+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:17:24.245 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:17:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:17:23.796+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:17:24.245 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:17:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:17:23.796+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:17:24.246 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:17:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:17:23.761+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:17:24.246 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:17:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:17:23.761+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:17:24.246 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:17:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:17:23.761+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:17:24.246 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:17:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:17:23.762+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:17:25.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:17:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:17:24.968+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:17:25.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:17:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:17:24.968+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:17:25.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:17:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:17:24.968+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:17:25.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:17:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:17:24.824+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:17:25.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:17:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:17:24.824+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:17:25.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:17:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:17:24.824+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:17:25.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:17:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:17:24.800+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:17:25.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:17:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:17:24.800+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:17:25.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:17:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:17:24.800+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:17:25.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:17:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:17:24.800+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:17:26.240 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:17:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:17:25.956+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:17:26.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:17:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:17:25.956+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:17:26.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:17:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:17:25.956+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:17:26.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:17:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:17:25.869+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:17:26.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:17:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:17:25.869+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:17:26.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:17:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:17:25.869+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:17:26.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:17:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:17:25.794+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:17:26.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:17:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:17:25.794+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:17:26.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:17:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:17:25.794+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:17:26.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:17:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:17:25.794+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:17:27.128 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:17:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:17:27.003+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 15427 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:17:27.129 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:17:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:17:26.918+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:17:27.129 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:17:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:17:26.918+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:17:27.129 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:17:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:17:26.918+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:17:27.130 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:17:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:17:26.867+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:17:27.130 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:17:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:17:26.867+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:17:27.130 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:17:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:17:26.867+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:17:27.130 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:17:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:17:26.810+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:17:27.130 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:17:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:17:26.810+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:17:27.130 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:17:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:17:26.810+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:17:27.130 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:17:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:17:26.810+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:17:28.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:17:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:17:27.879+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:17:28.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:17:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:17:27.879+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:17:28.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:17:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:17:27.879+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:17:28.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:17:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:17:27.965+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:17:28.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:17:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:17:27.965+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:17:28.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:17:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:17:27.965+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:17:28.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:17:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:17:27.788+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:17:28.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:17:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:17:27.788+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:17:28.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:17:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:17:27.788+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:17:28.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:17:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:17:27.788+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:17:29.240 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:17:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:17:28.871+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:17:29.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:17:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:17:28.871+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:17:29.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:17:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:17:28.871+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:17:29.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:17:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:17:28.946+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:17:29.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:17:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:17:28.946+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:17:29.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:17:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:17:28.946+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:17:29.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:17:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:17:28.767+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:17:29.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:17:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:17:28.767+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:17:29.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:17:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:17:28.767+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:17:29.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:17:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:17:28.767+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:17:30.179 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:17:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:17:29.896+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:17:30.179 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:17:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:17:29.896+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:17:30.179 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:17:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:17:29.896+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:17:30.180 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:17:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:17:29.967+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:17:30.180 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:17:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:17:29.967+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:17:30.180 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:17:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:17:29.967+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:17:30.181 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:17:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:17:29.784+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:17:30.181 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:17:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:17:29.784+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:17:30.181 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:17:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:17:29.784+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:17:30.181 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:17:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:17:29.784+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:17:31.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:17:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:17:31.002+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:17:31.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:17:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:17:31.002+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:17:31.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:17:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:17:31.002+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:17:31.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:17:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:17:30.886+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:17:31.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:17:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:17:30.886+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:17:31.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:17:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:17:30.886+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:17:31.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:17:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:17:30.780+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:17:31.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:17:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:17:30.780+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:17:31.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:17:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:17:30.780+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:17:31.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:17:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:17:30.780+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:17:31.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:17:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:17:31.856+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:17:31.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:17:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:17:31.856+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:17:31.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:17:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:17:31.856+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:17:31.992 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:17:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:17:31.953+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:17:31.992 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:17:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:17:31.953+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:17:31.992 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:17:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:17:31.953+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:17:31.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:17:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:17:31.741+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:17:31.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:17:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:17:31.741+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:17:31.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:17:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:17:31.741+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:17:31.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:17:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:17:31.741+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:17:32.500 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:17:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:17:32.006+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 15497 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:17:32.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:17:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:17:32.891+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:17:32.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:17:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:17:32.891+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:17:32.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:17:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:17:32.891+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:17:32.992 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:17:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:17:32.934+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:17:32.992 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:17:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:17:32.934+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:17:32.992 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:17:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:17:32.934+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:17:32.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:17:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:17:32.697+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:17:32.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:17:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:17:32.697+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:17:32.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:17:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:17:32.697+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:17:32.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:17:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:17:32.697+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:17:33.990 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:17:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:17:33.887+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:17:33.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:17:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:17:33.887+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:17:33.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:17:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:17:33.887+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:17:33.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:17:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:17:33.892+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:17:33.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:17:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:17:33.892+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:17:33.992 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:17:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:17:33.892+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:17:33.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:17:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:17:33.651+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:17:33.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:17:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:17:33.651+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:17:33.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:17:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:17:33.651+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:17:33.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:17:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:17:33.651+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:17:34.926 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:17:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:17:34.838+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:17:34.927 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:17:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:17:34.838+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:17:34.927 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:17:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:17:34.838+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:17:34.927 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:17:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:17:34.656+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:17:34.927 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:17:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:17:34.656+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:17:34.927 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:17:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:17:34.656+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:17:34.928 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:17:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:17:34.656+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:17:35.240 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:17:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:17:34.926+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:17:35.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:17:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:17:34.926+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:17:35.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:17:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:17:34.926+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:17:35.990 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:17:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:17:35.841+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:17:35.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:17:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:17:35.841+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:17:35.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:17:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:17:35.841+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:17:35.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:17:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:17:35.928+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:17:35.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:17:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:17:35.928+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:17:35.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:17:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:17:35.928+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:17:35.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:17:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:17:35.693+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:17:35.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:17:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:17:35.693+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:17:35.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:17:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:17:35.693+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:17:35.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:17:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:17:35.693+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:17:36.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:17:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:17:36.882+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:17:36.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:17:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:17:36.882+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:17:36.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:17:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:17:36.882+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:17:36.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:17:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:17:36.879+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:17:36.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:17:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:17:36.879+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:17:36.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:17:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:17:36.879+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:17:36.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:17:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:17:36.667+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:17:36.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:17:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:17:36.667+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:17:36.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:17:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:17:36.667+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:17:36.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:17:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:17:36.667+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:17:37.490 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:17:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:17:37.008+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 15567 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:17:37.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:17:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:17:37.897+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:17:37.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:17:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:17:37.897+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:17:37.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:17:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:17:37.897+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:17:37.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:17:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:17:37.898+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:17:37.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:17:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:17:37.898+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:17:37.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:17:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:17:37.898+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:17:37.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:17:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:17:37.667+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:17:37.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:17:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:17:37.667+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:17:37.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:17:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:17:37.667+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:17:37.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:17:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:17:37.667+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:17:38.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:17:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:17:38.899+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:17:38.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:17:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:17:38.899+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:17:38.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:17:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:17:38.899+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:17:38.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:17:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:17:38.872+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:17:38.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:17:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:17:38.872+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:17:38.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:17:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:17:38.872+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:17:38.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:17:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:17:38.659+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:17:38.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:17:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:17:38.659+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:17:38.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:17:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:17:38.659+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:17:38.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:17:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:17:38.660+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:17:39.919 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:17:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:17:39.865+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:17:39.919 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:17:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:17:39.865+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:17:39.919 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:17:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:17:39.865+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:17:39.920 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:17:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:17:39.659+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:17:39.920 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:17:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:17:39.659+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:17:39.920 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:17:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:17:39.659+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:17:39.920 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:17:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:17:39.659+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:17:40.179 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:17:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:17:39.919+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:17:40.180 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:17:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:17:39.919+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:17:40.180 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:17:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:17:39.919+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:17:40.861 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:17:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:17:40.609+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:17:40.862 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:17:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:17:40.609+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:17:40.862 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:17:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:17:40.609+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:17:40.862 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:17:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:17:40.609+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:17:41.240 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:17:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:17:40.862+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:17:41.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:17:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:17:40.862+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:17:41.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:17:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:17:40.862+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:08.947813+0000 (oldest deadline 2024-02-20T08:07:28.947813+0000) 2024-02-20T08:17:41.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:17:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:17:40.927+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:17:41.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:17:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:17:40.927+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:17:41.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:17:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:17:40.927+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:17:41.946 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:17:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:17:41.618+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:17:41.946 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:17:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:17:41.618+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:17:41.946 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:17:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:17:41.618+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:17:41.947 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:17:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:17:41.618+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:17:42.241 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:17:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:17:42.010+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 15637 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:17:42.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:17:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:17:41.946+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:17:42.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:17:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:17:41.946+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:17:42.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:17:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:17:41.946+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:17:42.990 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:17:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:17:42.623+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:17:42.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:17:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:17:42.623+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:17:42.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:17:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:17:42.623+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:17:42.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:17:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:17:42.623+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:17:43.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:17:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:17:42.990+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:17:43.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:17:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:17:42.990+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:17:43.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:17:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:17:42.990+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:17:43.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:17:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:17:43.652+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:17:43.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:17:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:17:43.652+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:17:43.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:17:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:17:43.652+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:17:43.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:17:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:17:43.652+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:17:44.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:17:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:17:43.998+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:17:44.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:17:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:17:43.999+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:17:44.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:17:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:17:43.999+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:17:44.990 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:17:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:17:44.634+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:17:44.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:17:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:17:44.634+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:17:44.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:17:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:17:44.634+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:17:44.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:17:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:17:44.634+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:17:45.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:17:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:17:45.035+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:17:45.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:17:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:17:45.035+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:17:45.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:17:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:17:45.035+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:17:45.990 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:17:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:17:45.639+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:17:45.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:17:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:17:45.639+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:17:45.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:17:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:17:45.639+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:17:45.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:17:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:17:45.639+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:17:46.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:17:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:17:46.076+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:17:46.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:17:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:17:46.076+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:17:46.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:17:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:17:46.076+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:17:46.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:17:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:17:46.655+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:17:46.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:17:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:17:46.655+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:17:46.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:17:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:17:46.655+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:17:46.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:17:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:17:46.655+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:17:47.490 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:17:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:17:47.012+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 15707 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:17:47.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:17:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:17:47.100+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:17:47.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:17:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:17:47.100+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:17:47.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:17:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:17:47.100+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:17:47.990 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:17:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:17:47.695+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:17:47.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:17:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:17:47.695+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:17:47.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:17:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:17:47.695+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:17:47.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:17:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:17:47.695+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:17:48.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:17:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:17:48.068+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:17:48.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:17:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:17:48.068+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:17:48.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:17:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:17:48.068+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:17:48.990 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:17:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:17:48.717+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:17:48.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:17:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:17:48.717+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:17:48.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:17:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:17:48.717+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:17:48.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:17:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:17:48.717+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:17:49.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:17:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:17:49.062+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:17:49.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:17:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:17:49.062+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:17:49.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:17:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:17:49.062+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:17:50.040 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:17:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:17:49.741+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:17:50.040 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:17:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:17:49.741+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:17:50.040 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:17:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:17:49.741+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:17:50.041 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:17:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:17:49.741+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:17:50.327 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:17:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:17:50.039+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:17:50.327 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:17:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:17:50.039+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:17:50.327 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:17:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:17:50.039+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:17:51.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:17:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:17:50.994+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:17:51.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:17:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:17:50.995+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:17:51.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:17:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:17:50.995+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:17:51.241 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:17:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:17:50.774+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:17:51.241 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:17:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:17:50.774+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:17:51.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:17:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:17:50.774+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:17:51.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:17:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:17:50.774+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:17:51.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:17:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:17:51.741+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:17:51.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:17:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:17:51.741+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:17:51.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:17:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:17:51.741+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:17:51.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:17:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:17:51.741+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:17:52.490 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:17:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:17:52.014+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 15777 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:17:52.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:17:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:17:52.039+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:17:52.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:17:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:17:52.039+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:17:52.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:17:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:17:52.039+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:17:52.990 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:17:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:17:52.728+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:17:52.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:17:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:17:52.728+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:17:52.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:17:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:17:52.728+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:17:52.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:17:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:17:52.728+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:17:53.399 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:17:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:17:53.082+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:17:53.399 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:17:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:17:53.082+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:17:53.399 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:17:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:17:53.082+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:19.284568+0000 (oldest deadline 2024-02-20T08:07:39.284568+0000) 2024-02-20T08:17:54.241 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:17:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:17:53.763+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:17:54.241 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:17:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:17:53.763+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:17:54.241 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:17:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:17:53.763+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:17:54.241 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:17:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:17:53.763+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:17:55.241 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:17:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:17:54.809+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:17:55.241 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:17:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:17:54.809+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:17:55.241 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:17:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:17:54.809+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:17:55.241 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:17:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:17:54.809+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:17:56.240 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:17:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:17:55.844+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:17:56.241 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:17:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:17:55.844+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:17:56.241 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:17:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:17:55.844+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:17:56.241 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:17:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:17:55.844+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:17:57.140 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:17:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:17:57.017+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 15847 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:17:57.141 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:17:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:17:56.819+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:17:57.141 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:17:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:17:56.819+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:17:57.141 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:17:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:17:56.819+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:17:57.141 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:17:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:17:56.819+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:17:58.241 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:17:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:17:57.784+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:17:58.241 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:17:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:17:57.784+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:17:58.241 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:17:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:17:57.784+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:17:58.241 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:17:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:17:57.784+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:17:59.241 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:17:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:17:58.795+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:17:59.241 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:17:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:17:58.795+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:17:59.241 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:17:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:17:58.795+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:17:59.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:17:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:17:58.795+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:18:00.181 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:17:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:17:59.755+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:18:00.182 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:17:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:17:59.755+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:18:00.182 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:17:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:17:59.755+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:18:00.182 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:17:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:17:59.755+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:18:00.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:18:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:18:00.730+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:18:00.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:18:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:18:00.730+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:18:00.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:18:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:18:00.730+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:18:00.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:18:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:18:00.730+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:18:01.990 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:18:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:18:01.777+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:18:01.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:18:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:18:01.777+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:18:01.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:18:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:18:01.777+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:18:01.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:18:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:18:01.777+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:18:02.490 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:18:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:18:02.019+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 15917 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:18:02.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:18:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:18:02.811+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:18:02.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:18:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:18:02.811+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:18:02.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:18:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:18:02.811+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:18:02.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:18:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:18:02.811+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:18:03.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:18:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:18:03.709+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:18:03.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:18:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:18:03.709+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:18:03.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:18:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:18:03.709+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:18:03.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:18:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:18:03.811+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:18:03.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:18:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:18:03.811+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:18:03.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:18:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:18:03.811+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:18:03.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:18:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:18:03.811+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:18:04.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:18:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:18:04.725+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:18:04.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:18:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:18:04.725+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:18:04.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:18:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:18:04.725+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:18:04.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:18:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:18:04.838+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:18:04.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:18:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:18:04.838+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:18:04.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:18:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:18:04.838+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:18:04.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:18:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:18:04.838+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:18:05.990 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:18:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:18:05.687+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:18:05.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:18:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:18:05.687+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:18:05.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:18:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:18:05.687+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:18:05.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:18:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:18:05.820+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:18:05.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:18:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:18:05.820+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:18:05.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:18:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:18:05.820+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:07:32.972047+0000 (oldest deadline 2024-02-20T08:07:52.972047+0000) 2024-02-20T08:18:05.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:18:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:18:05.820+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:18:06.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:18:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:18:06.776+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:18:06.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:18:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:18:06.643+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:18:06.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:18:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:18:06.643+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:18:06.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:18:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:18:06.643+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:18:07.490 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:18:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:18:07.022+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 15987 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:18:07.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:18:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:18:07.777+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:18:07.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:18:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:18:07.664+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:18:07.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:18:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:18:07.664+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:18:07.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:18:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:18:07.664+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:18:08.990 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:18:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:18:08.762+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:18:08.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:18:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:18:08.713+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:18:08.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:18:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:18:08.713+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:18:08.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:18:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:18:08.713+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:18:09.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:18:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:18:09.740+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:18:09.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:18:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:18:09.748+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:18:09.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:18:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:18:09.748+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:18:09.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:18:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:18:09.748+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:18:10.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:18:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:18:10.727+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:18:10.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:18:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:18:10.724+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:18:10.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:18:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:18:10.724+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:18:10.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:18:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:18:10.724+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:18:11.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:18:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:18:11.682+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:18:11.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:18:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:18:11.677+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:18:11.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:18:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:18:11.677+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:18:11.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:18:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:18:11.677+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:18:12.491 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:18:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:18:12.024+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 16057 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:18:12.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:18:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:18:12.655+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:18:12.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:18:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:18:12.636+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:18:12.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:18:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:18:12.636+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:18:12.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:18:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:18:12.636+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:18:13.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:18:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:18:13.649+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:18:13.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:18:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:18:13.589+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:18:13.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:18:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:18:13.589+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:18:13.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:18:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:18:13.589+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:18:14.990 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:18:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:18:14.644+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:18:14.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:18:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:18:14.625+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:18:14.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:18:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:18:14.625+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:18:14.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:18:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:18:14.625+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:18:15.600 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:18:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:18:15.347+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:18:15.600 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:18:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:18:15.347+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:18:15.601 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:18:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:18:15.347+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:18:15.990 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:18:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:18:15.600+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:18:15.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:18:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:18:15.652+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:18:15.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:18:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:18:15.652+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:18:15.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:18:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:18:15.652+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:18:16.610 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:18:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:18:16.301+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:18:16.610 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:18:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:18:16.301+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:18:16.611 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:18:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:18:16.301+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:18:16.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:18:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:18:16.613+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:18:16.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:18:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:18:16.610+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:18:16.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:18:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:18:16.610+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:18:16.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:18:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:18:16.610+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:18:17.284 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:18:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:18:17.027+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 16127 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:18:17.617 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:18:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:18:17.284+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:18:17.617 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:18:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:18:17.284+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:18:17.617 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:18:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:18:17.284+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:18:17.990 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:18:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:18:17.617+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:18:17.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:18:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:18:17.617+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:18:17.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:18:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:18:17.617+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:18:17.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:18:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:18:17.617+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:18:18.607 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:18:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:18:18.284+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:18:18.607 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:18:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:18:18.284+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:18:18.607 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:18:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:18:18.284+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:18:18.990 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:18:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:18:18.649+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:18:18.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:18:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:18:18.607+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:18:18.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:18:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:18:18.607+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:18:18.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:18:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:18:18.607+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:18:19.606 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:18:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:18:19.291+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:18:19.607 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:18:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:18:19.291+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:18:19.607 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:18:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:18:19.291+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:18:19.888 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:18:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:18:19.607+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:18:19.888 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:18:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:18:19.621+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:18:19.889 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:18:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:18:19.621+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:18:19.889 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:18:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:18:19.621+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:18:20.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:18:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:18:20.311+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:18:20.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:18:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:18:20.311+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:18:20.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:18:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:18:20.311+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:18:20.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:18:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:18:20.607+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:18:20.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:18:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:18:20.666+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:18:20.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:18:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:18:20.666+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:18:20.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:18:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:18:20.666+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:18:21.598 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:18:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:18:21.321+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:18:21.598 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:18:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:18:21.321+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:18:21.598 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:18:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:18:21.321+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:18:21.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:18:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:18:21.598+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:18:21.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:18:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:18:21.688+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:18:21.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:18:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:18:21.688+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:18:21.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:18:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:18:21.688+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:18:22.348 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:18:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:18:22.030+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 16197 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:18:22.647 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:18:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:18:22.591+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:18:22.647 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:18:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:18:22.348+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:18:22.647 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:18:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:18:22.348+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:18:22.647 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:18:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:18:22.348+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:18:22.990 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:18:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:18:22.647+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:18:22.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:18:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:18:22.647+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:18:22.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:18:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:18:22.647+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:18:23.646 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:18:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:18:23.550+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:18:23.646 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:18:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:18:23.392+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:18:23.646 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:18:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:18:23.392+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:18:23.647 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:18:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:18:23.392+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:18:23.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:18:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:18:23.646+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:18:23.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:18:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:18:23.646+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:18:23.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:18:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:18:23.646+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:18:24.490 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:18:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:18:24.358+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:18:24.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:18:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:18:24.358+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:18:24.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:18:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:18:24.358+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:18:24.990 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:18:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:18:24.512+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:18:24.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:18:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:18:24.643+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:18:24.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:18:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:18:24.643+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:18:24.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:18:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:18:24.643+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:18:25.681 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:18:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:18:25.545+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:18:25.681 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:18:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:18:25.311+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:18:25.681 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:18:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:18:25.311+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:18:25.682 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:18:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:18:25.311+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:18:25.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:18:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:18:25.681+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:18:25.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:18:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:18:25.681+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:18:25.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:18:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:18:25.681+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:18:26.696 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:18:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:18:26.543+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:18:26.699 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:18:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:18:26.351+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:18:26.704 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:18:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:18:26.351+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:18:26.704 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:18:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:18:26.351+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:18:26.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:18:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:18:26.694+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:18:26.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:18:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:18:26.694+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:18:26.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:18:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:18:26.694+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:18:27.384 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:18:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:18:27.032+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 16267 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:18:27.654 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:18:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:18:27.578+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:18:27.655 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:18:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:18:27.578+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:18:27.655 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:18:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:18:27.578+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:18:27.655 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:18:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:18:27.578+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:18:27.655 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:18:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:18:27.383+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:18:27.655 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:18:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:18:27.383+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:18:27.656 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:18:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:18:27.383+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:18:27.990 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:18:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:18:27.648+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:18:27.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:18:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:18:27.648+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:18:27.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:18:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:18:27.648+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:18:28.658 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:18:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:18:28.408+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:18:28.658 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:18:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:18:28.408+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:18:28.658 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:18:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:18:28.408+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:18:28.658 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:18:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:18:28.602+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:18:28.659 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:18:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:18:28.602+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:18:28.659 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:18:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:18:28.602+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:18:28.659 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:18:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:18:28.603+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:18:28.990 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:18:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:18:28.658+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:18:28.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:18:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:18:28.658+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:18:28.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:18:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:18:28.658+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:18:29.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:18:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:18:29.632+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:18:29.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:18:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:18:29.632+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:18:29.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:18:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:18:29.632+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:18:29.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:18:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:18:29.392+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:18:29.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:18:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:18:29.392+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:18:29.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:18:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:18:29.392+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:18:29.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:18:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:18:29.634+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:18:29.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:18:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:18:29.634+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:18:29.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:18:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:18:29.634+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:18:29.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:18:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:18:29.634+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:18:30.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:18:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:18:30.419+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:18:30.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:18:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:18:30.419+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:18:30.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:18:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:18:30.419+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:18:30.990 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:18:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:18:30.607+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:18:30.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:18:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:18:30.607+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:18:30.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:18:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:18:30.607+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:18:30.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:18:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:18:30.595+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:18:30.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:18:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:18:30.595+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:18:30.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:18:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:18:30.595+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:18:30.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:18:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:18:30.595+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:18:31.711 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:18:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:18:31.599+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:18:31.712 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:18:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:18:31.599+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:18:31.712 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:18:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:18:31.599+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:18:31.712 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:18:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:18:31.420+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:18:31.712 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:18:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:18:31.420+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:18:31.712 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:18:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:18:31.420+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:18:31.713 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:18:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:18:31.630+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:18:31.713 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:18:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:18:31.630+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:18:31.713 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:18:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:18:31.630+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:18:31.713 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:18:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:18:31.630+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:18:32.408 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:18:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:18:32.035+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 16338 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:18:32.667 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:18:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:18:32.631+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:18:32.667 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:18:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:18:32.631+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:18:32.667 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:18:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:18:32.631+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:18:32.668 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:18:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:18:32.409+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:18:32.668 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:18:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:18:32.409+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:18:32.668 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:18:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:18:32.409+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:18:32.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:18:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:18:32.667+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:18:32.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:18:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:18:32.667+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:18:32.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:18:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:18:32.667+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:18:32.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:18:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:18:32.667+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:18:33.628 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:18:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:18:33.370+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:18:33.628 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:18:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:18:33.370+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:18:33.628 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:18:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:18:33.370+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:18:33.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:18:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:18:33.628+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:18:33.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:18:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:18:33.628+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:18:33.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:18:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:18:33.628+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:18:33.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:18:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:18:33.646+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:18:33.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:18:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:18:33.646+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:18:33.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:18:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:18:33.646+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:18:33.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:18:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:18:33.646+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:18:34.607 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:18:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:18:34.338+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:18:34.608 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:18:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:18:34.338+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:18:34.608 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:18:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:18:34.338+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:18:34.990 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:18:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:18:34.608+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:18:34.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:18:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:18:34.608+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:18:34.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:18:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:18:34.608+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:18:34.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:18:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:18:34.627+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:18:34.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:18:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:18:34.627+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:18:34.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:18:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:18:34.627+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:18:34.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:18:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:18:34.627+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:18:35.561 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:18:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:18:35.295+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:18:35.561 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:18:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:18:35.295+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:18:35.561 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:18:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:18:35.295+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:18:35.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:18:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:18:35.561+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:18:35.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:18:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:18:35.561+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:18:35.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:18:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:18:35.561+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:18:35.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:18:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:18:35.629+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:18:35.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:18:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:18:35.629+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:18:35.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:18:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:18:35.629+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:18:35.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:18:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:18:35.629+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:18:36.541 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:18:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:18:36.277+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:18:36.542 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:18:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:18:36.277+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:18:36.542 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:18:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:18:36.277+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:18:36.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:18:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:18:36.541+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:18:36.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:18:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:18:36.541+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:18:36.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:18:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:18:36.541+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:18:36.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:18:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:18:36.672+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:18:36.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:18:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:18:36.672+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:18:36.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:18:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:18:36.672+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:18:36.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:18:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:18:36.672+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:18:37.491 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:18:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:18:37.036+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 16409 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:18:37.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:18:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:18:37.271+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:18:37.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:18:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:18:37.271+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:18:37.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:18:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:18:37.271+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:18:37.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:18:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:18:37.682+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:18:37.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:18:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:18:37.682+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:18:37.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:18:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:18:37.682+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:18:37.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:18:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:18:37.682+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:18:37.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:18:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:18:37.544+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:18:37.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:18:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:18:37.544+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:18:37.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:18:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:18:37.544+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:18:38.555 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:18:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:18:38.297+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:18:38.556 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:18:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:18:38.297+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:18:38.556 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:18:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:18:38.297+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:18:38.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:18:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:18:38.719+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:18:38.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:18:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:18:38.719+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:18:38.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:18:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:18:38.719+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:18:38.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:18:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:18:38.719+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:18:38.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:18:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:18:38.555+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:18:38.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:18:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:18:38.555+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:18:38.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:18:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:18:38.555+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:18:39.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:18:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:18:39.332+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:18:39.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:18:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:18:39.332+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:18:39.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:18:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:18:39.332+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:18:39.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:18:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:18:39.523+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:18:39.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:18:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:18:39.523+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:18:39.742 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:18:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:18:39.523+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:18:40.184 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:18:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:18:39.754+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:18:40.184 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:18:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:18:39.754+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:18:40.184 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:18:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:18:39.754+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:18:40.184 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:18:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:18:39.754+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:18:40.490 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:18:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:18:40.296+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:18:40.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:18:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:18:40.296+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:18:40.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:18:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:18:40.296+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:18:40.990 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:18:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:18:40.708+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:18:40.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:18:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:18:40.708+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:18:40.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:18:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:18:40.708+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:18:40.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:18:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:18:40.708+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:18:40.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:18:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:18:40.550+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:18:40.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:18:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:18:40.550+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:18:40.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:18:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:18:40.550+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:18:41.587 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:18:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:18:41.331+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:18:41.588 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:18:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:18:41.331+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:18:41.588 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:18:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:18:41.331+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:18:41.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:18:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:18:41.587+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:18:41.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:18:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:18:41.587+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:18:41.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:18:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:18:41.587+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:18:41.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:18:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:18:41.751+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:18:41.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:18:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:18:41.751+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:18:41.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:18:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:18:41.751+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:18:41.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:18:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:18:41.752+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:18:42.379 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:18:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:18:42.039+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 16480 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:18:42.710 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:18:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:18:42.606+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:18:42.711 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:18:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:18:42.606+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:18:42.711 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:18:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:18:42.606+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:18:42.711 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:18:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:18:42.379+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:18:42.711 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:18:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:18:42.379+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:18:42.711 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:18:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:18:42.379+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:18:42.990 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:18:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:18:42.710+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:18:42.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:18:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:18:42.710+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:18:42.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:18:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:18:42.710+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:18:42.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:18:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:18:42.710+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:18:43.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:18:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:18:43.358+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:18:43.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:18:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:18:43.358+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:18:43.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:18:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:18:43.358+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:18:43.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:18:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:18:43.599+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:18:43.742 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:18:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:18:43.599+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:18:43.742 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:18:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:18:43.599+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:18:44.240 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:18:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:18:43.751+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:18:44.241 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:18:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:18:43.751+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:18:44.241 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:18:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:18:43.751+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:18:44.241 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:18:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:18:43.751+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:18:44.632 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:18:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:18:44.342+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:18:44.633 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:18:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:18:44.342+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:18:44.633 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:18:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:18:44.342+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:18:44.990 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:18:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:18:44.631+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:18:44.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:18:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:18:44.631+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:18:44.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:18:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:18:44.631+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:18:44.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:18:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:18:44.765+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:18:44.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:18:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:18:44.765+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:18:44.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:18:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:18:44.765+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:18:44.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:18:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:18:44.765+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:18:45.638 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:18:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:18:45.300+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:18:45.638 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:18:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:18:45.300+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:18:45.638 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:18:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:18:45.300+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:18:45.990 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:18:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:18:45.637+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:18:45.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:18:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:18:45.637+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:18:45.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:18:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:18:45.637+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:18:45.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:18:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:18:45.794+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:18:45.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:18:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:18:45.794+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:18:45.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:18:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:18:45.794+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:18:45.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:18:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:18:45.794+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:18:46.662 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:18:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:18:46.317+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:18:46.662 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:18:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:18:46.317+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:18:46.662 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:18:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:18:46.317+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:18:46.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:18:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:18:46.661+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:18:46.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:18:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:18:46.661+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:18:46.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:18:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:18:46.661+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:18:46.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:18:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:18:46.838+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:18:46.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:18:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:18:46.838+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:18:46.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:18:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:18:46.838+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:18:46.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:18:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:18:46.838+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:18:47.491 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:18:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:18:47.042+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 16551 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:18:47.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:18:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:18:47.283+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:18:47.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:18:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:18:47.283+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:18:47.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:18:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:18:47.283+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:18:47.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:18:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:18:47.796+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:18:47.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:18:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:18:47.796+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:18:47.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:18:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:18:47.796+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:18:47.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:18:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:18:47.796+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:18:47.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:18:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:18:47.623+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:18:47.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:18:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:18:47.623+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:18:47.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:18:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:18:47.623+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:18:48.623 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:18:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:18:48.255+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:18:48.624 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:18:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:18:48.255+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:18:48.624 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:18:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:18:48.255+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:18:48.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:18:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:18:48.622+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:18:48.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:18:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:18:48.622+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:18:48.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:18:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:18:48.622+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:18:48.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:18:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:18:48.829+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:18:48.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:18:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:18:48.829+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:18:48.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:18:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:18:48.829+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:18:48.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:18:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:18:48.830+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:18:49.632 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:18:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:18:49.277+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:18:49.632 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:18:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:18:49.277+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:18:49.633 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:18:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:18:49.277+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:18:49.886 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:18:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:18:49.631+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:18:49.887 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:18:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:18:49.631+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:18:49.887 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:18:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:18:49.631+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:18:49.887 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:18:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:18:49.878+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:18:49.887 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:18:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:18:49.878+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:18:49.888 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:18:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:18:49.878+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:18:49.888 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:18:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:18:49.878+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:18:50.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:18:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:18:50.322+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:18:50.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:18:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:18:50.322+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:18:50.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:18:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:18:50.322+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:18:50.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:18:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:18:50.650+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:18:50.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:18:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:18:50.650+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:18:50.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:18:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:18:50.650+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:18:50.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:18:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:18:50.876+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:18:50.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:18:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:18:50.876+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:18:50.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:18:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:18:50.876+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:18:50.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:18:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:18:50.876+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:18:51.691 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:18:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:18:51.304+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:18:51.692 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:18:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:18:51.304+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:18:51.692 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:18:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:18:51.304+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:18:51.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:18:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:18:51.691+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:18:51.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:18:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:18:51.691+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:18:51.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:18:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:18:51.691+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:18:51.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:18:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:18:51.876+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:18:51.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:18:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:18:51.876+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:18:51.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:18:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:18:51.876+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:18:51.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:18:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:18:51.876+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:18:52.491 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:18:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:18:52.044+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 16622 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:18:52.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:18:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:18:52.257+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:18:52.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:18:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:18:52.257+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:18:52.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:18:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:18:52.257+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:18:52.990 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:18:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:18:52.678+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:18:52.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:18:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:18:52.678+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:18:52.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:18:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:18:52.678+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:18:52.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:18:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:18:52.897+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:18:52.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:18:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:18:52.897+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:18:52.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:18:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:18:52.897+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:18:52.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:18:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:18:52.898+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:18:53.719 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:18:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:18:53.258+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:18:53.719 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:18:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:18:53.258+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:18:53.719 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:18:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:18:53.258+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:18:53.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:18:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:18:53.718+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:18:53.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:18:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:18:53.718+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:18:53.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:18:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:18:53.718+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:18:53.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:18:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:18:53.892+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:18:53.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:18:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:18:53.892+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:18:53.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:18:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:18:53.892+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:18:53.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:18:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:18:53.892+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:18:54.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:18:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:18:54.227+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:18:54.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:18:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:18:54.227+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:18:54.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:18:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:18:54.227+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:18:55.194 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:18:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:18:54.860+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:18:55.194 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:18:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:18:54.860+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:18:55.194 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:18:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:18:54.860+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:18:55.194 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:18:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:18:54.860+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:18:55.195 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:18:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:18:54.749+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:18:55.195 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:18:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:18:54.749+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:18:55.195 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:18:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:18:54.749+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:18:55.490 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:18:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:18:55.193+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:18:55.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:18:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:18:55.193+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:18:55.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:18:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:18:55.193+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:18:55.990 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:18:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:18:55.731+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:18:55.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:18:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:18:55.731+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:18:55.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:18:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:18:55.731+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:18:55.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:18:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:18:55.898+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:18:55.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:18:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:18:55.898+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:18:55.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:18:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:18:55.898+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:18:55.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:18:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:18:55.898+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:18:56.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:18:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:18:56.206+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:18:56.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:18:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:18:56.206+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:18:56.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:18:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:18:56.206+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:18:57.048 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:18:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:18:56.926+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:18:57.048 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:18:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:18:56.926+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:18:57.048 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:18:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:18:56.926+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:18:57.048 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:18:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:18:56.926+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:18:57.049 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:18:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:18:56.741+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:18:57.049 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:18:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:18:56.741+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:18:57.049 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:18:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:18:56.741+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:18:57.490 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:18:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:18:57.047+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 16693 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:18:57.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:18:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:18:57.179+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:18:57.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:18:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:18:57.179+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:18:57.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:18:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:18:57.179+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:18:57.990 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:18:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:18:57.726+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:18:57.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:18:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:18:57.726+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:18:57.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:18:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:18:57.726+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:18:57.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:18:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:18:57.967+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:18:57.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:18:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:18:57.967+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:18:57.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:18:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:18:57.967+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:18:57.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:18:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:18:57.968+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:18:58.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:18:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:18:58.143+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:18:58.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:18:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:18:58.143+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:18:58.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:18:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:18:58.143+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:18:58.990 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:18:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:18:58.699+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:18:58.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:18:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:18:58.699+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:18:58.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:18:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:18:58.699+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:18:59.491 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:18:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:18:58.994+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:18:59.491 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:18:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:18:58.994+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:18:59.491 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:18:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:18:58.994+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:18:59.491 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:18:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:18:58.994+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:18:59.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:18:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:18:59.103+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:18:59.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:18:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:18:59.103+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:18:59.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:18:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:18:59.103+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:18:59.981 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:18:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:18:59.700+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:18:59.981 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:18:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:18:59.700+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:18:59.981 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:18:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:18:59.700+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:19:00.241 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:18:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:18:59.980+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:19:00.241 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:18:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:18:59.980+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:19:00.241 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:18:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:18:59.980+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:19:00.241 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:18:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:18:59.980+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:19:00.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:19:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:19:00.083+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:19:00.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:19:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:19:00.083+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:19:00.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:19:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:19:00.083+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:19:00.983 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:19:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:19:00.712+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:19:00.983 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:19:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:19:00.712+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:19:00.983 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:19:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:19:00.712+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:19:01.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:19:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:19:01.106+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:19:01.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:19:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:19:01.106+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:19:01.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:19:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:19:01.106+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:19:01.241 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:19:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:19:00.996+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:19:01.241 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:19:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:19:00.996+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:19:01.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:19:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:19:00.996+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:19:01.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:19:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:19:00.996+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:19:01.990 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:19:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:19:01.719+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:19:01.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:19:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:19:01.719+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:19:01.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:19:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:19:01.719+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:19:02.490 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:19:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:19:02.049+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 16764 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:19:02.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:19:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:19:02.135+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:19:02.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:19:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:19:02.135+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:19:02.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:19:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:19:02.135+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:19:02.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:19:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:19:02.034+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:19:02.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:19:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:19:02.034+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:19:02.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:19:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:19:02.034+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:19:02.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:19:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:19:02.034+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:19:02.990 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:19:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:19:02.686+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:19:02.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:19:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:19:02.686+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:19:02.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:19:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:19:02.686+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:19:03.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:19:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:19:03.157+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:19:03.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:19:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:19:03.157+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:19:03.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:19:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:19:03.157+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:19:03.491 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:19:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:19:02.996+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:19:03.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:19:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:19:02.996+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:19:03.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:19:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:19:02.996+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:19:03.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:19:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:19:02.996+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:19:03.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:19:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:19:03.689+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:19:03.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:19:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:19:03.689+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:19:03.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:19:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:19:03.689+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:19:04.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:19:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:19:04.143+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:19:04.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:19:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:19:04.143+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:19:04.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:19:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:19:04.143+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:19:04.491 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:19:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:19:04.003+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:19:04.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:19:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:19:04.003+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:19:04.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:19:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:19:04.003+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:19:04.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:19:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:19:04.003+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:19:04.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:19:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:19:04.677+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:19:04.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:19:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:19:04.677+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:19:04.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:19:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:19:04.677+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:19:05.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:19:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:19:05.145+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:19:05.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:19:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:19:05.145+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:19:05.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:19:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:19:05.145+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:19:05.491 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:19:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:19:05.013+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:19:05.491 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:19:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:19:05.013+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:19:05.491 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:19:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:19:05.013+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:19:05.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:19:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:19:05.013+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:19:05.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:19:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:19:05.630+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:19:05.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:19:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:19:05.630+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:19:05.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:19:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:19:05.630+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:19:06.490 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:19:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:19:06.122+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:19:06.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:19:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:19:06.122+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:19:06.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:19:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:19:06.122+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:19:06.491 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:19:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:19:06.050+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:19:06.491 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:19:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:19:06.050+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:19:06.491 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:19:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:19:06.050+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:19:06.491 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:19:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:19:06.050+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:19:06.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:19:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:19:06.624+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:19:06.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:19:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:19:06.624+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:19:06.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:19:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:19:06.624+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:19:07.491 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:19:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:19:07.053+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 16835 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:19:07.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:19:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:19:07.149+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:19:07.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:19:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:19:07.149+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:19:07.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:19:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:19:07.149+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:19:07.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:19:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:19:07.058+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:19:07.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:19:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:19:07.058+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:19:07.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:19:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:19:07.058+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:19:07.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:19:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:19:07.058+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:19:07.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:19:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:19:07.666+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:19:07.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:19:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:19:07.666+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:19:07.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:19:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:19:07.666+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:19:08.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:19:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:19:08.168+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:19:08.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:19:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:19:08.168+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:19:08.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:19:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:19:08.168+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:19:08.491 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:19:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:19:08.018+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:19:08.491 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:19:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:19:08.018+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:19:08.491 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:19:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:19:08.018+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:19:08.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:19:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:19:08.019+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:19:08.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:19:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:19:08.676+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:19:08.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:19:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:19:08.676+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:19:08.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:19:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:19:08.676+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:19:09.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:19:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:19:09.198+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:19:09.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:19:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:19:09.198+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:19:09.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:19:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:19:09.198+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:19:09.491 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:19:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:19:09.049+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:19:09.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:19:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:19:09.049+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:19:09.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:19:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:19:09.049+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:19:09.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:19:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:19:09.049+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:19:09.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:19:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:19:09.670+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:19:09.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:19:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:19:09.670+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:19:09.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:19:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:19:09.670+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:19:10.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:19:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:19:10.170+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:19:10.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:19:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:19:10.170+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:19:10.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:19:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:19:10.170+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:19:10.491 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:19:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:19:10.090+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:19:10.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:19:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:19:10.090+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:19:10.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:19:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:19:10.090+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:19:10.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:19:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:19:10.090+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:19:10.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:19:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:19:10.662+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:19:11.052 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:19:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:19:10.662+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:19:11.052 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:19:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:19:10.662+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:19:11.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:19:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:19:11.157+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:19:11.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:19:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:19:11.157+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:19:11.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:19:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:19:11.157+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:19:11.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:19:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:19:11.086+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:19:11.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:19:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:19:11.086+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:19:11.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:19:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:19:11.086+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:19:11.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:19:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:19:11.086+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:19:11.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:19:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:19:11.708+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:19:11.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:19:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:19:11.708+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:19:11.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:19:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:19:11.708+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:19:12.491 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:19:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:19:12.055+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 16906 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:19:12.491 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:19:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:19:12.097+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:19:12.491 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:19:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:19:12.097+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:19:12.491 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:19:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:19:12.097+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:19:12.491 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:19:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:19:12.097+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:19:12.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:19:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:19:12.133+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:19:12.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:19:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:19:12.133+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:19:12.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:19:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:19:12.133+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:19:12.990 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:19:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:19:12.715+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:19:12.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:19:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:19:12.715+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:19:12.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:19:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:19:12.715+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:19:13.491 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:19:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:19:13.092+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:19:13.491 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:19:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:19:13.092+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:19:13.491 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:19:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:19:13.092+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:19:13.491 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:19:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:19:13.092+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:19:13.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:19:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:19:13.122+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:19:13.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:19:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:19:13.122+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:19:13.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:19:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:19:13.122+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:19:13.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:19:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:19:13.666+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:19:13.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:19:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:19:13.666+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:19:13.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:19:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:19:13.666+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:19:14.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:19:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:19:14.092+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:19:14.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:19:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:19:14.092+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:19:14.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:19:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:19:14.092+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:19:14.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:19:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:19:14.079+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:19:14.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:19:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:19:14.079+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:19:14.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:19:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:19:14.079+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:19:14.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:19:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:19:14.079+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:19:14.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:19:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:19:14.686+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:19:14.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:19:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:19:14.686+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:19:14.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:19:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:19:14.686+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:19:15.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:19:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:19:15.049+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:19:15.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:19:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:19:15.049+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:19:15.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:19:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:19:15.049+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:19:15.491 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:19:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:19:15.086+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:19:15.491 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:19:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:19:15.086+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:19:15.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:19:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:19:15.086+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:19:15.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:19:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:19:15.086+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:19:15.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:19:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:19:15.668+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:19:15.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:19:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:19:15.668+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:19:15.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:19:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:19:15.668+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:19:16.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:19:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:19:16.090+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:19:16.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:19:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:19:16.090+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:19:16.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:19:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:19:16.090+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:19:16.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:19:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:19:16.117+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:19:16.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:19:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:19:16.117+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:19:16.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:19:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:19:16.117+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:19:16.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:19:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:19:16.117+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:19:16.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:19:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:19:16.632+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:19:16.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:19:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:19:16.632+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:19:16.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:19:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:19:16.632+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:19:17.491 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:19:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:19:17.058+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 16977 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:19:17.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:19:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:19:17.069+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:19:17.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:19:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:19:17.069+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:19:17.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:19:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:19:17.069+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:19:17.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:19:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:19:17.069+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:19:17.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:19:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:19:17.070+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:19:17.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:19:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:19:17.070+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:19:17.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:19:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:19:17.070+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:19:17.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:19:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:19:17.613+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:19:17.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:19:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:19:17.613+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:19:17.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:19:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:19:17.613+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:19:18.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:19:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:19:18.060+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:19:18.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:19:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:19:18.060+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:19:18.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:19:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:19:18.060+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:19:18.491 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:19:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:19:18.069+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:19:18.491 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:19:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:19:18.069+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:19:18.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:19:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:19:18.069+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:19:18.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:19:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:19:18.069+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:19:18.990 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:19:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:19:18.565+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:19:18.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:19:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:19:18.565+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:19:18.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:19:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:19:18.565+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:19:19.490 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:19:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:19:19.021+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:19:19.491 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:19:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:19:19.021+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:19:19.491 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:19:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:19:19.021+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:19:19.491 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:19:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:19:19.021+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:19:19.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:19:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:19:19.040+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:19:19.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:19:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:19:19.040+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:19:19.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:19:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:19:19.040+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:19:19.884 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:19:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:19:19.543+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:19:19.884 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:19:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:19:19.543+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:19:19.885 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:19:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:19:19.543+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:19:20.187 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:19:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:19:19.996+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:19:20.188 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:19:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:19:19.996+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:19:20.188 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:19:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:19:19.996+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:19:20.188 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:19:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:19:20.022+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:19:20.188 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:19:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:19:20.022+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:19:20.188 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:19:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:19:20.022+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:19:20.189 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:19:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:19:20.023+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:19:20.990 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:19:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:19:20.525+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:19:20.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:19:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:19:20.525+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:19:20.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:19:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:19:20.525+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:19:21.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:19:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:19:21.011+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:19:21.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:19:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:19:21.011+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:19:21.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:19:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:19:21.011+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:19:21.491 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:19:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:19:21.007+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:19:21.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:19:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:19:21.007+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:19:21.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:19:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:19:21.007+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:19:21.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:19:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:19:21.007+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:19:21.990 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:19:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:19:21.501+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:19:21.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:19:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:19:21.501+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:19:21.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:19:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:19:21.501+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:19:22.461 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:19:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:19:22.060+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 17048 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:19:22.462 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:19:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:19:22.041+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:19:22.462 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:19:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:19:22.041+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:19:22.462 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:19:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:19:22.041+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:19:22.462 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:19:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:19:22.041+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:19:22.462 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:19:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:19:22.005+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:19:22.462 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:19:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:19:22.005+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:19:22.462 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:19:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:19:22.005+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:19:22.740 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:19:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:19:22.461+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:19:22.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:19:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:19:22.461+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:19:22.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:19:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:19:22.461+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:19:23.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:19:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:19:22.961+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:19:23.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:19:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:19:22.961+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:19:23.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:19:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:19:22.961+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:19:23.241 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:19:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:19:23.045+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:19:23.241 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:19:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:19:23.045+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:19:23.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:19:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:19:23.045+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:19:23.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:19:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:19:23.046+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:19:23.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:19:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:19:23.464+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:19:23.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:19:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:19:23.464+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:19:23.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:19:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:19:23.464+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:19:24.241 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:19:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:19:24.033+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:19:24.241 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:19:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:19:24.033+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:19:24.241 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:19:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:19:24.033+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:19:24.241 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:19:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:19:24.033+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:19:24.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:19:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:19:23.966+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:19:24.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:19:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:19:23.966+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:19:24.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:19:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:19:23.966+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:19:24.740 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:19:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:19:24.446+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:19:24.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:19:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:19:24.446+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:19:24.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:19:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:19:24.446+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:19:25.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:19:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:19:24.919+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:19:25.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:19:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:19:24.920+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:19:25.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:19:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:19:24.920+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:19:25.241 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:19:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:19:25.037+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:19:25.241 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:19:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:19:25.037+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:19:25.241 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:19:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:19:25.037+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:19:25.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:19:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:19:25.037+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:19:25.740 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:19:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:19:25.464+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:19:25.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:19:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:19:25.464+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:19:25.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:19:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:19:25.464+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:19:26.240 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:19:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:19:25.900+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:19:26.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:19:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:19:25.900+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:19:26.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:19:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:19:25.900+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:19:26.241 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:19:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:19:26.033+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:19:26.241 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:19:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:19:26.033+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:19:26.241 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:19:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:19:26.033+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:19:26.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:19:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:19:26.033+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:19:26.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:19:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:19:26.479+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:19:26.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:19:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:19:26.479+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:19:26.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:19:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:19:26.479+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:19:27.241 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:19:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:19:27.063+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 17118 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:19:27.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:19:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:19:26.943+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:19:27.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:19:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:19:26.943+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:19:27.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:19:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:19:26.943+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:19:27.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:19:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:19:27.011+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:19:27.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:19:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:19:27.011+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:19:27.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:19:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:19:27.011+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:19:27.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:19:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:19:27.011+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:19:27.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:19:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:19:27.457+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:19:27.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:19:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:19:27.457+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:19:27.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:19:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:19:27.457+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:19:28.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:19:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:19:27.960+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:19:28.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:19:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:19:27.960+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:19:28.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:19:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:19:27.960+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:19:28.241 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:19:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:19:27.999+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:19:28.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:19:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:19:27.999+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:19:28.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:19:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:19:27.999+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:19:28.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:19:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:19:27.999+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:19:28.740 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:19:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:19:28.439+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:19:28.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:19:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:19:28.439+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:19:28.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:19:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:19:28.439+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:19:29.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:19:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:19:28.923+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:19:29.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:19:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:19:28.923+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:19:29.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:19:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:19:28.923+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:19:29.241 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:19:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:19:28.981+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:19:29.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:19:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:19:28.981+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:19:29.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:19:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:19:28.981+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:19:29.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:19:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:19:28.981+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:19:29.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:19:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:19:29.484+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:19:29.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:19:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:19:29.484+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:19:29.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:19:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:19:29.484+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:19:30.188 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:19:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:19:29.947+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:19:30.188 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:19:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:19:29.947+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:19:30.188 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:19:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:19:29.947+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:19:30.189 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:19:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:19:29.947+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:19:30.189 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:19:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:19:29.918+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:19:30.189 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:19:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:19:29.918+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:19:30.189 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:19:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:19:29.918+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:19:30.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:19:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:19:30.481+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:19:30.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:19:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:19:30.481+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:19:30.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:19:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:19:30.481+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:19:31.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:19:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:19:30.966+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:19:31.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:19:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:19:30.966+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:19:31.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:19:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:19:30.966+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:19:31.241 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:19:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:19:30.938+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:19:31.241 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:19:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:19:30.938+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:19:31.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:19:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:19:30.938+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:19:31.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:19:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:19:30.938+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:19:31.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:19:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:19:31.482+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:19:31.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:19:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:19:31.482+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:19:31.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:19:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:19:31.482+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:19:32.187 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:19:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:19:32.064+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 17190 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:19:32.188 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:19:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:19:31.920+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:19:32.188 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:19:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:19:31.920+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:19:32.188 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:19:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:19:31.920+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:19:32.188 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:19:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:19:31.965+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:19:32.189 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:19:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:19:31.965+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:19:32.189 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:19:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:19:31.965+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:19:32.189 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:19:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:19:31.965+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:19:32.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:19:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:19:32.465+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:19:32.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:19:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:19:32.465+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:19:32.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:19:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:19:32.465+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:19:33.241 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:19:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:19:33.010+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:19:33.241 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:19:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:19:33.010+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:19:33.241 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:19:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:19:33.010+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:19:33.241 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:19:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:19:33.010+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:19:33.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:19:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:19:32.942+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:19:33.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:19:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:19:32.942+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:19:33.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:19:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:19:32.942+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:19:33.983 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:19:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:19:33.502+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:19:33.983 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:19:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:19:33.502+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:19:33.983 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:19:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:19:33.502+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:19:34.240 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:19:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:19:33.991+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:19:34.241 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:19:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:19:33.991+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:19:34.241 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:19:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:19:33.991+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:19:34.241 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:19:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:19:33.991+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:19:34.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:19:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:19:33.982+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:19:34.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:19:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:19:33.982+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:19:34.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:19:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:19:33.982+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:19:34.740 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:19:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:19:34.472+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:19:34.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:19:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:19:34.472+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:19:34.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:19:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:19:34.472+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:19:35.241 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:19:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:19:34.949+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:19:35.241 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:19:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:19:34.949+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:19:35.241 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:19:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:19:34.949+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:19:35.241 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:19:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:19:34.949+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:19:35.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:19:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:19:34.951+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:19:35.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:19:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:19:34.952+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:19:35.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:19:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:19:34.952+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:19:35.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:19:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:19:35.484+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:19:35.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:19:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:19:35.484+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:19:35.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:19:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:19:35.484+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:19:36.241 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:19:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:19:35.899+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:19:36.241 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:19:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:19:35.899+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:19:36.241 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:19:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:19:35.899+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:19:36.241 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:19:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:19:35.899+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:19:36.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:19:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:19:35.938+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:19:36.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:19:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:19:35.938+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:19:36.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:19:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:19:35.938+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:19:36.740 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:19:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:19:36.468+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:19:36.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:19:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:19:36.468+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:19:36.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:19:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:19:36.468+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:19:37.190 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:19:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:19:37.066+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 17260 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:19:37.191 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:19:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:19:36.902+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:19:37.191 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:19:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:19:36.902+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:19:37.191 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:19:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:19:36.902+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:19:37.191 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:19:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:19:36.859+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:19:37.191 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:19:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:19:36.859+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:19:37.191 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:19:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:19:36.859+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:19:37.192 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:19:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:19:36.859+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:19:37.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:19:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:19:37.461+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:19:37.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:19:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:19:37.461+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:19:37.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:19:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:19:37.461+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:19:38.240 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:19:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:19:37.947+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:19:38.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:19:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:19:37.947+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:19:38.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:19:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:19:37.947+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:19:38.241 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:19:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:19:37.883+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:19:38.241 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:19:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:19:37.883+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:19:38.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:19:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:19:37.883+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:19:38.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:19:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:19:37.883+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:19:38.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:19:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:19:38.416+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:19:38.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:19:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:19:38.416+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:19:38.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:19:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:19:38.416+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:19:39.241 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:19:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:19:38.887+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:19:39.241 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:19:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:19:38.887+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:19:39.241 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:19:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:19:38.887+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:19:39.241 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:19:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:19:38.888+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:19:39.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:19:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:19:38.907+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:19:39.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:19:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:19:38.907+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:19:39.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:19:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:19:38.907+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:19:39.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:19:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:19:39.438+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:19:39.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:19:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:19:39.438+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:19:39.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:19:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:19:39.438+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:19:40.189 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:19:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:19:39.932+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:19:40.189 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:19:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:19:39.932+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:19:40.189 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:19:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:19:39.932+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:19:40.190 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:19:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:19:39.886+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:19:40.190 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:19:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:19:39.886+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:19:40.190 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:19:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:19:39.886+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:19:40.190 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:19:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:19:39.886+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:19:40.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:19:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:19:40.431+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:19:40.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:19:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:19:40.431+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:19:40.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:19:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:19:40.431+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:19:41.241 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:19:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:19:40.899+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:19:41.241 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:19:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:19:40.899+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:19:41.241 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:19:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:19:40.899+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:19:41.241 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:19:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:19:40.899+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:19:41.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:19:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:19:40.933+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:19:41.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:19:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:19:40.933+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:19:41.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:19:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:19:40.933+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:19:41.719 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:19:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:19:41.398+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:19:41.719 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:19:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:19:41.398+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:19:41.719 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:19:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:19:41.398+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:19:41.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:19:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:19:41.954+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:19:41.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:19:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:19:41.954+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:19:41.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:19:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:19:41.954+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:19:41.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:19:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:19:41.904+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:19:41.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:19:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:19:41.904+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:19:41.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:19:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:19:41.904+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:19:41.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:19:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:19:41.904+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:19:42.445 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:19:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:19:42.069+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 17329 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:19:42.744 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:19:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:19:42.445+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:19:42.744 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:19:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:19:42.445+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:19:42.744 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:19:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:19:42.445+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:19:43.241 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:19:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:19:42.866+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:19:43.241 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:19:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:19:42.866+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:19:43.241 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:19:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:19:42.866+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:19:43.241 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:19:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:19:42.866+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:19:43.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:19:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:19:42.917+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:19:43.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:19:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:19:42.917+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:19:43.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:19:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:19:42.917+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:19:43.740 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:19:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:19:43.431+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:19:43.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:19:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:19:43.431+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:19:43.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:19:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:19:43.431+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:19:44.241 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:19:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:19:43.913+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:19:44.241 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:19:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:19:43.913+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:19:44.241 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:19:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:19:43.913+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:19:44.241 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:19:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:19:43.913+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:19:44.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:19:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:19:43.886+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:19:44.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:19:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:19:43.886+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:19:44.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:19:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:19:43.886+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:19:44.740 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:19:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:19:44.417+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:19:44.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:19:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:19:44.417+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:19:44.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:19:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:19:44.417+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:19:45.241 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:19:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:19:44.908+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:19:45.241 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:19:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:19:44.908+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:19:45.241 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:19:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:19:44.908+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:19:45.241 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:19:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:19:44.908+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:19:45.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:19:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:19:44.882+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:19:45.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:19:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:19:44.882+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:19:45.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:19:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:19:44.882+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:19:45.740 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:19:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:19:45.404+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:19:45.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:19:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:19:45.404+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:19:45.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:19:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:19:45.404+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:19:46.241 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:19:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:19:45.868+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:19:46.241 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:19:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:19:45.868+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:19:46.241 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:19:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:19:45.868+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:19:46.241 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:19:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:19:45.868+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:19:46.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:19:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:19:45.845+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:19:46.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:19:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:19:45.845+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:19:46.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:19:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:19:45.845+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:19:46.740 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:19:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:19:46.444+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:19:46.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:19:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:19:46.444+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:19:46.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:19:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:19:46.444+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:19:47.073 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:19:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:19:46.814+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:19:47.073 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:19:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:19:46.814+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:19:47.073 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:19:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:19:46.815+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:19:47.073 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:19:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:19:46.854+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:19:47.073 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:19:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:19:46.854+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:19:47.074 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:19:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:19:46.854+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:19:47.074 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:19:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:19:46.855+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:19:47.459 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:19:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:19:47.071+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 17396 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:19:47.740 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:19:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:19:47.458+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:19:47.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:19:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:19:47.458+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:19:47.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:19:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:19:47.458+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:19:48.241 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:19:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:19:47.899+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:19:48.241 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:19:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:19:47.899+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:19:48.241 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:19:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:19:47.899+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:19:48.241 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:19:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:19:47.899+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:19:48.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:19:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:19:47.860+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:19:48.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:19:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:19:47.860+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:19:48.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:19:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:19:47.860+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:19:48.740 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:19:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:19:48.430+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:19:48.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:19:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:19:48.430+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:19:48.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:19:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:19:48.430+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:19:49.241 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:19:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:19:48.919+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:19:49.241 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:19:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:19:48.919+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:19:49.241 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:19:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:19:48.919+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:19:49.241 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:19:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:19:48.919+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:19:49.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:19:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:19:48.823+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:19:49.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:19:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:19:48.823+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:19:49.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:19:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:19:48.823+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:19:49.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:19:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:19:49.445+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:19:49.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:19:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:19:49.445+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:19:49.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:19:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:19:49.445+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:19:50.190 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:19:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:19:49.782+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:19:50.190 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:19:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:19:49.782+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:19:50.190 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:19:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:19:49.782+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:19:50.191 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:19:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:19:49.910+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:19:50.191 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:19:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:19:49.910+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:19:50.191 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:19:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:19:49.910+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:19:50.191 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:19:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:19:49.910+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:19:50.740 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:19:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:19:50.467+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:19:50.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:19:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:19:50.467+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:19:50.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:19:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:19:50.467+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:19:51.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:19:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:19:50.831+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:19:51.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:19:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:19:50.831+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:19:51.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:19:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:19:50.831+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:19:51.241 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:19:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:19:50.862+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:19:51.241 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:19:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:19:50.862+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:19:51.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:19:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:19:50.862+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:19:51.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:19:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:19:50.862+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:19:51.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:19:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:19:51.489+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:19:51.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:19:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:19:51.489+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:19:51.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:19:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:19:51.489+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:19:52.086 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:19:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:19:52.074+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 17465 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:19:52.086 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:19:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:19:51.854+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:19:52.086 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:19:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:19:51.854+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:19:52.086 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:19:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:19:51.854+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:19:52.087 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:19:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:19:51.829+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:19:52.087 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:19:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:19:51.829+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:19:52.087 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:19:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:19:51.829+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:19:52.087 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:19:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:19:51.829+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:19:52.841 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:19:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:19:52.536+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:19:52.841 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:19:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:19:52.536+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:19:52.841 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:19:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:19:52.536+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:19:53.207 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:19:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:19:52.840+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:19:53.207 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:19:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:19:52.840+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:19:53.207 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:19:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:19:52.840+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:19:53.208 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:19:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:19:52.877+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:19:53.208 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:19:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:19:52.877+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:19:53.208 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:19:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:19:52.877+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:19:53.208 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:19:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:19:52.877+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:19:53.828 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:19:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:19:53.563+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:19:53.828 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:19:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:19:53.563+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:19:53.828 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:19:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:19:53.563+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:19:54.240 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:19:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:19:53.855+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:19:54.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:19:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:19:53.855+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:19:54.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:19:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:19:53.855+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:19:54.241 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:19:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:19:53.828+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:19:54.241 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:19:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:19:53.828+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:19:54.241 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:19:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:19:53.828+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:19:54.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:19:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:19:53.828+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:19:54.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:19:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:19:54.824+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:19:54.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:19:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:19:54.824+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:19:54.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:19:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:19:54.824+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:19:54.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:19:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:19:54.824+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:19:54.992 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:19:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:19:54.820+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:19:54.992 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:19:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:19:54.820+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:19:54.992 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:19:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:19:54.820+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:19:54.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:19:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:19:54.603+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:19:54.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:19:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:19:54.603+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:19:54.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:19:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:19:54.603+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:19:55.990 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:19:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:19:55.587+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:19:55.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:19:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:19:55.587+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:19:55.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:19:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:19:55.587+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:19:55.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:19:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:19:55.798+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:19:55.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:19:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:19:55.798+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:19:55.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:19:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:19:55.798+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:19:55.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:19:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:19:55.798+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:19:55.992 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:19:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:19:55.800+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:19:55.992 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:19:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:19:55.800+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:19:55.992 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:19:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:19:55.800+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:19:56.843 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:19:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:19:56.565+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:19:56.843 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:19:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:19:56.565+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:19:56.844 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:19:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:19:56.565+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:19:56.844 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:19:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:19:56.797+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:19:56.844 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:19:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:19:56.797+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:19:56.844 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:19:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:19:56.797+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:19:56.844 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:19:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:19:56.797+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:19:57.099 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:19:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:19:57.077+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 17529 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:19:57.099 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:19:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:19:56.843+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:19:57.099 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:19:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:19:56.843+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:19:57.100 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:19:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:19:56.843+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:19:57.854 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:19:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:19:57.600+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:19:57.854 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:19:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:19:57.600+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:19:57.854 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:19:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:19:57.600+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:19:57.855 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:19:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:19:57.841+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:19:57.855 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:19:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:19:57.841+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:19:57.855 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:19:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:19:57.841+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:19:57.855 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:19:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:19:57.841+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:19:58.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:19:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:19:57.854+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:19:58.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:19:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:19:57.854+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:19:58.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:19:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:19:57.854+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:19:58.888 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:19:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:19:58.627+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:19:58.888 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:19:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:19:58.627+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:19:58.889 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:19:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:19:58.627+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:19:58.889 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:19:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:19:58.793+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:19:58.889 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:19:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:19:58.793+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:19:58.889 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:19:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:19:58.793+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:19:58.889 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:19:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:19:58.793+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:19:59.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:19:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:19:58.888+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:19:59.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:19:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:19:58.888+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:19:59.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:19:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:19:58.888+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:19:59.884 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:19:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:19:59.617+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:19:59.885 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:19:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:19:59.617+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:19:59.885 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:19:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:19:59.617+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:19:59.885 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:19:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:19:59.776+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:19:59.885 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:19:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:19:59.776+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:19:59.885 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:19:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:19:59.776+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:19:59.885 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:19:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:19:59.777+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:20:00.190 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:19:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:19:59.892+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:20:00.190 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:19:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:19:59.892+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:20:00.191 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:19:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:19:59.892+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:20:00.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:20:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:20:00.647+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:20:00.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:20:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:20:00.647+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:20:00.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:20:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:20:00.647+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:20:00.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:20:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:20:00.868+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:20:00.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:20:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:20:00.868+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:20:00.992 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:20:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:20:00.868+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:20:00.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:20:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:20:00.748+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:20:00.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:20:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:20:00.748+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:20:00.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:20:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:20:00.748+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:20:00.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:20:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:20:00.748+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:20:01.898 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:20:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:20:01.778+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:20:01.898 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:20:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:20:01.778+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:20:01.898 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:20:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:20:01.778+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:20:01.898 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:20:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:20:01.778+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:20:01.898 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:20:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:20:01.610+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:20:01.899 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:20:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:20:01.610+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:20:01.899 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:20:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:20:01.610+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:20:02.203 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:20:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:20:02.080+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 17595 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:20:02.203 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:20:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:20:01.898+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:20:02.204 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:20:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:20:01.898+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:20:02.204 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:20:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:20:01.898+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:20:02.857 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:20:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:20:02.581+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:20:02.858 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:20:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:20:02.581+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:20:02.858 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:20:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:20:02.581+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:20:02.858 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:20:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:20:02.805+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:20:02.858 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:20:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:20:02.805+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:20:02.858 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:20:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:20:02.805+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:20:02.859 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:20:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:20:02.805+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:20:03.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:20:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:20:02.857+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:20:03.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:20:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:20:02.857+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:20:03.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:20:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:20:02.857+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:20:03.898 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:20:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:20:03.618+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:20:03.898 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:20:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:20:03.618+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:20:03.898 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:20:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:20:03.618+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:20:03.899 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:20:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:20:03.812+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:20:03.899 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:20:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:20:03.812+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:20:03.899 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:20:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:20:03.812+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:20:03.900 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:20:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:20:03.812+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:20:04.240 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:20:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:20:03.898+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:20:04.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:20:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:20:03.898+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:20:04.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:20:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:20:03.898+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:20:04.934 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:20:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:20:04.578+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:20:04.934 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:20:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:20:04.578+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:20:04.935 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:20:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:20:04.578+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:20:04.935 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:20:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:20:04.813+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:20:04.935 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:20:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:20:04.813+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:20:04.936 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:20:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:20:04.813+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:20:04.936 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:20:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:20:04.813+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:20:05.240 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:20:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:20:04.934+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:20:05.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:20:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:20:04.934+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:20:05.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:20:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:20:04.934+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:20:05.802 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:20:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:20:05.547+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:20:05.802 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:20:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:20:05.547+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:20:05.803 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:20:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:20:05.547+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:20:06.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:20:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:20:05.948+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:20:06.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:20:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:20:05.948+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:20:06.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:20:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:20:05.948+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:20:06.241 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:20:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:20:05.802+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:20:06.241 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:20:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:20:05.802+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:20:06.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:20:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:20:05.802+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:20:06.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:20:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:20:05.802+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:20:06.804 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:20:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:20:06.552+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:20:06.804 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:20:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:20:06.552+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:20:06.804 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:20:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:20:06.552+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:20:07.083 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:20:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:20:06.963+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:20:07.083 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:20:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:20:06.963+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:20:07.084 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:20:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:20:06.963+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:20:07.084 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:20:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:20:06.804+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:20:07.084 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:20:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:20:06.804+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:20:07.084 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:20:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:20:06.804+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:20:07.084 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:20:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:20:06.804+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:20:07.490 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:20:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:20:07.083+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 17659 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:20:07.824 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:20:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:20:07.554+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:20:07.824 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:20:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:20:07.554+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:20:07.824 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:20:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:20:07.554+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:20:08.240 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:20:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:20:07.997+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:20:08.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:20:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:20:07.997+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:20:08.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:20:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:20:07.997+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:20:08.241 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:20:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:20:07.824+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:20:08.241 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:20:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:20:07.824+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:20:08.241 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:20:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:20:07.824+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:20:08.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:20:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:20:07.824+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:20:08.799 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:20:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:20:08.518+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:20:08.800 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:20:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:20:08.518+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:20:08.800 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:20:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:20:08.518+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:20:09.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:20:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:20:09.046+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:20:09.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:20:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:20:09.046+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:20:09.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:20:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:20:09.046+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:20:09.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:20:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:20:08.799+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:20:09.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:20:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:20:08.799+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:20:09.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:20:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:20:08.799+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:20:09.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:20:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:20:08.799+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:20:09.885 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:20:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:20:09.553+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:20:09.885 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:20:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:20:09.553+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:20:09.885 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:20:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:20:09.553+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:20:09.886 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:20:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:20:09.789+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:20:09.886 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:20:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:20:09.790+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:20:09.886 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:20:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:20:09.790+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:20:09.887 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:20:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:20:09.790+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:20:10.191 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:20:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:20:10.020+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:20:10.191 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:20:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:20:10.020+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:20:10.191 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:20:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:20:10.020+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:20:10.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:20:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:20:10.507+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:20:10.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:20:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:20:10.507+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:20:10.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:20:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:20:10.507+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:20:10.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:20:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:20:10.744+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:20:10.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:20:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:20:10.744+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:20:10.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:20:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:20:10.744+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:20:10.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:20:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:20:10.744+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:20:11.490 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:20:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:20:11.010+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:20:11.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:20:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:20:11.010+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:20:11.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:20:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:20:11.010+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:20:11.751 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:20:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:20:11.494+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:20:11.752 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:20:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:20:11.494+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:20:11.752 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:20:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:20:11.494+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:20:12.086 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:20:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:20:11.751+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:20:12.086 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:20:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:20:11.751+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:20:12.086 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:20:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:20:11.751+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:20:12.086 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:20:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:20:11.752+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:20:12.087 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:20:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:20:12.001+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:20:12.087 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:20:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:20:12.001+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:20:12.087 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:20:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:20:12.001+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:20:12.490 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:20:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:20:12.086+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 17725 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:20:12.798 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:20:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:20:12.495+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:20:12.799 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:20:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:20:12.495+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:20:12.799 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:20:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:20:12.495+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:20:13.241 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:20:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:20:12.798+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:20:13.241 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:20:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:20:12.798+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:20:13.241 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:20:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:20:12.798+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:20:13.241 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:20:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:20:12.799+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:20:13.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:20:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:20:12.986+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:20:13.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:20:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:20:12.986+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:20:13.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:20:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:20:12.986+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:20:13.845 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:20:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:20:13.523+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:20:13.845 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:20:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:20:13.523+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:20:13.845 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:20:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:20:13.523+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:20:14.241 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:20:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:20:13.845+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:20:14.241 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:20:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:20:13.845+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:20:14.241 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:20:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:20:13.845+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:20:14.241 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:20:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:20:13.845+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:20:14.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:20:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:20:14.006+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:20:14.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:20:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:20:14.006+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:20:14.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:20:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:20:14.006+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:20:14.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:20:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:20:14.795+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:20:14.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:20:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:20:14.795+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:20:14.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:20:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:20:14.795+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:20:14.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:20:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:20:14.795+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:20:14.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:20:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:20:14.557+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:20:14.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:20:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:20:14.557+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:20:14.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:20:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:20:14.557+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:20:15.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:20:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:20:15.013+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:20:15.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:20:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:20:15.013+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:20:15.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:20:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:20:15.013+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:20:15.990 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:20:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:20:15.585+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:20:15.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:20:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:20:15.585+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:20:15.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:20:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:20:15.585+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:20:15.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:20:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:20:15.804+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:20:15.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:20:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:20:15.804+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:20:15.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:20:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:20:15.804+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:20:15.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:20:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:20:15.804+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:20:16.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:20:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:20:16.034+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:20:16.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:20:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:20:16.034+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:20:16.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:20:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:20:16.034+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:20:16.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:20:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:20:16.560+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:20:16.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:20:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:20:16.560+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:20:16.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:20:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:20:16.560+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:20:16.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:20:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:20:16.783+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:20:16.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:20:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:20:16.783+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:20:16.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:20:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:20:16.783+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:20:16.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:20:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:20:16.783+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:20:17.491 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:20:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:20:17.088+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 17792 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:20:17.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:20:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:20:17.066+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:20:17.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:20:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:20:17.066+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:20:17.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:20:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:20:17.066+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:20:17.780 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:20:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:20:17.513+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:20:17.780 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:20:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:20:17.513+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:20:17.780 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:20:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:20:17.513+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:20:18.091 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:20:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:20:17.780+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:20:18.092 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:20:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:20:17.780+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:20:18.092 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:20:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:20:17.780+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:20:18.092 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:20:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:20:17.780+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:20:18.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:20:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:20:18.092+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:20:18.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:20:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:20:18.092+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:20:18.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:20:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:20:18.092+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:20:18.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:20:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:20:18.536+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:20:18.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:20:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:20:18.536+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:20:18.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:20:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:20:18.536+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:20:18.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:20:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:20:18.734+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:20:18.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:20:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:20:18.734+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:20:18.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:20:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:20:18.734+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:20:18.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:20:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:20:18.734+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:20:19.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:20:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:20:19.115+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:20:19.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:20:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:20:19.115+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:20:19.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:20:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:20:19.115+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:20:19.885 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:20:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:20:19.532+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:20:19.885 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:20:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:20:19.532+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:20:19.885 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:20:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:20:19.532+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:20:19.885 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:20:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:20:19.736+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:20:19.885 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:20:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:20:19.736+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:20:19.886 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:20:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:20:19.736+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:20:19.886 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:20:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:20:19.736+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:20:20.192 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:20:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:20:20.071+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:20:20.192 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:20:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:20:20.071+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:20:20.192 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:20:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:20:20.071+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:20:20.856 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:20:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:20:20.565+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:20:20.856 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:20:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:20:20.565+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:20:20.856 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:20:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:20:20.565+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:20:20.856 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:20:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:20:20.768+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:20:20.857 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:20:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:20:20.768+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:20:20.857 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:20:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:20:20.768+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:20:20.857 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:20:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:20:20.768+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:20:21.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:20:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:20:21.101+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:20:21.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:20:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:20:21.101+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:20:21.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:20:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:20:21.101+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:20:21.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:20:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:20:21.569+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:20:21.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:20:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:20:21.569+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:20:21.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:20:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:20:21.569+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:20:21.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:20:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:20:21.745+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:20:21.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:20:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:20:21.745+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:20:21.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:20:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:20:21.745+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:20:21.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:20:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:20:21.745+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:20:22.491 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:20:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:20:22.090+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 17861 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:20:22.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:20:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:20:22.140+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:20:22.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:20:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:20:22.140+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:20:22.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:20:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:20:22.140+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:20:22.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:20:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:20:22.599+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:20:22.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:20:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:20:22.599+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:20:22.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:20:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:20:22.599+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:20:22.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:20:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:20:22.732+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:20:22.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:20:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:20:22.732+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:20:22.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:20:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:20:22.732+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:20:22.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:20:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:20:22.732+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:20:23.490 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:20:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:20:23.096+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:20:23.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:20:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:20:23.096+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:20:23.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:20:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:20:23.096+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:20:23.927 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:20:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:20:23.624+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:20:23.927 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:20:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:20:23.624+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:20:23.927 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:20:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:20:23.624+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:20:23.928 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:20:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:20:23.739+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:20:23.928 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:20:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:20:23.739+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:20:23.928 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:20:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:20:23.739+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:20:23.928 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:20:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:20:23.739+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:20:24.240 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:20:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:20:24.049+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:20:24.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:20:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:20:24.049+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:20:24.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:20:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:20:24.049+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:20:24.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:20:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:20:24.709+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:20:24.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:20:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:20:24.709+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:20:24.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:20:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:20:24.709+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:20:24.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:20:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:20:24.709+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:20:24.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:20:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:20:24.652+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:20:24.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:20:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:20:24.652+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:20:24.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:20:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:20:24.652+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:20:25.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:20:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:20:25.087+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:20:25.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:20:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:20:25.087+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:20:25.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:20:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:20:25.087+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:20:25.990 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:20:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:20:25.676+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:20:25.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:20:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:20:25.676+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:20:25.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:20:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:20:25.676+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:20:25.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:20:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:20:25.676+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:20:25.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:20:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:20:25.655+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:20:25.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:20:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:20:25.655+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:20:25.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:20:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:20:25.655+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:20:26.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:20:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:20:26.108+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:20:26.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:20:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:20:26.108+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:20:26.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:20:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:20:26.108+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:20:26.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:20:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:20:26.683+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:20:26.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:20:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:20:26.683+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:20:26.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:20:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:20:26.683+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:20:26.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:20:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:20:26.671+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:20:26.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:20:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:20:26.671+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:20:26.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:20:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:20:26.671+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:20:26.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:20:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:20:26.671+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:20:27.491 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:20:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:20:27.093+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 17928 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:20:27.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:20:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:20:27.108+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:20:27.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:20:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:20:27.108+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:20:27.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:20:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:20:27.108+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:20:27.990 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:20:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:20:27.651+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:20:27.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:20:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:20:27.651+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:20:27.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:20:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:20:27.651+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:20:27.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:20:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:20:27.655+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:20:27.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:20:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:20:27.655+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:20:27.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:20:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:20:27.655+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:20:27.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:20:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:20:27.655+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:20:28.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:20:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:20:28.137+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:20:28.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:20:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:20:28.137+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:20:28.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:20:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:20:28.137+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:20:28.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:20:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:20:28.663+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:20:28.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:20:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:20:28.663+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:20:28.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:20:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:20:28.663+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:20:28.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:20:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:20:28.692+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:20:28.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:20:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:20:28.692+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:20:28.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:20:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:20:28.692+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:20:28.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:20:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:20:28.692+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:20:29.490 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:20:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:20:29.089+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:20:29.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:20:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:20:29.089+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:20:29.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:20:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:20:29.089+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:20:29.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:20:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:20:29.653+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:20:29.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:20:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:20:29.653+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:20:29.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:20:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:20:29.653+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:20:29.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:20:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:20:29.697+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:20:29.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:20:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:20:29.697+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:20:29.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:20:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:20:29.697+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:20:29.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:20:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:20:29.697+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:20:30.490 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:20:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:20:30.127+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:20:30.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:20:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:20:30.127+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:20:30.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:20:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:20:30.127+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:20:30.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:20:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:20:30.701+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:20:30.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:20:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:20:30.701+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:20:30.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:20:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:20:30.701+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:20:30.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:20:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:20:30.693+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:20:30.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:20:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:20:30.693+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:20:30.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:20:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:20:30.693+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:20:30.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:20:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:20:30.693+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:20:31.490 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:20:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:20:31.176+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:20:31.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:20:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:20:31.176+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:20:31.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:20:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:20:31.176+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:20:31.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:20:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:20:31.683+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:20:31.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:20:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:20:31.683+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:20:31.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:20:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:20:31.683+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:20:31.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:20:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:20:31.738+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:20:31.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:20:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:20:31.738+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:20:31.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:20:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:20:31.738+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:20:31.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:20:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:20:31.738+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:20:32.491 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:20:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:20:32.095+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 18000 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:20:32.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:20:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:20:32.161+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:20:32.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:20:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:20:32.161+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:20:32.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:20:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:20:32.161+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:20:32.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:20:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:20:32.719+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:20:32.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:20:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:20:32.719+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:20:32.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:20:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:20:32.719+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:20:32.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:20:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:20:32.716+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:20:32.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:20:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:20:32.716+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:20:32.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:20:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:20:32.716+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:20:32.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:20:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:20:32.716+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:20:33.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:20:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:20:33.187+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:20:33.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:20:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:20:33.187+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:20:33.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:20:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:20:33.187+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:20:33.990 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:20:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:20:33.699+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:20:33.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:20:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:20:33.699+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:20:33.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:20:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:20:33.699+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:20:33.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:20:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:20:33.759+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:20:33.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:20:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:20:33.759+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:20:33.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:20:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:20:33.759+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:20:33.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:20:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:20:33.759+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:20:34.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:20:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:20:34.232+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:20:34.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:20:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:20:34.232+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:20:34.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:20:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:20:34.232+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:20:34.990 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:20:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:20:34.730+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:20:34.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:20:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:20:34.730+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:20:34.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:20:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:20:34.730+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:20:34.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:20:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:20:34.795+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:20:34.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:20:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:20:34.795+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:20:34.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:20:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:20:34.795+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:20:34.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:20:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:20:34.795+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:20:35.490 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:20:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:20:35.198+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:20:35.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:20:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:20:35.198+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:20:35.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:20:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:20:35.198+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:20:36.156 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:20:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:20:35.767+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:20:36.156 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:20:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:20:35.767+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:20:36.156 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:20:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:20:35.767+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:20:36.156 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:20:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:20:35.767+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:20:36.157 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:20:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:20:35.749+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:20:36.157 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:20:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:20:35.749+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:20:36.157 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:20:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:20:35.749+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:20:36.490 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:20:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:20:36.155+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:20:36.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:20:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:20:36.155+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:20:36.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:20:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:20:36.155+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:20:37.099 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:20:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:20:36.775+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:20:37.100 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:20:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:20:36.775+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:20:37.100 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:20:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:20:36.775+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:20:37.100 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:20:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:20:36.758+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:20:37.100 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:20:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:20:36.758+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:20:37.100 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:20:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:20:36.758+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:20:37.100 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:20:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:20:36.758+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:20:37.421 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:20:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:20:37.098+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 18070 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:20:37.421 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:20:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:20:37.188+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:20:37.421 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:20:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:20:37.189+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:20:37.421 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:20:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:20:37.189+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:20:37.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:20:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:20:37.770+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:20:37.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:20:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:20:37.770+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:20:37.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:20:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:20:37.770+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:20:37.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:20:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:20:37.732+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:20:37.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:20:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:20:37.732+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:20:37.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:20:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:20:37.732+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:20:37.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:20:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:20:37.732+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:20:38.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:20:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:20:38.205+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:20:38.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:20:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:20:38.205+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:20:38.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:20:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:20:38.205+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:20:39.195 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:20:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:20:38.779+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:20:39.195 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:20:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:20:38.779+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:20:39.196 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:20:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:20:38.779+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:20:39.196 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:20:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:20:38.754+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:20:39.196 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:20:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:20:38.754+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:20:39.196 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:20:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:20:38.754+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:20:39.196 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:20:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:20:38.754+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:20:39.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:20:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:20:39.194+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:20:39.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:20:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:20:39.194+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:20:39.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:20:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:20:39.194+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:20:39.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:20:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:20:39.747+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:20:39.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:20:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:20:39.747+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:20:39.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:20:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:20:39.747+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:20:39.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:20:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:20:39.706+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:20:39.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:20:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:20:39.706+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:20:39.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:20:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:20:39.706+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:20:39.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:20:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:20:39.706+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:20:40.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:20:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:20:40.157+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:20:40.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:20:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:20:40.157+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:20:40.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:20:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:20:40.157+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:20:40.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:20:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:20:40.735+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:20:40.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:20:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:20:40.735+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:20:40.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:20:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:20:40.735+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:20:40.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:20:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:20:40.657+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:20:40.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:20:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:20:40.657+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:20:40.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:20:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:20:40.657+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:20:40.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:20:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:20:40.657+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:20:41.490 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:20:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:20:41.117+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:20:41.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:20:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:20:41.117+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:20:41.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:20:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:20:41.117+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:20:41.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:20:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:20:41.731+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:20:41.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:20:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:20:41.731+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:20:41.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:20:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:20:41.731+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:20:41.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:20:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:20:41.633+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:20:41.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:20:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:20:41.633+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:20:41.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:20:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:20:41.633+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:20:41.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:20:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:20:41.633+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:20:42.359 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:20:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:20:42.101+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 18142 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:20:42.359 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:20:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:20:42.073+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:20:42.359 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:20:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:20:42.073+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:20:42.360 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:20:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:20:42.073+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:20:42.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:20:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:20:42.585+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:20:42.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:20:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:20:42.585+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:20:42.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:20:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:20:42.585+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:20:42.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:20:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:20:42.585+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:20:43.029 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:20:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:20:42.776+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:20:43.030 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:20:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:20:42.776+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:20:43.030 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:20:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:20:42.776+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:20:43.490 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:20:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:20:43.029+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:20:43.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:20:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:20:43.029+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:20:43.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:20:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:20:43.029+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:20:43.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:20:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:20:43.745+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:20:43.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:20:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:20:43.745+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:20:43.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:20:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:20:43.745+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:20:43.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:20:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:20:43.548+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:20:43.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:20:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:20:43.548+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:20:43.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:20:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:20:43.548+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:20:43.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:20:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:20:43.548+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:20:44.490 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:20:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:20:43.999+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:20:44.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:20:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:20:43.999+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:20:44.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:20:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:20:43.999+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:20:44.990 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:20:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:20:44.720+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:20:44.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:20:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:20:44.720+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:20:44.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:20:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:20:44.720+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:20:44.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:20:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:20:44.542+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:20:44.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:20:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:20:44.542+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:20:44.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:20:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:20:44.542+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:20:44.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:20:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:20:44.542+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:20:45.490 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:20:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:20:45.043+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:20:45.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:20:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:20:45.043+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:20:45.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:20:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:20:45.043+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:20:45.990 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:20:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:20:45.734+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:20:45.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:20:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:20:45.734+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:20:45.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:20:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:20:45.734+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:20:45.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:20:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:20:45.576+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:20:45.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:20:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:20:45.576+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:20:45.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:20:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:20:45.576+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:20:45.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:20:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:20:45.576+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:20:46.490 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:20:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:20:46.029+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:20:46.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:20:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:20:46.029+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:20:46.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:20:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:20:46.029+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:20:46.990 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:20:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:20:46.775+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:20:46.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:20:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:20:46.775+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:20:46.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:20:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:20:46.775+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:20:46.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:20:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:20:46.527+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:20:46.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:20:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:20:46.527+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:20:46.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:20:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:20:46.527+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:20:46.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:20:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:20:46.527+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:20:47.490 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:20:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:20:47.103+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 18212 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:20:47.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:20:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:20:47.001+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:20:47.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:20:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:20:47.001+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:20:47.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:20:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:20:47.001+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:20:47.788 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:20:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:20:47.508+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:20:47.789 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:20:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:20:47.508+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:20:47.789 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:20:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:20:47.508+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:20:47.789 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:20:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:20:47.508+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:20:48.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:20:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:20:47.788+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:20:48.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:20:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:20:47.788+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:20:48.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:20:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:20:47.788+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:20:48.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:20:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:20:47.965+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:20:48.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:20:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:20:47.965+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:20:48.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:20:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:20:47.965+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:20:48.740 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:20:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:20:48.525+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:20:48.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:20:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:20:48.525+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:20:48.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:20:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:20:48.525+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:20:48.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:20:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:20:48.525+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:20:49.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:20:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:20:48.769+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:20:49.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:20:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:20:48.769+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:20:49.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:20:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:20:48.769+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:20:49.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:20:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:20:49.008+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:20:49.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:20:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:20:49.008+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:20:49.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:20:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:20:49.008+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:20:49.884 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:20:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:20:49.727+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:20:49.884 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:20:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:20:49.727+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:20:49.885 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:20:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:20:49.727+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:20:49.885 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:20:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:20:49.524+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:20:49.885 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:20:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:20:49.524+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:20:49.885 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:20:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:20:49.524+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:20:49.885 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:20:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:20:49.524+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:20:50.194 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:20:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:20:50.019+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:20:50.194 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:20:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:20:50.019+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:20:50.194 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:20:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:20:50.019+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:20:50.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:20:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:20:50.736+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:20:50.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:20:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:20:50.736+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:20:50.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:20:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:20:50.736+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:20:50.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:20:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:20:50.540+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:20:50.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:20:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:20:50.540+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:20:50.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:20:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:20:50.540+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:20:50.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:20:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:20:50.540+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:20:51.490 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:20:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:20:51.000+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:20:51.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:20:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:20:51.000+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:20:51.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:20:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:20:51.000+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:20:51.965 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:20:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:20:51.515+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:20:51.966 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:20:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:20:51.515+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:20:51.966 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:20:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:20:51.515+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:20:51.966 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:20:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:20:51.515+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:20:51.966 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:20:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:20:51.759+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:20:51.966 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:20:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:20:51.759+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:20:51.967 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:20:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:20:51.759+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:20:52.240 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:20:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:20:52.107+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 18284 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:20:52.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:20:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:20:51.965+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:20:52.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:20:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:20:51.965+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:20:52.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:20:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:20:51.965+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:20:52.798 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:20:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:20:52.500+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:20:52.798 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:20:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:20:52.500+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:20:52.799 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:20:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:20:52.500+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:20:52.799 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:20:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:20:52.501+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:20:53.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:20:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:20:52.798+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:20:53.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:20:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:20:52.798+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:20:53.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:20:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:20:52.798+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:20:53.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:20:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:20:52.991+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:20:53.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:20:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:20:52.991+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:20:53.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:20:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:20:52.991+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:20:53.805 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:20:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:20:53.500+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:20:53.805 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:20:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:20:53.500+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:20:53.805 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:20:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:20:53.500+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:20:53.805 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:20:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:20:53.500+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:20:54.240 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:20:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:20:53.804+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:20:54.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:20:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:20:53.804+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:20:54.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:20:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:20:53.804+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:20:54.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:20:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:20:54.011+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:20:54.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:20:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:20:54.011+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:20:54.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:20:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:20:54.011+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:20:54.740 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:20:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:20:54.487+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:20:54.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:20:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:20:54.487+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:20:54.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:20:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:20:54.487+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:20:54.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:20:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:20:54.487+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:20:55.240 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:20:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:20:54.832+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:20:55.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:20:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:20:54.832+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:20:55.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:20:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:20:54.832+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:20:55.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:20:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:20:55.038+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:20:55.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:20:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:20:55.038+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:20:55.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:20:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:20:55.038+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:20:55.841 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:20:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:20:55.517+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:20:55.841 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:20:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:20:55.517+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:20:55.841 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:20:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:20:55.517+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:20:55.841 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:20:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:20:55.517+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:20:56.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:20:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:20:55.841+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:20:56.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:20:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:20:55.841+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:20:56.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:20:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:20:55.841+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:20:56.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:20:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:20:56.050+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:20:56.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:20:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:20:56.050+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:20:56.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:20:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:20:56.050+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:20:56.857 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:20:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:20:56.563+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:20:56.857 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:20:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:20:56.563+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:20:56.858 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:20:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:20:56.563+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:20:56.858 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:20:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:20:56.563+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:20:57.110 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:20:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:20:56.857+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:20:57.110 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:20:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:20:56.857+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:20:57.110 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:20:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:20:56.857+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:20:57.111 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:20:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:20:57.076+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:20:57.111 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:20:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:20:57.076+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:20:57.111 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:20:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:20:57.076+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:20:57.490 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:20:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:20:57.110+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 18354 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:20:57.895 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:20:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:20:57.601+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:20:57.895 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:20:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:20:57.601+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:20:57.895 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:20:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:20:57.601+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:20:57.895 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:20:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:20:57.601+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:20:58.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:20:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:20:57.895+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:20:58.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:20:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:20:57.895+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:20:58.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:20:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:20:57.895+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:20:58.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:20:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:20:58.034+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:20:58.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:20:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:20:58.034+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:20:58.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:20:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:20:58.034+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:20:58.874 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:20:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:20:58.588+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:20:58.874 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:20:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:20:58.588+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:20:58.874 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:20:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:20:58.588+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:20:58.874 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:20:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:20:58.588+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:20:59.240 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:20:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:20:58.874+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:20:59.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:20:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:20:58.874+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:20:59.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:20:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:20:58.874+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:20:59.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:20:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:20:59.004+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:20:59.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:20:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:20:59.004+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:20:59.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:20:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:20:59.004+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:20:59.837 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:20:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:20:59.837+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:20:59.838 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:20:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:20:59.837+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:20:59.838 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:20:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:20:59.837+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:20:59.838 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:20:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:20:59.587+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:20:59.838 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:20:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:20:59.587+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:20:59.838 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:20:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:20:59.587+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:20:59.838 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:20:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:20:59.587+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:21:00.194 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:20:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:20:59.989+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:21:00.195 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:20:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:20:59.989+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:21:00.195 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:20:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:20:59.989+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:21:00.975 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:21:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:21:00.798+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:21:00.976 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:21:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:21:00.798+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:21:00.976 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:21:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:21:00.798+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:21:00.976 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:21:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:21:00.635+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:21:00.976 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:21:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:21:00.635+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:21:00.976 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:21:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:21:00.635+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:21:00.976 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:21:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:21:00.635+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:21:01.240 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:21:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:21:00.975+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:21:01.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:21:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:21:00.975+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:21:01.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:21:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:21:00.975+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:21:01.976 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:21:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:21:01.789+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:21:01.977 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:21:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:21:01.789+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:21:01.977 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:21:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:21:01.789+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:21:01.977 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:21:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:21:01.619+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:21:01.977 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:21:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:21:01.619+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:21:01.977 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:21:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:21:01.619+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:21:01.978 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:21:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:21:01.619+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:21:02.235 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:21:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:21:02.113+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 18426 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:21:02.235 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:21:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:21:01.976+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:21:02.235 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:21:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:21:01.976+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:21:02.235 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:21:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:21:01.976+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:21:02.974 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:21:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:21:02.826+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:21:02.974 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:21:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:21:02.826+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:21:02.975 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:21:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:21:02.826+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:21:02.975 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:21:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:21:02.624+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:21:02.975 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:21:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:21:02.624+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:21:02.975 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:21:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:21:02.624+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:21:02.975 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:21:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:21:02.624+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:21:03.240 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:21:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:21:02.974+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:21:03.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:21:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:21:02.974+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:21:03.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:21:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:21:02.974+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:21:03.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:21:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:21:03.837+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:21:03.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:21:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:21:03.837+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:21:03.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:21:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:21:03.837+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:21:03.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:21:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:21:03.620+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:21:03.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:21:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:21:03.620+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:21:03.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:21:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:21:03.620+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:21:03.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:21:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:21:03.620+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:21:04.490 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:21:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:21:04.003+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:21:04.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:21:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:21:04.003+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:21:04.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:21:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:21:04.003+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:21:04.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:21:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:21:04.835+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:21:04.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:21:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:21:04.835+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:21:04.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:21:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:21:04.835+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:21:04.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:21:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:21:04.628+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:21:04.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:21:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:21:04.628+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:21:04.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:21:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:21:04.628+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:21:04.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:21:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:21:04.628+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:21:05.490 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:21:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:21:05.047+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:21:05.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:21:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:21:05.047+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:21:05.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:21:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:21:05.047+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:21:05.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:21:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:21:05.816+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:21:05.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:21:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:21:05.816+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:21:05.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:21:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:21:05.816+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:21:05.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:21:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:21:05.595+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:21:05.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:21:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:21:05.595+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:21:05.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:21:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:21:05.595+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:21:05.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:21:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:21:05.595+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:21:06.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:21:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:21:06.016+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:21:06.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:21:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:21:06.016+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:21:06.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:21:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:21:06.016+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:21:06.857 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:21:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:21:06.618+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:21:06.857 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:21:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:21:06.618+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:21:06.857 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:21:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:21:06.618+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:21:06.857 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:21:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:21:06.618+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:21:07.116 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:21:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:21:06.987+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:21:07.116 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:21:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:21:06.987+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:21:07.116 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:21:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:21:06.987+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:21:07.116 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:21:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:21:06.857+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:21:07.117 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:21:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:21:06.857+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:21:07.117 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:21:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:21:06.857+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:21:07.490 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:21:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:21:07.116+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 18496 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:21:07.879 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:21:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:21:07.624+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:21:07.879 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:21:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:21:07.624+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:21:07.879 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:21:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:21:07.624+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:21:07.880 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:21:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:21:07.624+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:21:08.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:21:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:21:07.879+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:21:08.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:21:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:21:07.879+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:21:08.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:21:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:21:07.879+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:21:08.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:21:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:21:08.006+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:21:08.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:21:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:21:08.006+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:21:08.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:21:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:21:08.006+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:21:08.971 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:21:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:21:08.841+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:21:08.971 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:21:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:21:08.841+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:21:08.971 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:21:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:21:08.841+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:21:08.972 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:21:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:21:08.654+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:21:08.972 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:21:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:21:08.654+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:21:08.972 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:21:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:21:08.654+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:21:08.972 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:21:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:21:08.654+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:21:09.240 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:21:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:21:08.971+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:21:09.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:21:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:21:08.971+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:21:09.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:21:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:21:08.971+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:21:09.958 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:21:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:21:09.855+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:21:09.958 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:21:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:21:09.855+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:21:09.958 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:21:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:21:09.855+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:21:09.959 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:21:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:21:09.660+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:21:09.959 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:21:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:21:09.660+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:21:09.959 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:21:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:21:09.660+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:21:09.959 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:21:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:21:09.661+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:21:10.240 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:21:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:21:09.958+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:21:10.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:21:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:21:09.958+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:21:10.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:21:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:21:09.958+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:21:10.942 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:21:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:21:10.882+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:21:10.942 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:21:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:21:10.882+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:21:10.942 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:21:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:21:10.882+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:21:10.942 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:21:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:21:10.634+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:21:10.942 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:21:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:21:10.634+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:21:10.943 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:21:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:21:10.634+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:21:10.943 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:21:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:21:10.634+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:21:11.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:21:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:21:10.942+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:21:11.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:21:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:21:10.942+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:21:11.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:21:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:21:10.942+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:21:11.893 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:21:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:21:11.594+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:21:11.894 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:21:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:21:11.594+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:21:11.894 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:21:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:21:11.594+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:21:11.894 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:21:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:21:11.594+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:21:12.240 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:21:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:21:12.118+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 18568 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:21:12.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:21:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:21:11.893+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:21:12.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:21:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:21:11.893+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:21:12.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:21:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:21:11.893+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:21:12.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:21:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:21:11.915+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:21:12.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:21:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:21:11.915+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:21:12.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:21:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:21:11.915+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:21:12.900 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:21:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:21:12.620+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:21:12.900 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:21:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:21:12.620+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:21:12.900 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:21:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:21:12.620+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:21:12.900 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:21:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:21:12.620+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:21:13.240 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:21:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:21:12.954+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:21:13.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:21:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:21:12.954+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:21:13.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:21:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:21:12.954+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:21:13.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:21:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:21:12.899+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:21:13.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:21:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:21:12.899+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:21:13.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:21:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:21:12.899+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:21:13.899 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:21:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:21:13.635+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:21:13.899 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:21:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:21:13.635+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:21:13.899 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:21:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:21:13.635+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:21:13.899 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:21:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:21:13.635+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:21:14.240 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:21:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:21:13.989+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:21:14.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:21:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:21:13.989+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:21:14.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:21:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:21:13.989+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:21:14.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:21:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:21:13.898+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:21:14.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:21:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:21:13.898+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:21:14.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:21:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:21:13.898+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:21:14.974 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:21:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:21:14.642+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:21:14.974 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:21:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:21:14.642+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:21:14.974 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:21:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:21:14.642+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:21:14.974 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:21:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:21:14.642+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:21:14.975 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:21:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:21:14.869+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:21:14.975 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:21:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:21:14.869+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:21:14.975 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:21:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:21:14.869+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:21:15.240 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:21:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:21:14.973+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:21:15.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:21:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:21:14.973+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:21:15.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:21:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:21:14.973+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:21:15.924 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:21:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:21:15.903+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:21:15.924 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:21:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:21:15.903+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:21:15.924 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:21:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:21:15.903+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:21:15.925 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:21:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:21:15.659+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:21:15.925 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:21:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:21:15.659+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:21:15.925 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:21:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:21:15.659+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:21:15.925 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:21:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:21:15.659+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:21:16.240 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:21:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:21:15.923+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:21:16.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:21:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:21:15.923+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:21:16.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:21:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:21:15.923+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:21:16.915 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:21:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:21:16.653+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:21:16.915 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:21:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:21:16.653+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:21:16.915 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:21:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:21:16.653+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:21:16.915 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:21:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:21:16.653+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:21:17.240 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:21:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:21:17.120+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 18638 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:21:17.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:21:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:21:16.914+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:21:17.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:21:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:21:16.914+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:21:17.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:21:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:21:16.914+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:21:17.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:21:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:21:16.950+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:21:17.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:21:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:21:16.950+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:21:17.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:21:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:21:16.950+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:21:17.990 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:21:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:21:17.918+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:21:17.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:21:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:21:17.918+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:21:17.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:21:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:21:17.918+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:21:17.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:21:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:21:17.894+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:21:17.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:21:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:21:17.894+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:21:17.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:21:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:21:17.894+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:21:17.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:21:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:21:17.701+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:21:17.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:21:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:21:17.701+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:21:17.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:21:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:21:17.701+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:21:17.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:21:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:21:17.701+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:21:18.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:21:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:21:18.879+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:21:18.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:21:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:21:18.879+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:21:18.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:21:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:21:18.879+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:21:18.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:21:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:21:18.927+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:21:18.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:21:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:21:18.927+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:21:18.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:21:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:21:18.927+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:21:18.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:21:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:21:18.704+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:21:18.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:21:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:21:18.704+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:21:18.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:21:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:21:18.704+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:21:18.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:21:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:21:18.704+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:21:19.990 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:21:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:21:19.917+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:21:19.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:21:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:21:19.918+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:21:19.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:21:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:21:19.918+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:21:19.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:21:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:21:19.913+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:21:19.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:21:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:21:19.913+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:21:19.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:21:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:21:19.913+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:21:19.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:21:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:21:19.689+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:21:19.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:21:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:21:19.689+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:21:19.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:21:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:21:19.689+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:21:19.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:21:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:21:19.689+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:21:20.964 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:21:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:21:20.940+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:21:20.964 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:21:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:21:20.940+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:21:20.965 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:21:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:21:20.940+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:21:20.965 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:21:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:21:20.695+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:21:20.965 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:21:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:21:20.695+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:21:20.965 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:21:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:21:20.695+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:21:20.965 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:21:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:21:20.695+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:21:21.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:21:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:21:20.963+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:21:21.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:21:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:21:20.963+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:21:21.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:21:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:21:20.963+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:21:21.951 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:21:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:21:21.692+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:21:21.952 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:21:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:21:21.692+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:21:21.952 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:21:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:21:21.692+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:21:21.952 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:21:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:21:21.692+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:21:22.240 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:21:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:21:22.123+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 18710 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:21:22.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:21:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:21:21.990+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:21:22.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:21:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:21:21.990+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:21:22.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:21:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:21:21.990+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:21:22.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:21:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:21:21.950+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:21:22.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:21:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:21:21.950+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:21:22.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:21:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:21:21.950+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:21:22.945 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:21:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:21:22.678+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:21:22.945 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:21:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:21:22.678+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:21:22.945 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:21:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:21:22.678+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:21:22.945 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:21:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:21:22.678+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:21:23.240 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:21:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:21:22.945+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:21:23.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:21:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:21:22.945+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:21:23.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:21:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:21:22.945+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:21:23.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:21:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:21:22.983+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:21:23.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:21:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:21:22.983+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:21:23.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:21:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:21:22.983+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:21:23.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:21:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:21:23.713+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:21:23.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:21:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:21:23.713+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:21:23.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:21:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:21:23.713+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:21:23.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:21:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:21:23.713+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:21:24.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:21:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:21:24.033+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:21:24.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:21:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:21:24.033+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:21:24.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:21:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:21:24.033+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:21:24.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:21:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:21:23.990+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:21:24.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:21:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:21:23.990+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:21:24.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:21:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:21:23.990+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:21:24.986 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:21:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:21:24.951+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:21:24.987 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:21:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:21:24.951+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:21:24.987 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:21:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:21:24.951+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:21:24.987 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:21:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:21:24.734+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:21:24.987 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:21:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:21:24.734+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:21:24.987 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:21:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:21:24.734+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:21:24.987 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:21:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:21:24.734+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:21:25.240 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:21:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:21:24.986+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:21:25.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:21:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:21:24.986+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:21:25.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:21:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:21:24.986+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:21:26.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:21:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:21:25.966+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:21:26.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:21:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:21:25.966+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:21:26.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:21:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:21:25.966+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:21:26.241 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:21:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:21:25.777+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:21:26.241 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:21:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:21:25.777+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:21:26.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:21:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:21:25.777+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:21:26.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:21:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:21:25.777+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:21:26.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:21:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:21:25.971+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:21:26.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:21:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:21:25.971+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:21:26.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:21:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:21:25.971+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:21:27.006 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:21:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:21:26.940+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:21:27.006 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:21:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:21:26.940+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:21:27.006 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:21:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:21:26.940+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:21:27.006 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:21:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:21:26.742+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:21:27.007 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:21:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:21:26.742+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:21:27.007 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:21:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:21:26.742+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:21:27.007 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:21:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:21:26.742+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:21:27.490 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:21:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:21:27.126+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 18780 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:21:27.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:21:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:21:27.005+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:21:27.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:21:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:21:27.005+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:21:27.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:21:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:21:27.005+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:21:27.990 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:21:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:21:27.924+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:21:27.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:21:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:21:27.924+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:21:27.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:21:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:21:27.924+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:21:27.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:21:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:21:27.739+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:21:27.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:21:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:21:27.739+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:21:27.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:21:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:21:27.739+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:21:27.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:21:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:21:27.739+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:21:27.992 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:21:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:21:27.987+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:21:27.992 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:21:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:21:27.987+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:21:27.992 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:21:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:21:27.987+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:21:28.990 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:21:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:21:28.886+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:21:28.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:21:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:21:28.886+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:21:28.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:21:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:21:28.886+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:21:28.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:21:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:21:28.971+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:21:28.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:21:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:21:28.971+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:21:28.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:21:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:21:28.971+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:21:28.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:21:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:21:28.725+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:21:28.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:21:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:21:28.725+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:21:28.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:21:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:21:28.725+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:21:28.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:21:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:21:28.725+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:21:29.990 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:21:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:21:29.907+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:21:29.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:21:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:21:29.907+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:21:29.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:21:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:21:29.907+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:21:29.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:21:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:21:29.722+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:21:29.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:21:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:21:29.722+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:21:29.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:21:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:21:29.722+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:21:29.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:21:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:21:29.722+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:21:30.490 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:21:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:21:29.996+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:21:30.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:21:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:21:29.996+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:21:30.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:21:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:21:29.996+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:21:31.240 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:21:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:21:30.894+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:21:31.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:21:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:21:30.894+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:21:31.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:21:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:21:30.894+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:21:31.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:21:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:21:30.956+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:21:31.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:21:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:21:30.956+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:21:31.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:21:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:21:30.956+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:21:31.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:21:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:21:30.754+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:21:31.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:21:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:21:30.754+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:21:31.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:21:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:21:30.754+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:21:31.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:21:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:21:30.754+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:21:31.904 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:21:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:21:31.717+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:21:31.904 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:21:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:21:31.717+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:21:31.905 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:21:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:21:31.717+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:21:31.905 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:21:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:21:31.717+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:21:32.240 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:21:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:21:32.129+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 18852 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:21:32.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:21:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:21:31.904+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:21:32.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:21:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:21:31.904+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:21:32.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:21:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:21:31.904+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:21:32.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:21:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:21:31.934+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:21:32.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:21:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:21:31.934+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:21:32.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:21:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:21:31.934+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:21:32.990 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:21:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:21:32.883+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:21:32.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:21:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:21:32.883+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:21:32.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:21:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:21:32.883+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:21:32.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:21:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:21:32.927+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:21:32.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:21:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:21:32.927+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:21:32.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:21:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:21:32.927+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:21:32.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:21:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:21:32.684+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:21:32.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:21:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:21:32.684+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:21:32.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:21:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:21:32.684+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:21:32.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:21:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:21:32.684+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:21:33.975 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:21:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:21:33.877+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:21:33.975 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:21:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:21:33.877+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:21:33.975 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:21:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:21:33.877+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:21:33.975 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:21:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:21:33.637+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:21:33.976 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:21:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:21:33.637+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:21:33.976 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:21:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:21:33.637+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:21:33.976 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:21:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:21:33.637+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:21:34.240 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:21:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:21:33.973+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:21:34.240 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:21:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:21:33.973+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:21:34.240 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:21:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:21:33.973+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:21:34.859 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:21:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:21:34.614+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:21:34.860 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:21:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:21:34.614+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:21:34.860 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:21:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:21:34.614+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:21:34.860 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:21:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:21:34.614+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:21:35.240 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:21:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:21:34.859+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:21:35.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:21:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:21:34.859+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:21:35.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:21:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:21:34.859+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:21:35.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:21:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:21:34.930+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:21:35.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:21:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:21:34.930+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:21:35.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:21:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:21:34.930+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:21:35.907 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:21:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:21:35.643+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:21:35.907 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:21:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:21:35.643+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:21:35.907 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:21:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:21:35.643+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:21:35.907 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:21:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:21:35.644+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:21:36.240 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:21:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:21:35.907+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:21:36.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:21:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:21:35.907+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:21:36.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:21:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:21:35.907+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:21:36.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:21:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:21:35.907+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:21:36.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:21:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:21:35.907+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:21:36.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:21:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:21:35.907+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:21:36.935 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:21:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:21:36.907+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:21:36.936 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:21:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:21:36.907+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:21:36.936 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:21:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:21:36.907+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:21:36.936 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:21:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:21:36.662+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:21:36.936 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:21:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:21:36.662+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:21:36.936 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:21:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:21:36.662+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:21:36.937 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:21:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:21:36.662+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:21:37.240 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:21:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:21:37.132+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 18922 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:21:37.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:21:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:21:36.935+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:21:37.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:21:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:21:36.935+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:21:37.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:21:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:21:36.935+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:21:37.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:21:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:21:37.876+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:21:37.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:21:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:21:37.876+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:21:37.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:21:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:21:37.876+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:21:37.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:21:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:21:37.903+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:21:37.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:21:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:21:37.903+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:21:37.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:21:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:21:37.903+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:21:37.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:21:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:21:37.656+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:21:37.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:21:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:21:37.656+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:21:37.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:21:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:21:37.656+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:21:37.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:21:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:21:37.657+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:21:38.990 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:21:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:21:38.884+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:21:38.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:21:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:21:38.884+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:21:38.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:21:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:21:38.884+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:21:38.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:21:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:21:38.883+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:21:38.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:21:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:21:38.883+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:21:38.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:21:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:21:38.883+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:21:38.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:21:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:21:38.652+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:21:38.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:21:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:21:38.652+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:21:38.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:21:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:21:38.652+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:21:38.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:21:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:21:38.652+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:21:39.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:21:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:21:39.897+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:21:39.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:21:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:21:39.897+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:21:39.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:21:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:21:39.897+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:21:39.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:21:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:21:39.909+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:21:39.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:21:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:21:39.909+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:21:39.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:21:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:21:39.909+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:21:39.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:21:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:21:39.668+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:21:39.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:21:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:21:39.668+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:21:39.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:21:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:21:39.668+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:21:39.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:21:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:21:39.668+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:21:40.990 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:21:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:21:40.703+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:21:40.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:21:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:21:40.703+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:21:40.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:21:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:21:40.703+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:21:40.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:21:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:21:40.703+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:21:40.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:21:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:21:40.878+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:21:40.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:21:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:21:40.878+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:21:40.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:21:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:21:40.878+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:21:40.992 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:21:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:21:40.866+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:21:40.992 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:21:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:21:40.866+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:21:40.992 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:21:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:21:40.866+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:21:41.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:21:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:21:41.715+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:21:41.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:21:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:21:41.715+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:21:41.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:21:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:21:41.715+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:21:41.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:21:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:21:41.715+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:21:41.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:21:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:21:41.833+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:21:41.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:21:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:21:41.833+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:21:41.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:21:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:21:41.833+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:21:41.992 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:21:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:21:41.902+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:21:41.992 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:21:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:21:41.902+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:21:41.992 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:21:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:21:41.902+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:21:42.490 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:21:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:21:42.135+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 18994 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:21:43.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:21:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:21:42.850+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:21:43.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:21:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:21:42.850+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:21:43.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:21:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:21:42.850+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:21:43.241 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:21:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:21:42.762+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:21:43.241 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:21:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:21:42.762+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:21:43.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:21:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:21:42.762+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:21:43.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:21:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:21:42.762+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:21:43.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:21:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:21:42.864+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:21:43.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:21:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:21:42.864+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:21:43.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:21:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:21:42.864+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:21:44.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:21:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:21:43.804+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:21:44.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:21:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:21:43.804+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:21:44.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:21:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:21:43.804+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:21:44.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:21:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:21:43.828+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:21:44.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:21:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:21:43.828+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:21:44.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:21:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:21:43.828+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:21:44.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:21:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:21:43.777+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:21:44.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:21:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:21:43.777+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:21:44.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:21:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:21:43.777+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:21:44.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:21:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:21:43.777+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:21:45.240 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:21:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:21:44.756+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:21:45.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:21:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:21:44.756+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:21:45.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:21:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:21:44.756+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:21:45.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:21:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:21:44.874+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:21:45.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:21:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:21:44.874+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:21:45.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:21:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:21:44.874+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:21:45.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:21:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:21:44.797+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:21:45.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:21:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:21:44.797+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:21:45.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:21:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:21:44.797+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:21:45.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:21:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:21:44.797+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:21:46.240 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:21:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:21:45.844+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:21:46.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:21:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:21:45.844+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:21:46.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:21:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:21:45.844+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:21:46.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:21:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:21:45.753+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:21:46.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:21:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:21:45.753+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:21:46.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:21:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:21:45.753+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:21:46.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:21:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:21:45.749+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:21:46.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:21:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:21:45.749+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:21:46.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:21:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:21:45.749+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:21:46.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:21:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:21:45.749+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:21:46.990 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:21:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:21:46.756+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:21:46.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:21:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:21:46.756+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:21:46.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:21:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:21:46.756+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:21:46.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:21:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:21:46.804+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:21:46.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:21:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:21:46.804+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:21:46.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:21:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:21:46.804+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:21:46.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:21:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:21:46.727+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:21:46.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:21:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:21:46.727+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:21:46.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:21:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:21:46.727+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:21:46.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:21:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:21:46.727+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:21:47.490 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:21:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:21:47.139+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 19064 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:21:48.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:21:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:21:47.781+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:21:48.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:21:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:21:47.781+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:21:48.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:21:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:21:47.781+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:21:48.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:21:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:21:47.764+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:21:48.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:21:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:21:47.764+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:21:48.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:21:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:21:47.764+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:21:48.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:21:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:21:47.766+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:21:48.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:21:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:21:47.766+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:21:48.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:21:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:21:47.766+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:21:48.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:21:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:21:47.766+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:21:49.240 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:21:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:21:48.762+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:21:49.241 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:21:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:21:48.762+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:21:49.241 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:21:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:21:48.762+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:21:49.241 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:21:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:21:48.762+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:21:49.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:21:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:21:48.797+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:21:49.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:21:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:21:48.797+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:21:49.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:21:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:21:48.797+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:21:49.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:21:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:21:48.806+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:21:49.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:21:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:21:48.806+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:21:49.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:21:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:21:48.806+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:21:50.197 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:21:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:21:49.791+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:21:50.198 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:21:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:21:49.791+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:21:50.198 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:21:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:21:49.791+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:21:50.198 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:21:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:21:49.791+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:21:50.199 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:21:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:21:49.827+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:21:50.199 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:21:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:21:49.827+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:21:50.199 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:21:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:21:49.827+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:21:50.199 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:21:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:21:49.798+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:21:50.199 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:21:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:21:49.798+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:21:50.200 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:21:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:21:49.798+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:21:51.240 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:21:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:21:50.863+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:21:51.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:21:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:21:50.863+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:21:51.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:21:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:21:50.863+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:21:51.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:21:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:21:50.791+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:21:51.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:21:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:21:50.791+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:21:51.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:21:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:21:50.791+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:21:51.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:21:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:21:50.803+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:21:51.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:21:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:21:50.803+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:21:51.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:21:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:21:50.803+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:21:51.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:21:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:21:50.804+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:21:51.990 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:21:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:21:51.782+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:21:51.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:21:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:21:51.782+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:21:51.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:21:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:21:51.782+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:21:51.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:21:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:21:51.833+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:21:51.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:21:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:21:51.833+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:21:51.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:21:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:21:51.833+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:21:51.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:21:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:21:51.824+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:21:51.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:21:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:21:51.824+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:21:51.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:21:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:21:51.824+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:21:51.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:21:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:21:51.824+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:21:52.490 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:21:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:21:52.142+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 19136 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:21:53.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:21:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:21:52.781+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:21:53.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:21:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:21:52.781+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:21:53.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:21:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:21:52.781+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:21:53.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:21:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:21:52.867+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:21:53.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:21:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:21:52.867+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:21:53.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:21:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:21:52.867+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:21:53.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:21:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:21:52.836+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:21:53.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:21:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:21:52.836+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:21:53.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:21:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:21:52.836+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:21:53.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:21:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:21:52.836+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:21:53.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:21:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:21:53.738+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:21:53.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:21:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:21:53.738+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:21:53.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:21:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:21:53.738+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:21:53.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:21:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:21:53.902+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:21:53.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:21:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:21:53.902+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:21:53.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:21:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:21:53.902+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:21:53.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:21:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:21:53.822+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:21:53.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:21:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:21:53.822+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:21:53.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:21:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:21:53.822+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:21:53.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:21:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:21:53.822+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:21:54.990 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:21:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:21:54.722+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:21:54.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:21:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:21:54.722+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:21:54.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:21:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:21:54.722+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:21:54.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:21:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:21:54.916+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:21:54.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:21:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:21:54.916+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:21:54.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:21:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:21:54.916+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:21:54.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:21:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:21:54.828+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:21:54.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:21:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:21:54.828+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:21:54.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:21:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:21:54.828+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:21:54.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:21:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:21:54.828+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:21:56.234 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:21:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:21:55.783+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:21:56.234 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:21:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:21:55.783+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:21:56.234 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:21:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:21:55.783+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:21:56.235 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:21:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:21:55.783+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:21:56.235 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:21:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:21:55.771+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:21:56.235 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:21:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:21:55.771+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:21:56.235 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:21:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:21:55.771+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:21:56.236 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:21:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:21:55.886+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:21:56.236 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:21:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:21:55.886+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:21:56.236 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:21:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:21:55.886+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:21:56.888 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:21:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:21:56.752+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:21:56.888 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:21:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:21:56.752+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:21:56.888 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:21:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:21:56.752+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:21:56.888 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:21:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:21:56.768+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:21:56.889 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:21:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:21:56.768+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:21:56.889 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:21:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:21:56.768+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:21:56.889 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:21:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:21:56.768+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:21:57.143 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:21:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:21:56.888+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:21:57.144 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:21:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:21:56.888+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:21:57.144 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:21:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:21:56.888+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:21:57.490 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:21:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:21:57.144+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 19206 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:21:58.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:21:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:21:57.935+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:21:58.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:21:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:21:57.935+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:21:58.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:21:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:21:57.935+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:21:58.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:21:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:21:57.787+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:21:58.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:21:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:21:57.787+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:21:58.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:21:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:21:57.787+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:21:58.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:21:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:21:57.815+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:21:58.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:21:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:21:57.815+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:21:58.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:21:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:21:57.815+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:21:58.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:21:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:21:57.815+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:21:59.241 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:21:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:21:58.777+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:21:59.241 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:21:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:21:58.777+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:21:59.241 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:21:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:21:58.777+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:21:59.241 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:21:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:21:58.777+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:21:59.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:21:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:21:58.923+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:21:59.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:21:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:21:58.923+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:21:59.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:21:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:21:58.923+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:21:59.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:21:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:21:58.792+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:21:59.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:21:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:21:58.792+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:21:59.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:21:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:21:58.792+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:22:00.198 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:21:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:21:59.797+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:22:00.198 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:21:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:21:59.797+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:22:00.198 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:21:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:21:59.797+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:22:00.198 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:21:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:21:59.797+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:22:00.198 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:21:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:21:59.746+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:22:00.199 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:21:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:21:59.746+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:22:00.199 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:21:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:21:59.746+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:22:00.199 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:21:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:21:59.936+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:22:00.199 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:21:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:21:59.936+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:22:00.199 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:21:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:21:59.936+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:22:00.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:22:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:22:00.940+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:22:00.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:22:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:22:00.940+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:22:00.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:22:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:22:00.941+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:22:00.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:22:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:22:00.705+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:22:00.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:22:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:22:00.705+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:22:00.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:22:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:22:00.705+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:22:00.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:22:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:22:00.810+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:22:00.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:22:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:22:00.810+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:22:00.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:22:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:22:00.810+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:22:00.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:22:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:22:00.811+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:22:01.987 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:22:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:22:01.748+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:22:01.988 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:22:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:22:01.748+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:22:01.988 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:22:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:22:01.748+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:22:01.988 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:22:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:22:01.781+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:22:01.988 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:22:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:22:01.781+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:22:01.988 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:22:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:22:01.781+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:22:01.989 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:22:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:22:01.782+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:22:02.240 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:22:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:22:02.146+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 19278 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:22:02.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:22:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:22:01.988+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:22:02.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:22:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:22:01.988+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:22:02.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:22:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:22:01.988+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:22:02.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:22:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:22:02.972+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:22:02.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:22:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:22:02.972+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:22:02.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:22:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:22:02.972+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:22:02.992 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:22:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:22:02.736+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:22:02.992 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:22:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:22:02.736+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:22:02.992 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:22:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:22:02.736+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:22:02.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:22:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:22:02.781+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:22:02.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:22:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:22:02.781+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:22:02.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:22:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:22:02.781+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:22:02.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:22:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:22:02.781+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:22:04.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:22:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:22:03.958+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:22:04.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:22:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:22:03.958+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:22:04.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:22:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:22:03.958+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:22:04.241 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:22:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:22:03.805+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:22:04.241 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:22:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:22:03.805+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:22:04.241 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:22:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:22:03.805+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:22:04.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:22:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:22:03.805+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:22:04.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:22:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:22:03.740+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:22:04.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:22:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:22:03.740+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:22:04.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:22:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:22:03.740+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:22:04.967 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:22:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:22:04.764+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:22:04.968 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:22:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:22:04.764+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:22:04.968 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:22:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:22:04.764+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:22:04.968 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:22:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:22:04.765+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:22:04.968 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:22:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:22:04.710+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:22:04.968 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:22:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:22:04.710+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:22:04.968 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:22:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:22:04.710+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:22:05.240 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:22:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:22:04.967+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:22:05.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:22:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:22:04.967+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:22:05.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:22:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:22:04.967+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:22:05.990 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:22:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:22:05.783+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:22:05.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:22:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:22:05.783+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:22:05.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:22:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:22:05.783+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:22:05.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:22:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:22:05.783+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:22:05.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:22:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:22:05.739+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:22:05.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:22:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:22:05.739+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:22:05.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:22:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:22:05.739+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:22:05.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:22:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:22:05.956+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:22:05.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:22:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:22:05.956+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:22:05.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:22:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:22:05.956+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:22:06.997 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:22:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:22:06.761+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:22:06.997 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:22:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:22:06.761+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:22:06.998 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:22:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:22:06.761+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:22:06.998 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:22:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:22:06.744+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:22:06.998 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:22:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:22:06.744+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:22:06.998 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:22:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:22:06.744+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:22:06.998 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:22:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:22:06.744+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:22:07.250 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:22:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:22:07.148+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 19348 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:22:07.251 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:22:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:22:06.996+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:22:07.251 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:22:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:22:06.996+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:22:07.251 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:22:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:22:06.996+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:22:07.843 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:22:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:22:07.738+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:22:07.844 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:22:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:22:07.738+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:22:07.844 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:22:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:22:07.738+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:22:07.844 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:22:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:22:07.734+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:22:07.844 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:22:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:22:07.734+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:22:07.844 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:22:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:22:07.734+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:22:07.845 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:22:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:22:07.734+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:22:08.240 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:22:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:22:07.961+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:22:08.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:22:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:22:07.961+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:22:08.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:22:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:22:07.961+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:22:08.937 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:22:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:22:08.720+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:22:08.937 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:22:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:22:08.720+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:22:08.938 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:22:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:22:08.720+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:22:08.938 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:22:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:22:08.696+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:22:08.938 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:22:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:22:08.696+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:22:08.938 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:22:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:22:08.696+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:22:08.938 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:22:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:22:08.697+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:22:09.240 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:22:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:22:08.936+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:22:09.240 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:22:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:22:08.936+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:22:09.240 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:22:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:22:08.936+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:22:09.934 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:22:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:22:09.686+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:22:09.934 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:22:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:22:09.686+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:22:09.934 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:22:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:22:09.686+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:22:09.934 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:22:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:22:09.681+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:22:09.935 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:22:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:22:09.681+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:22:09.935 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:22:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:22:09.681+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:22:09.935 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:22:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:22:09.681+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:22:10.198 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:22:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:22:09.933+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:22:10.199 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:22:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:22:09.933+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:22:10.199 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:22:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:22:09.933+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:22:10.973 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:22:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:22:10.673+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:22:10.973 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:22:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:22:10.673+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:22:10.973 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:22:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:22:10.673+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:22:10.973 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:22:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:22:10.681+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:22:10.973 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:22:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:22:10.681+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:22:10.974 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:22:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:22:10.681+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:22:10.974 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:22:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:22:10.681+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:22:11.240 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:22:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:22:10.972+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:22:11.240 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:22:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:22:10.972+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:22:11.240 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:22:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:22:10.972+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:22:11.728 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:22:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:22:11.665+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:22:11.728 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:22:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:22:11.665+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:22:11.729 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:22:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:22:11.665+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:22:11.729 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:22:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:22:11.666+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:22:11.729 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:22:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:22:11.644+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:22:11.729 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:22:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:22:11.644+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:22:11.729 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:22:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:22:11.644+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:22:12.276 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:22:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:22:12.151+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 19420 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:22:12.277 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:22:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:22:12.008+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:22:12.277 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:22:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:22:12.008+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:22:12.277 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:22:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:22:12.008+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:22:12.974 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:22:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:22:12.594+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:22:12.975 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:22:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:22:12.594+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:22:12.975 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:22:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:22:12.594+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:22:12.975 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:22:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:22:12.710+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:22:12.975 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:22:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:22:12.710+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:22:12.975 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:22:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:22:12.710+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:22:12.976 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:22:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:22:12.710+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:22:13.240 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:22:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:22:12.973+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:22:13.240 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:22:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:22:12.973+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:22:13.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:22:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:22:12.973+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:22:13.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:22:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:22:13.545+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:22:13.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:22:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:22:13.545+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:22:13.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:22:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:22:13.545+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:22:13.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:22:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:22:13.724+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:22:13.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:22:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:22:13.724+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:22:13.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:22:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:22:13.724+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:22:13.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:22:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:22:13.724+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:22:14.490 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:22:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:22:14.000+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:22:14.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:22:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:22:14.000+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:22:14.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:22:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:22:14.000+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:22:14.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:22:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:22:14.516+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:22:14.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:22:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:22:14.516+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:22:14.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:22:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:22:14.516+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:22:14.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:22:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:22:14.710+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:22:14.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:22:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:22:14.710+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:22:14.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:22:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:22:14.710+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:22:14.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:22:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:22:14.710+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:22:15.468 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:22:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:22:15.002+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:22:15.468 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:22:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:22:15.002+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:22:15.469 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:22:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:22:15.002+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:22:15.725 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:22:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:22:15.468+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:22:15.725 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:22:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:22:15.468+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:22:15.725 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:22:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:22:15.468+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:22:15.990 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:22:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:22:15.725+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:22:15.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:22:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:22:15.725+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:22:15.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:22:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:22:15.725+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:22:15.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:22:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:22:15.725+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:22:16.452 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:22:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:22:16.042+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:22:16.452 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:22:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:22:16.042+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:22:16.452 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:22:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:22:16.042+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:22:16.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:22:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:22:16.451+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:22:16.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:22:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:22:16.451+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:22:16.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:22:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:22:16.451+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:22:17.083 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:22:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:22:16.774+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:22:17.083 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:22:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:22:16.774+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:22:17.083 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:22:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:22:16.774+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:22:17.084 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:22:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:22:16.775+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:22:17.458 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:22:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:22:17.154+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 19490 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:22:17.459 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:22:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:22:17.083+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:22:17.459 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:22:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:22:17.083+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:22:17.459 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:22:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:22:17.083+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:22:17.740 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:22:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:22:17.458+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:22:17.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:22:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:22:17.458+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:22:17.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:22:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:22:17.458+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:22:18.101 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:22:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:22:17.775+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:22:18.102 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:22:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:22:17.775+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:22:18.102 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:22:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:22:17.775+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:22:18.102 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:22:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:22:17.775+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:22:18.486 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:22:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:22:18.101+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:22:18.486 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:22:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:22:18.101+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:22:18.486 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:22:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:22:18.101+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:22:18.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:22:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:22:18.485+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:22:18.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:22:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:22:18.485+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:22:18.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:22:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:22:18.485+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:22:18.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:22:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:22:18.735+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:22:18.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:22:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:22:18.735+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:22:18.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:22:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:22:18.735+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:22:18.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:22:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:22:18.735+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:22:19.335 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:22:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:22:19.068+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:22:19.336 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:22:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:22:19.068+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:22:19.336 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:22:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:22:19.068+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:22:19.710 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:22:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:22:19.451+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:22:19.710 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:22:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:22:19.451+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:22:19.711 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:22:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:22:19.451+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:22:19.990 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:22:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:22:19.710+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:22:19.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:22:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:22:19.710+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:22:19.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:22:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:22:19.710+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:22:19.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:22:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:22:19.710+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:22:20.485 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:22:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:22:20.065+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:22:20.485 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:22:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:22:20.065+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:22:20.485 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:22:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:22:20.065+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:22:20.740 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:22:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:22:20.728+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:22:20.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:22:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:22:20.728+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:22:20.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:22:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:22:20.728+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:22:20.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:22:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:22:20.728+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:22:20.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:22:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:22:20.485+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:22:20.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:22:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:22:20.485+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:22:20.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:22:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:22:20.485+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:22:21.437 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:22:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:22:21.062+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:22:21.437 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:22:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:22:21.062+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:22:21.437 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:22:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:22:21.062+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:22:21.706 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:22:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:22:21.437+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:22:21.706 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:22:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:22:21.437+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:22:21.706 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:22:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:22:21.437+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:22:21.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:22:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:22:21.706+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:22:21.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:22:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:22:21.706+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:22:21.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:22:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:22:21.706+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:22:21.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:22:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:22:21.706+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:22:22.280 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:22:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:22:22.157+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 19562 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:22:22.281 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:22:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:22:22.019+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:22:22.281 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:22:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:22:22.019+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:22:22.281 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:22:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:22:22.019+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:22:22.696 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:22:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:22:22.424+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:22:22.696 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:22:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:22:22.424+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:22:22.696 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:22:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:22:22.424+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:22:22.981 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:22:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:22:22.695+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:22:22.982 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:22:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:22:22.695+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:22:22.982 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:22:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:22:22.695+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:22:22.982 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:22:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:22:22.695+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:22:23.240 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:22:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:22:22.981+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:22:23.240 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:22:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:22:22.981+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:22:23.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:22:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:22:22.981+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:22:23.735 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:22:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:22:23.472+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:22:23.735 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:22:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:22:23.472+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:22:23.735 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:22:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:22:23.472+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:22:23.736 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:22:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:22:23.686+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:22:23.736 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:22:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:22:23.686+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:22:23.736 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:22:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:22:23.686+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:22:23.736 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:22:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:22:23.686+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:22:24.490 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:22:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:22:23.996+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:22:24.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:22:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:22:23.996+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:22:24.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:22:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:22:23.996+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:22:24.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:22:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:22:24.704+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:22:24.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:22:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:22:24.704+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:22:24.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:22:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:22:24.704+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:22:24.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:22:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:22:24.704+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:22:24.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:22:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:22:24.510+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:22:24.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:22:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:22:24.510+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:22:24.992 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:22:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:22:24.510+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:22:25.490 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:22:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:22:25.018+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:22:25.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:22:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:22:25.018+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:22:25.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:22:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:22:25.018+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:22:25.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:22:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:22:25.734+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:22:25.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:22:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:22:25.734+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:22:25.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:22:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:22:25.734+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:22:25.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:22:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:22:25.734+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:22:25.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:22:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:22:25.514+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:22:25.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:22:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:22:25.514+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:22:25.992 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:22:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:22:25.514+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:22:26.490 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:22:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:22:26.049+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:22:26.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:22:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:22:26.049+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:22:26.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:22:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:22:26.049+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:22:26.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:22:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:22:26.762+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:22:26.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:22:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:22:26.762+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:22:26.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:22:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:22:26.762+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:22:26.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:22:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:22:26.762+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:22:26.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:22:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:22:26.522+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:22:26.992 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:22:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:22:26.522+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:22:26.992 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:22:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:22:26.522+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:22:27.491 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:22:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:22:27.160+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 19632 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:22:27.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:22:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:22:27.073+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:22:27.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:22:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:22:27.073+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:22:27.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:22:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:22:27.073+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:22:27.990 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:22:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:22:27.789+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:22:27.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:22:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:22:27.789+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:22:27.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:22:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:22:27.789+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:22:27.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:22:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:22:27.789+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:22:27.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:22:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:22:27.563+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:22:27.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:22:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:22:27.563+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:22:27.992 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:22:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:22:27.563+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:22:28.490 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:22:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:22:28.054+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:22:28.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:22:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:22:28.054+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:22:28.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:22:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:22:28.054+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:22:28.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:22:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:22:28.755+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:22:28.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:22:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:22:28.755+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:22:28.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:22:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:22:28.756+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:22:28.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:22:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:22:28.756+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:22:28.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:22:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:22:28.553+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:22:28.992 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:22:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:22:28.553+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:22:28.992 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:22:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:22:28.553+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:22:29.490 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:22:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:22:29.046+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:22:29.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:22:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:22:29.046+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:22:29.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:22:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:22:29.046+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:22:29.884 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:22:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:22:29.750+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:22:29.884 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:22:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:22:29.750+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:22:29.884 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:22:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:22:29.750+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:22:29.885 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:22:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:22:29.750+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:22:29.885 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:22:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:22:29.520+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:22:29.885 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:22:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:22:29.520+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:22:29.885 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:22:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:22:29.520+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:22:30.199 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:22:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:22:30.009+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:22:30.200 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:22:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:22:30.009+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:22:30.200 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:22:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:22:30.009+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:22:30.990 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:22:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:22:30.517+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:22:30.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:22:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:22:30.517+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:22:30.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:22:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:22:30.517+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:22:30.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:22:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:22:30.762+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:22:30.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:22:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:22:30.762+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:22:30.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:22:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:22:30.762+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:22:30.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:22:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:22:30.762+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:22:31.482 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:22:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:22:31.044+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:22:31.482 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:22:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:22:31.044+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:22:31.482 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:22:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:22:31.044+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:22:31.740 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:22:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:22:31.481+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:22:31.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:22:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:22:31.481+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:22:31.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:22:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:22:31.481+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:22:32.164 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:22:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:22:32.006+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:22:32.165 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:22:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:22:32.006+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:22:32.165 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:22:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:22:32.006+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:22:32.165 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:22:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:22:31.768+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:22:32.165 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:22:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:22:31.768+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:22:32.165 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:22:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:22:31.768+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:22:32.165 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:22:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:22:31.769+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:22:32.490 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:22:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:22:32.163+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 19703 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:22:32.789 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:22:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:22:32.518+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:22:32.789 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:22:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:22:32.518+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:22:32.789 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:22:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:22:32.518+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:22:33.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:22:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:22:32.997+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:22:33.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:22:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:22:32.997+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:22:33.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:22:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:22:32.997+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:22:33.241 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:22:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:22:32.788+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:22:33.241 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:22:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:22:32.788+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:22:33.241 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:22:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:22:32.788+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:22:33.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:22:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:22:32.788+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:22:33.970 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:22:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:22:33.564+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:22:33.970 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:22:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:22:33.564+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:22:33.970 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:22:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:22:33.564+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:22:33.970 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:22:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:22:33.751+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:22:33.970 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:22:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:22:33.751+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:22:33.970 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:22:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:22:33.751+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:22:33.971 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:22:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:22:33.751+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:22:34.240 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:22:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:22:33.969+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:22:34.240 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:22:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:22:33.969+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:22:34.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:22:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:22:33.969+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:22:34.941 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:22:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:22:34.545+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:22:34.941 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:22:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:22:34.545+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:22:34.941 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:22:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:22:34.545+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:22:34.942 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:22:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:22:34.749+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:22:34.942 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:22:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:22:34.749+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:22:34.942 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:22:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:22:34.749+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:22:34.942 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:22:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:22:34.749+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:22:35.240 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:22:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:22:34.941+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:22:35.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:22:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:22:34.941+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:22:35.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:22:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:22:34.941+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:22:35.982 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:22:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:22:35.514+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:22:35.982 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:22:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:22:35.514+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:22:35.982 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:22:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:22:35.514+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:22:35.982 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:22:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:22:35.715+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:22:35.983 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:22:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:22:35.715+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:22:35.983 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:22:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:22:35.715+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:22:35.983 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:22:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:22:35.715+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:22:36.240 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:22:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:22:35.981+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:22:36.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:22:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:22:35.981+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:22:36.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:22:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:22:35.981+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:22:36.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:22:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:22:36.709+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:22:36.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:22:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:22:36.709+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:22:36.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:22:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:22:36.709+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:22:36.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:22:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:22:36.709+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:22:36.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:22:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:22:36.473+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:22:36.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:22:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:22:36.473+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:22:36.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:22:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:22:36.473+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:22:37.167 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:22:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:22:36.989+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:22:37.168 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:22:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:22:36.989+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:22:37.168 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:22:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:22:36.989+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:22:37.425 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:22:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:22:37.166+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 19774 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:22:37.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:22:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:22:37.670+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:22:37.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:22:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:22:37.670+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:22:37.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:22:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:22:37.670+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:22:37.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:22:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:22:37.670+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:22:37.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:22:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:22:37.425+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:22:37.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:22:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:22:37.425+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:22:37.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:22:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:22:37.425+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:22:38.432 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:22:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:22:38.016+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:22:38.432 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:22:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:22:38.016+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:22:38.432 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:22:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:22:38.016+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:22:38.713 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:22:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:22:38.432+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:22:38.713 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:22:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:22:38.432+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:22:38.714 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:22:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:22:38.432+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:22:38.990 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:22:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:22:38.713+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:22:38.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:22:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:22:38.713+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:22:38.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:22:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:22:38.713+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:22:38.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:22:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:22:38.713+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:22:39.441 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:22:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:22:39.057+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:22:39.441 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:22:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:22:39.057+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:22:39.441 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:22:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:22:39.057+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:22:39.710 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:22:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:22:39.440+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:22:39.710 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:22:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:22:39.440+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:22:39.710 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:22:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:22:39.440+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:22:39.990 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:22:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:22:39.710+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:22:39.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:22:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:22:39.710+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:22:39.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:22:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:22:39.710+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:22:39.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:22:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:22:39.710+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:22:40.426 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:22:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:22:40.074+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:22:40.426 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:22:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:22:40.074+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:22:40.427 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:22:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:22:40.074+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:22:40.723 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:22:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:22:40.426+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:22:40.723 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:22:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:22:40.426+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:22:40.723 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:22:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:22:40.426+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:22:40.990 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:22:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:22:40.723+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:22:40.990 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:22:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:22:40.723+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:22:40.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:22:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:22:40.723+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:22:40.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:22:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:22:40.723+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:22:41.431 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:22:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:22:41.081+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:22:41.432 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:22:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:22:41.081+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:22:41.432 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:22:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:22:41.081+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:22:41.708 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:22:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:22:41.431+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:22:41.708 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:22:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:22:41.431+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:22:41.708 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:22:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:22:41.431+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:22:41.990 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:22:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:22:41.708+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:22:41.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:22:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:22:41.708+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:22:41.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:22:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:22:41.708+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:22:41.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:22:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:22:41.708+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:22:42.439 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:22:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:22:42.171+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 19845 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:22:42.439 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:22:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:22:42.117+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:22:42.439 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:22:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:22:42.117+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:22:42.439 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:22:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:22:42.117+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:22:42.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:22:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:22:42.674+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:22:42.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:22:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:22:42.674+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:22:42.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:22:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:22:42.674+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:22:42.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:22:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:22:42.674+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:22:42.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:22:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:22:42.439+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:22:42.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:22:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:22:42.439+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:22:42.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:22:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:22:42.439+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:22:43.406 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:22:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:22:43.094+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:22:43.406 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:22:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:22:43.094+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:22:43.406 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:22:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:22:43.094+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:22:43.692 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:22:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:22:43.406+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:22:43.693 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:22:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:22:43.406+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:22:43.693 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:22:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:22:43.406+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:22:43.990 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:22:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:22:43.692+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:22:43.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:22:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:22:43.692+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:22:43.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:22:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:22:43.692+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:22:43.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:22:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:22:43.692+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:22:44.422 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:22:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:22:44.128+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:22:44.423 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:22:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:22:44.128+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:22:44.423 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:22:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:22:44.128+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:22:44.707 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:22:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:22:44.422+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:22:44.707 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:22:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:22:44.422+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:22:44.707 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:22:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:22:44.422+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:22:44.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:22:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:22:44.707+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:22:44.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:22:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:22:44.707+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:22:44.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:22:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:22:44.707+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:22:44.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:22:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:22:44.707+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:22:45.438 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:22:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:22:45.161+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:22:45.439 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:22:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:22:45.161+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:22:45.439 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:22:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:22:45.161+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:22:45.715 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:22:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:22:45.438+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:22:45.716 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:22:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:22:45.438+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:22:45.716 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:22:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:22:45.438+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:22:45.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:22:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:22:45.716+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:22:45.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:22:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:22:45.716+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:22:45.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:22:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:22:45.716+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:22:45.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:22:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:22:45.716+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:22:46.472 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:22:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:22:46.144+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:22:46.472 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:22:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:22:46.144+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:22:46.472 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:22:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:22:46.144+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:22:46.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:22:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:22:46.700+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:22:46.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:22:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:22:46.700+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:22:46.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:22:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:22:46.700+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:22:46.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:22:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:22:46.700+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:22:46.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:22:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:22:46.472+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:22:46.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:22:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:22:46.472+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:22:46.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:22:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:22:46.472+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:22:47.461 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:22:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:22:47.173+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 19916 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:22:47.461 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:22:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:22:47.129+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:22:47.462 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:22:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:22:47.129+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:22:47.462 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:22:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:22:47.129+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:22:47.723 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:22:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:22:47.461+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:22:47.723 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:22:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:22:47.461+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:22:47.723 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:22:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:22:47.461+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:22:47.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:22:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:22:47.723+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:22:47.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:22:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:22:47.723+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:22:47.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:22:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:22:47.723+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:22:47.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:22:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:22:47.723+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:22:48.463 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:22:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:22:48.177+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:22:48.463 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:22:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:22:48.177+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:22:48.463 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:22:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:22:48.177+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:22:48.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:22:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:22:48.704+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:22:48.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:22:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:22:48.704+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:22:48.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:22:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:22:48.704+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:22:48.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:22:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:22:48.704+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:22:48.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:22:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:22:48.463+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:22:48.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:22:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:22:48.463+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:22:48.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:22:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:22:48.463+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:22:49.490 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:22:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:22:49.193+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:22:49.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:22:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:22:49.193+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:22:49.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:22:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:22:49.193+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:22:49.885 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:22:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:22:49.683+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:22:49.885 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:22:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:22:49.683+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:22:49.885 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:22:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:22:49.683+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:22:49.885 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:22:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:22:49.683+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:22:49.886 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:22:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:22:49.493+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:22:49.886 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:22:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:22:49.493+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:22:49.886 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:22:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:22:49.493+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:22:50.487 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:22:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:22:50.227+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:22:50.487 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:22:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:22:50.227+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:22:50.487 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:22:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:22:50.227+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:22:50.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:22:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:22:50.487+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:22:50.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:22:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:22:50.487+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:22:50.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:22:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:22:50.487+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:22:50.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:22:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:22:50.696+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:22:50.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:22:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:22:50.696+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:22:50.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:22:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:22:50.696+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:22:50.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:22:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:22:50.696+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:22:51.515 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:22:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:22:51.241+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:22:51.516 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:22:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:22:51.241+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:22:51.516 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:22:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:22:51.241+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:22:51.990 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:22:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:22:51.515+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:22:51.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:22:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:22:51.515+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:22:51.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:22:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:22:51.515+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:22:51.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:22:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:22:51.718+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:22:51.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:22:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:22:51.718+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:22:51.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:22:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:22:51.718+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:22:51.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:22:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:22:51.718+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:22:52.491 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:22:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:22:52.176+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 19987 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:22:52.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:22:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:22:52.270+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:22:52.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:22:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:22:52.270+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:22:52.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:22:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:22:52.270+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:22:52.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:22:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:22:52.733+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:22:52.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:22:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:22:52.733+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:22:52.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:22:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:22:52.733+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:22:52.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:22:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:22:52.733+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:22:52.992 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:22:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:22:52.540+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:22:52.992 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:22:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:22:52.540+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:22:52.992 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:22:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:22:52.540+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:22:53.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:22:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:22:53.303+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:22:53.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:22:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:22:53.303+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:22:53.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:22:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:22:53.303+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:22:53.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:22:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:22:53.522+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:22:53.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:22:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:22:53.522+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:22:53.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:22:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:22:53.522+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:22:54.241 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:22:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:22:53.781+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:22:54.241 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:22:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:22:53.781+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:22:54.241 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:22:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:22:53.781+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:22:54.241 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:22:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:22:53.781+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:22:54.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:22:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:22:54.307+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:22:54.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:22:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:22:54.307+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:22:54.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:22:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:22:54.307+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:22:54.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:22:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:22:54.527+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:22:54.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:22:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:22:54.527+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:22:54.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:22:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:22:54.527+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:22:55.241 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:22:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:22:54.794+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:22:55.241 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:22:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:22:54.794+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:22:55.241 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:22:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:22:54.794+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:22:55.241 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:22:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:22:54.794+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:22:55.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:22:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:22:55.347+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:22:55.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:22:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:22:55.347+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:22:55.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:22:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:22:55.347+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:22:55.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:22:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:22:55.571+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:22:55.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:22:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:22:55.571+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:22:55.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:22:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:22:55.571+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:22:56.240 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:22:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:22:55.827+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:22:56.241 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:22:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:22:55.827+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:22:56.241 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:22:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:22:55.827+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:22:56.241 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:22:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:22:55.827+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:22:56.613 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:22:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:22:56.342+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:22:56.614 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:22:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:22:56.342+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:22:56.614 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:22:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:22:56.342+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:22:56.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:22:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:22:56.613+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:22:56.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:22:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:22:56.613+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:22:56.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:22:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:22:56.613+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:22:56.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:22:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:22:56.833+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:22:56.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:22:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:22:56.833+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:22:56.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:22:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:22:56.833+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:22:56.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:22:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:22:56.833+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:22:57.491 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:22:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:22:57.179+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 20058 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:22:57.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:22:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:22:57.300+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:22:57.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:22:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:22:57.300+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:22:57.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:22:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:22:57.300+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:22:57.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:22:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:22:57.628+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:22:57.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:22:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:22:57.628+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:22:57.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:22:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:22:57.628+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:22:57.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:22:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:22:57.824+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:22:57.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:22:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:22:57.824+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:22:57.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:22:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:22:57.824+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:22:57.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:22:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:22:57.824+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:22:58.665 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:22:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:22:58.259+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:22:58.665 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:22:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:22:58.260+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:22:58.665 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:22:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:22:58.260+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:22:58.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:22:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:22:58.664+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:22:58.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:22:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:22:58.664+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:22:58.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:22:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:22:58.664+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:22:58.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:22:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:22:58.813+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:22:58.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:22:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:22:58.813+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:22:58.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:22:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:22:58.813+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:22:58.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:22:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:22:58.814+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:22:59.630 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:22:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:22:59.305+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:22:59.630 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:22:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:22:59.305+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:22:59.630 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:22:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:22:59.305+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:22:59.884 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:22:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:22:59.811+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:22:59.885 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:22:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:22:59.811+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:22:59.885 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:22:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:22:59.811+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:22:59.885 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:22:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:22:59.811+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:22:59.885 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:22:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:22:59.629+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:22:59.885 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:22:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:22:59.629+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:22:59.886 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:22:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:22:59.629+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:23:00.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:23:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:23:00.352+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:23:00.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:23:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:23:00.352+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:23:00.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:23:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:23:00.352+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:23:00.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:23:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:23:00.671+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:23:00.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:23:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:23:00.671+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:23:00.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:23:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:23:00.671+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:23:00.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:23:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:23:00.777+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:23:00.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:23:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:23:00.777+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:23:00.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:23:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:23:00.777+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:23:00.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:23:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:23:00.777+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:23:01.648 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:23:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:23:01.393+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:23:01.649 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:23:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:23:01.393+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:23:01.649 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:23:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:23:01.393+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:23:01.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:23:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:23:01.818+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:23:01.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:23:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:23:01.818+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:23:01.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:23:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:23:01.818+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:23:01.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:23:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:23:01.818+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:23:01.992 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:23:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:23:01.647+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:23:01.992 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:23:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:23:01.647+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:23:01.992 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:23:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:23:01.647+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:23:02.491 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:23:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:23:02.181+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 20129 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:23:02.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:23:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:23:02.419+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:23:02.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:23:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:23:02.419+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:23:02.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:23:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:23:02.419+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:23:02.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:23:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:23:02.812+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:23:02.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:23:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:23:02.812+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:23:02.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:23:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:23:02.812+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:23:02.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:23:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:23:02.812+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:23:02.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:23:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:23:02.685+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:23:02.992 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:23:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:23:02.685+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:23:02.992 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:23:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:23:02.685+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:23:03.721 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:23:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:23:03.445+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:23:03.722 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:23:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:23:03.445+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:23:03.722 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:23:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:23:03.445+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:23:03.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:23:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:23:03.770+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:23:03.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:23:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:23:03.770+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:23:03.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:23:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:23:03.770+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:23:03.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:23:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:23:03.770+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:23:03.992 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:23:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:23:03.721+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:23:03.992 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:23:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:23:03.721+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:23:03.992 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:23:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:23:03.721+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:23:04.715 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:23:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:23:04.396+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:23:04.716 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:23:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:23:04.396+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:23:04.716 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:23:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:23:04.396+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:23:04.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:23:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:23:04.714+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:23:04.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:23:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:23:04.714+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:23:04.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:23:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:23:04.714+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:23:04.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:23:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:23:04.811+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:23:04.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:23:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:23:04.811+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:23:04.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:23:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:23:04.811+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:23:04.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:23:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:23:04.811+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:23:05.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:23:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:23:05.424+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:23:05.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:23:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:23:05.424+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:23:05.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:23:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:23:05.424+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:23:06.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:23:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:23:05.741+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:23:06.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:23:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:23:05.741+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:23:06.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:23:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:23:05.741+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:23:06.241 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:23:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:23:05.850+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:23:06.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:23:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:23:05.850+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:23:06.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:23:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:23:05.850+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:23:06.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:23:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:23:05.850+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:23:06.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:23:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:23:06.379+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:23:06.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:23:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:23:06.379+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:23:06.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:23:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:23:06.379+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:23:07.186 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:23:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:23:06.773+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:23:07.186 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:23:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:23:06.773+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:23:07.186 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:23:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:23:06.773+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:23:07.186 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:23:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:23:06.820+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:23:07.186 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:23:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:23:06.820+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:23:07.186 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:23:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:23:06.820+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:23:07.187 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:23:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:23:06.820+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:23:07.491 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:23:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:23:07.184+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 20200 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:23:07.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:23:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:23:07.414+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:23:07.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:23:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:23:07.414+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:23:07.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:23:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:23:07.414+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:23:07.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:23:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:23:07.778+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:23:07.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:23:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:23:07.778+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:23:07.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:23:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:23:07.778+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:23:07.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:23:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:23:07.778+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:23:07.992 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:23:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:23:07.735+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:23:07.992 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:23:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:23:07.735+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:23:07.992 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:23:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:23:07.735+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:23:08.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:23:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:23:08.388+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:23:08.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:23:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:23:08.388+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:23:08.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:23:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:23:08.388+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:23:09.241 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:23:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:23:08.816+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:23:09.241 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:23:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:23:08.816+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:23:09.241 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:23:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:23:08.816+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:23:09.241 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:23:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:23:08.816+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:23:09.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:23:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:23:08.767+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:23:09.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:23:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:23:08.767+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:23:09.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:23:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:23:08.767+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:23:09.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:23:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:23:09.403+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:23:09.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:23:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:23:09.403+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:23:09.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:23:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:23:09.403+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:23:10.202 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:23:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:23:09.758+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:23:10.203 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:23:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:23:09.758+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:23:10.203 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:23:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:23:09.758+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:23:10.203 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:23:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:23:09.843+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:23:10.203 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:23:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:23:09.843+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:23:10.203 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:23:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:23:09.843+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:23:10.204 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:23:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:23:09.843+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:23:10.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:23:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:23:10.395+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:23:10.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:23:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:23:10.395+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:23:10.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:23:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:23:10.395+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:23:11.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:23:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:23:10.792+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:23:11.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:23:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:23:10.792+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:23:11.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:23:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:23:10.792+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:23:11.241 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:23:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:23:10.861+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:23:11.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:23:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:23:10.861+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:23:11.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:23:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:23:10.861+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:23:11.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:23:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:23:10.861+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:23:11.734 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:23:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:23:11.351+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:23:11.734 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:23:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:23:11.351+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:23:11.734 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:23:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:23:11.351+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:23:11.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:23:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:23:11.824+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:23:11.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:23:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:23:11.824+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:23:11.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:23:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:23:11.824+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:23:11.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:23:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:23:11.844+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:23:11.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:23:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:23:11.844+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:23:11.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:23:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:23:11.844+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:23:11.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:23:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:23:11.844+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:23:12.491 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:23:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:23:12.187+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 20271 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:23:12.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:23:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:23:12.355+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:23:12.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:23:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:23:12.355+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:23:12.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:23:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:23:12.355+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:23:13.241 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:23:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:23:12.878+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:23:13.241 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:23:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:23:12.878+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:23:13.241 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:23:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:23:12.878+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:23:13.241 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:23:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:23:12.878+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:23:13.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:23:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:23:12.778+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:23:13.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:23:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:23:12.778+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:23:13.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:23:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:23:12.778+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:23:13.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:23:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:23:13.335+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:23:13.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:23:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:23:13.335+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:23:13.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:23:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:23:13.335+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:23:14.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:23:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:23:13.757+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:23:14.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:23:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:23:13.757+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:23:14.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:23:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:23:13.757+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:23:14.241 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:23:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:23:13.910+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:23:14.241 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:23:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:23:13.910+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:23:14.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:23:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:23:13.910+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:23:14.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:23:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:23:13.910+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:23:14.735 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:23:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:23:14.332+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:23:14.735 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:23:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:23:14.332+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:23:14.735 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:23:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:23:14.332+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:23:14.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:23:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:23:14.734+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:23:14.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:23:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:23:14.734+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:23:14.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:23:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:23:14.734+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:23:14.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:23:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:23:14.918+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:23:14.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:23:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:23:14.918+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:23:14.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:23:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:23:14.918+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:23:14.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:23:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:23:14.918+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:23:15.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:23:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:23:15.339+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:23:15.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:23:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:23:15.339+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:23:15.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:23:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:23:15.339+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:23:16.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:23:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:23:15.766+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:23:16.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:23:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:23:15.766+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:23:16.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:23:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:23:15.766+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:23:16.241 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:23:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:23:15.951+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:23:16.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:23:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:23:15.951+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:23:16.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:23:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:23:15.951+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:23:16.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:23:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:23:15.951+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:23:16.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:23:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:23:16.310+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:23:16.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:23:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:23:16.310+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:23:16.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:23:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:23:16.310+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:23:17.192 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:23:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:23:16.927+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:23:17.192 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:23:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:23:16.927+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:23:17.192 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:23:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:23:16.927+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:23:17.192 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:23:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:23:16.927+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:23:17.193 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:23:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:23:16.752+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:23:17.193 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:23:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:23:16.752+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:23:17.193 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:23:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:23:16.752+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:23:17.491 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:23:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:23:17.190+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 20373 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:23:17.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:23:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:23:17.295+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:23:17.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:23:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:23:17.295+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:23:17.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:23:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:23:17.295+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:23:18.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:23:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:23:17.796+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:23:18.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:23:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:23:17.796+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:23:18.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:23:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:23:17.796+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:23:18.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:23:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:23:17.942+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:23:18.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:23:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:23:17.942+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:23:18.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:23:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:23:17.942+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:23:18.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:23:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:23:17.942+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:23:18.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:23:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:23:18.327+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:23:18.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:23:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:23:18.327+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:23:18.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:23:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:23:18.327+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:23:19.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:23:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:23:18.812+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:23:19.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:23:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:23:18.812+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:23:19.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:23:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:23:18.812+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:23:19.241 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:23:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:23:18.946+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:23:19.241 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:23:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:23:18.946+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:23:19.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:23:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:23:18.946+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:23:19.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:23:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:23:18.946+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:23:19.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:23:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:23:19.334+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:23:19.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:23:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:23:19.334+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:23:19.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:23:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:23:19.334+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:23:20.203 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:23:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:23:19.770+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:23:20.203 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:23:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:23:19.770+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:23:20.203 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:23:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:23:19.770+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:23:20.203 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:23:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:23:19.961+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:23:20.204 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:23:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:23:19.961+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:23:20.204 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:23:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:23:19.961+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:23:20.204 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:23:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:23:19.961+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:23:20.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:23:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:23:20.357+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:23:20.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:23:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:23:20.357+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:23:20.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:23:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:23:20.357+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:23:21.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:23:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:23:20.754+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:23:21.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:23:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:23:20.754+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:23:21.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:23:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:23:20.754+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:23:21.241 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:23:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:23:20.965+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:23:21.241 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:23:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:23:20.965+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:23:21.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:23:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:23:20.965+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:23:21.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:23:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:23:20.965+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:23:21.734 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:23:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:23:21.358+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:23:21.734 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:23:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:23:21.358+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:23:21.735 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:23:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:23:21.358+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:23:21.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:23:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:23:21.777+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:23:21.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:23:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:23:21.777+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:23:21.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:23:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:23:21.777+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:23:22.317 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:23:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:23:22.194+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 20444 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:23:22.317 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:23:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:23:21.997+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:23:22.317 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:23:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:23:21.997+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:23:22.317 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:23:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:23:21.997+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:23:22.317 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:23:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:23:21.997+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:23:22.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:23:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:23:22.336+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:23:22.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:23:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:23:22.337+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:23:22.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:23:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:23:22.337+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:23:23.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:23:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:23:22.741+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:23:23.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:23:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:23:22.741+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:23:23.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:23:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:23:22.741+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:23:23.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:23:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:23:22.973+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:23:23.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:23:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:23:22.973+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:23:23.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:23:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:23:22.973+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:23:23.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:23:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:23:22.973+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:23:23.736 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:23:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:23:23.307+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:23:23.736 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:23:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:23:23.307+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:23:23.736 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:23:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:23:23.307+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:23:23.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:23:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:23:23.734+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:23:23.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:23:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:23:23.734+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:23:23.992 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:23:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:23:23.734+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:23:24.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:23:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:23:24.267+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:23:24.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:23:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:23:24.267+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:23:24.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:23:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:23:24.267+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:23:24.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:23:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:23:24.020+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:23:24.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:23:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:23:24.020+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:23:24.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:23:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:23:24.020+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:23:24.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:23:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:23:24.020+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:23:25.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:23:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:23:24.782+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:23:25.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:23:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:23:24.782+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:23:25.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:23:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:23:24.782+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:23:25.241 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:23:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:23:24.984+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:23:25.241 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:23:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:23:24.984+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:23:25.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:23:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:23:24.984+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:23:25.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:23:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:23:24.984+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:23:25.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:23:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:23:25.255+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:23:25.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:23:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:23:25.255+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:23:25.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:23:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:23:25.255+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:23:26.235 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:23:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:23:25.790+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:23:26.235 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:23:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:23:25.790+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:23:26.235 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:23:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:23:25.790+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:23:26.235 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:23:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:23:25.964+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:23:26.236 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:23:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:23:25.964+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:23:26.236 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:23:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:23:25.964+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:23:26.236 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:23:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:23:25.964+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:23:26.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:23:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:23:26.234+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:23:26.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:23:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:23:26.234+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:23:26.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:23:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:23:26.234+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:23:27.200 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:23:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:23:26.817+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:23:27.200 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:23:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:23:26.817+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:23:27.201 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:23:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:23:26.817+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:23:27.201 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:23:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:23:26.995+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:23:27.201 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:23:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:23:26.995+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:23:27.201 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:23:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:23:26.995+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:23:27.201 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:23:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:23:26.995+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:23:27.491 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:23:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:23:27.196+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 20515 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:23:27.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:23:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:23:27.202+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:23:27.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:23:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:23:27.202+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:23:27.492 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:23:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:23:27.202+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:23:28.230 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:23:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:23:27.843+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:23:28.230 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:23:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:23:27.843+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:23:28.230 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:23:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:23:27.843+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:23:28.231 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:23:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:23:27.974+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:23:28.231 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:23:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:23:27.974+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:23:28.231 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:23:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:23:27.974+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:23:28.231 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:23:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:23:27.974+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:23:28.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:23:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:23:28.229+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:23:28.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:23:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:23:28.229+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:23:28.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:23:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:23:28.229+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:23:29.202 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:23:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:23:28.825+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:23:29.202 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:23:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:23:28.825+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:23:29.202 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:23:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:23:28.825+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:23:29.202 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:23:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:23:29.007+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:23:29.203 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:23:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:23:29.007+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:23:29.203 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:23:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:23:29.007+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:23:29.203 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:23:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:23:29.007+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:23:29.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:23:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:23:29.201+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:23:29.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:23:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:23:29.201+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:23:29.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:23:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:23:29.201+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:23:30.185 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:23:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:23:29.828+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:23:30.185 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:23:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:23:29.828+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:23:30.185 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:23:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:23:29.828+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:23:30.185 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:23:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:23:29.964+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:23:30.185 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:23:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:23:29.964+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:23:30.185 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:23:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:23:29.964+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:23:30.186 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:23:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:23:29.964+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:23:30.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:23:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:23:30.184+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:23:30.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:23:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:23:30.184+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:23:30.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:23:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:23:30.184+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:23:31.137 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:23:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:23:30.828+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:23:31.137 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:23:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:23:30.828+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:23:31.138 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:23:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:23:30.828+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:23:31.138 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:23:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:23:30.918+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:23:31.138 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:23:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:23:30.918+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:23:31.138 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:23:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:23:30.918+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:23:31.138 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:23:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:23:30.918+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:23:31.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:23:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:23:31.137+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:23:31.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:23:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:23:31.137+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:23:31.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:23:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:23:31.137+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:23:31.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:23:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:23:31.811+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:23:31.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:23:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:23:31.811+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:23:31.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:23:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:23:31.811+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:23:31.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:23:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:23:31.936+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:23:31.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:23:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:23:31.936+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:23:31.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:23:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:23:31.936+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:23:31.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:23:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:23:31.936+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:23:32.491 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:23:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:23:32.200+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 20586 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:23:32.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:23:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:23:32.111+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:23:32.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:23:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:23:32.111+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:23:32.492 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:23:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:23:32.111+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:23:33.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:23:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:23:33.067+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:23:33.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:23:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:23:33.067+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:23:33.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:23:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:23:33.067+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:23:33.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:23:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:23:32.856+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:23:33.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:23:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:23:32.856+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:23:33.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:23:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:23:32.856+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:23:33.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:23:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:23:32.959+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:23:33.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:23:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:23:32.959+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:23:33.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:23:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:23:32.959+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:23:33.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:23:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:23:32.959+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:23:34.241 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:23:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:23:34.003+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:23:34.241 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:23:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:23:34.003+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:23:34.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:23:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:23:34.003+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:23:34.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:23:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:23:34.004+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:23:34.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:23:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:23:34.089+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:23:34.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:23:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:23:34.089+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:23:34.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:23:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:23:34.089+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:23:34.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:23:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:23:33.848+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:23:34.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:23:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:23:33.848+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:23:34.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:23:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:23:33.848+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:23:35.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:23:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:23:35.044+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:23:35.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:23:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:23:35.044+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:23:35.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:23:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:23:35.044+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:23:35.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:23:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:23:34.803+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:23:35.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:23:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:23:34.803+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:23:35.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:23:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:23:34.803+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:23:35.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:23:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:23:34.989+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:23:35.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:23:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:23:34.989+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:23:35.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:23:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:23:34.989+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:23:35.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:23:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:23:34.989+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:23:36.017 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:23:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:23:35.755+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:23:36.018 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:23:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:23:35.755+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:23:36.018 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:23:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:23:35.755+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:23:36.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:23:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:23:36.063+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:23:36.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:23:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:23:36.063+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:23:36.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:23:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:23:36.063+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:23:36.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:23:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:23:36.017+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:23:36.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:23:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:23:36.017+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:23:36.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:23:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:23:36.017+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:23:36.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:23:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:23:36.017+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:23:37.049 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:23:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:23:36.775+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:23:37.049 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:23:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:23:36.775+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:23:37.049 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:23:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:23:36.775+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:23:37.326 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:23:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:23:37.204+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 20657 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:23:37.327 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:23:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:23:37.108+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:23:37.327 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:23:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:23:37.108+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:23:37.327 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:23:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:23:37.108+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:23:37.327 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:23:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:23:37.049+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:23:37.327 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:23:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:23:37.049+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:23:37.327 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:23:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:23:37.049+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:23:37.328 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:23:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:23:37.049+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:23:38.130 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:23:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:23:37.787+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:23:38.130 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:23:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:23:37.787+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:23:38.130 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:23:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:23:37.787+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:23:38.130 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:23:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:23:38.007+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:23:38.130 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:23:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:23:38.007+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:23:38.131 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:23:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:23:38.007+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:23:38.131 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:23:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:23:38.007+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:23:38.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:23:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:23:38.129+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:23:38.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:23:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:23:38.129+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:23:38.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:23:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:23:38.129+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:23:38.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:23:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:23:38.738+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:23:38.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:23:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:23:38.738+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:23:38.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:23:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:23:38.738+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:23:38.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:23:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:23:38.977+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:23:38.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:23:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:23:38.977+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:23:38.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:23:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:23:38.977+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:23:38.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:23:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:23:38.977+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:23:39.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:23:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:23:39.152+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:23:39.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:23:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:23:39.152+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:23:39.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:23:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:23:39.152+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:23:39.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:23:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:23:39.717+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:23:39.992 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:23:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:23:39.717+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:23:39.992 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:23:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:23:39.717+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:23:39.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:23:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:23:39.942+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:23:39.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:23:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:23:39.942+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:23:39.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:23:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:23:39.942+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:23:39.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:23:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:23:39.942+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:23:40.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:23:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:23:40.130+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:23:40.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:23:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:23:40.130+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:23:40.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:23:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:23:40.130+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:23:40.902 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:23:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:23:40.702+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:23:40.902 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:23:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:23:40.702+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:23:40.902 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:23:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:23:40.702+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:23:41.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:23:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:23:41.107+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:23:41.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:23:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:23:41.107+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:23:41.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:23:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:23:41.107+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:23:41.241 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:23:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:23:40.901+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:23:41.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:23:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:23:40.901+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:23:41.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:23:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:23:40.901+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:23:41.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:23:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:23:40.902+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:23:41.948 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:23:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:23:41.671+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:23:41.949 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:23:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:23:41.671+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:23:41.949 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:23:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:23:41.671+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:23:42.208 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:23:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:23:42.064+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:23:42.208 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:23:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:23:42.064+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:23:42.208 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:23:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:23:42.064+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:23:42.208 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:23:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:23:41.948+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:23:42.208 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:23:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:23:41.948+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:23:42.209 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:23:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:23:41.948+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:23:42.209 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:23:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:23:41.948+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:23:42.491 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:23:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:23:42.207+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 20728 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:23:42.978 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:23:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:23:42.636+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:23:42.978 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:23:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:23:42.636+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:23:42.979 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:23:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:23:42.636+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:23:43.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:23:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:23:43.051+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:23:43.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:23:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:23:43.051+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:23:43.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:23:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:23:43.051+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:23:43.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:23:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:23:42.978+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:23:43.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:23:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:23:42.978+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:23:43.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:23:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:23:42.978+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:23:43.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:23:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:23:42.978+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:23:43.948 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:23:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:23:43.620+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:23:43.948 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:23:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:23:43.620+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:23:43.948 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:23:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:23:43.620+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:23:44.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:23:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:23:44.096+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:23:44.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:23:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:23:44.096+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:23:44.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:23:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:23:44.096+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:23:44.241 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:23:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:23:43.948+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:23:44.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:23:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:23:43.948+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:23:44.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:23:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:23:43.948+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:23:44.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:23:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:23:43.948+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:23:44.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:23:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:23:44.614+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:23:44.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:23:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:23:44.614+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:23:44.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:23:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:23:44.614+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:23:45.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:23:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:23:45.091+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:23:45.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:23:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:23:45.091+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:23:45.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:23:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:23:45.091+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:23:45.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:23:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:23:44.993+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:23:45.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:23:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:23:44.993+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:23:45.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:23:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:23:44.993+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:23:45.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:23:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:23:44.993+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:23:45.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:23:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:23:45.592+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:23:45.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:23:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:23:45.592+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:23:45.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:23:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:23:45.592+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:23:46.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:23:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:23:46.060+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:23:46.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:23:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:23:46.060+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:23:46.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:23:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:23:46.060+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:23:46.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:23:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:23:46.012+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:23:46.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:23:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:23:46.012+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:23:46.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:23:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:23:46.012+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:23:46.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:23:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:23:46.012+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:23:46.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:23:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:23:46.556+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:23:46.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:23:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:23:46.556+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:23:46.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:23:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:23:46.556+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:23:47.335 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:23:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:23:47.210+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 20799 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:23:47.335 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:23:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:23:47.088+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:23:47.335 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:23:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:23:47.088+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:23:47.335 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:23:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:23:47.088+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:23:47.336 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:23:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:23:47.022+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:23:47.336 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:23:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:23:47.022+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:23:47.336 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:23:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:23:47.022+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:23:47.336 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:23:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:23:47.022+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:23:47.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:23:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:23:47.510+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:23:47.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:23:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:23:47.510+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:23:47.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:23:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:23:47.510+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:23:48.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:23:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:23:48.054+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:23:48.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:23:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:23:48.054+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:23:48.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:23:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:23:48.054+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:23:48.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:23:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:23:47.983+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:23:48.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:23:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:23:47.983+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:23:48.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:23:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:23:47.983+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:23:48.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:23:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:23:47.983+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:23:48.949 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:23:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:23:48.511+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:23:48.950 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:23:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:23:48.511+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:23:48.950 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:23:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:23:48.511+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:23:49.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:23:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:23:49.022+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:23:49.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:23:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:23:49.022+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:23:49.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:23:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:23:49.022+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:23:49.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:23:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:23:48.949+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:23:49.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:23:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:23:48.949+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:23:49.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:23:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:23:48.949+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:23:49.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:23:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:23:48.949+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:23:49.885 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:23:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:23:49.561+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:23:49.885 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:23:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:23:49.561+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:23:49.885 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:23:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:23:49.561+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:23:50.205 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:23:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:23:49.933+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:23:50.206 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:23:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:23:49.933+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:23:50.206 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:23:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:23:49.933+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:23:50.206 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:23:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:23:49.933+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:23:50.206 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:23:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:23:50.058+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:23:50.206 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:23:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:23:50.058+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:23:50.206 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:23:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:23:50.058+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:23:50.904 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:23:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:23:50.584+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:23:50.904 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:23:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:23:50.584+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:23:50.904 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:23:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:23:50.584+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:23:51.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:23:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:23:51.040+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:23:51.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:23:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:23:51.040+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:23:51.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:23:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:23:51.040+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:23:51.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:23:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:23:50.903+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:23:51.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:23:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:23:50.903+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:23:51.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:23:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:23:50.903+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:23:51.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:23:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:23:50.903+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:23:51.867 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:23:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:23:51.556+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:23:51.867 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:23:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:23:51.556+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:23:51.867 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:23:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:23:51.556+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:23:52.214 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:23:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:23:52.040+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:23:52.214 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:23:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:23:52.040+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:23:52.214 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:23:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:23:52.040+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:23:52.215 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:23:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:23:51.866+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:23:52.215 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:23:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:23:51.866+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:23:52.215 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:23:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:23:51.866+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:23:52.215 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:23:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:23:51.866+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:23:52.491 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:23:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:23:52.212+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 20870 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:23:52.824 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:23:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:23:52.564+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:23:52.824 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:23:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:23:52.564+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:23:52.824 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:23:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:23:52.564+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:23:53.241 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:23:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:23:52.908+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:23:53.241 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:23:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:23:52.908+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:23:53.241 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:23:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:23:52.908+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:23:53.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:23:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:23:52.908+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:23:53.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:23:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:23:53.062+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:23:53.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:23:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:23:53.062+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:23:53.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:23:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:23:53.062+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:23:53.874 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:23:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:23:53.576+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:23:53.875 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:23:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:23:53.576+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:23:53.875 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:23:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:23:53.576+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:23:54.241 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:23:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:23:53.873+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:23:54.241 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:23:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:23:53.873+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:23:54.241 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:23:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:23:53.873+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:23:54.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:23:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:23:53.873+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:23:54.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:23:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:23:54.017+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:23:54.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:23:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:23:54.017+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:23:54.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:23:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:23:54.017+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:23:54.906 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:23:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:23:54.581+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:23:54.906 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:23:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:23:54.581+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:23:54.906 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:23:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:23:54.581+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:23:55.241 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:23:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:23:54.905+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:23:55.241 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:23:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:23:54.905+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:23:55.241 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:23:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:23:54.905+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:23:55.241 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:23:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:23:54.905+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:23:55.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:23:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:23:55.021+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:23:55.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:23:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:23:55.021+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:23:55.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:23:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:23:55.021+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:23:55.908 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:23:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:23:55.536+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:23:55.908 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:23:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:23:55.536+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:23:55.908 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:23:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:23:55.536+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:23:56.241 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:23:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:23:55.907+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:23:56.241 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:23:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:23:55.907+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:23:56.241 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:23:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:23:55.907+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:23:56.241 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:23:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:23:55.907+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:23:56.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:23:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:23:56.009+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:23:56.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:23:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:23:56.009+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:23:56.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:23:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:23:56.009+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:23:56.891 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:23:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:23:56.491+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:23:56.891 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:23:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:23:56.491+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:23:56.891 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:23:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:23:56.491+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:23:57.217 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:23:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:23:56.969+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:23:57.217 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:23:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:23:56.969+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:23:57.218 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:23:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:23:56.969+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:23:57.218 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:23:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:23:56.890+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:23:57.218 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:23:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:23:56.890+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:23:57.218 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:23:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:23:56.890+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:23:57.218 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:23:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:23:56.890+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:23:57.491 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:23:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:23:57.216+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 20941 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:23:57.897 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:23:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:23:57.500+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:23:57.898 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:23:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:23:57.500+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:23:57.898 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:23:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:23:57.500+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:23:58.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:23:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:23:57.964+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:23:58.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:23:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:23:57.964+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:23:58.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:23:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:23:57.964+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:23:58.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:23:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:23:57.896+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:23:58.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:23:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:23:57.896+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:23:58.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:23:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:23:57.896+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:23:58.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:23:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:23:57.896+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:23:58.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:23:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:23:58.452+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:23:58.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:23:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:23:58.452+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:23:58.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:23:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:23:58.452+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:23:59.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:23:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:23:58.971+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:23:59.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:23:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:23:58.971+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:23:59.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:23:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:23:58.971+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:23:59.241 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:23:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:23:58.883+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:23:59.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:23:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:23:58.883+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:23:59.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:23:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:23:58.883+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:23:59.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:23:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:23:58.883+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:23:59.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:23:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:23:59.442+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:23:59.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:23:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:23:59.442+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:23:59.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:23:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:23:59.442+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:24:00.206 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:23:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:23:59.929+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:24:00.206 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:23:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:23:59.929+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:24:00.206 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:23:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:23:59.929+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:24:00.206 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:23:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:23:59.835+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:24:00.207 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:23:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:23:59.835+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:24:00.207 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:23:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:23:59.835+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:24:00.207 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:23:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:23:59.835+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:24:00.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:24:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:24:00.439+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:24:00.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:24:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:24:00.439+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:24:00.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:24:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:24:00.439+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:24:01.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:24:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:24:00.893+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:24:01.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:24:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:24:00.893+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:24:01.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:24:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:24:00.893+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:24:01.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:24:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:24:00.797+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:24:01.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:24:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:24:00.797+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:24:01.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:24:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:24:00.797+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:24:01.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:24:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:24:00.797+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:24:01.736 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:24:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:24:01.476+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:24:01.736 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:24:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:24:01.476+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:24:01.737 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:24:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:24:01.476+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:24:01.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:24:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:24:01.851+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:24:01.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:24:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:24:01.851+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:24:01.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:24:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:24:01.851+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:24:01.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:24:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:24:01.806+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:24:01.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:24:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:24:01.806+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:24:01.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:24:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:24:01.806+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:24:01.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:24:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:24:01.807+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:24:02.342 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:24:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:24:02.219+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 21012 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:24:02.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:24:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:24:02.512+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:24:02.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:24:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:24:02.512+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:24:02.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:24:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:24:02.512+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:24:03.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:24:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:24:02.873+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:24:03.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:24:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:24:02.873+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:24:03.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:24:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:24:02.873+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:24:03.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:24:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:24:02.776+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:24:03.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:24:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:24:02.776+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:24:03.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:24:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:24:02.776+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:24:03.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:24:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:24:02.776+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:24:03.830 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:24:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:24:03.498+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:24:03.830 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:24:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:24:03.498+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:24:03.830 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:24:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:24:03.498+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:24:03.830 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:24:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:24:03.732+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:24:03.830 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:24:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:24:03.732+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:24:03.830 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:24:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:24:03.732+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:24:03.831 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:24:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:24:03.732+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:24:04.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:24:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:24:03.828+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:24:04.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:24:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:24:03.828+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:24:04.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:24:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:24:03.828+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:24:04.832 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:24:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:24:04.539+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:24:04.832 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:24:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:24:04.539+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:24:04.833 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:24:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:24:04.539+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:24:04.833 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:24:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:24:04.742+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:24:04.833 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:24:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:24:04.742+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:24:04.833 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:24:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:24:04.742+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:24:04.833 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:24:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:24:04.742+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:24:05.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:24:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:24:04.831+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:24:05.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:24:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:24:04.831+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:24:05.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:24:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:24:04.831+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:24:05.766 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:24:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:24:05.502+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:24:05.766 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:24:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:24:05.502+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:24:05.766 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:24:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:24:05.502+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:24:06.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:24:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:24:05.862+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:24:06.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:24:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:24:05.862+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:24:06.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:24:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:24:05.862+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:24:06.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:24:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:24:05.764+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:24:06.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:24:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:24:05.764+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:24:06.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:24:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:24:05.764+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:24:06.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:24:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:24:05.765+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:24:06.876 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:24:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:24:06.764+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:24:06.877 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:24:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:24:06.764+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:24:06.877 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:24:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:24:06.764+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:24:06.877 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:24:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:24:06.764+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:24:06.877 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:24:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:24:06.548+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:24:06.877 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:24:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:24:06.548+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:24:06.877 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:24:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:24:06.548+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:24:07.223 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:24:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:24:06.875+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:24:07.224 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:24:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:24:06.875+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:24:07.224 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:24:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:24:06.875+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:24:07.491 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:24:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:24:07.222+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 21083 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:24:07.898 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:24:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:24:07.788+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:24:07.898 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:24:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:24:07.788+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:24:07.898 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:24:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:24:07.788+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:24:07.898 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:24:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:24:07.788+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:24:07.899 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:24:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:24:07.576+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:24:07.899 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:24:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:24:07.576+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:24:07.899 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:24:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:24:07.576+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:24:08.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:24:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:24:07.896+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:24:08.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:24:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:24:07.896+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:24:08.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:24:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:24:07.896+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:24:08.824 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:24:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:24:08.539+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:24:08.824 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:24:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:24:08.539+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:24:08.825 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:24:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:24:08.539+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:24:09.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:24:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:24:08.940+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:24:09.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:24:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:24:08.940+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:24:09.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:24:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:24:08.940+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:24:09.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:24:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:24:08.823+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:24:09.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:24:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:24:08.823+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:24:09.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:24:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:24:08.823+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:24:09.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:24:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:24:08.823+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:24:09.885 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:24:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:24:09.781+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:24:09.885 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:24:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:24:09.781+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:24:09.886 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:24:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:24:09.781+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:24:09.886 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:24:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:24:09.781+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:24:09.886 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:24:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:24:09.572+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:24:09.886 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:24:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:24:09.572+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:24:09.886 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:24:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:24:09.572+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:24:10.207 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:24:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:24:09.905+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:24:10.207 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:24:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:24:09.905+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:24:10.207 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:24:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:24:09.905+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:24:10.938 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:24:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:24:10.574+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:24:10.938 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:24:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:24:10.574+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:24:10.938 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:24:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:24:10.574+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:24:10.938 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:24:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:24:10.774+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:24:10.939 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:24:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:24:10.774+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:24:10.939 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:24:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:24:10.774+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:24:10.939 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:24:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:24:10.774+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:24:11.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:24:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:24:10.937+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:24:11.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:24:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:24:10.937+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:24:11.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:24:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:24:10.937+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:24:11.951 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:24:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:24:11.535+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:24:11.951 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:24:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:24:11.535+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:24:11.951 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:24:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:24:11.535+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:24:11.951 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:24:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:24:11.757+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:24:11.951 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:24:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:24:11.757+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:24:11.952 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:24:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:24:11.757+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:24:11.952 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:24:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:24:11.757+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:24:12.227 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:24:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:24:11.950+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:24:12.227 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:24:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:24:11.950+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:24:12.227 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:24:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:24:11.950+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:24:12.491 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:24:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:24:12.225+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 21154 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:24:12.908 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:24:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:24:12.571+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:24:12.908 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:24:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:24:12.571+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:24:12.908 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:24:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:24:12.571+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:24:12.908 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:24:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:24:12.793+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:24:12.908 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:24:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:24:12.793+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:24:12.909 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:24:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:24:12.793+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:24:12.909 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:24:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:24:12.793+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:24:13.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:24:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:24:12.907+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:24:13.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:24:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:24:12.907+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:24:13.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:24:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:24:12.907+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:24:13.816 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:24:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:24:13.815+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:24:13.817 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:24:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:24:13.815+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:24:13.817 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:24:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:24:13.815+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:24:13.817 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:24:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:24:13.815+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:24:13.817 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:24:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:24:13.565+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:24:13.817 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:24:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:24:13.565+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:24:13.817 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:24:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:24:13.565+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:24:14.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:24:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:24:13.865+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:24:14.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:24:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:24:13.865+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:24:14.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:24:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:24:13.865+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:24:14.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:24:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:24:14.547+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:24:14.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:24:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:24:14.547+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:24:14.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:24:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:24:14.547+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:24:15.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:24:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:24:14.848+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:24:15.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:24:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:24:14.848+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:24:15.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:24:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:24:14.848+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:24:15.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:24:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:24:14.765+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:24:15.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:24:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:24:14.765+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:24:15.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:24:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:24:14.765+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:24:15.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:24:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:24:14.766+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:24:15.809 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:24:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:24:15.546+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:24:15.810 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:24:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:24:15.546+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:24:15.810 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:24:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:24:15.546+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:24:15.810 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:24:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:24:15.725+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:24:15.810 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:24:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:24:15.725+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:24:15.810 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:24:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:24:15.725+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:24:15.810 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:24:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:24:15.725+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:24:16.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:24:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:24:15.809+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:24:16.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:24:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:24:15.809+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:24:16.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:24:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:24:15.809+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:24:16.857 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:24:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:24:16.689+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:24:16.857 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:24:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:24:16.689+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:24:16.857 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:24:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:24:16.689+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:24:16.858 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:24:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:24:16.689+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:24:16.858 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:24:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:24:16.540+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:24:16.858 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:24:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:24:16.540+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:24:16.858 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:24:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:24:16.540+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:24:17.230 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:24:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:24:16.855+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:24:17.230 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:24:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:24:16.855+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:24:17.230 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:24:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:24:16.855+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:24:17.491 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:24:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:24:17.228+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 21225 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:24:17.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:24:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:24:17.810+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:24:17.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:24:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:24:17.810+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:24:17.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:24:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:24:17.810+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:24:17.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:24:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:24:17.703+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:24:17.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:24:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:24:17.703+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:24:17.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:24:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:24:17.703+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:24:17.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:24:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:24:17.703+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:24:17.992 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:24:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:24:17.584+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:24:17.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:24:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:24:17.584+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:24:17.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:24:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:24:17.584+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:24:18.850 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:24:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:24:18.731+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:24:18.850 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:24:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:24:18.731+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:24:18.850 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:24:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:24:18.731+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:24:18.850 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:24:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:24:18.731+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:24:18.851 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:24:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:24:18.542+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:24:18.851 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:24:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:24:18.542+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:24:18.851 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:24:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:24:18.542+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:24:19.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:24:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:24:18.849+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:24:19.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:24:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:24:18.849+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:24:19.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:24:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:24:18.849+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:24:19.843 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:24:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:24:19.742+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:24:19.843 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:24:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:24:19.742+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:24:19.843 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:24:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:24:19.742+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:24:19.844 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:24:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:24:19.742+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:24:19.844 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:24:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:24:19.526+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:24:19.844 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:24:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:24:19.526+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:24:19.844 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:24:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:24:19.526+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:24:20.207 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:24:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:24:19.842+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:24:20.207 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:24:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:24:19.842+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:24:20.208 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:24:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:24:19.842+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:24:20.782 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:24:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:24:20.510+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:24:20.782 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:24:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:24:20.510+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:24:20.782 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:24:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:24:20.510+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:24:21.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:24:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:24:20.884+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:24:21.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:24:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:24:20.884+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:24:21.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:24:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:24:20.885+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:24:21.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:24:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:24:20.781+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:24:21.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:24:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:24:20.781+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:24:21.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:24:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:24:20.781+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:24:21.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:24:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:24:20.781+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:24:21.888 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:24:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:24:21.765+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:24:21.888 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:24:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:24:21.765+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:24:21.888 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:24:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:24:21.765+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:24:21.888 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:24:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:24:21.765+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:24:21.888 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:24:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:24:21.527+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:24:21.889 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:24:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:24:21.527+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:24:21.889 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:24:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:24:21.527+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:24:22.233 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:24:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:24:21.887+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:24:22.233 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:24:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:24:21.887+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:24:22.233 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:24:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:24:21.887+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:24:22.491 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:24:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:24:22.231+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 21296 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:24:22.894 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:24:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:24:22.541+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:24:22.894 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:24:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:24:22.541+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:24:22.894 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:24:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:24:22.541+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:24:22.895 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:24:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:24:22.748+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:24:22.895 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:24:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:24:22.748+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:24:22.895 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:24:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:24:22.748+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:24:22.895 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:24:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:24:22.748+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:24:23.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:24:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:24:22.893+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:24:23.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:24:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:24:22.893+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:24:23.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:24:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:24:22.893+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:24:23.770 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:24:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:24:23.502+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:24:23.771 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:24:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:24:23.502+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:24:23.771 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:24:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:24:23.502+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:24:24.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:24:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:24:23.942+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:24:24.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:24:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:24:23.942+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:24:24.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:24:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:24:23.942+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:24:24.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:24:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:24:23.770+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:24:24.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:24:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:24:23.770+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:24:24.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:24:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:24:23.770+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:24:24.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:24:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:24:23.770+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:24:24.777 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:24:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:24:24.505+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:24:24.777 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:24:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:24:24.505+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:24:24.777 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:24:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:24:24.505+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:24:25.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:24:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:24:24.934+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:24:25.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:24:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:24:24.934+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:24:25.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:24:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:24:24.934+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:24:25.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:24:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:24:24.776+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:24:25.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:24:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:24:24.776+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:24:25.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:24:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:24:24.776+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:24:25.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:24:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:24:24.776+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:24:25.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:24:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:24:25.478+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:24:25.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:24:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:24:25.478+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:24:25.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:24:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:24:25.478+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:24:26.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:24:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:24:25.970+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:24:26.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:24:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:24:25.970+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:24:26.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:24:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:24:25.970+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:24:26.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:24:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:24:25.774+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:24:26.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:24:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:24:25.774+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:24:26.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:24:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:24:25.774+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:24:26.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:24:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:24:25.774+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:24:26.734 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:24:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:24:26.441+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:24:26.734 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:24:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:24:26.441+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:24:26.734 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:24:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:24:26.441+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:24:26.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:24:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:24:26.957+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:24:26.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:24:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:24:26.957+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:24:26.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:24:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:24:26.957+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:24:26.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:24:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:24:26.733+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:24:26.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:24:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:24:26.733+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:24:26.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:24:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:24:26.733+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:24:26.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:24:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:24:26.733+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:24:27.491 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:24:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:24:27.236+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 21367 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:24:27.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:24:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:24:27.419+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:24:27.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:24:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:24:27.419+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:24:27.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:24:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:24:27.419+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:24:28.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:24:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:24:28.005+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:24:28.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:24:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:24:28.005+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:24:28.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:24:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:24:28.005+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:24:28.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:24:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:24:27.770+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:24:28.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:24:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:24:27.770+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:24:28.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:24:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:24:27.770+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:24:28.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:24:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:24:27.770+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:24:28.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:24:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:24:28.407+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:24:28.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:24:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:24:28.407+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:24:28.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:24:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:24:28.407+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:24:29.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:24:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:24:29.023+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:24:29.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:24:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:24:29.023+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:24:29.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:24:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:24:29.023+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:24:29.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:24:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:24:28.809+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:24:29.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:24:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:24:28.809+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:24:29.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:24:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:24:28.809+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:24:29.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:24:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:24:28.809+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:24:29.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:24:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:24:29.448+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:24:29.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:24:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:24:29.448+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:24:29.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:24:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:24:29.448+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:24:30.031 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:24:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:24:29.763+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:24:30.031 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:24:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:24:29.763+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:24:30.031 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:24:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:24:29.763+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:24:30.032 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:24:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:24:29.763+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:24:30.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:24:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:24:30.030+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:24:30.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:24:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:24:30.030+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:24:30.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:24:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:24:30.030+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:24:30.751 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:24:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:24:30.491+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:24:30.751 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:24:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:24:30.491+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:24:30.751 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:24:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:24:30.491+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:24:31.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:24:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:24:31.000+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:24:31.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:24:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:24:31.000+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:24:31.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:24:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:24:31.000+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:24:31.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:24:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:24:30.751+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:24:31.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:24:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:24:30.751+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:24:31.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:24:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:24:30.751+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:24:31.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:24:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:24:30.751+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:24:31.961 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:24:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:24:31.506+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:24:31.961 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:24:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:24:31.506+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:24:31.961 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:24:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:24:31.506+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:24:31.961 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:24:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:24:31.745+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:24:31.961 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:24:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:24:31.745+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:24:31.962 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:24:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:24:31.745+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:24:31.962 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:24:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:24:31.746+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:24:32.240 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:24:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:24:31.960+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:24:32.240 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:24:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:24:31.960+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:24:32.240 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:24:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:24:31.960+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:24:32.491 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:24:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:24:32.239+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 21438 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:24:32.958 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:24:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:24:32.514+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:24:32.958 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:24:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:24:32.514+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:24:32.958 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:24:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:24:32.514+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:24:32.959 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:24:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:24:32.735+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:24:32.959 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:24:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:24:32.735+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:24:32.959 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:24:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:24:32.735+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:24:32.959 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:24:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:24:32.735+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:24:33.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:24:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:24:32.957+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:24:33.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:24:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:24:32.957+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:24:33.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:24:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:24:32.957+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:24:33.987 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:24:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:24:33.691+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:24:33.987 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:24:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:24:33.691+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:24:33.987 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:24:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:24:33.691+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:24:33.987 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:24:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:24:33.691+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:24:33.987 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:24:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:24:33.543+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:24:33.988 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:24:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:24:33.543+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:24:33.988 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:24:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:24:33.543+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:24:34.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:24:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:24:33.986+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:24:34.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:24:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:24:33.986+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:24:34.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:24:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:24:33.986+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:24:34.950 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:24:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:24:34.711+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:24:34.950 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:24:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:24:34.711+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:24:34.951 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:24:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:24:34.711+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:24:34.951 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:24:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:24:34.711+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:24:34.951 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:24:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:24:34.533+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:24:34.951 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:24:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:24:34.533+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:24:34.951 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:24:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:24:34.533+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:24:35.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:24:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:24:34.950+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:24:35.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:24:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:24:34.950+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:24:35.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:24:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:24:34.950+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:24:35.908 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:24:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:24:35.572+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:24:35.909 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:24:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:24:35.572+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:24:35.909 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:24:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:24:35.572+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:24:35.909 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:24:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:24:35.697+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:24:35.909 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:24:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:24:35.697+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:24:35.909 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:24:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:24:35.697+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:24:35.909 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:24:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:24:35.697+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:24:36.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:24:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:24:35.908+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:24:36.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:24:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:24:35.908+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:24:36.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:24:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:24:35.908+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:24:36.907 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:24:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:24:36.584+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:24:36.907 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:24:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:24:36.584+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:24:36.907 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:24:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:24:36.584+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:24:36.907 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:24:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:24:36.733+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:24:36.908 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:24:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:24:36.733+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:24:36.908 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:24:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:24:36.733+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:24:36.908 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:24:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:24:36.733+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:24:37.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:24:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:24:36.906+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:24:37.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:24:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:24:36.906+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:24:37.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:24:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:24:36.906+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:24:37.611 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:24:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:24:37.242+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 21509 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:24:37.891 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:24:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:24:37.611+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:24:37.892 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:24:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:24:37.611+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:24:37.892 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:24:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:24:37.611+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:24:37.892 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:24:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:24:37.699+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:24:37.892 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:24:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:24:37.699+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:24:37.892 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:24:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:24:37.699+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:24:37.892 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:24:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:24:37.699+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:24:38.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:24:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:24:37.891+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:24:38.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:24:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:24:37.891+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:24:38.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:24:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:24:37.891+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:24:38.939 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:24:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:24:38.602+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:24:38.939 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:24:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:24:38.602+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:24:38.939 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:24:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:24:38.602+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:24:38.940 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:24:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:24:38.659+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:24:38.940 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:24:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:24:38.659+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:24:38.940 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:24:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:24:38.659+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:24:38.940 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:24:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:24:38.659+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:24:39.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:24:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:24:38.938+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:24:39.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:24:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:24:38.938+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:24:39.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:24:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:24:38.938+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:24:39.966 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:24:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:24:39.648+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:24:39.966 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:24:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:24:39.648+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:24:39.967 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:24:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:24:39.648+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:24:39.967 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:24:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:24:39.705+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:24:39.967 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:24:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:24:39.705+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:24:39.967 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:24:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:24:39.705+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:24:39.967 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:24:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:24:39.705+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:24:40.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:24:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:24:39.965+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:24:40.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:24:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:24:39.965+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:24:40.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:24:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:24:39.965+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:24:40.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:24:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:24:40.695+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:24:40.992 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:24:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:24:40.695+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:24:40.992 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:24:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:24:40.695+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:24:40.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:24:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:24:40.702+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:24:40.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:24:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:24:40.702+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:24:40.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:24:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:24:40.702+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:24:40.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:24:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:24:40.703+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:24:41.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:24:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:24:40.995+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:24:41.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:24:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:24:40.995+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:24:41.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:24:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:24:40.995+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:24:41.974 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:24:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:24:41.718+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:24:41.975 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:24:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:24:41.718+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:24:41.975 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:24:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:24:41.718+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:24:41.975 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:24:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:24:41.662+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:24:41.975 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:24:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:24:41.662+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:24:41.975 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:24:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:24:41.662+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:24:41.975 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:24:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:24:41.662+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:24:42.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:24:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:24:41.974+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:24:42.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:24:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:24:41.974+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:24:42.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:24:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:24:41.974+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:24:42.652 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:24:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:24:42.245+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 21580 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:24:42.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:24:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:24:42.651+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:24:42.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:24:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:24:42.651+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:24:42.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:24:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:24:42.651+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:24:42.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:24:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:24:42.651+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:24:42.992 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:24:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:24:42.764+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:24:42.992 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:24:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:24:42.764+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:24:42.992 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:24:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:24:42.764+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:24:43.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:24:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:24:42.998+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:24:43.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:24:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:24:42.998+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:24:43.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:24:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:24:42.998+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:24:43.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:24:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:24:43.690+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:24:43.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:24:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:24:43.690+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:24:43.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:24:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:24:43.690+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:24:43.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:24:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:24:43.690+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:24:43.992 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:24:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:24:43.784+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:24:43.992 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:24:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:24:43.784+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:24:43.992 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:24:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:24:43.784+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:24:44.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:24:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:24:44.036+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:24:44.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:24:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:24:44.036+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:24:44.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:24:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:24:44.036+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:24:44.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:24:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:24:44.699+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:24:44.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:24:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:24:44.699+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:24:44.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:24:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:24:44.699+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:24:44.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:24:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:24:44.699+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:24:44.992 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:24:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:24:44.766+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:24:44.992 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:24:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:24:44.766+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:24:44.992 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:24:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:24:44.766+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:24:45.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:24:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:24:45.073+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:24:45.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:24:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:24:45.073+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:24:45.492 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:24:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:24:45.073+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:24:45.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:24:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:24:45.717+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:24:45.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:24:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:24:45.717+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:24:45.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:24:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:24:45.717+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:24:45.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:24:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:24:45.717+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:24:45.992 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:24:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:24:45.764+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:24:45.992 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:24:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:24:45.764+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:24:45.992 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:24:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:24:45.764+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:24:46.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:24:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:24:46.025+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:24:46.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:24:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:24:46.025+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:24:46.492 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:24:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:24:46.025+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:24:46.977 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:24:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:24:46.681+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:24:46.977 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:24:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:24:46.681+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:24:46.977 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:24:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:24:46.681+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:24:46.977 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:24:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:24:46.681+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:24:46.977 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:24:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:24:46.722+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:24:46.978 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:24:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:24:46.722+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:24:46.978 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:24:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:24:46.722+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:24:47.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:24:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:24:46.975+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:24:47.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:24:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:24:46.975+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:24:47.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:24:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:24:46.975+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:24:47.686 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:24:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:24:47.248+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 21651 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:24:47.944 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:24:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:24:47.685+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:24:47.944 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:24:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:24:47.685+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:24:47.944 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:24:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:24:47.685+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:24:47.944 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:24:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:24:47.702+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:24:47.945 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:24:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:24:47.702+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:24:47.945 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:24:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:24:47.702+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:24:47.945 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:24:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:24:47.702+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:24:48.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:24:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:24:47.942+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:24:48.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:24:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:24:47.942+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:24:48.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:24:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:24:47.942+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:24:48.969 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:24:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:24:48.681+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:24:48.969 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:24:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:24:48.681+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:24:48.969 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:24:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:24:48.681+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:24:48.970 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:24:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:24:48.733+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:24:48.970 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:24:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:24:48.733+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:24:48.970 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:24:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:24:48.733+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:24:48.970 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:24:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:24:48.733+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:24:49.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:24:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:24:48.968+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:24:49.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:24:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:24:48.968+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:24:49.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:24:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:24:48.968+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:24:49.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:24:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:24:49.726+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:24:49.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:24:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:24:49.726+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:24:49.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:24:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:24:49.726+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:24:49.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:24:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:24:49.726+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:24:49.992 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:24:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:24:49.730+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:24:49.992 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:24:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:24:49.730+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:24:49.992 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:24:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:24:49.730+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:24:49.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:24:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:24:49.956+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:24:49.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:24:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:24:49.956+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:24:49.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:24:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:24:49.956+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:24:50.943 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:24:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:24:50.686+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:24:50.943 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:24:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:24:50.686+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:24:50.943 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:24:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:24:50.686+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:24:50.943 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:24:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:24:50.699+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:24:50.944 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:24:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:24:50.699+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:24:50.944 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:24:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:24:50.699+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:24:50.944 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:24:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:24:50.699+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:24:51.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:24:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:24:50.942+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:24:51.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:24:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:24:50.942+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:24:51.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:24:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:24:50.942+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:24:51.912 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:24:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:24:51.651+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:24:51.912 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:24:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:24:51.651+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:24:51.912 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:24:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:24:51.651+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:24:51.913 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:24:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:24:51.746+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:24:51.913 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:24:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:24:51.746+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:24:51.913 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:24:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:24:51.746+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:24:51.913 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:24:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:24:51.746+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:24:52.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:24:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:24:51.911+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:24:52.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:24:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:24:51.911+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:24:52.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:24:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:24:51.911+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:24:52.684 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:24:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:24:52.251+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 21722 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:24:52.944 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:24:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:24:52.767+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:24:52.945 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:24:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:24:52.767+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:24:52.945 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:24:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:24:52.767+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:24:52.945 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:24:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:24:52.767+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:24:52.945 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:24:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:24:52.682+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:24:52.945 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:24:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:24:52.682+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:24:52.946 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:24:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:24:52.682+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:24:53.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:24:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:24:52.943+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:24:53.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:24:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:24:52.943+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:24:53.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:24:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:24:52.943+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:24:53.981 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:24:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:24:53.747+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:24:53.982 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:24:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:24:53.747+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:24:53.982 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:24:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:24:53.747+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:24:53.982 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:24:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:24:53.747+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:24:53.982 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:24:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:24:53.678+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:24:53.982 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:24:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:24:53.678+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:24:53.982 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:24:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:24:53.678+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:24:54.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:24:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:24:53.980+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:24:54.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:24:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:24:53.980+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:24:54.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:24:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:24:53.980+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:24:54.963 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:24:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:24:54.702+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:24:54.963 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:24:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:24:54.702+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:24:54.963 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:24:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:24:54.702+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:24:54.964 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:24:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:24:54.702+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:24:54.964 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:24:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:24:54.724+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:24:54.964 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:24:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:24:54.724+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:24:54.964 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:24:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:24:54.724+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:24:55.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:24:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:24:54.961+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:24:55.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:24:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:24:54.961+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:24:55.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:24:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:24:54.961+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:24:55.978 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:24:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:24:55.652+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:24:55.978 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:24:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:24:55.652+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:24:55.979 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:24:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:24:55.652+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:24:55.979 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:24:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:24:55.652+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:24:55.979 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:24:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:24:55.684+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:24:55.979 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:24:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:24:55.684+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:24:55.979 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:24:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:24:55.684+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:24:56.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:24:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:24:55.977+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:24:56.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:24:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:24:55.977+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:24:56.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:24:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:24:55.977+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:24:56.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:24:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:24:56.673+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:24:56.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:24:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:24:56.673+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:24:56.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:24:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:24:56.673+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:24:56.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:24:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:24:56.670+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:24:56.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:24:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:24:56.670+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:24:56.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:24:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:24:56.670+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:24:56.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:24:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:24:56.670+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:24:57.266 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:24:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:24:57.253+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 21793 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:24:57.267 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:24:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:24:57.010+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:24:57.267 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:24:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:24:57.010+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:24:57.267 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:24:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:24:57.010+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:24:57.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:24:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:24:57.666+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:24:57.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:24:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:24:57.666+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:24:57.992 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:24:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:24:57.666+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:24:57.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:24:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:24:57.663+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:24:57.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:24:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:24:57.663+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:24:57.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:24:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:24:57.663+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:24:57.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:24:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:24:57.663+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:24:58.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:24:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:24:58.049+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:24:58.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:24:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:24:58.049+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:24:58.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:24:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:24:58.049+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:24:58.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:24:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:24:58.698+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:24:58.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:24:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:24:58.698+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:24:58.992 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:24:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:24:58.698+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:24:58.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:24:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:24:58.693+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:24:58.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:24:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:24:58.693+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:24:58.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:24:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:24:58.693+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:24:58.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:24:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:24:58.693+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:24:59.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:24:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:24:59.094+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:24:59.492 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:24:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:24:59.094+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:24:59.492 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:24:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:24:59.094+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:24:59.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:24:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:24:59.654+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:24:59.992 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:24:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:24:59.654+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:24:59.992 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:24:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:24:59.654+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:24:59.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:24:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:24:59.685+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:24:59.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:24:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:24:59.685+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:24:59.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:24:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:24:59.685+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:24:59.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:24:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:24:59.685+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:25:00.407 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:25:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:25:00.114+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:25:00.408 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:25:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:25:00.114+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:25:00.408 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:25:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:25:00.114+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:25:00.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:25:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:25:00.700+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:25:00.992 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:25:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:25:00.700+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:25:00.992 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:25:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:25:00.700+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:25:00.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:25:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:25:00.657+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:25:00.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:25:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:25:00.657+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:25:00.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:25:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:25:00.657+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:25:00.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:25:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:25:00.657+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:25:01.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:25:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:25:01.128+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:25:01.492 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:25:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:25:01.128+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:25:01.492 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:25:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:25:01.128+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:25:01.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:25:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:25:01.676+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:25:01.992 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:25:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:25:01.676+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:25:01.992 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:25:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:25:01.676+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:25:01.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:25:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:25:01.704+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:25:01.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:25:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:25:01.704+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:25:01.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:25:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:25:01.704+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:25:01.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:25:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:25:01.704+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:25:02.382 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:25:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:25:02.257+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 21864 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:25:02.382 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:25:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:25:02.119+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:25:02.382 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:25:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:25:02.119+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:25:02.382 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:25:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:25:02.119+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:25:02.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:25:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:25:02.752+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:25:02.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:25:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:25:02.752+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:25:02.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:25:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:25:02.752+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:25:02.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:25:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:25:02.752+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:25:02.992 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:25:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:25:02.711+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:25:02.992 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:25:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:25:02.711+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:25:02.992 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:25:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:25:02.711+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:25:03.480 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:25:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:25:03.142+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:25:03.480 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:25:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:25:03.142+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:25:03.480 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:25:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:25:03.142+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:25:03.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:25:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:25:03.711+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:25:03.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:25:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:25:03.711+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:25:03.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:25:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:25:03.711+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:25:04.154 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:25:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:25:03.778+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:25:04.154 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:25:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:25:03.778+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:25:04.154 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:25:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:25:03.778+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:25:04.154 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:25:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:25:03.778+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:25:04.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:25:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:25:04.152+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:25:04.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:25:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:25:04.152+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:25:04.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:25:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:25:04.152+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:25:04.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:25:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:25:04.727+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:25:04.992 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:25:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:25:04.727+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:25:04.992 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:25:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:25:04.727+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:25:04.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:25:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:25:04.823+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:25:04.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:25:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:25:04.823+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:25:04.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:25:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:25:04.823+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:25:04.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:25:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:25:04.823+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:25:05.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:25:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:25:05.170+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:25:05.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:25:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:25:05.170+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:25:05.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:25:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:25:05.170+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:25:05.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:25:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:25:05.680+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:25:05.992 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:25:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:25:05.680+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:25:05.992 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:25:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:25:05.680+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:25:05.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:25:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:25:05.803+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:25:05.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:25:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:25:05.803+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:25:05.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:25:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:25:05.803+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:25:05.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:25:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:25:05.803+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:25:06.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:25:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:25:06.207+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:25:06.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:25:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:25:06.207+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:25:06.492 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:25:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:25:06.207+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:25:06.847 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:25:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:25:06.665+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:25:06.847 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:25:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:25:06.665+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:25:06.847 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:25:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:25:06.665+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:25:07.180 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:25:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:25:06.845+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:25:07.180 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:25:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:25:06.845+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:25:07.180 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:25:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:25:06.845+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:25:07.180 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:25:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:25:06.845+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:25:07.491 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:25:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:25:07.260+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 21935 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:25:07.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:25:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:25:07.179+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:25:07.492 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:25:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:25:07.179+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:25:07.492 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:25:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:25:07.179+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:25:07.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:25:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:25:07.660+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:25:07.992 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:25:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:25:07.660+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:25:07.992 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:25:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:25:07.660+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:25:07.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:25:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:25:07.818+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:25:07.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:25:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:25:07.818+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:25:07.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:25:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:25:07.818+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:25:07.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:25:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:25:07.818+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:25:08.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:25:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:25:08.187+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:25:08.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:25:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:25:08.187+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:25:08.492 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:25:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:25:08.187+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:25:08.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:25:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:25:08.623+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:25:08.992 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:25:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:25:08.623+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:25:08.992 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:25:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:25:08.623+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:25:08.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:25:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:25:08.842+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:25:08.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:25:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:25:08.842+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:25:08.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:25:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:25:08.842+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:25:08.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:25:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:25:08.842+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:25:09.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:25:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:25:09.150+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:25:09.492 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:25:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:25:09.150+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:25:09.492 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:25:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:25:09.150+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:25:09.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:25:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:25:09.662+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:25:09.992 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:25:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:25:09.662+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:25:09.992 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:25:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:25:09.662+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:25:09.995 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:25:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:25:09.824+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:25:09.996 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:25:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:25:09.824+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:25:09.996 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:25:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:25:09.824+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:25:09.996 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:25:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:25:09.824+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:25:10.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:25:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:25:10.108+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:25:10.492 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:25:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:25:10.109+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:25:10.492 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:25:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:25:10.109+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:25:10.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:25:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:25:10.668+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:25:10.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:25:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:25:10.668+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:25:10.992 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:25:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:25:10.668+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:25:10.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:25:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:25:10.867+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:25:10.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:25:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:25:10.867+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:25:10.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:25:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:25:10.867+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:25:10.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:25:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:25:10.868+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:25:11.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:25:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:25:11.100+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:25:11.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:25:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:25:11.100+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:25:11.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:25:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:25:11.100+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:25:11.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:25:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:25:11.653+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:25:11.992 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:25:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:25:11.653+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:25:11.992 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:25:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:25:11.653+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:25:11.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:25:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:25:11.873+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:25:11.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:25:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:25:11.873+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:25:11.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:25:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:25:11.873+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:25:11.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:25:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:25:11.873+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:25:12.491 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:25:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:25:12.263+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 22006 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:25:12.492 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:25:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:25:12.146+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:25:12.492 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:25:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:25:12.147+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:25:12.492 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:25:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:25:12.147+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:25:12.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:25:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:25:12.619+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:25:12.992 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:25:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:25:12.619+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:25:12.992 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:25:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:25:12.619+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:25:12.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:25:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:25:12.849+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:25:12.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:25:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:25:12.849+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:25:12.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:25:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:25:12.849+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:25:12.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:25:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:25:12.849+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:25:13.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:25:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:25:13.160+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:25:13.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:25:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:25:13.160+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:25:13.492 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:25:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:25:13.160+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:25:13.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:25:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:25:13.620+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:25:13.992 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:25:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:25:13.620+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:25:13.992 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:25:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:25:13.620+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:25:13.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:25:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:25:13.827+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:25:13.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:25:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:25:13.827+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:25:13.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:25:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:25:13.827+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:25:13.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:25:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:25:13.827+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:25:14.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:25:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:25:14.181+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:25:14.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:25:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:25:14.181+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:25:14.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:25:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:25:14.181+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:25:14.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:25:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:25:14.620+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:25:14.992 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:25:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:25:14.620+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:25:14.992 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:25:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:25:14.620+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:25:14.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:25:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:25:14.832+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:25:14.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:25:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:25:14.832+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:25:14.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:25:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:25:14.832+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:25:14.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:25:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:25:14.832+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:25:15.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:25:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:25:15.150+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:25:15.492 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:25:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:25:15.150+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:25:15.492 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:25:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:25:15.150+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:25:15.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:25:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:25:15.658+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:25:15.992 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:25:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:25:15.658+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:25:15.992 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:25:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:25:15.658+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:25:15.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:25:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:25:15.808+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:25:15.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:25:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:25:15.808+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:25:15.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:25:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:25:15.808+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:25:15.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:25:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:25:15.808+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:25:16.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:25:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:25:16.144+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:25:16.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:25:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:25:16.144+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:25:16.492 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:25:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:25:16.144+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:25:16.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:25:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:25:16.665+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:25:16.992 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:25:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:25:16.665+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:25:16.992 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:25:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:25:16.665+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:25:16.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:25:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:25:16.790+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:25:16.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:25:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:25:16.790+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:25:16.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:25:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:25:16.790+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:25:16.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:25:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:25:16.790+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:25:17.491 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:25:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:25:17.268+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 22077 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:25:17.492 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:25:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:25:17.167+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:25:17.492 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:25:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:25:17.167+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:25:17.492 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:25:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:25:17.167+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:25:17.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:25:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:25:17.659+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:25:17.992 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:25:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:25:17.659+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:25:17.992 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:25:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:25:17.659+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:25:17.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:25:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:25:17.787+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:25:17.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:25:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:25:17.787+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:25:17.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:25:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:25:17.787+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:25:17.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:25:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:25:17.787+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:25:18.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:25:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:25:18.180+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:25:18.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:25:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:25:18.180+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:25:18.492 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:25:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:25:18.180+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:25:18.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:25:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:25:18.676+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:25:18.992 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:25:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:25:18.676+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:25:18.992 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:25:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:25:18.676+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:25:18.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:25:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:25:18.817+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:25:18.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:25:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:25:18.817+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:25:18.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:25:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:25:18.817+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:25:18.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:25:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:25:18.817+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:25:19.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:25:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:25:19.145+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:25:19.492 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:25:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:25:19.145+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:25:19.492 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:25:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:25:19.145+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:25:19.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:25:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:25:19.662+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:25:19.992 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:25:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:25:19.662+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:25:19.992 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:25:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:25:19.662+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:25:19.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:25:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:25:19.855+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:25:19.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:25:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:25:19.855+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:25:19.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:25:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:25:19.855+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:25:19.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:25:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:25:19.855+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:25:20.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:25:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:25:20.117+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:25:20.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:25:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:25:20.117+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:25:20.492 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:25:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:25:20.117+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:25:20.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:25:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:25:20.631+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:25:20.992 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:25:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:25:20.631+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:25:20.992 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:25:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:25:20.631+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:25:20.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:25:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:25:20.860+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:25:20.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:25:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:25:20.860+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:25:20.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:25:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:25:20.860+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:25:20.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:25:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:25:20.860+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:25:21.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:25:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:25:21.129+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:25:21.492 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:25:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:25:21.129+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:25:21.492 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:25:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:25:21.129+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:25:21.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:25:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:25:21.668+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:25:21.992 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:25:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:25:21.668+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:25:21.992 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:25:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:25:21.668+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:25:21.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:25:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:25:21.897+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:25:21.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:25:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:25:21.897+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:25:21.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:25:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:25:21.897+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:25:21.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:25:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:25:21.897+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:25:22.394 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:25:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:25:22.271+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 22148 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:25:22.395 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:25:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:25:22.122+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:25:22.395 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:25:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:25:22.122+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:25:22.395 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:25:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:25:22.122+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:25:22.924 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:25:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:25:22.656+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:25:22.924 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:25:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:25:22.656+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:25:22.924 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:25:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:25:22.656+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:25:23.241 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:25:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:25:22.923+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:25:23.241 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:25:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:25:22.923+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:25:23.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:25:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:25:22.923+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:25:23.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:25:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:25:22.923+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:25:23.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:25:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:25:23.091+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:25:23.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:25:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:25:23.091+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:25:23.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:25:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:25:23.091+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:25:23.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:25:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:25:23.912+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:25:23.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:25:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:25:23.912+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:25:23.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:25:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:25:23.912+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:25:23.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:25:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:25:23.912+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:25:23.992 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:25:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:25:23.666+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:25:23.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:25:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:25:23.666+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:25:23.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:25:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:25:23.666+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:25:24.492 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:25:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:25:24.097+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:25:24.495 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:25:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:25:24.097+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:25:24.495 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:25:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:25:24.097+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:25:24.887 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:25:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:25:24.632+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:25:24.887 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:25:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:25:24.632+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:25:24.888 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:25:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:25:24.632+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:25:25.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:25:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:25:25.069+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:25:25.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:25:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:25:25.069+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:25:25.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:25:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:25:25.069+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:25:25.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:25:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:25:24.886+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:25:25.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:25:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:25:24.886+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:25:25.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:25:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:25:24.886+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:25:25.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:25:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:25:24.886+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:25:25.869 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:25:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:25:25.616+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:25:25.869 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:25:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:25:25.616+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:25:25.870 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:25:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:25:25.616+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:25:26.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:25:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:25:26.066+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:25:26.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:25:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:25:26.066+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:25:26.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:25:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:25:26.066+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:25:26.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:25:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:25:25.868+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:25:26.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:25:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:25:25.868+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:25:26.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:25:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:25:25.868+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:25:26.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:25:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:25:25.869+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:25:26.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:25:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:25:26.865+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:25:26.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:25:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:25:26.865+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:25:26.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:25:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:25:26.865+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:25:26.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:25:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:25:26.865+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:25:26.992 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:25:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:25:26.654+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:25:26.992 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:25:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:25:26.654+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:25:26.992 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:25:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:25:26.654+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:25:27.298 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:25:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:25:27.274+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 22219 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:25:27.298 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:25:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:25:27.041+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:25:27.298 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:25:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:25:27.041+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:25:27.299 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:25:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:25:27.041+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:25:27.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:25:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:25:27.880+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:25:27.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:25:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:25:27.880+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:25:27.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:25:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:25:27.880+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:25:27.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:25:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:25:27.880+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:25:27.992 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:25:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:25:27.668+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:25:27.992 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:25:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:25:27.668+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:25:27.992 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:25:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:25:27.668+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:25:28.492 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:25:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:25:28.051+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:25:28.492 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:25:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:25:28.051+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:25:28.492 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:25:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:25:28.051+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:25:28.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:25:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:25:28.874+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:25:28.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:25:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:25:28.874+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:25:28.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:25:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:25:28.874+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:25:28.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:25:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:25:28.874+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:25:28.992 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:25:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:25:28.651+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:25:28.992 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:25:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:25:28.651+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:25:28.992 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:25:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:25:28.651+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:25:29.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:25:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:25:29.002+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:25:29.492 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:25:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:25:29.002+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:25:29.492 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:25:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:25:29.002+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:25:29.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:25:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:25:29.869+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:25:29.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:25:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:25:29.870+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:25:29.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:25:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:25:29.870+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:25:29.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:25:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:25:29.870+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:25:29.992 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:25:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:25:29.673+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:25:29.992 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:25:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:25:29.673+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:25:29.992 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:25:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:25:29.673+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:25:30.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:25:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:25:30.007+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:25:30.492 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:25:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:25:30.007+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:25:30.492 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:25:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:25:30.007+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:25:30.975 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:25:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:25:30.652+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:25:30.975 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:25:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:25:30.652+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:25:30.975 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:25:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:25:30.652+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:25:30.975 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:25:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:25:30.823+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:25:30.975 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:25:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:25:30.823+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:25:30.976 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:25:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:25:30.823+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:25:30.976 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:25:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:25:30.823+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:25:31.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:25:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:25:30.974+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:25:31.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:25:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:25:30.974+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:25:31.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:25:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:25:30.974+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:25:31.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:25:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:25:31.777+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:25:31.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:25:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:25:31.777+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:25:31.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:25:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:25:31.777+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:25:31.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:25:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:25:31.777+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:25:31.992 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:25:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:25:31.633+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:25:31.992 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:25:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:25:31.633+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:25:31.992 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:25:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:25:31.633+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:25:32.278 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:25:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:25:31.993+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:25:32.278 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:25:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:25:31.993+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:25:32.279 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:25:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:25:31.993+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:25:32.607 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:25:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:25:32.277+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 22290 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:25:32.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:25:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:25:32.606+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:25:32.992 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:25:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:25:32.606+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:25:32.992 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:25:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:25:32.606+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:25:32.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:25:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:25:32.804+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:25:32.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:25:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:25:32.804+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:25:32.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:25:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:25:32.804+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:25:32.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:25:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:25:32.804+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:25:33.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:25:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:25:33.000+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:25:33.492 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:25:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:25:33.000+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:25:33.492 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:25:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:25:33.000+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:25:33.988 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:25:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:25:33.772+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:25:33.988 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:25:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:25:33.772+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:25:33.988 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:25:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:25:33.772+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:25:33.988 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:25:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:25:33.773+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:25:33.988 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:25:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:25:33.559+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:25:33.989 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:25:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:25:33.559+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:25:33.989 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:25:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:25:33.559+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:25:34.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:25:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:25:33.987+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:25:34.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:25:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:25:33.987+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:25:34.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:25:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:25:33.987+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:25:34.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:25:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:25:34.571+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:25:34.992 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:25:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:25:34.571+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:25:34.992 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:25:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:25:34.571+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:25:34.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:25:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:25:34.800+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:25:34.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:25:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:25:34.800+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:25:34.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:25:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:25:34.800+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:25:34.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:25:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:25:34.801+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:25:35.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:25:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:25:35.002+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:25:35.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:25:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:25:35.002+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:25:35.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:25:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:25:35.002+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:25:35.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:25:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:25:35.585+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:25:35.992 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:25:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:25:35.585+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:25:35.992 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:25:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:25:35.585+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:25:35.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:25:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:25:35.795+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:25:35.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:25:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:25:35.795+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:25:35.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:25:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:25:35.795+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:25:35.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:25:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:25:35.795+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:25:36.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:25:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:25:36.044+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:25:36.492 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:25:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:25:36.044+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:25:36.492 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:25:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:25:36.044+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:25:36.783 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:25:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:25:36.585+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:25:36.783 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:25:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:25:36.585+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:25:36.784 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:25:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:25:36.585+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:25:37.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:25:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:25:36.994+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:25:37.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:25:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:25:36.994+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:25:37.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:25:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:25:36.994+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:25:37.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:25:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:25:36.782+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:25:37.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:25:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:25:36.782+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:25:37.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:25:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:25:36.782+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:25:37.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:25:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:25:36.782+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:25:37.556 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:25:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:25:37.280+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 22361 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:25:37.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:25:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:25:37.554+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:25:37.992 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:25:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:25:37.554+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:25:37.992 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:25:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:25:37.554+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:25:37.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:25:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:25:37.770+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:25:37.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:25:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:25:37.770+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:25:37.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:25:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:25:37.770+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:25:37.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:25:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:25:37.770+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:25:38.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:25:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:25:38.025+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:25:38.492 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:25:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:25:38.025+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:25:38.492 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:25:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:25:38.025+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:25:38.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:25:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:25:38.556+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:25:38.992 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:25:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:25:38.556+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:25:38.992 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:25:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:25:38.556+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:25:38.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:25:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:25:38.765+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:25:38.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:25:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:25:38.765+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:25:38.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:25:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:25:38.765+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:25:38.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:25:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:25:38.765+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:25:39.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:25:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:25:39.024+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:25:39.492 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:25:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:25:39.024+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:25:39.492 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:25:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:25:39.024+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:25:39.885 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:25:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:25:39.551+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:25:39.886 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:25:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:25:39.551+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:25:39.886 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:25:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:25:39.551+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:25:39.886 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:25:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:25:39.769+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:25:39.886 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:25:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:25:39.769+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:25:39.887 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:25:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:25:39.769+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:25:39.887 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:25:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:25:39.769+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:25:40.214 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:25:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:25:40.051+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:25:40.215 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:25:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:25:40.051+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:25:40.215 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:25:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:25:40.051+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:25:40.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:25:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:25:40.558+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:25:40.992 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:25:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:25:40.558+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:25:40.992 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:25:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:25:40.558+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:25:40.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:25:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:25:40.769+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:25:40.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:25:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:25:40.769+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:25:40.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:25:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:25:40.769+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:25:40.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:25:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:25:40.770+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:25:41.491 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:25:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:25:41.025+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:25:41.492 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:25:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:25:41.025+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:25:41.492 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:25:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:25:41.025+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:25:41.992 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:25:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:25:41.575+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:25:41.992 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:25:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:25:41.575+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:25:41.992 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:25:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:25:41.575+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:25:41.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:25:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:25:41.723+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:25:41.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:25:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:25:41.723+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:25:41.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:25:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:25:41.723+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:25:41.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:25:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:25:41.723+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:25:42.285 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:25:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:25:41.996+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:25:42.285 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:25:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:25:41.996+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:25:42.285 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:25:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:25:41.996+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:25:42.574 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:25:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:25:42.283+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 22432 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:25:42.978 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:25:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:25:42.572+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:25:42.979 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:25:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:25:42.572+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:25:42.979 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:25:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:25:42.572+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:25:42.979 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:25:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:25:42.749+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:25:42.979 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:25:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:25:42.749+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:25:42.979 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:25:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:25:42.749+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:25:42.979 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:25:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:25:42.750+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:25:43.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:25:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:25:42.977+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:25:43.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:25:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:25:42.977+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:25:43.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:25:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:25:42.977+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:25:43.726 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:25:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:25:43.552+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:25:43.726 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:25:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:25:43.552+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:25:43.726 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:25:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:25:43.552+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:25:43.977 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:25:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:25:43.724+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:25:43.977 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:25:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:25:43.724+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:25:43.977 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:25:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:25:43.724+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:25:43.977 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:25:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:25:43.724+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:25:44.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:25:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:25:43.975+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:25:44.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:25:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:25:43.975+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:25:44.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:25:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:25:43.975+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:25:44.937 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:25:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:25:44.767+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:25:44.938 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:25:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:25:44.767+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:25:44.938 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:25:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:25:44.767+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:25:44.938 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:25:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:25:44.767+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:25:44.938 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:25:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:25:44.547+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:25:44.939 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:25:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:25:44.547+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:25:44.939 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:25:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:25:44.547+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:25:45.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:25:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:25:44.936+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:25:45.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:25:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:25:44.936+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:25:45.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:25:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:25:44.936+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:25:45.944 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:25:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:25:45.804+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:25:45.944 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:25:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:25:45.804+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:25:45.944 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:25:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:25:45.804+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:25:45.944 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:25:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:25:45.804+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:25:45.945 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:25:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:25:45.558+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:25:45.945 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:25:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:25:45.558+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:25:45.945 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:25:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:25:45.558+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:25:46.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:25:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:25:45.942+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:25:46.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:25:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:25:45.942+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:25:46.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:25:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:25:45.942+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:25:46.969 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:25:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:25:46.791+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:25:46.969 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:25:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:25:46.791+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:25:46.969 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:25:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:25:46.791+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:25:46.970 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:25:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:25:46.792+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:25:46.970 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:25:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:25:46.578+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:25:46.970 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:25:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:25:46.578+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:25:46.970 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:25:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:25:46.578+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:25:47.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:25:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:25:46.967+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:25:47.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:25:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:25:46.967+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:25:47.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:25:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:25:46.967+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:25:47.574 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:25:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:25:47.286+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 22503 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:25:47.966 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:25:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:25:47.573+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:25:47.966 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:25:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:25:47.573+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:25:47.967 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:25:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:25:47.573+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:25:47.967 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:25:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:25:47.747+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:25:47.967 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:25:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:25:47.747+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:25:47.967 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:25:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:25:47.747+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:25:47.967 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:25:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:25:47.747+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:25:48.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:25:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:25:47.964+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:25:48.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:25:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:25:47.964+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:25:48.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:25:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:25:47.964+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:25:48.989 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:25:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:25:48.607+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:25:48.990 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:25:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:25:48.607+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:25:48.990 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:25:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:25:48.607+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:25:48.990 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:25:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:25:48.770+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:25:48.990 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:25:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:25:48.770+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:25:48.990 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:25:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:25:48.770+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:25:48.990 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:25:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:25:48.770+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:25:49.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:25:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:25:48.988+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:25:49.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:25:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:25:48.988+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:25:49.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:25:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:25:48.988+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:25:49.884 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:25:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:25:49.812+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:25:49.885 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:25:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:25:49.812+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:25:49.885 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:25:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:25:49.812+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:25:49.885 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:25:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:25:49.812+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:25:49.885 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:25:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:25:49.600+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:25:49.885 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:25:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:25:49.600+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:25:49.885 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:25:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:25:49.600+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:25:50.215 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:25:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:25:49.963+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:25:50.216 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:25:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:25:49.963+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:25:50.216 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:25:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:25:49.963+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:25:50.937 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:25:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:25:50.617+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:25:50.937 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:25:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:25:50.617+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:25:50.937 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:25:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:25:50.617+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:25:50.938 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:25:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:25:50.793+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:25:50.938 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:25:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:25:50.793+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:25:50.938 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:25:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:25:50.793+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:25:50.938 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:25:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:25:50.794+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:25:51.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:25:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:25:50.936+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:25:51.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:25:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:25:50.936+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:25:51.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:25:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:25:50.936+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:25:51.956 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:25:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:25:51.758+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:25:51.956 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:25:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:25:51.758+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:25:51.956 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:25:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:25:51.758+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:25:51.957 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:25:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:25:51.758+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:25:51.957 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:25:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:25:51.664+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:25:51.957 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:25:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:25:51.664+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:25:51.957 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:25:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:25:51.664+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:25:52.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:25:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:25:51.954+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:25:52.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:25:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:25:51.954+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:25:52.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:25:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:25:51.954+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:25:52.632 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:25:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:25:52.289+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 22574 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:25:52.922 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:25:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:25:52.674+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:25:52.922 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:25:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:25:52.674+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:25:52.922 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:25:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:25:52.674+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:25:52.923 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:25:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:25:52.784+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:25:52.923 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:25:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:25:52.784+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:25:52.923 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:25:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:25:52.784+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:25:52.923 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:25:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:25:52.784+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:25:53.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:25:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:25:52.920+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:25:53.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:25:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:25:52.920+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:25:53.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:25:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:25:52.920+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:25:53.933 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:25:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:25:53.659+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:25:53.933 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:25:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:25:53.659+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:25:53.933 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:25:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:25:53.659+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:25:53.933 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:25:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:25:53.737+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:25:53.934 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:25:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:25:53.737+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:25:53.934 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:25:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:25:53.737+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:25:53.934 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:25:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:25:53.737+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:25:54.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:25:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:25:53.931+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:25:54.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:25:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:25:53.931+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:25:54.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:25:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:25:53.931+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:25:54.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:25:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:25:54.902+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:25:54.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:25:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:25:54.902+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:25:54.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:25:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:25:54.902+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:25:54.992 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:25:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:25:54.658+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:25:54.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:25:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:25:54.658+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:25:54.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:25:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:25:54.658+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:25:54.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:25:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:25:54.757+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:25:54.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:25:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:25:54.757+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:25:54.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:25:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:25:54.757+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:25:54.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:25:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:25:54.757+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:25:55.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:25:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:25:55.886+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:25:55.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:25:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:25:55.886+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:25:55.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:25:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:25:55.886+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:25:55.992 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:25:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:25:55.681+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:25:55.992 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:25:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:25:55.681+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:25:55.992 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:25:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:25:55.681+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:25:55.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:25:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:25:55.789+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:25:55.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:25:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:25:55.789+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:25:55.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:25:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:25:55.789+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:25:55.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:25:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:25:55.789+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:25:56.927 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:25:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:25:56.656+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:25:56.927 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:25:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:25:56.656+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:25:56.927 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:25:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:25:56.656+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:25:56.927 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:25:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:25:56.831+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:25:56.927 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:25:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:25:56.831+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:25:56.927 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:25:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:25:56.831+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:25:56.928 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:25:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:25:56.831+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:25:57.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:25:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:25:56.925+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:25:57.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:25:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:25:56.925+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:25:57.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:25:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:25:56.925+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:25:57.698 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:25:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:25:57.293+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 22644 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:25:57.969 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:25:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:25:57.696+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:25:57.969 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:25:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:25:57.696+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:25:57.969 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:25:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:25:57.696+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:25:57.970 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:25:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:25:57.784+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:25:57.970 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:25:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:25:57.784+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:25:57.970 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:25:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:25:57.784+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:25:57.970 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:25:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:25:57.784+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:25:58.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:25:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:25:57.968+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:25:58.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:25:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:25:57.968+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:25:58.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:25:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:25:57.968+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:25:58.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:25:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:25:58.963+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:25:58.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:25:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:25:58.963+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:25:58.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:25:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:25:58.963+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:25:58.992 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:25:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:25:58.716+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:25:58.992 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:25:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:25:58.716+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:25:58.992 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:25:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:25:58.716+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:25:58.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:25:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:25:58.808+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:25:58.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:25:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:25:58.808+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:25:58.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:25:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:25:58.808+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:25:58.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:25:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:25:58.808+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:25:59.979 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:25:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:25:59.669+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:25:59.979 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:25:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:25:59.669+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:25:59.980 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:25:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:25:59.669+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:25:59.980 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:25:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:25:59.849+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:25:59.980 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:25:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:25:59.849+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:25:59.980 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:25:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:25:59.849+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:25:59.980 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:25:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:25:59.849+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:26:00.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:25:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:25:59.977+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:26:00.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:25:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:25:59.977+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:26:00.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:25:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:25:59.977+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:26:00.980 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:26:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:26:00.629+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:26:00.981 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:26:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:26:00.629+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:26:00.981 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:26:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:26:00.629+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:26:00.981 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:26:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:26:00.814+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:26:00.981 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:26:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:26:00.814+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:26:00.981 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:26:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:26:00.814+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:26:00.981 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:26:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:26:00.814+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:26:01.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:26:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:26:00.978+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:26:01.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:26:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:26:00.978+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:26:01.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:26:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:26:00.978+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:26:01.848 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:26:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:26:01.582+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:26:01.848 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:26:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:26:01.582+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:26:01.848 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:26:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:26:01.582+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:26:02.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:26:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:26:01.931+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:26:02.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:26:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:26:01.931+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:26:02.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:26:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:26:01.931+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:26:02.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:26:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:26:01.861+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:26:02.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:26:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:26:01.861+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:26:02.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:26:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:26:01.861+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:26:02.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:26:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:26:01.861+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:26:02.573 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:26:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:26:02.296+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 22716 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:26:02.969 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:26:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:26:02.572+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:26:02.970 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:26:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:26:02.572+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:26:02.970 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:26:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:26:02.572+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:26:02.970 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:26:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:26:02.818+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:26:02.970 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:26:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:26:02.818+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:26:02.970 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:26:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:26:02.818+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:26:02.970 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:26:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:26:02.818+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:26:03.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:26:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:26:02.968+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:26:03.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:26:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:26:02.968+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:26:03.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:26:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:26:02.968+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:26:03.859 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:26:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:26:03.525+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:26:03.860 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:26:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:26:03.525+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:26:03.860 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:26:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:26:03.525+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:26:04.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:26:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:26:04.015+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:26:04.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:26:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:26:04.015+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:26:04.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:26:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:26:04.015+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:26:04.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:26:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:26:03.858+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:26:04.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:26:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:26:03.858+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:26:04.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:26:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:26:03.858+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:26:04.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:26:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:26:03.858+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:26:04.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:26:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:26:04.477+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:26:04.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:26:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:26:04.477+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:26:04.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:26:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:26:04.477+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:26:05.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:26:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:26:05.034+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:26:05.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:26:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:26:05.034+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:26:05.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:26:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:26:05.034+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:26:05.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:26:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:26:04.894+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:26:05.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:26:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:26:04.894+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:26:05.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:26:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:26:04.894+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:26:05.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:26:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:26:04.894+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:26:05.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:26:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:26:05.487+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:26:05.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:26:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:26:05.487+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:26:05.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:26:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:26:05.487+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:26:06.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:26:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:26:06.046+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:26:06.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:26:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:26:06.046+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:26:06.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:26:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:26:06.046+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:26:06.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:26:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:26:05.862+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:26:06.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:26:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:26:05.862+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:26:06.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:26:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:26:05.862+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:26:06.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:26:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:26:05.862+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:26:06.913 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:26:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:26:06.510+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:26:06.914 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:26:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:26:06.510+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:26:06.914 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:26:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:26:06.510+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:26:07.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:26:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:26:07.076+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:26:07.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:26:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:26:07.076+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:26:07.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:26:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:26:07.076+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:26:07.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:26:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:26:06.912+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:26:07.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:26:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:26:06.912+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:26:07.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:26:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:26:06.912+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:26:07.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:26:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:26:06.912+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:26:07.741 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:26:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:26:07.300+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 22786 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:26:07.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:26:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:26:07.533+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:26:07.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:26:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:26:07.533+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:26:07.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:26:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:26:07.533+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:26:08.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:26:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:26:08.069+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:26:08.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:26:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:26:08.069+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:26:08.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:26:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:26:08.070+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:26:08.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:26:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:26:07.939+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:26:08.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:26:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:26:07.939+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:26:08.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:26:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:26:07.939+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:26:08.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:26:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:26:07.939+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:26:08.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:26:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:26:08.487+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:26:08.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:26:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:26:08.487+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:26:08.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:26:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:26:08.487+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:26:09.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:26:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:26:09.024+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:26:09.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:26:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:26:09.024+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:26:09.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:26:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:26:09.024+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:26:09.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:26:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:26:08.921+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:26:09.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:26:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:26:08.921+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:26:09.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:26:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:26:08.921+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:26:09.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:26:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:26:08.921+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:26:09.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:26:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:26:09.455+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:26:09.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:26:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:26:09.455+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:26:09.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:26:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:26:09.455+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:26:10.216 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:26:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:26:10.000+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:26:10.216 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:26:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:26:10.000+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:26:10.217 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:26:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:26:10.000+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:26:10.217 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:26:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:26:09.891+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:26:10.217 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:26:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:26:09.891+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:26:10.217 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:26:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:26:09.891+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:26:10.217 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:26:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:26:09.892+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:26:10.843 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:26:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:26:10.503+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:26:10.843 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:26:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:26:10.503+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:26:10.843 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:26:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:26:10.503+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:26:11.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:26:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:26:10.961+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:26:11.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:26:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:26:10.961+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:26:11.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:26:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:26:10.961+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:26:11.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:26:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:26:10.842+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:26:11.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:26:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:26:10.842+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:26:11.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:26:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:26:10.842+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:26:11.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:26:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:26:10.842+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:26:11.850 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:26:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:26:11.503+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:26:11.851 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:26:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:26:11.504+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:26:11.851 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:26:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:26:11.504+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:26:12.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:26:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:26:11.982+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:26:12.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:26:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:26:11.982+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:26:12.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:26:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:26:11.982+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:26:12.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:26:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:26:11.849+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:26:12.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:26:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:26:11.849+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:26:12.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:26:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:26:11.849+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:26:12.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:26:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:26:11.849+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:26:12.741 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:26:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:26:12.304+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 22858 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:26:12.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:26:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:26:12.482+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:26:12.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:26:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:26:12.482+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:26:12.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:26:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:26:12.482+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:26:13.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:26:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:26:12.955+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:26:13.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:26:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:26:12.955+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:26:13.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:26:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:26:12.955+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:26:13.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:26:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:26:12.830+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:26:13.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:26:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:26:12.830+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:26:13.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:26:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:26:12.830+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:26:13.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:26:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:26:12.830+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:26:13.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:26:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:26:13.471+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:26:13.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:26:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:26:13.471+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:26:13.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:26:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:26:13.471+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:26:14.136 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:26:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:26:13.942+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:26:14.136 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:26:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:26:13.942+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:26:14.136 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:26:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:26:13.942+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:26:14.137 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:26:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:26:13.822+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:26:14.137 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:26:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:26:13.822+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:26:14.137 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:26:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:26:13.822+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:26:14.137 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:26:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:26:13.822+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:26:14.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:26:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:26:14.468+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:26:14.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:26:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:26:14.468+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:26:14.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:26:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:26:14.468+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:26:15.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:26:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:26:14.970+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:26:15.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:26:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:26:14.970+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:26:15.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:26:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:26:14.970+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:26:15.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:26:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:26:14.861+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:26:15.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:26:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:26:14.861+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:26:15.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:26:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:26:14.861+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:26:15.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:26:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:26:14.861+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:26:15.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:26:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:26:15.450+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:26:15.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:26:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:26:15.450+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:26:15.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:26:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:26:15.450+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:26:16.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:26:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:26:15.961+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:26:16.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:26:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:26:15.961+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:26:16.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:26:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:26:15.961+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:26:16.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:26:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:26:15.900+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:26:16.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:26:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:26:15.900+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:26:16.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:26:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:26:15.900+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:26:16.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:26:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:26:15.900+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:26:16.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:26:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:26:16.442+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:26:16.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:26:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:26:16.442+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:26:16.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:26:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:26:16.442+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:26:17.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:26:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:26:16.930+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:26:17.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:26:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:26:16.930+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:26:17.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:26:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:26:16.930+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:26:17.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:26:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:26:16.906+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:26:17.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:26:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:26:16.906+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:26:17.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:26:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:26:16.906+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:26:17.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:26:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:26:16.907+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:26:17.742 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:26:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:26:17.307+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 22928 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:26:17.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:26:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:26:17.470+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:26:17.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:26:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:26:17.470+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:26:17.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:26:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:26:17.470+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:26:18.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:26:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:26:17.922+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:26:18.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:26:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:26:17.922+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:26:18.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:26:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:26:17.922+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:26:18.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:26:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:26:17.928+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:26:18.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:26:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:26:17.928+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:26:18.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:26:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:26:17.928+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:26:18.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:26:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:26:17.929+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:26:18.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:26:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:26:18.483+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:26:18.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:26:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:26:18.483+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:26:18.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:26:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:26:18.483+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:26:19.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:26:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:26:18.935+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:26:19.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:26:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:26:18.935+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:26:19.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:26:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:26:18.935+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:26:19.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:26:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:26:18.896+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:26:19.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:26:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:26:18.896+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:26:19.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:26:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:26:18.896+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:26:19.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:26:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:26:18.896+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:26:19.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:26:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:26:19.490+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:26:19.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:26:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:26:19.490+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:26:19.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:26:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:26:19.490+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:26:20.217 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:26:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:26:19.892+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:26:20.217 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:26:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:26:19.892+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:26:20.217 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:26:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:26:19.892+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:26:20.217 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:26:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:26:19.946+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:26:20.218 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:26:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:26:19.946+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:26:20.218 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:26:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:26:19.946+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:26:20.218 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:26:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:26:19.946+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:26:20.852 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:26:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:26:20.524+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:26:20.852 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:26:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:26:20.524+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:26:20.852 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:26:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:26:20.524+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:26:21.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:26:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:26:20.851+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:26:21.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:26:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:26:20.851+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:26:21.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:26:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:26:20.851+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:26:21.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:26:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:26:20.898+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:26:21.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:26:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:26:20.898+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:26:21.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:26:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:26:20.898+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:26:21.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:26:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:26:20.898+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:26:21.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:26:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:26:21.480+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:26:21.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:26:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:26:21.480+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:26:21.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:26:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:26:21.480+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:26:22.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:26:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:26:21.836+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:26:22.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:26:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:26:21.836+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:26:22.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:26:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:26:21.836+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:26:22.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:26:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:26:21.870+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:26:22.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:26:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:26:21.870+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:26:22.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:26:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:26:21.870+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:26:22.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:26:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:26:21.870+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:26:22.741 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:26:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:26:22.310+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 23000 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:26:22.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:26:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:26:22.467+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:26:22.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:26:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:26:22.467+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:26:22.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:26:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:26:22.467+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:26:23.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:26:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:26:22.860+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:26:23.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:26:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:26:22.860+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:26:23.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:26:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:26:22.860+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:26:23.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:26:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:26:22.824+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:26:23.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:26:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:26:22.824+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:26:23.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:26:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:26:22.824+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:26:23.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:26:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:26:22.824+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:26:23.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:26:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:26:23.465+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:26:23.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:26:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:26:23.465+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:26:23.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:26:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:26:23.465+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:26:24.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:26:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:26:23.831+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:26:24.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:26:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:26:23.831+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:26:24.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:26:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:26:23.831+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:26:24.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:26:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:26:23.856+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:26:24.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:26:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:26:23.856+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:26:24.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:26:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:26:23.856+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:26:24.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:26:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:26:23.856+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:26:24.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:26:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:26:24.466+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:26:24.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:26:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:26:24.466+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:26:24.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:26:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:26:24.466+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:26:25.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:26:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:26:24.858+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:26:25.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:26:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:26:24.858+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:26:25.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:26:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:26:24.858+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:26:25.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:26:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:26:24.843+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:26:25.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:26:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:26:24.843+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:26:25.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:26:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:26:24.843+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:26:25.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:26:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:26:24.843+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:26:25.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:26:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:26:25.467+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:26:25.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:26:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:26:25.467+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:26:25.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:26:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:26:25.467+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:26:26.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:26:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:26:25.860+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:26:26.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:26:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:26:25.860+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:26:26.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:26:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:26:25.860+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:26:26.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:26:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:26:25.799+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:26:26.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:26:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:26:25.799+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:26:26.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:26:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:26:25.799+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:26:26.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:26:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:26:25.799+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:26:26.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:26:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:26:26.445+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:26:26.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:26:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:26:26.445+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:26:26.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:26:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:26:26.445+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:26:27.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:26:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:26:26.903+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:26:27.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:26:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:26:26.903+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:26:27.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:26:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:26:26.903+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:26:27.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:26:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:26:26.809+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:26:27.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:26:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:26:26.809+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:26:27.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:26:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:26:26.809+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:26:27.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:26:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:26:26.809+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:26:27.742 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:26:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:26:27.314+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 23070 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:26:27.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:26:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:26:27.454+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:26:27.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:26:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:26:27.454+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:26:27.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:26:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:26:27.454+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:26:28.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:26:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:26:27.906+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:26:28.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:26:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:26:27.906+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:26:28.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:26:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:26:27.906+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:26:28.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:26:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:26:27.844+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:26:28.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:26:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:26:27.844+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:26:28.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:26:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:26:27.844+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:26:28.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:26:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:26:27.844+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:26:28.880 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:26:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:26:28.501+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:26:28.881 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:26:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:26:28.501+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:26:28.881 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:26:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:26:28.501+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:26:29.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:26:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:26:28.919+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:26:29.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:26:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:26:28.919+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:26:29.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:26:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:26:28.919+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:26:29.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:26:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:26:28.880+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:26:29.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:26:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:26:28.880+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:26:29.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:26:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:26:28.880+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:26:29.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:26:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:26:28.880+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:26:29.883 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:26:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:26:29.532+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:26:29.883 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:26:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:26:29.532+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:26:29.883 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:26:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:26:29.532+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:26:30.218 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:26:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:26:29.965+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:26:30.218 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:26:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:26:29.965+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:26:30.218 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:26:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:26:29.965+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:26:30.218 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:26:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:26:29.881+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:26:30.219 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:26:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:26:29.881+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:26:30.219 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:26:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:26:29.881+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:26:30.219 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:26:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:26:29.881+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:26:30.927 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:26:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:26:30.539+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:26:30.928 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:26:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:26:30.539+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:26:30.928 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:26:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:26:30.539+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:26:31.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:26:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:26:30.959+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:26:31.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:26:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:26:30.959+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:26:31.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:26:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:26:30.959+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:26:31.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:26:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:26:30.926+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:26:31.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:26:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:26:30.926+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:26:31.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:26:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:26:30.926+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:26:31.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:26:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:26:30.926+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:26:31.927 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:26:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:26:31.569+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:26:31.928 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:26:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:26:31.569+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:26:31.928 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:26:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:26:31.569+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:26:32.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:26:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:26:31.944+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:26:32.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:26:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:26:31.944+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:26:32.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:26:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:26:31.944+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:26:32.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:26:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:26:31.923+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:26:32.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:26:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:26:31.923+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:26:32.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:26:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:26:31.923+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:26:32.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:26:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:26:31.923+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:26:32.741 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:26:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:26:32.316+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 23142 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:26:32.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:26:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:26:32.546+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:26:32.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:26:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:26:32.546+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:26:32.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:26:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:26:32.546+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:26:33.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:26:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:26:32.966+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:26:33.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:26:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:26:32.966+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:26:33.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:26:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:26:32.966+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:26:33.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:26:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:26:32.911+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:26:33.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:26:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:26:32.911+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:26:33.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:26:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:26:32.911+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:26:33.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:26:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:26:32.912+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:26:33.944 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:26:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:26:33.540+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:26:33.944 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:26:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:26:33.540+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:26:33.944 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:26:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:26:33.540+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:26:34.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:26:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:26:33.942+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:26:34.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:26:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:26:33.942+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:26:34.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:26:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:26:33.942+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:26:34.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:26:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:26:33.957+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:26:34.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:26:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:26:33.957+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:26:34.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:26:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:26:33.957+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:26:34.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:26:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:26:33.957+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:26:34.899 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:26:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:26:34.557+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:26:34.900 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:26:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:26:34.557+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:26:34.900 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:26:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:26:34.557+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:26:35.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:26:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:26:34.897+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:26:35.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:26:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:26:34.897+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:26:35.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:26:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:26:34.897+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:26:35.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:26:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:26:34.985+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:26:35.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:26:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:26:34.985+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:26:35.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:26:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:26:34.985+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:26:35.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:26:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:26:34.985+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:26:35.912 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:26:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:26:35.515+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:26:35.912 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:26:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:26:35.515+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:26:35.913 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:26:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:26:35.515+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:26:36.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:26:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:26:35.910+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:26:36.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:26:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:26:35.910+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:26:36.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:26:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:26:35.910+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:26:36.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:26:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:26:36.011+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:26:36.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:26:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:26:36.011+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:26:36.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:26:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:26:36.011+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:26:36.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:26:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:26:36.011+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:26:36.962 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:26:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:26:36.524+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:26:36.962 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:26:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:26:36.524+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:26:36.962 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:26:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:26:36.524+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:26:37.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:26:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:26:36.960+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:26:37.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:26:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:26:36.960+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:26:37.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:26:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:26:36.960+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:26:37.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:26:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:26:37.012+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:26:37.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:26:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:26:37.012+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:26:37.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:26:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:26:37.012+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:26:37.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:26:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:26:37.012+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:26:37.742 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:26:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:26:37.320+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 23212 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:26:37.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:26:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:26:37.501+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:26:37.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:26:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:26:37.501+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:26:37.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:26:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:26:37.501+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:26:38.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:26:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:26:37.953+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:26:38.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:26:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:26:37.953+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:26:38.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:26:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:26:37.953+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:26:38.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:26:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:26:38.061+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:26:38.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:26:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:26:38.061+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:26:38.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:26:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:26:38.061+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:26:38.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:26:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:26:38.061+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:26:38.949 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:26:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:26:38.500+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:26:38.949 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:26:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:26:38.500+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:26:38.949 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:26:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:26:38.500+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:26:39.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:26:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:26:38.947+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:26:39.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:26:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:26:38.947+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:26:39.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:26:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:26:38.947+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:26:39.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:26:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:26:39.035+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:26:39.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:26:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:26:39.035+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:26:39.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:26:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:26:39.035+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:26:39.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:26:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:26:39.035+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:26:39.885 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:26:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:26:39.522+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:26:39.885 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:26:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:26:39.522+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:26:39.885 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:26:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:26:39.522+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:26:40.218 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:26:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:26:39.911+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:26:40.219 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:26:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:26:39.911+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:26:40.219 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:26:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:26:39.911+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:26:40.219 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:26:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:26:39.988+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:26:40.219 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:26:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:26:39.988+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:26:40.219 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:26:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:26:39.988+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:26:40.219 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:26:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:26:39.988+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:26:40.937 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:26:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:26:40.505+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:26:40.937 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:26:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:26:40.505+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:26:40.937 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:26:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:26:40.505+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:26:41.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:26:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:26:40.935+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:26:41.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:26:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:26:40.935+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:26:41.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:26:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:26:40.935+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:26:41.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:26:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:26:40.944+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:26:41.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:26:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:26:40.944+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:26:41.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:26:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:26:40.944+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:26:41.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:26:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:26:40.945+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:26:41.751 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:26:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:26:41.490+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:26:41.751 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:26:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:26:41.490+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:26:41.751 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:26:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:26:41.490+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:26:42.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:26:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:26:41.906+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:26:42.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:26:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:26:41.906+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:26:42.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:26:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:26:41.906+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:26:42.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:26:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:26:41.953+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:26:42.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:26:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:26:41.953+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:26:42.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:26:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:26:41.953+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:26:42.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:26:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:26:41.953+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:26:42.743 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:26:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:26:42.323+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 23284 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:26:42.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:26:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:26:42.511+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:26:42.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:26:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:26:42.511+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:26:42.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:26:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:26:42.511+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:26:43.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:26:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:26:42.938+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:26:43.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:26:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:26:42.938+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:26:43.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:26:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:26:42.938+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:26:43.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:26:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:26:42.912+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:26:43.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:26:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:26:42.912+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:26:43.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:26:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:26:42.912+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:26:43.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:26:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:26:42.913+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:26:43.945 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:26:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:26:43.527+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:26:43.946 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:26:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:26:43.527+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:26:43.946 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:26:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:26:43.527+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:26:44.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:26:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:26:43.944+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:26:44.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:26:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:26:43.944+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:26:44.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:26:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:26:43.944+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:26:44.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:26:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:26:43.948+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:26:44.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:26:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:26:43.948+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:26:44.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:26:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:26:43.948+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:26:44.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:26:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:26:43.949+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:26:44.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:26:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:26:44.488+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:26:44.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:26:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:26:44.488+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:26:44.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:26:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:26:44.488+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:26:45.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:26:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:26:44.914+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:26:45.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:26:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:26:44.914+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:26:45.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:26:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:26:44.914+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:26:45.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:26:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:26:44.903+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:26:45.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:26:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:26:44.903+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:26:45.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:26:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:26:44.903+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:26:45.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:26:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:26:44.903+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:26:45.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:26:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:26:45.479+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:26:45.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:26:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:26:45.479+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:26:45.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:26:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:26:45.479+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:26:46.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:26:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:26:45.906+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:26:46.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:26:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:26:45.906+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:26:46.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:26:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:26:45.906+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:26:46.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:26:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:26:45.883+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:26:46.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:26:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:26:45.883+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:26:46.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:26:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:26:45.883+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:26:46.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:26:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:26:45.883+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:26:46.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:26:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:26:46.447+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:26:46.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:26:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:26:46.447+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:26:46.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:26:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:26:46.447+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:26:47.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:26:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:26:46.945+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:26:47.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:26:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:26:46.945+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:26:47.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:26:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:26:46.945+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:26:47.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:26:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:26:46.915+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:26:47.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:26:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:26:46.915+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:26:47.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:26:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:26:46.915+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:26:47.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:26:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:26:46.915+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:26:47.742 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:26:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:26:47.327+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 23354 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:26:47.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:26:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:26:47.438+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:26:47.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:26:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:26:47.438+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:26:47.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:26:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:26:47.438+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:26:48.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:26:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:26:47.896+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:26:48.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:26:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:26:47.896+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:26:48.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:26:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:26:47.896+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:26:48.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:26:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:26:47.904+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:26:48.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:26:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:26:47.904+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:26:48.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:26:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:26:47.904+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:26:48.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:26:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:26:47.904+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:26:48.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:26:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:26:48.458+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:26:48.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:26:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:26:48.458+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:26:48.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:26:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:26:48.458+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:26:49.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:26:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:26:48.886+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:26:49.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:26:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:26:48.886+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:26:49.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:26:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:26:48.886+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:26:49.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:26:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:26:48.891+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:26:49.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:26:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:26:48.891+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:26:49.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:26:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:26:48.891+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:26:49.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:26:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:26:48.891+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:26:49.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:26:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:26:49.480+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:26:49.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:26:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:26:49.480+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:26:49.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:26:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:26:49.480+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:26:50.219 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:26:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:26:49.862+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:26:50.219 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:26:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:26:49.862+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:26:50.220 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:26:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:26:49.862+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:26:50.220 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:26:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:26:49.920+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:26:50.220 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:26:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:26:49.920+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:26:50.220 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:26:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:26:49.920+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:26:50.220 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:26:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:26:49.920+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:26:50.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:26:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:26:50.437+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:26:50.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:26:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:26:50.437+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:26:50.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:26:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:26:50.437+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:26:51.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:26:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:26:50.845+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:26:51.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:26:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:26:50.845+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:26:51.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:26:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:26:50.845+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:26:51.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:26:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:26:50.932+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:26:51.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:26:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:26:50.932+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:26:51.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:26:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:26:50.932+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:26:51.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:26:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:26:50.933+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:26:51.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:26:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:26:51.461+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:26:51.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:26:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:26:51.461+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:26:51.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:26:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:26:51.461+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:26:52.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:26:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:26:51.845+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:26:52.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:26:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:26:51.845+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:26:52.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:26:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:26:51.845+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:26:52.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:26:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:26:51.914+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:26:52.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:26:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:26:51.914+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:26:52.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:26:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:26:51.914+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:26:52.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:26:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:26:51.914+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:26:52.742 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:26:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:26:52.330+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 23426 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:26:52.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:26:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:26:52.498+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:26:52.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:26:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:26:52.498+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:26:52.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:26:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:26:52.498+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:26:53.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:26:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:26:52.848+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:26:53.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:26:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:26:52.848+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:26:53.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:26:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:26:52.848+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:26:53.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:26:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:26:52.874+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:26:53.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:26:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:26:52.874+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:26:53.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:26:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:26:52.874+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:26:53.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:26:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:26:52.874+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:26:53.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:26:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:26:53.470+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:26:53.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:26:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:26:53.470+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:26:53.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:26:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:26:53.470+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:26:54.072 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:26:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:26:53.801+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:26:54.073 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:26:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:26:53.801+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:26:54.073 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:26:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:26:53.801+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:26:54.073 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:26:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:26:53.851+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:26:54.073 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:26:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:26:53.851+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:26:54.073 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:26:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:26:53.851+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:26:54.073 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:26:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:26:53.851+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:26:54.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:26:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:26:54.480+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:26:54.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:26:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:26:54.480+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:26:54.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:26:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:26:54.480+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:26:55.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:26:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:26:54.807+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:26:55.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:26:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:26:54.807+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:26:55.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:26:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:26:54.807+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:26:55.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:26:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:26:54.823+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:26:55.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:26:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:26:54.823+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:26:55.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:26:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:26:54.823+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:26:55.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:26:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:26:54.823+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:26:55.799 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:26:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:26:55.524+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:26:55.799 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:26:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:26:55.524+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:26:55.799 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:26:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:26:55.524+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:26:56.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:26:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:26:55.825+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:26:56.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:26:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:26:55.825+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:26:56.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:26:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:26:55.825+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:26:56.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:26:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:26:55.797+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:26:56.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:26:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:26:55.797+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:26:56.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:26:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:26:55.797+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:26:56.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:26:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:26:55.797+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:26:56.821 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:26:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:26:56.774+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:26:56.822 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:26:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:26:56.774+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:26:56.822 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:26:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:26:56.774+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:26:56.822 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:26:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:26:56.774+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:26:56.822 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:26:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:26:56.570+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:26:56.822 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:26:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:26:56.570+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:26:56.823 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:26:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:26:56.570+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:26:57.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:26:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:26:56.820+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:26:57.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:26:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:26:56.820+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:26:57.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:26:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:26:56.820+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:26:57.605 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:26:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:26:57.334+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 23496 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:26:57.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:26:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:26:57.837+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:26:57.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:26:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:26:57.837+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:26:57.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:26:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:26:57.837+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:26:57.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:26:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:26:57.780+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:26:57.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:26:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:26:57.780+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:26:57.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:26:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:26:57.780+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:26:57.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:26:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:26:57.780+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:26:57.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:26:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:26:57.604+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:26:57.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:26:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:26:57.604+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:26:57.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:26:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:26:57.604+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:26:58.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:26:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:26:58.819+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:26:58.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:26:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:26:58.819+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:26:58.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:26:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:26:58.819+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:26:58.992 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:26:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:26:58.580+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:26:58.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:26:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:26:58.580+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:26:58.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:26:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:26:58.580+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:26:58.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:26:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:26:58.744+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:26:58.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:26:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:26:58.744+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:26:58.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:26:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:26:58.744+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:26:58.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:26:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:26:58.744+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:26:59.858 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:26:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:26:59.769+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:26:59.858 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:26:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:26:59.769+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:26:59.858 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:26:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:26:59.769+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:26:59.858 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:26:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:26:59.769+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:26:59.859 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:26:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:26:59.581+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:26:59.859 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:26:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:26:59.581+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:26:59.859 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:26:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:26:59.581+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:27:00.220 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:26:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:26:59.856+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:27:00.220 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:26:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:26:59.856+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:27:00.220 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:26:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:26:59.856+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:27:00.892 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:27:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:27:00.533+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:27:00.892 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:27:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:27:00.533+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:27:00.892 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:27:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:27:00.533+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:27:00.892 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:27:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:27:00.753+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:27:00.893 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:27:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:27:00.753+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:27:00.893 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:27:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:27:00.753+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:27:00.893 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:27:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:27:00.753+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:27:01.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:27:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:27:00.890+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:27:01.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:27:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:27:00.890+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:27:01.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:27:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:27:00.890+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:27:01.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:27:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:27:01.486+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:27:01.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:27:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:27:01.486+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:27:01.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:27:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:27:01.486+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:27:02.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:27:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:27:01.880+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:27:02.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:27:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:27:01.880+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:27:02.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:27:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:27:01.880+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:27:02.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:27:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:27:01.785+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:27:02.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:27:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:27:01.785+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:27:02.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:27:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:27:01.785+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:27:02.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:27:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:27:01.785+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:27:02.742 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:27:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:27:02.339+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 23568 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:27:02.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:27:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:27:02.477+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:27:02.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:27:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:27:02.477+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:27:02.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:27:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:27:02.477+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:27:03.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:27:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:27:02.919+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:27:03.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:27:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:27:02.919+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:27:03.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:27:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:27:02.919+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:27:03.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:27:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:27:02.744+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:27:03.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:27:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:27:02.744+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:27:03.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:27:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:27:02.744+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:27:03.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:27:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:27:02.744+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:27:03.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:27:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:27:03.465+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:27:03.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:27:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:27:03.465+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:27:03.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:27:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:27:03.465+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:27:04.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:27:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:27:03.749+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:27:04.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:27:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:27:03.749+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:27:04.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:27:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:27:03.749+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:27:04.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:27:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:27:03.749+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:27:04.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:27:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:27:03.916+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:27:04.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:27:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:27:03.916+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:27:04.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:27:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:27:03.916+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:27:04.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:27:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:27:04.472+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:27:04.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:27:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:27:04.472+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:27:04.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:27:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:27:04.472+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:27:05.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:27:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:27:04.748+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:27:05.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:27:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:27:04.748+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:27:05.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:27:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:27:04.748+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:27:05.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:27:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:27:04.748+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:27:05.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:27:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:27:04.915+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:27:05.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:27:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:27:04.915+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:27:05.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:27:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:27:04.915+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:27:05.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:27:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:27:05.473+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:27:05.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:27:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:27:05.473+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:27:05.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:27:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:27:05.473+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:27:05.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:27:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:27:05.700+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:27:05.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:27:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:27:05.700+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:27:05.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:27:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:27:05.700+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:27:05.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:27:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:27:05.701+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:27:06.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:27:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:27:05.963+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:27:06.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:27:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:27:05.963+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:27:06.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:27:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:27:05.963+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:27:06.726 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:27:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:27:06.433+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:27:06.727 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:27:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:27:06.433+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:27:06.727 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:27:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:27:06.433+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:27:06.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:27:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:27:06.955+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:27:06.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:27:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:27:06.956+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:27:06.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:27:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:27:06.956+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:27:06.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:27:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:27:06.725+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:27:06.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:27:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:27:06.725+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:27:06.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:27:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:27:06.725+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:27:06.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:27:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:27:06.725+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:27:07.738 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:27:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:27:07.342+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 23638 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:27:07.738 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:27:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:27:07.482+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:27:07.738 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:27:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:27:07.482+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:27:07.738 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:27:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:27:07.482+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:27:07.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:27:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:27:07.737+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:27:08.025 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:27:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:27:07.737+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:27:08.025 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:27:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:27:07.737+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:27:08.025 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:27:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:27:07.737+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:27:08.025 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:27:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:27:07.920+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:27:08.025 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:27:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:27:07.920+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:27:08.025 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:27:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:27:07.920+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:27:08.727 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:27:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:27:08.497+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:27:08.728 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:27:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:27:08.497+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:27:08.728 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:27:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:27:08.497+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:27:08.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:27:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:27:08.932+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:27:08.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:27:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:27:08.932+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:27:08.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:27:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:27:08.932+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:27:08.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:27:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:27:08.726+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:27:08.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:27:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:27:08.726+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:27:08.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:27:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:27:08.726+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:27:08.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:27:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:27:08.726+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:27:09.885 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:27:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:27:09.531+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:27:09.886 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:27:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:27:09.531+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:27:09.886 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:27:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:27:09.531+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:27:09.886 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:27:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:27:09.771+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:27:09.886 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:27:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:27:09.771+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:27:09.886 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:27:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:27:09.771+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:27:09.886 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:27:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:27:09.771+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:27:10.221 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:27:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:27:09.893+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:27:10.221 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:27:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:27:09.893+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:27:10.221 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:27:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:27:09.893+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:27:10.872 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:27:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:27:10.770+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:27:10.872 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:27:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:27:10.770+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:27:10.872 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:27:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:27:10.770+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:27:10.872 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:27:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:27:10.770+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:27:10.873 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:27:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:27:10.562+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:27:10.873 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:27:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:27:10.562+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:27:10.873 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:27:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:27:10.562+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:27:11.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:27:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:27:10.871+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:27:11.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:27:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:27:10.871+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:27:11.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:27:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:27:10.871+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:27:11.878 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:27:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:27:11.750+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:27:11.879 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:27:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:27:11.750+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:27:11.879 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:27:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:27:11.750+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:27:11.879 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:27:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:27:11.750+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:27:11.879 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:27:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:27:11.516+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:27:11.879 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:27:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:27:11.516+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:27:11.880 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:27:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:27:11.516+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:27:12.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:27:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:27:11.877+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:27:12.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:27:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:27:11.877+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:27:12.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:27:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:27:11.877+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:27:12.742 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:27:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:27:12.345+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 23710 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:27:12.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:27:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:27:12.491+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:27:12.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:27:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:27:12.491+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:27:12.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:27:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:27:12.491+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:27:13.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:27:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:27:12.842+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:27:13.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:27:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:27:12.842+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:27:13.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:27:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:27:12.842+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:27:13.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:27:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:27:12.780+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:27:13.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:27:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:27:12.780+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:27:13.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:27:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:27:12.780+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:27:13.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:27:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:27:12.780+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:27:13.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:27:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:27:13.477+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:27:13.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:27:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:27:13.477+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:27:13.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:27:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:27:13.477+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:27:14.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:27:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:27:13.819+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:27:14.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:27:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:27:13.819+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:27:14.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:27:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:27:13.819+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:27:14.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:27:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:27:13.756+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:27:14.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:27:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:27:13.756+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:27:14.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:27:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:27:13.756+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:27:14.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:27:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:27:13.756+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:27:14.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:27:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:27:14.484+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:27:14.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:27:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:27:14.484+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:27:14.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:27:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:27:14.484+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:27:15.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:27:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:27:14.784+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:27:15.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:27:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:27:14.784+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:27:15.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:27:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:27:14.784+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:27:15.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:27:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:27:14.784+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:27:15.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:27:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:27:14.812+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:27:15.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:27:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:27:14.812+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:27:15.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:27:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:27:14.812+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:27:15.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:27:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:27:15.469+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:27:15.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:27:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:27:15.469+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:27:15.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:27:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:27:15.469+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:27:16.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:27:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:27:15.799+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:27:16.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:27:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:27:15.799+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:27:16.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:27:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:27:15.799+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:27:16.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:27:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:27:15.800+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:27:16.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:27:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:27:15.789+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:27:16.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:27:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:27:15.789+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:27:16.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:27:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:27:15.789+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:27:16.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:27:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:27:16.471+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:27:16.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:27:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:27:16.471+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:27:16.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:27:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:27:16.471+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:27:17.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:27:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:27:16.808+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:27:17.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:27:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:27:16.808+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:27:17.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:27:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:27:16.808+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:27:17.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:27:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:27:16.808+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:27:17.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:27:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:27:16.775+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:27:17.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:27:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:27:16.775+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:27:17.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:27:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:27:16.775+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:27:17.742 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:27:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:27:17.349+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 23780 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:27:17.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:27:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:27:17.483+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:27:17.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:27:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:27:17.483+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:27:17.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:27:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:27:17.483+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:27:18.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:27:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:27:17.805+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:27:18.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:27:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:27:17.805+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:27:18.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:27:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:27:17.805+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:27:18.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:27:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:27:17.805+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:27:18.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:27:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:27:17.796+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:27:18.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:27:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:27:17.796+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:27:18.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:27:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:27:17.796+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:27:18.793 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:27:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:27:18.767+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:27:18.794 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:27:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:27:18.767+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:27:18.794 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:27:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:27:18.767+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:27:18.794 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:27:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:27:18.767+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:27:18.794 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:27:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:27:18.524+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:27:18.794 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:27:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:27:18.524+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:27:18.794 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:27:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:27:18.524+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:27:19.086 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:27:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:27:18.792+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:27:19.086 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:27:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:27:18.792+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:27:19.086 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:27:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:27:18.792+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:27:19.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:27:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:27:19.551+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:27:19.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:27:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:27:19.551+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:27:19.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:27:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:27:19.551+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:27:20.046 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:27:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:27:19.802+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:27:20.046 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:27:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:27:19.802+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:27:20.046 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:27:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:27:19.802+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:27:20.046 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:27:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:27:19.759+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:27:20.047 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:27:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:27:19.759+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:27:20.047 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:27:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:27:19.759+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:27:20.047 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:27:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:27:19.759+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:27:20.992 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:27:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:27:20.590+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:27:20.992 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:27:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:27:20.590+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:27:20.992 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:27:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:27:20.590+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:27:20.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:27:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:27:20.781+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:27:20.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:27:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:27:20.781+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:27:20.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:27:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:27:20.781+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:27:20.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:27:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:27:20.770+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:27:20.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:27:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:27:20.770+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:27:20.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:27:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:27:20.770+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:27:20.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:27:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:27:20.770+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:27:21.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:27:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:27:21.814+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:27:21.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:27:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:27:21.814+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:27:21.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:27:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:27:21.814+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:27:21.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:27:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:27:21.814+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:27:21.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:27:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:27:21.794+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:27:21.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:27:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:27:21.794+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:27:21.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:27:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:27:21.794+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:27:21.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:27:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:27:21.583+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:27:21.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:27:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:27:21.583+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:27:21.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:27:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:27:21.583+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:27:22.609 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:27:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:27:22.351+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 23852 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:27:22.992 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:27:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:27:22.607+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:27:22.992 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:27:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:27:22.607+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:27:22.992 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:27:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:27:22.607+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:27:22.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:27:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:27:22.770+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:27:22.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:27:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:27:22.770+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:27:22.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:27:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:27:22.770+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:27:22.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:27:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:27:22.770+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:27:22.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:27:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:27:22.816+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:27:22.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:27:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:27:22.816+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:27:22.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:27:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:27:22.816+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:27:23.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:27:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:27:23.818+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:27:23.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:27:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:27:23.818+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:27:23.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:27:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:27:23.818+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:27:23.992 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:27:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:27:23.594+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:27:23.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:27:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:27:23.594+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:27:23.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:27:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:27:23.594+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:27:23.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:27:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:27:23.721+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:27:23.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:27:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:27:23.721+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:27:23.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:27:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:27:23.721+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:27:23.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:27:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:27:23.721+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:27:24.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:27:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:27:24.771+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:27:24.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:27:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:27:24.771+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:27:24.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:27:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:27:24.771+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:27:24.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:27:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:27:24.640+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:27:24.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:27:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:27:24.640+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:27:24.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:27:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:27:24.640+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:27:24.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:27:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:27:24.714+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:27:24.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:27:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:27:24.714+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:27:24.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:27:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:27:24.714+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:27:24.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:27:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:27:24.714+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:27:25.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:27:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:27:25.770+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:27:25.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:27:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:27:25.770+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:27:25.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:27:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:27:25.770+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:27:25.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:27:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:27:25.666+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:27:25.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:27:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:27:25.666+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:27:25.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:27:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:27:25.666+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:27:25.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:27:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:27:25.723+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:27:25.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:27:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:27:25.723+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:27:25.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:27:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:27:25.723+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:27:25.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:27:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:27:25.723+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:27:26.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:27:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:27:26.725+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:27:26.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:27:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:27:26.725+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:27:26.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:27:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:27:26.725+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:27:26.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:27:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:27:26.665+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:27:26.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:27:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:27:26.666+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:27:26.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:27:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:27:26.666+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:27:26.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:27:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:27:26.767+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:27:26.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:27:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:27:26.767+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:27:26.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:27:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:27:26.767+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:27:26.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:27:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:27:26.767+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:27:27.698 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:27:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:27:27.355+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 23922 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:27:27.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:27:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:27:27.761+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:27:27.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:27:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:27:27.761+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:27:27.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:27:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:27:27.761+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:27:27.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:27:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:27:27.761+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:27:27.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:27:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:27:27.696+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:27:27.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:27:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:27:27.696+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:27:27.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:27:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:27:27.696+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:27:27.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:27:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:27:27.704+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:27:27.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:27:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:27:27.704+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:27:27.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:27:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:27:27.704+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:27:28.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:27:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:27:28.704+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:27:28.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:27:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:27:28.704+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:27:28.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:27:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:27:28.704+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:27:28.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:27:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:27:28.745+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:27:28.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:27:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:27:28.745+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:27:28.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:27:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:27:28.745+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:27:28.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:27:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:27:28.745+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:27:28.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:27:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:27:28.720+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:27:28.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:27:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:27:28.720+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:27:28.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:27:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:27:28.720+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:27:29.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:27:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:27:29.732+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:27:29.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:27:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:27:29.732+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:27:29.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:27:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:27:29.732+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:27:29.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:27:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:27:29.706+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:27:29.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:27:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:27:29.706+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:27:29.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:27:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:27:29.706+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:27:29.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:27:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:27:29.706+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:27:29.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:27:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:27:29.690+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:27:29.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:27:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:27:29.690+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:27:29.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:27:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:27:29.690+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:27:30.763 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:27:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:27:30.699+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:27:30.764 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:27:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:27:30.699+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:27:30.764 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:27:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:27:30.699+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:27:30.764 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:27:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:27:30.728+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:27:30.764 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:27:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:27:30.728+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:27:30.765 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:27:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:27:30.728+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:27:30.765 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:27:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:27:30.657+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:27:30.765 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:27:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:27:30.657+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:27:30.765 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:27:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:27:30.657+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:27:30.765 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:27:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:27:30.657+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:27:31.992 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:27:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:27:31.777+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:27:31.992 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:27:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:27:31.778+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:27:31.992 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:27:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:27:31.778+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:27:31.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:27:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:27:31.620+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:27:31.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:27:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:27:31.620+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:27:31.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:27:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:27:31.620+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:27:31.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:27:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:27:31.621+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:27:31.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:27:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:27:31.732+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:27:31.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:27:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:27:31.732+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:27:31.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:27:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:27:31.732+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:27:32.742 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:27:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:27:32.359+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 23994 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:27:32.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:27:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:27:32.582+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:27:32.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:27:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:27:32.582+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:27:32.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:27:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:27:32.582+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:27:32.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:27:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:27:32.582+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:27:33.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:27:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:27:32.800+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:27:33.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:27:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:27:32.800+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:27:33.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:27:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:27:32.800+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:27:33.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:27:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:27:32.772+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:27:33.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:27:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:27:32.772+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:27:33.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:27:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:27:32.772+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:27:33.992 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:27:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:27:33.806+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:27:33.992 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:27:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:27:33.806+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:27:33.992 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:27:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:27:33.806+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:27:33.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:27:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:27:33.585+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:27:33.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:27:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:27:33.585+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:27:33.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:27:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:27:33.585+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:27:33.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:27:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:27:33.586+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:27:33.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:27:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:27:33.755+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:27:33.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:27:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:27:33.755+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:27:33.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:27:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:27:33.755+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:27:34.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:27:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:27:34.617+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:27:34.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:27:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:27:34.617+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:27:34.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:27:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:27:34.617+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:27:34.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:27:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:27:34.617+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:27:34.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:27:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:27:34.773+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:27:34.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:27:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:27:34.773+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:27:34.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:27:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:27:34.773+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:27:34.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:27:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:27:34.748+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:27:34.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:27:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:27:34.748+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:27:34.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:27:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:27:34.748+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:27:35.992 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:27:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:27:35.768+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:27:35.992 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:27:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:27:35.768+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:27:35.992 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:27:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:27:35.768+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:27:35.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:27:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:27:35.721+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:27:35.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:27:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:27:35.721+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:27:35.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:27:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:27:35.721+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:27:35.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:27:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:27:35.612+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:27:35.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:27:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:27:35.612+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:27:35.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:27:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:27:35.612+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:27:35.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:27:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:27:35.612+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:27:36.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:27:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:27:36.747+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:27:36.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:27:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:27:36.747+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:27:36.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:27:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:27:36.747+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:27:36.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:27:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:27:36.646+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:27:36.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:27:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:27:36.646+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:27:36.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:27:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:27:36.646+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:27:36.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:27:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:27:36.646+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:27:36.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:27:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:27:36.737+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:27:36.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:27:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:27:36.737+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:27:36.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:27:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:27:36.737+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:27:37.675 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:27:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:27:37.362+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 24064 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:27:37.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:27:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:27:37.704+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:27:37.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:27:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:27:37.704+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:27:37.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:27:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:27:37.704+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:27:37.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:27:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:27:37.700+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:27:37.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:27:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:27:37.700+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:27:37.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:27:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:27:37.700+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:27:37.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:27:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:27:37.672+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:27:37.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:27:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:27:37.672+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:27:37.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:27:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:27:37.672+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:27:37.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:27:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:27:37.673+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:27:38.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:27:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:27:38.742+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:27:38.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:27:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:27:38.743+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:27:38.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:27:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:27:38.743+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:27:38.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:27:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:27:38.640+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:27:38.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:27:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:27:38.640+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:27:38.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:27:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:27:38.640+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:27:38.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:27:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:27:38.640+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:27:38.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:27:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:27:38.735+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:27:38.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:27:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:27:38.735+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:27:38.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:27:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:27:38.735+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:27:39.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:27:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:27:39.711+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:27:39.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:27:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:27:39.711+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:27:39.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:27:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:27:39.711+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:27:39.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:27:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:27:39.715+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:27:39.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:27:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:27:39.715+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:27:39.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:27:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:27:39.715+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:27:39.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:27:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:27:39.686+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:27:39.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:27:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:27:39.686+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:27:39.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:27:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:27:39.686+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:27:39.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:27:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:27:39.686+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:27:40.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:27:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:27:40.738+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:27:40.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:27:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:27:40.738+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:27:40.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:27:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:27:40.738+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:27:40.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:27:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:27:40.669+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:27:40.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:27:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:27:40.669+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:27:40.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:27:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:27:40.669+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:27:40.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:27:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:27:40.669+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:27:40.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:27:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:27:40.675+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:27:40.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:27:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:27:40.675+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:27:40.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:27:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:27:40.675+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:27:41.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:27:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:27:41.760+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:27:41.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:27:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:27:41.760+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:27:41.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:27:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:27:41.760+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:27:41.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:27:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:27:41.681+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:27:41.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:27:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:27:41.681+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:27:41.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:27:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:27:41.681+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:27:41.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:27:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:27:41.681+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:27:41.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:27:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:27:41.649+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:27:41.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:27:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:27:41.649+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:27:41.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:27:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:27:41.649+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:27:42.638 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:27:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:27:42.366+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 24136 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:27:42.992 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:27:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:27:42.667+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:27:42.992 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:27:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:27:42.667+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:27:42.992 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:27:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:27:42.667+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:27:42.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:27:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:27:42.770+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:27:42.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:27:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:27:42.770+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:27:42.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:27:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:27:42.770+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:27:42.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:27:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:27:42.637+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:27:42.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:27:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:27:42.637+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:27:42.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:27:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:27:42.637+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:27:42.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:27:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:27:42.637+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:27:43.992 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:27:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:27:43.622+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:27:43.992 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:27:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:27:43.622+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:27:43.992 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:27:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:27:43.622+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:27:43.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:27:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:27:43.784+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:27:43.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:27:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:27:43.784+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:27:43.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:27:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:27:43.784+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:27:43.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:27:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:27:43.624+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:27:43.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:27:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:27:43.624+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:27:43.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:27:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:27:43.624+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:27:43.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:27:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:27:43.624+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:27:44.992 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:27:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:27:44.578+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:27:44.992 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:27:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:27:44.578+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:27:44.992 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:27:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:27:44.578+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:27:44.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:27:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:27:44.771+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:27:44.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:27:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:27:44.771+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:27:44.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:27:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:27:44.771+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:27:44.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:27:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:27:44.581+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:27:44.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:27:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:27:44.581+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:27:44.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:27:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:27:44.581+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:27:44.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:27:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:27:44.581+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:27:45.813 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:27:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:27:45.543+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:27:45.814 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:27:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:27:45.543+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:27:45.814 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:27:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:27:45.543+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:27:45.814 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:27:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:27:45.617+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:27:45.814 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:27:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:27:45.617+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:27:45.814 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:27:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:27:45.617+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:27:45.815 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:27:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:27:45.617+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:27:46.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:27:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:27:45.812+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:27:46.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:27:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:27:45.812+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:27:46.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:27:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:27:45.812+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:27:46.817 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:27:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:27:46.541+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:27:46.817 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:27:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:27:46.541+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:27:46.817 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:27:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:27:46.541+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:27:46.817 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:27:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:27:46.595+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:27:46.817 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:27:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:27:46.595+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:27:46.818 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:27:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:27:46.595+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:27:46.818 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:27:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:27:46.595+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:27:47.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:27:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:27:46.815+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:27:47.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:27:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:27:46.815+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:27:47.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:27:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:27:46.815+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:27:47.742 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:27:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:27:47.369+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 24206 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:27:47.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:27:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:27:47.521+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:27:47.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:27:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:27:47.521+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:27:47.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:27:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:27:47.521+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:27:47.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:27:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:27:47.605+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:27:47.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:27:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:27:47.605+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:27:47.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:27:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:27:47.605+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:27:47.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:27:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:27:47.605+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:27:48.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:27:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:27:47.861+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:27:48.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:27:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:27:47.861+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:27:48.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:27:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:27:47.861+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:27:48.825 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:27:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:27:48.513+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:27:48.826 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:27:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:27:48.513+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:27:48.826 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:27:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:27:48.513+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:27:48.826 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:27:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:27:48.624+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:27:48.826 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:27:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:27:48.624+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:27:48.826 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:27:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:27:48.624+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:27:48.826 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:27:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:27:48.624+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:27:49.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:27:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:27:48.822+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:27:49.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:27:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:27:48.822+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:27:49.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:27:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:27:48.822+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:27:49.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:27:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:27:49.665+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:27:49.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:27:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:27:49.665+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:27:49.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:27:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:27:49.665+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:27:49.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:27:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:27:49.665+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:27:49.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:27:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:27:49.469+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:27:49.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:27:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:27:49.469+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:27:49.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:27:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:27:49.469+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:27:50.224 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:27:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:27:49.827+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:27:50.224 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:27:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:27:49.827+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:27:50.224 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:27:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:27:49.827+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:27:50.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:27:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:27:50.467+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:27:50.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:27:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:27:50.467+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:27:50.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:27:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:27:50.467+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:27:50.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:27:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:27:50.854+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:27:50.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:27:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:27:50.854+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:27:50.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:27:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:27:50.854+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:27:50.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:27:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:27:50.657+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:27:50.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:27:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:27:50.657+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:27:50.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:27:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:27:50.657+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:27:50.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:27:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:27:50.657+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:27:51.756 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:27:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:27:51.504+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:27:51.756 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:27:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:27:51.504+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:27:51.757 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:27:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:27:51.504+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:27:51.757 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:27:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:27:51.641+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:27:51.757 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:27:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:27:51.641+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:27:51.757 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:27:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:27:51.641+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:27:51.757 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:27:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:27:51.641+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:27:52.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:27:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:27:51.885+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:27:52.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:27:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:27:51.885+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:27:52.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:27:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:27:51.885+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:27:52.667 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:27:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:27:52.373+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 24278 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:27:52.667 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:27:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:27:52.505+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:27:52.668 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:27:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:27:52.505+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:27:52.668 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:27:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:27:52.505+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:27:52.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:27:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:27:52.836+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:27:52.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:27:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:27:52.836+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:27:52.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:27:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:27:52.836+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:27:52.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:27:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:27:52.666+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:27:52.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:27:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:27:52.666+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:27:52.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:27:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:27:52.666+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:27:52.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:27:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:27:52.666+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:27:53.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:27:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:27:53.470+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:27:53.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:27:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:27:53.470+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:27:53.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:27:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:27:53.470+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:27:53.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:27:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:27:53.670+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:27:53.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:27:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:27:53.670+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:27:53.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:27:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:27:53.670+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:27:53.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:27:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:27:53.670+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:27:54.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:27:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:27:53.842+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:27:54.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:27:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:27:53.842+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:27:54.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:27:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:27:53.842+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:27:54.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:27:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:27:54.626+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:27:54.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:27:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:27:54.626+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:27:54.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:27:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:27:54.626+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:27:54.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:27:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:27:54.626+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:27:54.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:27:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:27:54.441+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:27:54.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:27:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:27:54.441+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:27:54.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:27:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:27:54.441+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:27:55.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:27:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:27:54.884+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:27:55.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:27:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:27:54.884+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:27:55.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:27:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:27:54.884+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:27:55.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:27:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:27:55.440+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:27:55.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:27:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:27:55.440+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:27:55.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:27:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:27:55.440+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:27:55.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:27:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:27:55.623+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:27:55.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:27:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:27:55.623+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:27:55.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:27:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:27:55.623+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:27:55.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:27:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:27:55.623+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:27:56.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:27:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:27:55.860+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:27:56.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:27:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:27:55.860+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:27:56.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:27:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:27:55.860+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:27:56.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:27:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:27:56.472+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:27:56.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:27:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:27:56.472+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:27:56.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:27:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:27:56.472+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:27:56.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:27:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:27:56.667+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:27:56.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:27:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:27:56.667+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:27:56.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:27:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:27:56.667+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:27:56.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:27:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:27:56.668+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:27:57.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:27:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:27:56.830+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:27:57.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:27:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:27:56.830+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:27:57.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:27:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:27:56.830+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:27:57.683 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:27:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:27:57.378+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 24348 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:27:57.683 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:27:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:27:57.458+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:27:57.683 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:27:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:27:57.458+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:27:57.683 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:27:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:27:57.458+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:27:57.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:27:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:27:57.831+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:27:57.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:27:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:27:57.831+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:27:57.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:27:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:27:57.831+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:27:57.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:27:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:27:57.681+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:27:57.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:27:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:27:57.681+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:27:57.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:27:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:27:57.681+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:27:57.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:27:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:27:57.682+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:27:58.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:27:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:27:58.482+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:27:58.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:27:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:27:58.482+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:27:58.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:27:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:27:58.482+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:27:58.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:27:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:27:58.663+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:27:58.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:27:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:27:58.663+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:27:58.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:27:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:27:58.663+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:27:58.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:27:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:27:58.663+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:27:59.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:27:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:27:58.862+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:27:59.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:27:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:27:58.862+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:27:59.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:27:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:27:58.862+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:27:59.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:27:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:27:59.455+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:27:59.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:27:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:27:59.455+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:27:59.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:27:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:27:59.455+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:27:59.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:27:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:27:59.632+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:27:59.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:27:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:27:59.632+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:27:59.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:27:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:27:59.632+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:27:59.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:27:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:27:59.632+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:28:00.225 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:27:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:27:59.853+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:28:00.225 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:27:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:27:59.853+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:28:00.225 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:27:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:27:59.853+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:28:00.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:28:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:28:00.441+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:28:00.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:28:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:28:00.441+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:28:00.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:28:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:28:00.442+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:28:00.890 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:28:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:28:00.606+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:28:00.891 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:28:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:28:00.606+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:28:00.891 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:28:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:28:00.606+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:28:00.891 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:28:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:28:00.606+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:28:01.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:28:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:28:00.889+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:28:01.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:28:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:28:00.889+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:28:01.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:28:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:28:00.889+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:28:01.656 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:28:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:28:01.579+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:28:01.657 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:28:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:28:01.579+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:28:01.657 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:28:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:28:01.579+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:28:01.657 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:28:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:28:01.579+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:28:01.657 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:28:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:28:01.401+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:28:01.657 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:28:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:28:01.401+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:28:01.657 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:28:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:28:01.401+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:28:01.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:28:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:28:01.858+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:28:01.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:28:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:28:01.858+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:28:01.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:28:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:28:01.858+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:28:02.742 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:28:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:28:02.381+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 24420 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:28:02.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:28:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:28:02.435+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:28:02.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:28:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:28:02.435+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:28:02.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:28:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:28:02.435+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:28:02.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:28:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:28:02.610+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:28:02.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:28:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:28:02.610+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:28:02.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:28:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:28:02.610+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:28:02.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:28:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:28:02.611+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:28:03.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:28:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:28:02.850+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:28:03.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:28:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:28:02.850+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:28:03.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:28:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:28:02.850+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:28:03.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:28:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:28:03.406+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:28:03.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:28:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:28:03.406+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:28:03.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:28:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:28:03.406+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:28:03.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:28:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:28:03.580+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:28:03.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:28:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:28:03.580+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:28:03.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:28:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:28:03.580+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:28:03.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:28:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:28:03.580+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:28:04.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:28:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:28:03.814+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:28:04.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:28:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:28:03.814+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:28:04.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:28:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:28:03.814+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:28:04.728 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:28:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:28:04.406+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:28:04.728 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:28:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:28:04.406+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:28:04.729 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:28:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:28:04.406+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:28:04.729 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:28:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:28:04.542+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:28:04.729 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:28:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:28:04.542+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:28:04.729 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:28:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:28:04.542+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:28:04.729 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:28:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:28:04.542+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:28:04.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:28:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:28:04.804+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:28:04.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:28:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:28:04.804+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:28:04.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:28:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:28:04.804+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:28:05.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:28:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:28:05.550+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:28:05.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:28:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:28:05.550+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:28:05.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:28:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:28:05.550+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:28:05.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:28:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:28:05.550+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:28:05.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:28:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:28:05.368+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:28:05.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:28:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:28:05.368+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:28:05.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:28:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:28:05.368+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:28:06.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:28:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:28:05.817+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:28:06.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:28:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:28:05.817+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:28:06.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:28:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:28:05.817+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:28:06.533 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:28:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:28:06.391+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:28:06.533 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:28:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:28:06.391+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:28:06.533 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:28:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:28:06.391+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:28:06.829 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:28:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:28:06.532+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:28:06.829 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:28:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:28:06.532+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:28:06.829 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:28:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:28:06.532+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:28:06.829 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:28:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:28:06.532+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:28:07.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:28:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:28:06.827+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:28:07.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:28:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:28:06.827+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:28:07.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:28:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:28:06.827+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:28:07.742 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:28:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:28:07.385+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 24490 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:28:07.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:28:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:28:07.409+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:28:07.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:28:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:28:07.409+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:28:07.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:28:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:28:07.409+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:28:07.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:28:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:28:07.550+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:28:07.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:28:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:28:07.550+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:28:07.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:28:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:28:07.550+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:28:07.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:28:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:28:07.550+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:28:08.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:28:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:28:07.868+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:28:08.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:28:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:28:07.868+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:28:08.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:28:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:28:07.868+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:28:08.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:28:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:28:08.419+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:28:08.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:28:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:28:08.419+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:28:08.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:28:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:28:08.419+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:28:08.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:28:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:28:08.542+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:28:08.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:28:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:28:08.542+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:28:08.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:28:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:28:08.542+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:28:08.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:28:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:28:08.542+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:28:09.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:28:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:28:08.910+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:28:09.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:28:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:28:08.910+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:28:09.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:28:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:28:08.910+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:28:09.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:28:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:28:09.441+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:28:09.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:28:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:28:09.441+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:28:09.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:28:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:28:09.441+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:28:09.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:28:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:28:09.559+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:28:09.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:28:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:28:09.559+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:28:09.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:28:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:28:09.559+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:28:09.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:28:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:28:09.559+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:28:10.225 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:28:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:28:09.928+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:28:10.226 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:28:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:28:09.928+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:28:10.226 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:28:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:28:09.928+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:28:10.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:28:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:28:10.600+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:28:10.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:28:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:28:10.600+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:28:10.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:28:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:28:10.600+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:28:10.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:28:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:28:10.600+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:28:10.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:28:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:28:10.490+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:28:10.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:28:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:28:10.490+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:28:10.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:28:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:28:10.490+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:28:11.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:28:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:28:10.941+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:28:11.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:28:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:28:10.941+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:28:11.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:28:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:28:10.941+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:28:11.757 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:28:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:28:11.492+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:28:11.757 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:28:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:28:11.492+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:28:11.758 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:28:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:28:11.492+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:28:11.758 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:28:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:28:11.560+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:28:11.758 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:28:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:28:11.560+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:28:11.758 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:28:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:28:11.560+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:28:11.758 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:28:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:28:11.560+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:28:12.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:28:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:28:11.911+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:28:12.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:28:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:28:11.911+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:28:12.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:28:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:28:11.911+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:28:12.742 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:28:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:28:12.388+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 24562 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:28:12.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:28:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:28:12.496+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:28:12.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:28:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:28:12.496+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:28:12.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:28:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:28:12.496+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:28:12.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:28:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:28:12.586+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:28:12.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:28:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:28:12.586+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:28:12.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:28:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:28:12.586+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:28:12.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:28:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:28:12.586+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:28:13.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:28:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:28:12.938+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:28:13.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:28:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:28:12.938+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:28:13.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:28:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:28:12.938+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:28:13.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:28:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:28:13.567+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:28:13.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:28:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:28:13.567+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:28:13.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:28:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:28:13.567+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:28:13.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:28:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:28:13.567+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:28:13.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:28:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:28:13.463+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:28:13.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:28:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:28:13.463+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:28:13.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:28:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:28:13.463+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:28:14.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:28:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:28:13.899+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:28:14.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:28:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:28:13.899+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:28:14.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:28:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:28:13.899+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:28:14.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:28:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:28:14.580+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:28:14.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:28:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:28:14.581+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:28:14.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:28:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:28:14.581+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:28:14.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:28:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:28:14.581+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:28:14.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:28:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:28:14.417+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:28:14.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:28:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:28:14.417+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:28:14.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:28:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:28:14.417+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:28:15.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:28:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:28:14.861+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:28:15.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:28:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:28:14.861+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:28:15.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:28:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:28:14.861+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:28:15.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:28:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:28:15.579+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:28:15.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:28:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:28:15.579+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:28:15.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:28:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:28:15.579+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:28:15.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:28:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:28:15.579+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:28:15.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:28:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:28:15.382+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:28:15.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:28:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:28:15.382+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:28:15.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:28:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:28:15.382+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:28:16.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:28:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:28:15.845+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:28:16.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:28:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:28:15.845+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:28:16.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:28:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:28:15.845+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:28:16.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:28:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:28:16.416+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:28:16.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:28:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:28:16.416+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:28:16.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:28:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:28:16.416+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:28:16.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:28:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:28:16.536+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:28:16.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:28:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:28:16.537+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:28:16.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:28:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:28:16.537+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:28:16.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:28:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:28:16.537+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:28:17.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:28:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:28:16.834+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:28:17.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:28:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:28:16.834+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:28:17.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:28:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:28:16.834+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:28:17.742 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:28:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:28:17.391+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 24632 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:28:17.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:28:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:28:17.548+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:28:17.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:28:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:28:17.548+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:28:17.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:28:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:28:17.548+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:28:17.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:28:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:28:17.548+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:28:17.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:28:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:28:17.376+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:28:17.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:28:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:28:17.376+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:28:17.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:28:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:28:17.376+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:28:18.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:28:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:28:17.787+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:28:18.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:28:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:28:17.787+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:28:18.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:28:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:28:17.787+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:43.542463+0000 (oldest deadline 2024-02-20T08:18:03.542463+0000) 2024-02-20T08:28:18.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:28:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:28:18.595+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:28:18.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:28:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:28:18.595+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:28:18.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:28:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:28:18.595+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:28:18.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:28:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:28:18.595+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:28:18.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:28:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:28:18.392+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:28:18.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:28:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:28:18.392+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:28:18.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:28:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:28:18.392+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:28:19.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:28:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:28:19.359+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:28:19.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:28:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:28:19.359+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:28:19.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:28:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:28:19.359+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:28:19.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:28:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:28:19.568+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:28:19.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:28:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:28:19.568+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:28:19.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:28:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:28:19.568+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:28:19.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:28:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:28:19.568+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:28:20.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:28:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:28:20.327+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:28:20.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:28:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:28:20.327+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:28:20.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:28:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:28:20.327+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:28:20.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:28:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:28:20.548+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:28:20.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:28:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:28:20.548+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:28:20.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:28:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:28:20.548+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:28:20.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:28:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:28:20.548+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:28:21.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:28:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:28:21.342+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:28:21.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:28:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:28:21.342+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:28:21.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:28:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:28:21.342+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:28:21.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:28:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:28:21.505+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:28:21.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:28:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:28:21.505+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:28:21.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:28:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:28:21.505+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:28:21.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:28:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:28:21.505+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:28:22.742 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:28:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:28:22.395+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 24704 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:28:22.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:28:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:28:22.302+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:28:22.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:28:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:28:22.302+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:28:22.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:28:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:28:22.302+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:28:22.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:28:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:28:22.508+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:28:22.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:28:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:28:22.508+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:28:22.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:28:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:28:22.508+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:28:22.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:28:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:28:22.508+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:28:23.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:28:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:28:23.264+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:28:23.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:28:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:28:23.264+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:28:23.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:28:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:28:23.264+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:28:23.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:28:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:28:23.556+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:28:23.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:28:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:28:23.556+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:28:23.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:28:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:28:23.556+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:28:23.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:28:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:28:23.556+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:28:24.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:28:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:28:24.295+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:28:24.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:28:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:28:24.295+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:28:24.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:28:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:28:24.295+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:28:24.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:28:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:28:24.525+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:28:24.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:28:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:28:24.525+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:28:24.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:28:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:28:24.525+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:28:24.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:28:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:28:24.525+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:28:25.556 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:28:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:28:25.262+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:28:25.556 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:28:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:28:25.262+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:28:25.557 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:28:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:28:25.262+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:28:25.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:28:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:28:25.554+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:28:25.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:28:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:28:25.554+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:28:25.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:28:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:28:25.554+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:28:25.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:28:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:28:25.554+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:28:26.574 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:28:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:28:26.273+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:28:26.574 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:28:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:28:26.273+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:28:26.574 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:28:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:28:26.273+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:17:55.268028+0000 (oldest deadline 2024-02-20T08:18:15.268028+0000) 2024-02-20T08:28:26.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:28:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:28:26.572+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:28:26.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:28:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:28:26.572+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:28:26.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:28:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:28:26.572+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:28:26.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:28:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:28:26.572+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:28:27.742 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:28:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:28:27.398+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 24774 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:28:27.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:28:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:28:27.531+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:28:27.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:28:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:28:27.531+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:28:27.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:28:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:28:27.531+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:28:27.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:28:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:28:27.531+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:28:28.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:28:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:28:28.528+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:28:28.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:28:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:28:28.528+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:28:28.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:28:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:28:28.528+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:28:28.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:28:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:28:28.528+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:28:29.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:28:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:28:29.484+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:28:29.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:28:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:28:29.484+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:28:29.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:28:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:28:29.484+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:28:29.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:28:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:28:29.484+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:28:30.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:28:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:28:30.513+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:28:30.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:28:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:28:30.513+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:28:30.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:28:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:28:30.513+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:28:30.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:28:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:28:30.513+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:28:31.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:28:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:28:31.473+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:28:31.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:28:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:28:31.473+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:28:31.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:28:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:28:31.473+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:28:31.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:28:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:28:31.473+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:28:32.742 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:28:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:28:32.402+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 24846 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:28:32.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:28:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:28:32.456+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:28:32.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:28:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:28:32.456+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:28:32.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:28:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:28:32.456+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:28:32.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:28:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:28:32.456+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:28:33.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:28:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:28:33.476+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:28:33.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:28:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:28:33.476+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:28:33.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:28:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:28:33.476+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:28:33.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:28:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:28:33.476+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:28:34.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:28:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:28:34.452+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:28:34.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:28:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:28:34.452+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:28:34.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:28:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:28:34.452+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:28:34.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:28:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:28:34.452+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:28:35.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:28:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:28:35.406+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:28:35.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:28:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:28:35.406+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:28:35.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:28:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:28:35.406+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:28:35.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:28:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:28:35.406+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:28:36.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:28:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:28:36.414+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:28:36.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:28:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:28:36.414+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:28:36.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:28:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:28:36.414+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:28:36.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:28:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:28:36.414+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:28:37.742 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:28:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:28:37.405+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 24916 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:28:37.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:28:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:28:37.431+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:28:37.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:28:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:28:37.431+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:28:37.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:28:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:28:37.431+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:28:37.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:28:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:28:37.431+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:28:38.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:28:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:28:38.435+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:28:38.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:28:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:28:38.435+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:28:38.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:28:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:28:38.435+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:28:38.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:28:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:28:38.435+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:28:39.742 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:28:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:28:39.641+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:28:39.742 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:28:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:28:39.641+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:28:39.742 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:28:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:28:39.641+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:28:39.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:28:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:28:39.482+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:28:39.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:28:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:28:39.482+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:28:39.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:28:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:28:39.482+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:28:39.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:28:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:28:39.482+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:28:40.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:28:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:28:40.633+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:28:40.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:28:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:28:40.633+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:28:40.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:28:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:28:40.633+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:28:40.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:28:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:28:40.531+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:28:40.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:28:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:28:40.531+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:28:40.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:28:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:28:40.531+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:28:40.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:28:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:28:40.531+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:28:41.759 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:28:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:28:41.599+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:28:41.759 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:28:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:28:41.599+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:28:41.759 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:28:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:28:41.599+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:28:41.759 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:28:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:28:41.494+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:28:41.760 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:28:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:28:41.494+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:28:41.760 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:28:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:28:41.494+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:18:07.257505+0000 (oldest deadline 2024-02-20T08:18:27.257505+0000) 2024-02-20T08:28:41.760 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:28:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:28:41.494+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:28:42.742 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:28:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:28:42.409+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 24988 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:28:42.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:28:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:28:42.478+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:28:42.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:28:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:28:42.632+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:28:42.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:28:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:28:42.632+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:28:42.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:28:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:28:42.632+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:28:43.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:28:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:28:43.454+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:28:43.742 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:28:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:28:43.625+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:28:43.742 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:28:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:28:43.625+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:28:43.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:28:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:28:43.625+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:28:44.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:28:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:28:44.421+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:28:44.742 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:28:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:28:44.610+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:28:44.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:28:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:28:44.610+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:28:44.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:28:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:28:44.610+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:28:45.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:28:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:28:45.441+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:28:45.742 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:28:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:28:45.619+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:28:45.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:28:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:28:45.619+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:28:45.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:28:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:28:45.619+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:28:46.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:28:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:28:46.436+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:28:46.742 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:28:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:28:46.588+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:28:46.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:28:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:28:46.588+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:28:46.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:28:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:28:46.588+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:28:47.742 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:28:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:28:47.414+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 25058 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:28:47.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:28:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:28:47.437+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:28:47.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:28:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:28:47.620+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:28:47.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:28:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:28:47.620+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:28:47.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:28:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:28:47.620+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:28:48.443 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:28:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:28:48.186+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:28:48.444 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:28:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:28:48.186+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:28:48.444 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:28:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:28:48.186+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:28:48.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:28:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:28:48.442+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:28:48.742 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:28:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:28:48.587+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:28:48.742 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:28:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:28:48.587+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:28:48.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:28:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:28:48.587+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:28:49.437 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:28:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:28:49.174+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:28:49.438 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:28:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:28:49.174+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:28:49.438 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:28:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:28:49.174+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:28:49.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:28:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:28:49.436+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:28:49.742 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:28:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:28:49.634+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:28:49.742 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:28:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:28:49.634+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:28:49.742 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:28:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:28:49.634+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:28:50.450 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:28:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:28:50.160+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:28:50.450 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:28:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:28:50.160+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:28:50.450 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:28:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:28:50.160+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:28:50.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:28:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:28:50.448+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:28:50.742 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:28:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:28:50.639+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:28:50.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:28:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:28:50.639+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:28:50.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:28:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:28:50.639+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:28:51.452 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:28:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:28:51.111+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:28:51.453 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:28:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:28:51.111+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:28:51.453 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:28:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:28:51.111+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:28:51.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:28:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:28:51.451+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:28:51.742 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:28:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:28:51.637+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:28:51.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:28:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:28:51.637+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:28:51.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:28:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:28:51.637+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:28:52.419 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:28:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:28:52.080+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:28:52.420 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:28:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:28:52.080+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:28:52.420 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:28:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:28:52.080+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:28:52.742 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:28:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:28:52.418+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 25130 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:28:52.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:28:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:28:52.484+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:28:52.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:28:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:28:52.653+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:28:52.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:28:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:28:52.653+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:28:52.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:28:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:28:52.653+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:28:53.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:28:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:28:53.055+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:28:53.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:28:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:28:53.055+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:28:53.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:28:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:28:53.055+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:28:53.880 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:28:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:28:53.510+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:28:53.881 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:28:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:28:53.671+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:28:53.881 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:28:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:28:53.671+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:28:53.881 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:28:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:28:53.671+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:28:54.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:28:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:28:54.021+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:28:54.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:28:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:28:54.021+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:28:54.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:28:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:28:54.021+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:28:54.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:28:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:28:54.478+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:28:54.742 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:28:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:28:54.621+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:28:54.742 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:28:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:28:54.621+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:28:54.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:28:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:28:54.621+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:28:55.490 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:28:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:28:55.063+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:28:55.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:28:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:28:55.063+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:28:55.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:28:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:28:55.063+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:28:55.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:28:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:28:55.489+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:28:55.742 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:28:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:28:55.585+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:28:55.742 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:28:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:28:55.585+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:28:55.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:28:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:28:55.585+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:28:56.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:28:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:28:56.048+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:28:56.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:28:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:28:56.048+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:28:56.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:28:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:28:56.048+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:28:56.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:28:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:28:56.532+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:28:56.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:28:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:28:56.559+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:28:56.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:28:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:28:56.559+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:28:56.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:28:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:28:56.560+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:28:57.423 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:28:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:28:57.049+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:28:57.423 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:28:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:28:57.049+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:28:57.424 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:28:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:28:57.049+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:28:57.742 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:28:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:28:57.422+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 25200 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:28:57.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:28:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:28:57.513+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:28:57.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:28:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:28:57.553+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:28:57.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:28:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:28:57.553+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:28:57.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:28:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:28:57.553+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:28:58.479 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:28:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:28:58.011+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:28:58.479 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:28:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:28:58.011+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:28:58.479 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:28:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:28:58.011+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:28:58.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:28:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:28:58.477+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:28:58.742 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:28:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:28:58.554+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:28:58.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:28:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:28:58.554+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:28:58.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:28:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:28:58.554+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:28:59.454 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:28:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:28:59.008+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:28:59.454 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:28:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:28:59.008+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:28:59.454 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:28:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:28:59.008+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:28:59.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:28:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:28:59.453+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:28:59.742 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:28:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:28:59.526+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:28:59.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:28:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:28:59.526+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:28:59.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:28:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:28:59.526+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:29:00.229 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:29:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:29:00.036+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:29:00.229 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:29:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:29:00.036+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:29:00.229 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:29:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:29:00.036+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:29:00.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:29:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:29:00.440+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:29:00.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:29:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:29:00.569+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:29:00.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:29:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:29:00.569+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:29:00.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:29:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:29:00.569+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:29:01.423 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:29:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:29:01.085+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:29:01.423 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:29:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:29:01.085+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:29:01.423 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:29:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:29:01.085+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:29:01.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:29:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:29:01.422+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:29:01.742 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:29:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:29:01.602+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:29:01.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:29:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:29:01.602+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:29:01.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:29:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:29:01.602+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:29:02.427 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:29:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:29:02.128+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:29:02.427 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:29:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:29:02.128+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:29:02.427 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:29:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:29:02.128+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:29:02.743 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:29:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:29:02.425+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 25272 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:29:02.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:29:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:29:02.429+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:29:02.744 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:29:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:29:02.610+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:29:02.744 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:29:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:29:02.610+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:29:02.744 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:29:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:29:02.610+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:29:03.455 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:29:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:29:03.161+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:29:03.455 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:29:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:29:03.161+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:29:03.456 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:29:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:29:03.161+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:29:03.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:29:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:29:03.453+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:29:03.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:29:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:29:03.636+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:29:03.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:29:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:29:03.636+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:29:03.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:29:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:29:03.636+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:29:04.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:29:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:29:04.180+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:29:04.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:29:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:29:04.180+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:29:04.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:29:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:29:04.180+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:29:04.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:29:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:29:04.499+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:29:04.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:29:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:29:04.600+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:29:04.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:29:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:29:04.600+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:29:04.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:29:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:29:04.600+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:29:05.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:29:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:29:05.142+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:29:05.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:29:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:29:05.142+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:29:05.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:29:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:29:05.142+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:29:05.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:29:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:29:05.601+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:29:05.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:29:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:29:05.601+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:29:05.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:29:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:29:05.601+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:29:05.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:29:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:29:05.548+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:29:05.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:29:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:29:05.548+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:29:05.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:29:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:29:05.548+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:29:05.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:29:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:29:05.548+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:29:06.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:29:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:29:06.144+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:29:06.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:29:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:29:06.144+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:29:06.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:29:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:29:06.144+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:29:06.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:29:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:29:06.607+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:29:06.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:29:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:29:06.607+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:29:06.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:29:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:29:06.607+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:29:06.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:29:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:29:06.573+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:29:06.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:29:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:29:06.573+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:29:06.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:29:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:29:06.573+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:29:06.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:29:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:29:06.574+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:29:07.431 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:29:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:29:07.137+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:29:07.431 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:29:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:29:07.137+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:29:07.431 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:29:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:29:07.137+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:29:07.742 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:29:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:29:07.428+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 25342 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:29:07.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:29:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:29:07.583+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:29:07.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:29:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:29:07.583+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:29:07.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:29:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:29:07.583+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:29:07.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:29:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:29:07.560+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:29:07.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:29:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:29:07.560+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:29:07.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:29:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:29:07.560+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:29:07.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:29:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:29:07.560+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:29:08.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:29:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:29:08.109+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:29:08.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:29:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:29:08.109+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:29:08.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:29:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:29:08.109+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:29:08.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:29:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:29:08.622+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:29:08.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:29:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:29:08.622+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:29:08.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:29:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:29:08.622+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:29:08.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:29:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:29:08.548+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:29:08.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:29:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:29:08.548+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:29:08.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:29:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:29:08.548+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:29:08.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:29:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:29:08.548+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:29:09.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:29:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:29:09.084+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:29:09.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:29:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:29:09.084+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:29:09.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:29:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:29:09.084+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:29:09.886 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:29:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:29:09.585+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:29:09.886 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:29:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:29:09.585+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:29:09.887 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:29:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:29:09.585+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:29:09.887 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:29:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:29:09.560+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:29:09.887 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:29:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:29:09.560+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:29:09.887 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:29:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:29:09.560+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:29:09.887 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:29:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:29:09.560+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:29:10.229 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:29:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:29:10.088+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:29:10.229 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:29:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:29:10.088+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:29:10.229 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:29:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:29:10.088+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:29:10.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:29:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:29:10.564+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:29:10.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:29:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:29:10.564+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:29:10.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:29:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:29:10.564+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:29:10.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:29:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:29:10.537+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:29:10.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:29:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:29:10.537+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:29:10.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:29:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:29:10.537+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:29:10.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:29:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:29:10.537+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:29:11.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:29:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:29:11.062+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:29:11.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:29:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:29:11.062+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:29:11.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:29:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:29:11.062+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:29:11.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:29:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:29:11.525+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:29:11.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:29:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:29:11.525+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:29:11.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:29:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:29:11.525+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:29:11.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:29:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:29:11.514+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:29:11.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:29:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:29:11.514+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:29:11.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:29:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:29:11.514+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:29:11.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:29:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:29:11.514+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:29:12.434 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:29:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:29:12.074+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:29:12.434 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:29:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:29:12.074+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:29:12.435 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:29:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:29:12.074+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:29:12.742 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:29:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:29:12.432+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 25413 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:29:12.742 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:29:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:29:12.495+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:29:12.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:29:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:29:12.495+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:29:12.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:29:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:29:12.495+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:29:12.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:29:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:29:12.513+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:29:12.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:29:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:29:12.513+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:29:12.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:29:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:29:12.513+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:29:12.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:29:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:29:12.513+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:29:13.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:29:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:29:13.076+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:29:13.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:29:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:29:13.076+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:29:13.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:29:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:29:13.076+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:29:13.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:29:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:29:13.526+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:29:13.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:29:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:29:13.526+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:29:13.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:29:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:29:13.526+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:29:13.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:29:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:29:13.543+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:29:13.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:29:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:29:13.543+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:29:13.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:29:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:29:13.543+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:29:13.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:29:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:29:13.544+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:29:14.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:29:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:29:14.126+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:29:14.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:29:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:29:14.126+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:29:14.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:29:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:29:14.126+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:29:14.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:29:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:29:14.540+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:29:14.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:29:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:29:14.540+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:29:14.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:29:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:29:14.540+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:29:14.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:29:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:29:14.550+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:29:14.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:29:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:29:14.550+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:29:14.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:29:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:29:14.550+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:29:14.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:29:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:29:14.550+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:29:15.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:29:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:29:15.118+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:29:15.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:29:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:29:15.118+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:29:15.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:29:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:29:15.118+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:29:15.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:29:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:29:15.555+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:29:15.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:29:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:29:15.555+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:29:15.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:29:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:29:15.555+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:29:15.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:29:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:29:15.569+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:29:15.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:29:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:29:15.569+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:29:15.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:29:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:29:15.569+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:29:15.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:29:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:29:15.569+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:29:16.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:29:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:29:16.137+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:29:16.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:29:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:29:16.137+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:29:16.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:29:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:29:16.137+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:29:16.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:29:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:29:16.583+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:29:16.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:29:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:29:16.583+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:29:16.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:29:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:29:16.583+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:29:16.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:29:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:29:16.610+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:29:16.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:29:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:29:16.610+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:29:16.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:29:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:29:16.610+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:29:16.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:29:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:29:16.610+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:29:17.438 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:29:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:29:17.142+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:29:17.438 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:29:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:29:17.142+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:29:17.439 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:29:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:29:17.142+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:29:17.742 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:29:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:29:17.436+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 25484 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:29:17.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:29:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:29:17.549+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:29:17.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:29:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:29:17.549+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:29:17.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:29:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:29:17.549+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:29:17.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:29:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:29:17.615+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:29:17.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:29:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:29:17.615+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:29:17.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:29:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:29:17.615+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:29:17.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:29:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:29:17.615+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:29:18.496 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:29:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:29:18.137+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:29:18.496 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:29:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:29:18.137+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:29:18.496 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:29:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:29:18.137+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:29:18.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:29:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:29:18.550+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:29:18.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:29:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:29:18.550+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:29:18.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:29:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:29:18.550+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:29:18.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:29:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:29:18.585+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:29:18.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:29:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:29:18.585+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:29:18.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:29:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:29:18.585+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:29:18.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:29:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:29:18.585+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:29:19.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:29:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:29:19.121+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:29:19.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:29:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:29:19.121+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:29:19.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:29:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:29:19.121+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:29:19.885 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:29:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:29:19.562+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:29:19.885 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:29:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:29:19.562+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:29:19.886 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:29:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:29:19.562+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:29:19.886 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:29:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:29:19.571+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:29:19.886 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:29:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:29:19.571+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:29:19.886 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:29:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:29:19.571+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:29:19.886 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:29:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:29:19.571+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:29:20.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:29:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:29:20.133+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:29:20.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:29:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:29:20.134+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:29:20.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:29:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:29:20.134+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:29:20.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:29:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:29:20.588+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:29:20.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:29:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:29:20.588+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:29:20.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:29:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:29:20.588+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:29:20.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:29:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:29:20.540+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:29:20.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:29:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:29:20.540+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:29:20.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:29:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:29:20.540+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:29:20.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:29:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:29:20.540+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:29:21.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:29:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:29:21.087+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:29:21.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:29:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:29:21.087+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:29:21.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:29:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:29:21.087+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:29:21.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:29:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:29:21.638+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:29:21.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:29:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:29:21.638+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:29:21.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:29:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:29:21.638+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:29:21.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:29:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:29:21.518+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:29:21.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:29:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:29:21.518+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:29:21.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:29:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:29:21.518+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:29:21.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:29:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:29:21.518+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:29:22.442 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:29:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:29:22.128+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:29:22.442 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:29:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:29:22.128+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:29:22.443 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:29:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:29:22.128+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:29:22.742 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:29:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:29:22.440+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 25555 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:29:22.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:29:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:29:22.668+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:29:22.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:29:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:29:22.668+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:29:22.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:29:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:29:22.668+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:29:22.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:29:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:29:22.487+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:29:22.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:29:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:29:22.487+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:29:22.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:29:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:29:22.487+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:29:22.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:29:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:29:22.487+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:29:23.473 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:29:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:29:23.103+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:29:23.473 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:29:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:29:23.103+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:29:23.474 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:29:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:29:23.103+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:29:23.742 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:29:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:29:23.690+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:29:23.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:29:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:29:23.690+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:29:23.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:29:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:29:23.690+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:29:23.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:29:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:29:23.471+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:29:23.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:29:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:29:23.471+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:29:23.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:29:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:29:23.471+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:29:23.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:29:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:29:23.471+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:29:24.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:29:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:29:24.072+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:29:24.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:29:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:29:24.072+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:29:24.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:29:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:29:24.072+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:29:24.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:29:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:29:24.718+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:29:24.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:29:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:29:24.718+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:29:24.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:29:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:29:24.718+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:29:24.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:29:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:29:24.493+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:29:24.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:29:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:29:24.493+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:29:24.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:29:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:29:24.493+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:29:24.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:29:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:29:24.494+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:29:25.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:29:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:29:25.086+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:29:25.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:29:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:29:25.086+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:29:25.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:29:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:29:25.086+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:29:25.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:29:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:29:25.724+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:29:25.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:29:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:29:25.724+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:29:25.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:29:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:29:25.724+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:29:25.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:29:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:29:25.508+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:29:25.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:29:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:29:25.508+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:29:25.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:29:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:29:25.508+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:29:25.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:29:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:29:25.508+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:29:26.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:29:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:29:26.120+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:29:26.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:29:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:29:26.120+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:29:26.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:29:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:29:26.120+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:29:26.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:29:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:29:26.722+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:29:26.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:29:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:29:26.722+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:29:26.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:29:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:29:26.722+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:29:26.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:29:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:29:26.497+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:29:26.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:29:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:29:26.497+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:29:26.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:29:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:29:26.497+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:29:26.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:29:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:29:26.497+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:29:27.446 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:29:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:29:27.080+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:29:27.446 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:29:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:29:27.080+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:29:27.446 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:29:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:29:27.080+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:29:27.716 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:29:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:29:27.443+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 25626 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:29:27.717 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:29:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:29:27.471+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:29:27.717 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:29:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:29:27.472+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:29:27.717 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:29:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:29:27.472+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:29:27.717 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:29:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:29:27.472+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:29:27.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:29:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:29:27.715+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:29:27.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:29:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:29:27.715+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:29:27.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:29:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:29:27.715+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:29:28.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:29:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:29:28.043+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:29:28.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:29:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:29:28.043+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:29:28.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:29:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:29:28.043+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:29:28.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:29:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:29:28.745+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:29:28.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:29:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:29:28.745+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:29:28.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:29:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:29:28.745+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:29:28.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:29:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:29:28.504+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:29:28.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:29:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:29:28.504+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:29:28.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:29:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:29:28.504+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:29:28.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:29:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:29:28.504+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:29:29.471 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:29:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:29:29.039+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:29:29.472 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:29:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:29:29.039+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:29:29.472 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:29:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:29:29.039+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:29:29.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:29:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:29:29.470+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:29:29.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:29:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:29:29.470+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:29:29.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:29:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:29:29.470+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:29:29.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:29:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:29:29.470+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:29:30.081 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:29:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:29:29.761+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:29:30.082 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:29:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:29:29.761+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:29:30.082 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:29:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:29:29.761+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:29:30.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:29:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:29:30.080+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:29:30.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:29:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:29:30.080+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:29:30.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:29:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:29:30.080+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:29:30.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:29:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:29:30.738+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:29:30.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:29:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:29:30.738+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:29:30.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:29:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:29:30.738+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:29:30.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:29:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:29:30.510+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:29:30.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:29:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:29:30.510+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:29:30.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:29:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:29:30.510+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:29:30.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:29:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:29:30.510+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:29:31.479 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:29:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:29:31.073+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:29:31.479 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:29:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:29:31.073+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:29:31.479 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:29:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:29:31.073+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:29:31.742 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:29:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:29:31.715+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:29:31.742 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:29:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:29:31.715+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:29:31.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:29:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:29:31.715+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:29:31.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:29:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:29:31.477+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:29:31.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:29:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:29:31.477+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:29:31.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:29:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:29:31.477+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:29:31.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:29:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:29:31.477+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:29:32.438 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:29:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:29:32.111+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:29:32.438 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:29:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:29:32.111+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:29:32.438 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:29:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:29:32.111+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:29:32.742 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:29:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:29:32.447+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 25697 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:29:32.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:29:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:29:32.435+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:29:32.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:29:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:29:32.435+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:29:32.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:29:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:29:32.436+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:29:32.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:29:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:29:32.436+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:29:33.112 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:29:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:29:32.742+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:29:33.112 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:29:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:29:32.742+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:29:33.113 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:29:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:29:32.742+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:29:33.390 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:29:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:29:33.111+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:29:33.391 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:29:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:29:33.111+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:29:33.391 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:29:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:29:33.111+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:29:33.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:29:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:29:33.389+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:29:33.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:29:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:29:33.389+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:29:33.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:29:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:29:33.389+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:29:33.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:29:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:29:33.389+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:29:34.114 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:29:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:29:33.745+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:29:34.115 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:29:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:29:33.745+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:29:34.115 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:29:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:29:33.745+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:29:34.404 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:29:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:29:34.112+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:29:34.404 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:29:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:29:34.112+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:29:34.404 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:29:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:29:34.112+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:29:34.738 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:29:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:29:34.402+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:29:34.739 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:29:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:29:34.402+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:29:34.739 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:29:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:29:34.402+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:29:34.739 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:29:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:29:34.402+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:29:34.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:29:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:29:34.736+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:29:34.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:29:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:29:34.736+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:29:34.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:29:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:29:34.736+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:29:35.447 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:29:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:29:35.092+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:29:35.447 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:29:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:29:35.092+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:29:35.447 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:29:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:29:35.092+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:29:35.742 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:29:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:29:35.689+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:29:35.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:29:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:29:35.689+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:29:35.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:29:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:29:35.689+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:29:35.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:29:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:29:35.445+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:29:35.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:29:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:29:35.445+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:29:35.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:29:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:29:35.445+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:29:35.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:29:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:29:35.445+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:29:36.445 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:29:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:29:36.045+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:29:36.445 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:29:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:29:36.045+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:29:36.445 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:29:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:29:36.045+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:29:36.695 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:29:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:29:36.692+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:29:36.695 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:29:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:29:36.692+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:29:36.695 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:29:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:29:36.692+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:29:36.696 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:29:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:29:36.443+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:29:36.696 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:29:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:29:36.443+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:29:36.696 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:29:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:29:36.443+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:29:36.696 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:29:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:29:36.443+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:29:37.436 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:29:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:29:37.088+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:29:37.437 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:29:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:29:37.088+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:29:37.437 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:29:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:29:37.088+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:29:37.698 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:29:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:29:37.451+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 25768 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:29:37.699 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:29:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:29:37.435+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:29:37.699 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:29:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:29:37.435+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:29:37.699 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:29:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:29:37.435+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:29:37.699 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:29:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:29:37.435+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:29:37.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:29:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:29:37.697+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:29:37.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:29:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:29:37.697+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:29:37.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:29:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:29:37.697+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:29:38.483 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:29:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:29:38.065+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:29:38.483 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:29:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:29:38.065+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:29:38.483 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:29:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:29:38.065+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:29:38.742 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:29:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:29:38.650+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:29:38.742 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:29:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:29:38.650+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:29:38.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:29:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:29:38.650+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:29:38.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:29:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:29:38.481+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:29:38.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:29:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:29:38.481+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:29:38.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:29:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:29:38.481+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:29:38.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:29:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:29:38.481+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:29:39.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:29:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:29:39.097+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:29:39.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:29:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:29:39.097+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:29:39.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:29:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:29:39.097+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:29:39.886 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:29:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:29:39.503+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:29:39.886 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:29:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:29:39.503+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:29:39.886 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:29:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:29:39.503+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:29:39.886 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:29:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:29:39.503+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:29:39.886 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:29:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:29:39.629+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:29:39.886 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:29:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:29:39.629+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:29:39.887 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:29:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:29:39.629+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:29:40.231 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:29:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:29:40.070+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:29:40.231 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:29:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:29:40.070+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:29:40.231 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:29:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:29:40.070+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:29:40.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:29:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:29:40.461+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:29:40.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:29:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:29:40.461+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:29:40.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:29:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:29:40.461+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:29:40.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:29:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:29:40.461+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:29:40.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:29:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:29:40.623+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:29:40.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:29:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:29:40.623+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:29:40.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:29:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:29:40.623+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:29:41.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:29:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:29:41.037+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:29:41.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:29:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:29:41.037+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:29:41.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:29:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:29:41.037+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:29:41.764 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:29:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:29:41.506+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:29:41.764 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:29:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:29:41.506+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:29:41.764 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:29:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:29:41.506+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:29:41.764 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:29:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:29:41.506+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:29:41.765 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:29:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:29:41.629+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:29:41.765 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:29:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:29:41.629+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:29:41.765 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:29:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:29:41.629+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:29:42.457 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:29:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:29:42.086+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:29:42.457 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:29:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:29:42.086+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:29:42.458 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:29:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:29:42.086+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:29:42.742 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:29:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:29:42.456+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 25839 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:29:42.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:29:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:29:42.649+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:29:42.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:29:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:29:42.649+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:29:42.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:29:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:29:42.649+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:29:42.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:29:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:29:42.509+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:29:42.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:29:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:29:42.509+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:29:42.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:29:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:29:42.509+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:29:42.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:29:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:29:42.509+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:29:43.484 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:29:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:29:43.096+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:29:43.484 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:29:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:29:43.096+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:29:43.485 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:29:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:29:43.096+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:29:43.742 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:29:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:29:43.609+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:29:43.742 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:29:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:29:43.609+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:29:43.742 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:29:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:29:43.609+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:29:43.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:29:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:29:43.482+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:29:43.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:29:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:29:43.482+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:29:43.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:29:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:29:43.482+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:29:43.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:29:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:29:43.483+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:29:44.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:29:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:29:44.091+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:29:44.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:29:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:29:44.091+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:29:44.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:29:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:29:44.091+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:29:44.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:29:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:29:44.503+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:29:44.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:29:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:29:44.503+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:29:44.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:29:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:29:44.503+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:29:44.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:29:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:29:44.503+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:29:44.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:29:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:29:44.606+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:29:44.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:29:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:29:44.606+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:29:44.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:29:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:29:44.606+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:29:45.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:29:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:29:45.058+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:29:45.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:29:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:29:45.058+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:29:45.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:29:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:29:45.058+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:29:45.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:29:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:29:45.521+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:29:45.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:29:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:29:45.521+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:29:45.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:29:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:29:45.521+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:29:45.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:29:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:29:45.521+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:29:45.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:29:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:29:45.637+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:29:45.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:29:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:29:45.637+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:29:45.995 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:29:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:29:45.637+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:29:46.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:29:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:29:46.045+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:29:46.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:29:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:29:46.045+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:29:46.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:29:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:29:46.045+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:29:46.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:29:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:29:46.504+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:29:46.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:29:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:29:46.504+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:29:46.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:29:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:29:46.504+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:29:46.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:29:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:29:46.504+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:29:46.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:29:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:29:46.602+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:29:46.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:29:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:29:46.602+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:29:46.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:29:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:29:46.602+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:29:47.461 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:29:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:29:47.085+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:29:47.461 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:29:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:29:47.085+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:29:47.462 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:29:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:29:47.085+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:29:47.742 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:29:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:29:47.460+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 25910 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:29:47.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:29:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:29:47.519+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:29:47.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:29:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:29:47.519+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:29:47.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:29:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:29:47.519+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:29:47.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:29:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:29:47.520+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:29:47.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:29:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:29:47.601+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:29:47.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:29:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:29:47.601+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:29:47.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:29:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:29:47.601+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:29:48.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:29:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:29:48.095+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:29:48.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:29:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:29:48.095+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:29:48.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:29:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:29:48.095+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:29:48.995 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:29:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:29:48.503+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:29:48.995 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:29:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:29:48.503+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:29:48.995 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:29:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:29:48.503+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:29:48.995 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:29:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:29:48.503+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:29:48.996 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:29:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:29:48.568+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:29:48.996 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:29:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:29:48.568+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:29:48.996 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:29:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:29:48.568+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:29:49.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:29:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:29:49.106+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:29:49.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:29:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:29:49.106+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:29:49.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:29:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:29:49.106+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:29:49.885 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:29:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:29:49.596+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:29:49.885 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:29:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:29:49.596+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:29:49.886 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:29:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:29:49.596+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:29:49.886 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:29:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:29:49.505+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:29:49.886 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:29:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:29:49.505+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:29:49.886 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:29:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:29:49.505+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:29:49.886 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:29:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:29:49.505+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:29:50.231 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:29:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:29:50.061+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:29:50.231 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:29:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:29:50.061+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:29:50.232 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:29:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:29:50.061+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:29:50.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:29:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:29:50.477+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:29:50.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:29:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:29:50.477+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:29:50.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:29:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:29:50.477+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:29:50.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:29:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:29:50.477+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:29:50.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:29:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:29:50.568+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:29:50.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:29:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:29:50.568+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:29:50.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:29:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:29:50.568+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:29:51.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:29:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:29:51.102+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:29:51.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:29:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:29:51.102+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:29:51.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:29:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:29:51.102+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:29:51.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:29:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:29:51.549+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:29:51.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:29:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:29:51.549+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:29:51.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:29:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:29:51.549+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:29:51.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:29:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:29:51.523+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:29:51.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:29:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:29:51.523+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:29:51.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:29:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:29:51.523+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:29:51.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:29:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:29:51.523+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:29:52.465 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:29:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:29:52.122+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:29:52.465 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:29:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:29:52.122+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:29:52.466 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:29:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:29:52.122+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:29:52.742 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:29:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:29:52.463+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 25981 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:29:52.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:29:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:29:52.500+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:29:52.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:29:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:29:52.500+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:29:52.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:29:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:29:52.500+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:29:52.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:29:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:29:52.533+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:29:52.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:29:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:29:52.533+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:29:52.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:29:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:29:52.533+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:29:52.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:29:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:29:52.533+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:29:53.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:29:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:29:53.094+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:29:53.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:29:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:29:53.094+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:29:53.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:29:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:29:53.094+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:29:53.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:29:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:29:53.537+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:29:53.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:29:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:29:53.537+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:29:53.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:29:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:29:53.537+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:29:53.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:29:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:29:53.545+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:29:53.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:29:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:29:53.545+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:29:53.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:29:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:29:53.545+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:29:53.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:29:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:29:53.545+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:29:54.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:29:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:29:54.062+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:29:54.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:29:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:29:54.062+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:29:54.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:29:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:29:54.062+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:29:54.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:29:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:29:54.495+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:29:54.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:29:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:29:54.495+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:29:54.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:29:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:29:54.495+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:29:54.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:29:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:29:54.540+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:29:54.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:29:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:29:54.540+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:29:54.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:29:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:29:54.540+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:29:54.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:29:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:29:54.540+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:29:55.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:29:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:29:55.063+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:29:55.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:29:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:29:55.063+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:29:55.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:29:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:29:55.063+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:29:55.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:29:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:29:55.507+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:29:55.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:29:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:29:55.507+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:29:55.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:29:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:29:55.507+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:29:55.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:29:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:29:55.518+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:29:55.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:29:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:29:55.518+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:29:55.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:29:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:29:55.518+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:29:55.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:29:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:29:55.518+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:29:56.462 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:29:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:29:56.054+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:29:56.463 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:29:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:29:56.054+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:29:56.463 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:29:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:29:56.054+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:29:56.742 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:29:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:29:56.461+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:29:56.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:29:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:29:56.461+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:29:56.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:29:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:29:56.461+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:29:56.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:29:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:29:56.491+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:29:56.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:29:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:29:56.491+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:29:56.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:29:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:29:56.491+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:29:56.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:29:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:29:56.491+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:29:57.451 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:29:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:29:57.068+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:29:57.451 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:29:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:29:57.068+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:29:57.451 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:29:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:29:57.068+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:29:57.742 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:29:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:29:57.467+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 26052 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:29:57.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:29:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:29:57.449+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:29:57.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:29:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:29:57.449+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:29:57.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:29:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:29:57.449+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:29:57.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:29:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:29:57.465+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:29:57.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:29:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:29:57.465+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:29:57.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:29:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:29:57.465+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:29:57.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:29:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:29:57.466+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:29:58.449 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:29:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:29:58.105+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:29:58.450 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:29:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:29:58.105+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:29:58.450 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:29:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:29:58.105+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:29:58.742 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:29:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:29:58.447+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:29:58.742 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:29:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:29:58.447+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:29:58.742 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:29:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:29:58.447+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:29:58.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:29:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:29:58.450+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:29:58.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:29:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:29:58.450+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:29:58.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:29:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:29:58.450+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:29:58.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:29:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:29:58.450+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:29:59.480 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:29:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:29:59.114+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:29:59.480 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:29:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:29:59.114+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:29:59.480 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:29:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:29:59.114+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:29:59.742 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:29:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:29:59.492+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:29:59.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:29:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:29:59.492+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:29:59.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:29:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:29:59.492+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:29:59.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:29:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:29:59.478+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:29:59.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:29:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:29:59.478+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:29:59.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:29:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:29:59.478+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:29:59.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:29:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:29:59.478+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:30:00.441 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:30:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:30:00.159+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:30:00.442 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:30:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:30:00.159+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:30:00.442 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:30:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:30:00.159+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:30:00.742 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:30:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:30:00.541+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:30:00.742 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:30:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:30:00.541+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:30:00.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:30:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:30:00.541+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:30:00.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:30:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:30:00.438+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:30:00.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:30:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:30:00.438+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:30:00.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:30:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:30:00.438+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:30:00.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:30:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:30:00.439+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:30:01.467 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:30:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:30:01.136+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:30:01.468 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:30:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:30:01.136+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:30:01.468 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:30:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:30:01.136+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:30:01.742 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:30:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:30:01.525+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:30:01.742 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:30:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:30:01.525+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:30:01.742 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:30:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:30:01.525+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:30:01.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:30:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:30:01.477+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:30:01.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:30:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:30:01.477+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:30:01.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:30:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:30:01.477+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:30:01.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:30:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:30:01.477+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:30:02.473 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:30:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:30:02.164+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:30:02.473 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:30:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:30:02.164+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:30:02.473 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:30:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:30:02.164+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:30:02.742 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:30:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:30:02.470+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 26123 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:30:02.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:30:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:30:02.508+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:30:02.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:30:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:30:02.508+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:30:02.744 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:30:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:30:02.508+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:30:02.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:30:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:30:02.513+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:30:02.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:30:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:30:02.513+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:30:02.745 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:30:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:30:02.513+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:30:02.745 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:30:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:30:02.514+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:30:03.479 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:30:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:30:03.148+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:30:03.479 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:30:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:30:03.148+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:30:03.480 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:30:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:30:03.148+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:30:03.742 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:30:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:30:03.481+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:30:03.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:30:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:30:03.481+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:30:03.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:30:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:30:03.481+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:30:03.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:30:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:30:03.477+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:30:03.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:30:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:30:03.477+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:30:03.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:30:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:30:03.477+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:30:03.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:30:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:30:03.477+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:30:04.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:30:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:30:04.102+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:30:04.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:30:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:30:04.102+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:30:04.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:30:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:30:04.102+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:30:04.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:30:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:30:04.498+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:30:04.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:30:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:30:04.498+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:30:04.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:30:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:30:04.498+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:30:04.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:30:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:30:04.505+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:30:04.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:30:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:30:04.505+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:30:04.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:30:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:30:04.505+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:30:04.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:30:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:30:04.505+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:30:05.476 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:30:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:30:05.144+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:30:05.477 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:30:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:30:05.144+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:30:05.477 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:30:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:30:05.144+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:30:05.742 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:30:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:30:05.499+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:30:05.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:30:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:30:05.499+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:30:05.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:30:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:30:05.499+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:30:05.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:30:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:30:05.474+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:30:05.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:30:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:30:05.474+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:30:05.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:30:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:30:05.474+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:30:05.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:30:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:30:05.474+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:30:06.490 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:30:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:30:06.113+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:30:06.490 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:30:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:30:06.113+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:30:06.490 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:30:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:30:06.113+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:30:06.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:30:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:30:06.503+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:30:06.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:30:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:30:06.503+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:30:06.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:30:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:30:06.503+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:30:06.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:30:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:30:06.504+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:30:06.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:30:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:30:06.487+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:30:06.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:30:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:30:06.487+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:30:06.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:30:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:30:06.487+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:30:07.473 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:30:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:30:07.137+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:30:07.473 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:30:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:30:07.137+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:30:07.473 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:30:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:30:07.137+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:30:07.742 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:30:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:30:07.474+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 26194 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:30:07.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:30:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:30:07.470+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:30:07.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:30:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:30:07.470+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:30:07.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:30:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:30:07.470+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:30:07.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:30:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:30:07.529+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:30:07.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:30:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:30:07.529+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:30:07.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:30:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:30:07.529+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:30:07.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:30:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:30:07.529+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:30:08.492 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:30:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:30:08.422+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:30:08.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:30:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:30:08.422+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:30:08.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:30:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:30:08.422+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:30:08.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:30:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:30:08.176+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:30:08.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:30:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:30:08.176+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:30:08.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:30:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:30:08.176+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:30:08.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:30:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:30:08.545+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:30:08.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:30:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:30:08.545+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:30:08.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:30:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:30:08.545+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:30:08.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:30:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:30:08.545+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:30:09.492 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:30:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:30:09.430+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:30:09.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:30:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:30:09.430+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:30:09.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:30:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:30:09.430+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:30:09.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:30:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:30:09.214+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:30:09.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:30:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:30:09.214+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:30:09.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:30:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:30:09.214+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:30:09.886 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:30:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:30:09.565+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:30:09.886 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:30:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:30:09.565+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:30:09.886 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:30:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:30:09.565+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:30:09.886 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:30:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:30:09.565+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:30:10.421 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:30:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:30:10.209+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:30:10.422 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:30:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:30:10.209+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:30:10.422 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:30:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:30:10.209+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:30:10.422 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:30:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:30:10.418+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:30:10.422 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:30:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:30:10.418+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:30:10.422 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:30:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:30:10.418+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:30:10.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:30:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:30:10.604+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:30:10.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:30:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:30:10.604+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:30:10.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:30:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:30:10.604+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:30:10.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:30:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:30:10.604+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:30:11.492 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:30:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:30:11.414+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:30:11.492 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:30:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:30:11.414+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:30:11.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:30:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:30:11.414+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:30:11.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:30:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:30:11.202+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:30:11.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:30:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:30:11.202+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:30:11.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:30:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:30:11.202+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:30:11.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:30:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:30:11.563+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:30:11.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:30:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:30:11.563+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:30:11.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:30:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:30:11.563+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:30:11.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:30:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:30:11.563+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:30:12.481 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:30:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:30:12.167+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:30:12.481 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:30:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:30:12.167+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:30:12.481 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:30:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:30:12.167+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:30:12.481 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:30:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:30:12.387+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:30:12.482 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:30:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:30:12.387+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:30:12.482 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:30:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:30:12.387+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:30:12.742 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:30:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:30:12.478+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 26265 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:30:12.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:30:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:30:12.553+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:30:12.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:30:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:30:12.553+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:30:12.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:30:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:30:12.553+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:30:12.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:30:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:30:12.553+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:30:13.403 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:30:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:30:13.120+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:30:13.403 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:30:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:30:13.120+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:30:13.403 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:30:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:30:13.120+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:30:13.742 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:30:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:30:13.401+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:30:13.742 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:30:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:30:13.401+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:30:13.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:30:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:30:13.401+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:30:13.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:30:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:30:13.507+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:30:13.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:30:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:30:13.507+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:30:13.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:30:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:30:13.507+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:30:13.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:30:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:30:13.507+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:30:14.469 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:30:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:30:14.398+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:30:14.469 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:30:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:30:14.398+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:30:14.469 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:30:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:30:14.398+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:30:14.469 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:30:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:30:14.148+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:30:14.469 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:30:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:30:14.148+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:30:14.470 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:30:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:30:14.148+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:30:14.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:30:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:30:14.466+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:30:14.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:30:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:30:14.466+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:30:14.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:30:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:30:14.466+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:30:14.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:30:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:30:14.466+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:30:15.461 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:30:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:30:15.386+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:30:15.462 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:30:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:30:15.386+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:30:15.462 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:30:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:30:15.386+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:30:15.462 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:30:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:30:15.160+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:30:15.462 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:30:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:30:15.160+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:30:15.462 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:30:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:30:15.160+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:30:15.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:30:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:30:15.459+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:30:15.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:30:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:30:15.459+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:30:15.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:30:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:30:15.459+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:30:15.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:30:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:30:15.459+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:30:16.486 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:30:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:30:16.340+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:30:16.486 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:30:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:30:16.340+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:30:16.486 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:30:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:30:16.340+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:30:16.487 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:30:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:30:16.135+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:30:16.487 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:30:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:30:16.135+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:30:16.487 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:30:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:30:16.135+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:30:16.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:30:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:30:16.484+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:30:16.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:30:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:30:16.484+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:30:16.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:30:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:30:16.484+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:30:16.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:30:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:30:16.484+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:30:17.453 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:30:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:30:17.150+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:30:17.453 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:30:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:30:17.150+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:30:17.454 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:30:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:30:17.150+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:30:17.454 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:30:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:30:17.334+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:30:17.454 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:30:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:30:17.334+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:30:17.454 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:30:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:30:17.334+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:30:17.742 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:30:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:30:17.482+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 26336 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:30:17.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:30:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:30:17.451+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:30:17.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:30:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:30:17.451+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:30:17.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:30:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:30:17.451+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:30:17.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:30:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:30:17.451+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:30:18.486 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:30:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:30:18.180+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:30:18.486 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:30:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:30:18.180+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:30:18.486 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:30:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:30:18.180+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:30:18.486 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:30:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:30:18.368+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:30:18.486 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:30:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:30:18.368+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:30:18.487 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:30:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:30:18.368+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:30:18.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:30:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:30:18.484+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:30:18.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:30:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:30:18.484+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:30:18.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:30:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:30:18.484+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:30:18.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:30:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:30:18.484+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:30:19.455 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:30:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:30:19.138+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:30:19.455 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:30:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:30:19.138+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:30:19.455 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:30:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:30:19.138+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:30:19.456 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:30:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:30:19.327+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:30:19.456 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:30:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:30:19.327+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:30:19.456 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:30:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:30:19.327+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:30:19.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:30:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:30:19.453+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:30:19.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:30:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:30:19.453+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:30:19.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:30:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:30:19.453+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:30:19.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:30:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:30:19.453+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:30:20.233 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:30:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:30:20.124+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:30:20.233 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:30:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:30:20.124+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:30:20.234 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:30:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:30:20.124+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:30:20.492 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:30:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:30:20.347+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:30:20.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:30:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:30:20.347+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:30:20.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:30:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:30:20.347+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:30:20.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:30:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:30:20.406+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:30:20.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:30:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:30:20.406+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:30:20.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:30:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:30:20.406+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:30:20.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:30:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:30:20.406+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:30:21.492 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:30:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:30:21.360+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:30:21.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:30:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:30:21.360+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:30:21.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:30:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:30:21.360+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:30:21.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:30:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:30:21.148+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:30:21.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:30:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:30:21.148+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:30:21.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:30:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:30:21.148+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:30:21.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:30:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:30:21.389+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:30:21.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:30:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:30:21.389+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:30:21.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:30:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:30:21.389+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:30:21.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:30:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:30:21.389+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:30:22.489 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:30:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:30:22.373+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:30:22.489 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:30:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:30:22.373+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:30:22.489 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:30:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:30:22.373+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:30:22.490 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:30:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:30:22.369+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:30:22.490 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:30:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:30:22.369+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:30:22.490 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:30:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:30:22.369+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:30:22.490 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:30:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:30:22.369+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:30:22.490 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:30:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:30:22.188+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:30:22.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:30:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:30:22.188+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:30:22.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:30:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:30:22.188+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:30:22.742 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:30:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:30:22.486+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 26404 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:30:23.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:30:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:30:23.165+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:30:23.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:30:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:30:23.165+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:30:23.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:30:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:30:23.165+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:30:23.742 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:30:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:30:23.356+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:30:23.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:30:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:30:23.356+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:30:23.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:30:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:30:23.356+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:30:23.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:30:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:30:23.350+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:30:23.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:30:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:30:23.350+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:30:23.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:30:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:30:23.350+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:30:23.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:30:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:30:23.350+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:30:24.408 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:30:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:30:24.391+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:30:24.409 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:30:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:30:24.391+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:30:24.409 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:30:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:30:24.391+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:30:24.409 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:30:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:30:24.391+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:30:24.409 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:30:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:30:24.146+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:30:24.409 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:30:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:30:24.146+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:30:24.410 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:30:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:30:24.146+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:30:24.742 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:30:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:30:24.406+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:30:24.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:30:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:30:24.406+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:30:24.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:30:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:30:24.406+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:30:25.375 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:30:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:30:25.109+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:30:25.375 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:30:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:30:25.109+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:30:25.375 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:30:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:30:25.109+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:30:25.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:30:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:30:25.373+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:30:25.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:30:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:30:25.373+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:30:25.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:30:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:30:25.373+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:30:25.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:30:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:30:25.373+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:30:25.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:30:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:30:25.434+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:30:25.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:30:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:30:25.434+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:30:25.744 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:30:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:30:25.434+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:30:26.456 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:30:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:30:26.339+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:30:26.456 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:30:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:30:26.339+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:30:26.456 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:30:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:30:26.339+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:30:26.457 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:30:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:30:26.339+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:30:26.457 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:30:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:30:26.099+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:30:26.457 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:30:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:30:26.099+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:30:26.457 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:30:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:30:26.099+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:30:26.742 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:30:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:30:26.454+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:30:26.742 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:30:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:30:26.454+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:30:26.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:30:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:30:26.454+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:30:27.475 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:30:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:30:27.310+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:30:27.476 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:30:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:30:27.310+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:30:27.476 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:30:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:30:27.310+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:30:27.476 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:30:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:30:27.310+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:30:27.476 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:30:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:30:27.139+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:30:27.476 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:30:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:30:27.139+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:30:27.476 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:30:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:30:27.139+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:30:27.742 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:30:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:30:27.490+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 26468 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:30:27.742 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:30:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:30:27.473+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:30:27.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:30:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:30:27.473+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:30:27.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:30:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:30:27.473+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:30:28.430 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:30:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:30:28.172+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:30:28.430 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:30:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:30:28.172+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:30:28.430 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:30:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:30:28.172+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:30:28.430 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:30:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:30:28.331+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:30:28.430 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:30:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:30:28.331+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:30:28.431 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:30:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:30:28.331+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:30:28.431 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:30:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:30:28.331+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:30:28.742 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:30:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:30:28.427+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:30:28.742 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:30:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:30:28.427+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:30:28.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:30:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:30:28.427+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:30:29.479 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:30:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:30:29.195+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:30:29.479 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:30:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:30:29.195+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:30:29.479 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:30:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:30:29.195+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:30:29.480 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:30:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:30:29.297+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:30:29.480 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:30:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:30:29.297+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:30:29.480 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:30:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:30:29.297+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:30:29.480 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:30:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:30:29.297+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:30:29.742 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:30:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:30:29.477+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:30:29.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:30:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:30:29.477+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:30:29.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:30:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:30:29.477+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:30:30.437 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:30:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:30:30.177+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:30:30.438 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:30:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:30:30.177+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:30:30.438 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:30:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:30:30.177+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:30:30.438 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:30:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:30:30.314+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:30:30.438 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:30:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:30:30.314+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:30:30.438 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:30:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:30:30.315+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:30:30.439 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:30:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:30:30.315+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:30:30.742 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:30:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:30:30.436+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:30:30.742 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:30:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:30:30.436+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:30:30.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:30:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:30:30.436+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:30:31.485 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:30:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:30:31.288+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:30:31.486 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:30:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:30:31.288+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:30:31.486 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:30:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:30:31.288+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:30:31.486 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:30:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:30:31.288+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:30:31.486 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:30:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:30:31.212+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:30:31.486 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:30:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:30:31.212+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:30:31.486 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:30:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:30:31.212+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:30:31.742 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:30:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:30:31.484+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:30:31.742 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:30:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:30:31.484+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:30:31.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:30:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:30:31.484+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:30:32.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:30:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:30:32.217+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:30:32.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:30:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:30:32.217+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:30:32.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:30:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:30:32.217+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:30:32.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:30:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:30:32.253+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:30:32.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:30:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:30:32.253+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:30:32.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:30:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:30:32.253+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:30:32.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:30:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:30:32.253+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:30:32.992 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:30:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:30:32.495+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 26533 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:30:32.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:30:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:30:32.507+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:30:32.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:30:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:30:32.507+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:30:32.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:30:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:30:32.507+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:30:33.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:30:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:30:33.233+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:30:33.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:30:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:30:33.233+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:30:33.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:30:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:30:33.233+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:30:33.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:30:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:30:33.274+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:30:33.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:30:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:30:33.274+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:30:33.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:30:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:30:33.274+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:30:33.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:30:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:30:33.274+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:30:33.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:30:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:30:33.546+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:30:33.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:30:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:30:33.546+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:30:33.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:30:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:30:33.546+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:30:34.516 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:30:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:30:34.278+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:30:34.516 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:30:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:30:34.278+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:30:34.516 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:30:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:30:34.278+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:30:34.516 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:30:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:30:34.278+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:30:34.517 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:30:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:30:34.241+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:30:34.517 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:30:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:30:34.241+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:30:34.517 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:30:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:30:34.241+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:30:34.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:30:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:30:34.514+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:30:34.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:30:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:30:34.514+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:30:34.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:30:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:30:34.514+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:30:35.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:30:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:30:35.195+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:30:35.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:30:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:30:35.195+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:30:35.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:30:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:30:35.195+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:30:35.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:30:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:30:35.234+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:30:35.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:30:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:30:35.234+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:30:35.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:30:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:30:35.234+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:30:35.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:30:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:30:35.234+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:30:35.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:30:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:30:35.491+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:30:35.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:30:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:30:35.491+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:30:35.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:30:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:30:35.491+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:30:36.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:30:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:30:36.150+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:30:36.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:30:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:30:36.150+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:30:36.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:30:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:30:36.150+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:30:36.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:30:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:30:36.189+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:30:36.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:30:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:30:36.189+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:30:36.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:30:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:30:36.189+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:30:36.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:30:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:30:36.189+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:30:36.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:30:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:30:36.536+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:30:36.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:30:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:30:36.536+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:30:36.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:30:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:30:36.536+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:30:37.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:30:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:30:37.131+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:30:37.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:30:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:30:37.131+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:30:37.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:30:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:30:37.131+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:30:37.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:30:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:30:37.144+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:30:37.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:30:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:30:37.144+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:30:37.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:30:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:30:37.144+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:30:37.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:30:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:30:37.144+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:30:37.992 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:30:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:30:37.499+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 26598 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:30:37.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:30:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:30:37.517+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:30:37.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:30:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:30:37.517+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:30:37.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:30:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:30:37.517+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:30:38.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:30:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:30:38.099+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:30:38.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:30:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:30:38.099+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:30:38.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:30:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:30:38.099+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:30:38.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:30:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:30:38.141+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:30:38.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:30:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:30:38.141+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:30:38.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:30:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:30:38.141+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:30:38.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:30:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:30:38.141+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:30:38.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:30:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:30:38.556+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:30:38.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:30:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:30:38.556+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:30:38.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:30:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:30:38.556+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:30:39.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:30:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:30:39.141+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:30:39.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:30:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:30:39.141+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:30:39.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:30:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:30:39.141+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:30:39.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:30:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:30:39.103+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:30:39.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:30:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:30:39.103+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:30:39.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:30:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:30:39.103+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:30:39.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:30:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:30:39.103+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:30:39.886 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:30:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:30:39.587+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:30:39.886 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:30:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:30:39.587+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:30:39.886 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:30:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:30:39.587+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:30:40.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:30:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:30:40.141+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:30:40.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:30:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:30:40.141+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:30:40.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:30:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:30:40.141+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:30:40.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:30:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:30:40.141+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:30:40.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:30:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:30:40.144+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:30:40.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:30:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:30:40.144+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:30:40.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:30:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:30:40.144+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:30:40.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:30:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:30:40.554+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:30:40.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:30:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:30:40.554+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:30:40.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:30:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:30:40.554+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:30:41.401 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:30:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:30:41.120+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:30:41.401 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:30:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:30:41.120+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:30:41.401 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:30:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:30:41.120+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:30:41.401 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:30:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:30:41.190+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:30:41.402 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:30:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:30:41.190+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:30:41.402 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:30:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:30:41.190+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:30:41.402 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:30:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:30:41.190+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:30:41.742 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:30:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:30:41.568+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:30:41.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:30:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:30:41.568+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:30:41.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:30:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:30:41.568+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:30:42.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:30:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:30:42.093+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:30:42.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:30:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:30:42.093+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:30:42.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:30:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:30:42.093+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:30:42.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:30:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:30:42.190+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:30:42.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:30:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:30:42.190+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:30:42.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:30:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:30:42.190+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:30:42.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:30:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:30:42.190+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:30:42.992 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:30:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:30:42.503+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 26662 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:30:42.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:30:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:30:42.572+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:30:42.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:30:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:30:42.572+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:30:42.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:30:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:30:42.572+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:30:43.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:30:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:30:43.063+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:30:43.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:30:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:30:43.063+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:30:43.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:30:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:30:43.063+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:30:43.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:30:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:30:43.224+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:30:43.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:30:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:30:43.224+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:30:43.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:30:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:30:43.225+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:30:43.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:30:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:30:43.225+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:30:43.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:30:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:30:43.539+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:30:43.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:30:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:30:43.539+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:30:43.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:30:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:30:43.539+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:30:44.473 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:30:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:30:44.063+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:30:44.473 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:30:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:30:44.063+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:30:44.473 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:30:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:30:44.063+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:30:44.474 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:30:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:30:44.188+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:30:44.474 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:30:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:30:44.188+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:30:44.474 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:30:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:30:44.188+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:30:44.474 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:30:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:30:44.188+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:30:44.742 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:30:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:30:44.491+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:30:44.742 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:30:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:30:44.491+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:30:44.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:30:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:30:44.491+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:30:45.450 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:30:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:30:45.040+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:30:45.450 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:30:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:30:45.040+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:30:45.451 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:30:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:30:45.040+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:30:45.451 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:30:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:30:45.174+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:30:45.451 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:30:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:30:45.174+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:30:45.451 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:30:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:30:45.174+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:30:45.451 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:30:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:30:45.175+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:30:45.742 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:30:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:30:45.448+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:30:45.742 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:30:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:30:45.448+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:30:45.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:30:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:30:45.448+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:30:46.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:30:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:30:46.014+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:30:46.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:30:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:30:46.014+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:30:46.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:30:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:30:46.014+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:30:46.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:30:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:30:46.174+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:30:46.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:30:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:30:46.174+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:30:46.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:30:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:30:46.174+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:30:46.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:30:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:30:46.174+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:30:46.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:30:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:30:46.498+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:30:46.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:30:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:30:46.498+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:30:46.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:30:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:30:46.498+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:30:47.496 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:30:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:30:47.042+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:30:47.496 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:30:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:30:47.042+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:30:47.496 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:30:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:30:47.042+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:30:47.497 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:30:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:30:47.163+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:30:47.497 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:30:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:30:47.163+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:30:47.497 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:30:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:30:47.163+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:30:47.497 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:30:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:30:47.163+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:30:47.992 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:30:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:30:47.507+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 26727 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:30:47.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:30:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:30:47.515+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:30:47.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:30:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:30:47.515+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:30:47.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:30:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:30:47.515+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:30:48.479 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:30:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:30:48.061+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:30:48.480 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:30:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:30:48.061+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:30:48.480 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:30:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:30:48.061+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:30:48.480 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:30:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:30:48.197+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:30:48.480 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:30:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:30:48.197+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:30:48.480 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:30:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:30:48.197+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:30:48.480 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:30:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:30:48.197+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:30:48.742 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:30:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:30:48.478+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:30:48.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:30:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:30:48.478+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:30:48.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:30:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:30:48.478+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:30:49.475 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:30:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:30:49.222+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:30:49.475 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:30:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:30:49.222+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:30:49.476 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:30:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:30:49.222+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:30:49.476 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:30:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:30:49.222+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:30:49.476 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:30:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:30:49.028+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:30:49.476 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:30:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:30:49.028+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:30:49.476 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:30:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:30:49.028+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:30:49.742 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:30:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:30:49.474+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:30:49.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:30:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:30:49.474+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:30:49.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:30:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:30:49.474+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:30:50.235 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:30:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:30:50.055+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:30:50.235 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:30:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:30:50.055+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:30:50.235 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:30:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:30:50.055+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:30:50.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:30:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:30:50.256+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:30:50.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:30:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:30:50.256+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:30:50.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:30:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:30:50.256+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:30:50.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:30:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:30:50.256+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:30:50.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:30:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:30:50.492+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:30:50.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:30:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:30:50.492+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:30:50.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:30:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:30:50.492+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:30:51.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:30:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:30:51.260+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:30:51.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:30:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:30:51.260+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:30:51.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:30:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:30:51.260+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:30:51.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:30:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:30:51.260+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:30:51.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:30:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:30:51.033+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:30:51.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:30:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:30:51.033+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:30:51.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:30:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:30:51.033+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:30:51.769 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:30:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:30:51.490+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:30:51.769 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:30:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:30:51.490+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:30:51.770 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:30:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:30:51.490+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:30:52.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:30:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:30:52.025+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:30:52.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:30:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:30:52.025+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:30:52.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:30:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:30:52.025+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:30:52.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:30:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:30:52.268+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:30:52.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:30:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:30:52.268+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:30:52.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:30:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:30:52.268+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:30:52.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:30:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:30:52.268+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:30:52.992 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:30:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:30:52.510+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 26795 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:30:52.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:30:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:30:52.495+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:30:52.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:30:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:30:52.495+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:30:52.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:30:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:30:52.495+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:30:53.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:30:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:30:53.074+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:30:53.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:30:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:30:53.074+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:30:53.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:30:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:30:53.074+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:30:53.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:30:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:30:53.229+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:30:53.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:30:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:30:53.229+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:30:53.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:30:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:30:53.229+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:30:53.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:30:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:30:53.229+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:30:53.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:30:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:30:53.493+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:30:53.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:30:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:30:53.493+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:30:53.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:30:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:30:53.493+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:30:54.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:30:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:30:54.032+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:30:54.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:30:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:30:54.032+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:30:54.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:30:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:30:54.032+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:30:54.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:30:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:30:54.212+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:30:54.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:30:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:30:54.212+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:30:54.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:30:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:30:54.212+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:30:54.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:30:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:30:54.212+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:30:54.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:30:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:30:54.519+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:30:54.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:30:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:30:54.519+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:30:54.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:30:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:30:54.519+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:30:55.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:30:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:30:55.024+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:30:55.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:30:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:30:55.024+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:30:55.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:30:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:30:55.024+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:30:55.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:30:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:30:55.236+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:30:55.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:30:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:30:55.236+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:30:55.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:30:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:30:55.236+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:30:55.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:30:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:30:55.236+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:30:55.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:30:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:30:55.558+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:30:55.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:30:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:30:55.558+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:30:55.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:30:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:30:55.558+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:30:56.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:30:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:30:55.993+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:30:56.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:30:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:30:55.993+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:30:56.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:30:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:30:55.993+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:30:56.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:30:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:30:56.206+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:30:56.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:30:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:30:56.206+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:30:56.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:30:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:30:56.206+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:30:56.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:30:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:30:56.207+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:30:56.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:30:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:30:56.549+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:30:56.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:30:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:30:56.549+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:30:56.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:30:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:30:56.549+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:30:57.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:30:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:30:57.191+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:30:57.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:30:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:30:57.191+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:30:57.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:30:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:30:57.191+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:30:57.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:30:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:30:57.191+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:30:57.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:30:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:30:57.018+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:30:57.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:30:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:30:57.018+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:30:57.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:30:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:30:57.018+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:30:57.992 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:30:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:30:57.514+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 26863 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:30:57.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:30:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:30:57.557+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:30:57.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:30:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:30:57.557+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:30:57.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:30:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:30:57.557+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:30:58.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:30:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:30:58.034+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:30:58.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:30:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:30:58.034+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:30:58.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:30:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:30:58.034+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:30:58.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:30:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:30:58.205+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:30:58.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:30:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:30:58.205+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:30:58.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:30:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:30:58.205+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:30:58.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:30:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:30:58.206+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:30:58.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:30:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:30:58.518+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:30:58.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:30:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:30:58.518+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:30:58.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:30:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:30:58.518+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:30:59.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:30:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:30:59.226+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:30:59.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:30:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:30:59.226+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:30:59.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:30:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:30:59.226+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:30:59.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:30:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:30:59.226+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:30:59.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:30:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:30:59.076+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:30:59.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:30:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:30:59.076+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:30:59.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:30:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:30:59.076+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:30:59.886 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:30:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:30:59.543+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:30:59.886 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:30:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:30:59.543+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:30:59.886 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:30:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:30:59.543+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:31:00.235 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:31:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:31:00.065+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:31:00.236 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:31:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:31:00.065+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:31:00.236 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:31:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:31:00.065+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:31:00.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:31:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:31:00.257+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:31:00.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:31:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:31:00.257+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:31:00.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:31:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:31:00.257+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:31:00.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:31:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:31:00.257+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:31:00.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:31:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:31:00.579+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:31:00.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:31:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:31:00.579+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:31:00.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:31:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:31:00.579+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:31:01.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:31:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:31:01.108+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:31:01.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:31:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:31:01.108+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:31:01.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:31:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:31:01.108+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:31:01.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:31:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:31:01.212+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:31:01.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:31:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:31:01.212+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:31:01.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:31:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:31:01.212+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:31:01.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:31:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:31:01.212+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:31:01.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:31:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:31:01.586+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:31:01.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:31:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:31:01.586+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:31:01.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:31:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:31:01.586+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:31:02.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:31:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:31:02.150+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:31:02.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:31:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:31:02.150+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:31:02.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:31:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:31:02.150+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:31:02.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:31:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:31:02.222+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:31:02.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:31:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:31:02.222+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:31:02.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:31:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:31:02.222+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:31:02.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:31:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:31:02.222+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:31:02.905 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:31:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:31:02.518+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 26931 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:31:02.905 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:31:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:31:02.572+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:31:02.905 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:31:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:31:02.572+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:31:02.905 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:31:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:31:02.572+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:31:03.218 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:31:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:31:03.127+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:31:03.218 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:31:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:31:03.127+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:31:03.219 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:31:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:31:03.127+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:31:03.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:31:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:31:03.216+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:31:03.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:31:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:31:03.216+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:31:03.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:31:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:31:03.216+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:31:03.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:31:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:31:03.216+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:31:03.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:31:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:31:03.596+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:31:03.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:31:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:31:03.596+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:31:03.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:31:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:31:03.596+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:31:04.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:31:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:31:04.140+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:31:04.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:31:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:31:04.140+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:31:04.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:31:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:31:04.140+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:31:04.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:31:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:31:04.253+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:31:04.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:31:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:31:04.253+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:31:04.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:31:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:31:04.253+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:31:04.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:31:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:31:04.253+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:31:04.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:31:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:31:04.549+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:31:04.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:31:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:31:04.549+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:31:04.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:31:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:31:04.549+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:31:05.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:31:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:31:05.123+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:31:05.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:31:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:31:05.123+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:31:05.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:31:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:31:05.123+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:31:05.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:31:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:31:05.259+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:31:05.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:31:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:31:05.259+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:31:05.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:31:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:31:05.259+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:31:05.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:31:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:31:05.259+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:31:05.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:31:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:31:05.553+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:31:05.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:31:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:31:05.553+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:31:05.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:31:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:31:05.553+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:31:06.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:31:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:31:06.150+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:31:06.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:31:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:31:06.150+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:31:06.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:31:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:31:06.150+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:31:06.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:31:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:31:06.259+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:31:06.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:31:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:31:06.259+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:31:06.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:31:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:31:06.259+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:31:06.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:31:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:31:06.259+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:31:06.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:31:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:31:06.572+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:31:06.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:31:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:31:06.572+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:31:06.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:31:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:31:06.572+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:31:07.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:31:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:31:07.117+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:31:07.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:31:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:31:07.117+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:31:07.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:31:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:31:07.117+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:31:07.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:31:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:31:07.236+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:31:07.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:31:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:31:07.236+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:31:07.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:31:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:31:07.236+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:31:07.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:31:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:31:07.236+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:31:07.992 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:31:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:31:07.522+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 27001 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:31:07.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:31:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:31:07.570+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:31:07.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:31:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:31:07.570+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:31:07.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:31:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:31:07.570+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:31:08.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:31:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:31:08.080+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:31:08.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:31:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:31:08.080+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:31:08.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:31:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:31:08.080+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:31:08.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:31:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:31:08.206+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:31:08.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:31:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:31:08.206+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:31:08.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:31:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:31:08.206+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:31:08.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:31:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:31:08.206+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:31:08.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:31:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:31:08.536+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:31:08.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:31:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:31:08.536+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:31:08.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:31:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:31:08.536+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:31:09.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:31:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:31:09.129+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:31:09.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:31:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:31:09.129+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:31:09.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:31:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:31:09.129+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:31:09.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:31:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:31:09.234+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:31:09.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:31:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:31:09.234+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:31:09.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:31:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:31:09.234+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:31:09.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:31:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:31:09.234+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:31:09.886 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:31:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:31:09.499+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:31:09.887 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:31:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:31:09.499+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:31:09.887 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:31:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:31:09.499+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:31:10.217 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:31:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:31:10.126+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:31:10.217 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:31:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:31:10.126+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:31:10.218 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:31:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:31:10.126+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:31:10.485 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:31:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:31:10.215+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:31:10.486 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:31:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:31:10.215+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:31:10.486 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:31:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:31:10.215+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:31:10.486 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:31:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:31:10.215+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:31:10.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:31:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:31:10.483+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:31:10.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:31:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:31:10.483+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:31:10.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:31:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:31:10.483+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:31:11.479 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:31:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:31:11.148+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:31:11.479 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:31:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:31:11.148+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:31:11.479 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:31:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:31:11.148+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:31:11.479 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:31:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:31:11.248+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:31:11.479 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:31:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:31:11.248+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:31:11.479 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:31:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:31:11.248+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:31:11.480 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:31:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:31:11.248+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:31:11.742 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:31:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:31:11.476+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:31:11.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:31:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:31:11.476+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:31:11.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:31:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:31:11.476+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:31:12.434 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:31:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:31:12.155+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:31:12.434 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:31:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:31:12.155+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:31:12.434 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:31:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:31:12.155+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:31:12.435 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:31:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:31:12.264+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:31:12.435 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:31:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:31:12.264+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:31:12.435 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:31:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:31:12.264+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:31:12.435 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:31:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:31:12.264+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:31:12.742 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:31:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:31:12.526+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 27072 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:31:12.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:31:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:31:12.431+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:31:12.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:31:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:31:12.431+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:31:12.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:31:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:31:12.431+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:31:13.437 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:31:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:31:13.266+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:31:13.437 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:31:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:31:13.266+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:31:13.437 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:31:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:31:13.266+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:31:13.438 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:31:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:31:13.266+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:31:13.438 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:31:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:31:13.137+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:31:13.438 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:31:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:31:13.137+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:31:13.438 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:31:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:31:13.137+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:31:13.742 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:31:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:31:13.434+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:31:13.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:31:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:31:13.434+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:31:13.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:31:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:31:13.434+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:31:14.487 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:31:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:31:14.242+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:31:14.487 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:31:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:31:14.242+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:31:14.488 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:31:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:31:14.242+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:31:14.488 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:31:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:31:14.242+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:31:14.488 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:31:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:31:14.137+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:31:14.488 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:31:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:31:14.137+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:31:14.488 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:31:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:31:14.137+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:31:14.742 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:31:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:31:14.482+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:31:14.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:31:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:31:14.482+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:31:14.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:31:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:31:14.482+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:31:15.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:31:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:31:15.292+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:31:15.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:31:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:31:15.292+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:31:15.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:31:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:31:15.292+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:31:15.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:31:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:31:15.292+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:31:15.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:31:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:31:15.153+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:31:15.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:31:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:31:15.153+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:31:15.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:31:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:31:15.153+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:31:15.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:31:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:31:15.505+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:31:15.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:31:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:31:15.505+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:31:15.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:31:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:31:15.505+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:31:16.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:31:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:31:16.250+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:31:16.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:31:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:31:16.250+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:31:16.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:31:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:31:16.250+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:31:16.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:31:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:31:16.250+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:31:16.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:31:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:31:16.111+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:31:16.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:31:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:31:16.111+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:31:16.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:31:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:31:16.111+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:31:16.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:31:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:31:16.539+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:31:16.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:31:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:31:16.539+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:31:16.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:31:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:31:16.539+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:31:17.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:31:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:31:17.091+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:31:17.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:31:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:31:17.091+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:31:17.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:31:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:31:17.091+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:31:17.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:31:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:31:17.276+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:31:17.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:31:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:31:17.276+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:31:17.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:31:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:31:17.276+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:31:17.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:31:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:31:17.276+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:31:17.992 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:31:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:31:17.530+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 27142 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:31:17.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:31:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:31:17.553+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:31:17.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:31:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:31:17.553+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:31:17.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:31:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:31:17.553+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:31:18.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:31:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:31:18.289+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:31:18.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:31:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:31:18.289+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:31:18.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:31:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:31:18.289+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:31:18.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:31:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:31:18.289+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:31:18.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:31:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:31:18.128+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:31:18.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:31:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:31:18.128+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:31:18.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:31:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:31:18.128+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:31:18.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:31:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:31:18.510+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:31:18.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:31:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:31:18.510+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:31:18.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:31:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:31:18.510+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:31:19.465 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:31:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:31:19.127+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:31:19.465 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:31:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:31:19.127+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:31:19.465 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:31:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:31:19.127+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:31:19.466 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:31:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:31:19.320+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:31:19.466 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:31:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:31:19.320+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:31:19.466 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:31:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:31:19.320+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:31:19.466 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:31:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:31:19.320+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:31:19.742 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:31:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:31:19.462+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:31:19.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:31:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:31:19.462+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:31:19.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:31:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:31:19.462+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:31:20.236 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:31:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:31:20.093+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:31:20.236 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:31:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:31:20.093+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:31:20.237 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:31:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:31:20.093+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:31:20.487 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:31:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:31:20.355+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:31:20.488 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:31:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:31:20.355+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:31:20.488 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:31:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:31:20.355+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:31:20.488 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:31:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:31:20.355+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:31:20.742 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:31:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:31:20.486+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:31:20.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:31:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:31:20.486+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:31:20.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:31:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:31:20.486+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:31:21.337 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:31:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:31:21.047+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:31:21.337 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:31:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:31:21.047+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:31:21.337 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:31:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:31:21.047+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:31:21.742 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:31:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:31:21.473+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:31:21.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:31:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:31:21.473+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:31:21.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:31:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:31:21.473+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:31:21.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:31:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:31:21.357+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:31:21.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:31:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:31:21.357+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:31:21.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:31:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:31:21.357+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:31:21.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:31:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:31:21.357+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:31:22.364 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:31:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:31:22.091+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:31:22.365 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:31:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:31:22.091+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:31:22.365 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:31:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:31:22.091+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:31:22.658 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:31:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:31:22.535+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 27213 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:31:22.659 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:31:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:31:22.436+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:31:22.659 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:31:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:31:22.436+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:31:22.659 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:31:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:31:22.436+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:31:22.660 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:31:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:31:22.362+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:31:22.660 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:31:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:31:22.363+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:31:22.660 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:31:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:31:22.363+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:31:22.660 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:31:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:31:22.363+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:31:23.388 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:31:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:31:23.059+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:31:23.388 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:31:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:31:23.059+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:31:23.388 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:31:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:31:23.059+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:31:23.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:31:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:31:23.397+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:31:23.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:31:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:31:23.397+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:31:23.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:31:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:31:23.397+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:31:23.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:31:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:31:23.386+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:31:23.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:31:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:31:23.386+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:31:23.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:31:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:31:23.386+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:31:23.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:31:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:31:23.386+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:31:24.354 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:31:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:31:24.048+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:31:24.354 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:31:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:31:24.048+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:31:24.354 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:31:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:31:24.048+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:31:24.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:31:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:31:24.352+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:31:24.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:31:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:31:24.352+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:31:24.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:31:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:31:24.352+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:31:24.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:31:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:31:24.352+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:31:24.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:31:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:31:24.375+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:31:24.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:31:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:31:24.375+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:31:24.744 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:31:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:31:24.375+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:31:25.351 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:31:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:31:25.078+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:31:25.351 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:31:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:31:25.078+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:31:25.351 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:31:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:31:25.078+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:31:25.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:31:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:31:25.387+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:31:25.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:31:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:31:25.387+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:31:25.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:31:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:31:25.387+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:31:25.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:31:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:31:25.388+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:31:25.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:31:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:31:25.349+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:31:25.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:31:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:31:25.349+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:31:25.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:31:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:31:25.349+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:31:26.380 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:31:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:31:26.110+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:31:26.380 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:31:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:31:26.110+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:31:26.380 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:31:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:31:26.110+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:31:26.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:31:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:31:26.403+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:31:26.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:31:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:31:26.403+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:31:26.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:31:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:31:26.403+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:31:26.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:31:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:31:26.403+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:31:26.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:31:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:31:26.378+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:31:26.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:31:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:31:26.378+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:31:26.744 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:31:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:31:26.378+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:31:27.425 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:31:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:31:27.158+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:31:27.425 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:31:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:31:27.158+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:31:27.425 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:31:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:31:27.158+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:31:27.425 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:31:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:31:27.400+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:31:27.426 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:31:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:31:27.400+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:31:27.426 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:31:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:31:27.400+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:31:27.426 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:31:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:31:27.400+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:31:27.742 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:31:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:31:27.539+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 27284 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:31:27.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:31:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:31:27.423+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:31:27.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:31:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:31:27.423+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:31:27.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:31:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:31:27.423+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:31:28.472 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:31:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:31:28.172+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:31:28.472 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:31:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:31:28.172+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:31:28.472 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:31:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:31:28.173+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:31:28.473 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:31:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:31:28.420+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:31:28.473 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:31:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:31:28.420+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:31:28.473 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:31:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:31:28.420+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:31:28.473 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:31:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:31:28.420+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:31:28.742 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:31:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:31:28.470+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:31:28.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:31:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:31:28.470+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:31:28.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:31:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:31:28.470+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:31:29.390 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:31:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:31:29.137+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:31:29.391 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:31:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:31:29.137+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:31:29.391 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:31:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:31:29.137+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:31:29.742 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:31:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:31:29.449+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:31:29.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:31:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:31:29.449+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:31:29.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:31:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:31:29.449+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:31:29.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:31:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:31:29.389+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:31:29.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:31:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:31:29.389+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:31:29.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:31:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:31:29.389+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:31:29.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:31:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:31:29.389+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:31:30.237 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:31:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:31:30.098+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:31:30.237 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:31:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:31:30.098+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:31:30.237 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:31:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:31:30.098+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:31:30.492 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:31:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:31:30.467+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:31:30.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:31:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:31:30.467+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:31:30.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:31:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:31:30.467+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:31:30.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:31:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:31:30.364+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:31:30.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:31:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:31:30.364+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:31:30.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:31:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:31:30.364+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:31:30.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:31:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:31:30.364+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:31:31.406 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:31:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:31:31.136+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:31:31.406 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:31:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:31:31.136+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:31:31.406 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:31:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:31:31.136+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:31:31.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:31:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:31:31.404+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:31:31.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:31:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:31:31.404+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:31:31.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:31:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:31:31.404+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:31:31.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:31:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:31:31.404+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:31:31.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:31:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:31:31.508+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:31:31.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:31:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:31:31.508+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:31:31.744 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:31:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:31:31.508+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:31:32.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:31:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:31:32.160+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:31:32.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:31:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:31:32.160+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:31:32.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:31:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:31:32.160+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:31:32.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:31:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:31:32.385+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:31:32.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:31:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:31:32.385+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:31:32.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:31:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:31:32.385+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:31:32.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:31:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:31:32.385+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:31:32.992 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:31:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:31:32.543+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 27355 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:31:32.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:31:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:31:32.504+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:31:32.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:31:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:31:32.504+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:31:32.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:31:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:31:32.504+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:31:33.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:31:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:31:33.196+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:31:33.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:31:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:31:33.196+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:31:33.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:31:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:31:33.196+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:31:33.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:31:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:31:33.386+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:31:33.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:31:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:31:33.386+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:31:33.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:31:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:31:33.386+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:31:33.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:31:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:31:33.387+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:31:33.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:31:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:31:33.509+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:31:33.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:31:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:31:33.509+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:31:33.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:31:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:31:33.509+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:31:34.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:31:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:31:34.163+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:31:34.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:31:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:31:34.163+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:31:34.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:31:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:31:34.163+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:31:34.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:31:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:31:34.359+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:31:34.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:31:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:31:34.359+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:31:34.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:31:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:31:34.359+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:31:34.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:31:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:31:34.359+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:31:34.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:31:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:31:34.512+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:31:34.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:31:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:31:34.512+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:31:34.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:31:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:31:34.512+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:31:35.036 ERROR:paramiko.transport:Socket exception: No route to host (113) 2024-02-20T08:31:35.041 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-20T08:31:35.041 INFO:tasks.workunit:Stopping ['suites/fsstress.sh'] on client.1... 2024-02-20T08:31:35.042 ERROR:teuthology.run_tasks:Manager failed: kclient Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_batrick_ceph_3d72bcfd7affbb4a923c191d212814b4934cfeaa/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/github.com_batrick_ceph_3d72bcfd7affbb4a923c191d212814b4934cfeaa/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/github.com_batrick_ceph_3d72bcfd7affbb4a923c191d212814b4934cfeaa/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 450, in run r.execute() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 100, in execute self.client.exec_command(self.command) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/client.py", line 560, in exec_command chan = self._transport.open_session(timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 959, in open_session return self.open_channel( File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 1090, in open_channel raise e File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2159, in run ptype, m = self.packetizer.read_message() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 463, in read_message header = self.read_all(self.__block_size_in, check_rekey=True) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 308, in read_all x = self.__socket.recv(n) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/gevent/_socketcommon.py", line 660, in recv return self._sock.recv(*args) OSError: [Errno 113] No route to host 2024-02-20T08:31:35.152 DEBUG:teuthology.run_tasks:Unwinding manager full_sequential_finally 2024-02-20T08:31:35.164 INFO:teuthology.task.full_sequential_finally:In full_sequential_finally, running task sleep... 2024-02-20T08:31:35.165 INFO:teuthology.task.sleep:Sleeping for 3600 seconds 2024-02-20T08:31:35.467 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:31:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:31:35.130+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:31:35.467 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:31:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:31:35.130+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:31:35.467 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:31:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:31:35.130+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:31:35.467 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:31:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:31:35.310+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:31:35.468 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:31:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:31:35.310+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:31:35.468 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:31:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:31:35.310+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:31:35.468 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:31:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:31:35.310+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:31:35.744 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:31:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:31:35.465+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:31:35.745 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:31:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:31:35.465+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:31:35.745 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:31:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:31:35.465+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:31:36.459 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:31:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:31:36.132+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:31:36.459 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:31:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:31:36.132+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:31:36.459 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:31:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:31:36.132+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:31:36.460 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:31:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:31:36.310+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:31:36.460 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:31:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:31:36.310+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:31:36.460 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:31:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:31:36.310+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:31:36.460 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:31:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:31:36.310+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:31:36.742 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:31:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:31:36.457+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:31:36.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:31:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:31:36.457+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:31:36.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:31:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:31:36.457+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:31:37.445 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:31:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:31:37.131+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:31:37.446 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:31:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:31:37.131+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:31:37.446 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:31:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:31:37.131+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:31:37.446 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:31:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:31:37.323+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:31:37.446 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:31:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:31:37.323+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:31:37.446 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:31:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:31:37.323+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:31:37.446 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:31:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:31:37.323+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:31:37.742 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:31:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:31:37.547+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 27426 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:31:37.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:31:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:31:37.444+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:31:37.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:31:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:31:37.444+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:31:37.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:31:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:31:37.444+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:31:38.457 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:31:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:31:38.091+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:31:38.457 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:31:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:31:38.091+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:31:38.457 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:31:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:31:38.091+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:31:38.458 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:31:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:31:38.314+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:31:38.458 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:31:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:31:38.314+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:31:38.458 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:31:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:31:38.314+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:31:38.458 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:31:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:31:38.314+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:31:38.742 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:31:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:31:38.455+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:31:38.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:31:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:31:38.455+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:31:38.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:31:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:31:38.455+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:31:39.455 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:31:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:31:39.087+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:31:39.455 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:31:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:31:39.087+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:31:39.456 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:31:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:31:39.087+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:31:39.456 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:31:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:31:39.298+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:31:39.456 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:31:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:31:39.298+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:31:39.456 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:31:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:31:39.298+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:31:39.456 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:31:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:31:39.298+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:31:39.742 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:31:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:31:39.453+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:31:39.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:31:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:31:39.453+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:31:39.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:31:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:31:39.453+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:31:40.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:31:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:31:40.136+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:31:40.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:31:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:31:40.136+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:31:40.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:31:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:31:40.136+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:31:40.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:31:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:31:40.298+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:31:40.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:31:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:31:40.298+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:31:40.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:31:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:31:40.298+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:31:40.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:31:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:31:40.298+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:31:40.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:31:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:31:40.493+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:31:40.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:31:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:31:40.494+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:31:40.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:31:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:31:40.494+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:31:41.462 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:31:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:31:41.315+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:31:41.462 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:31:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:31:41.315+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:31:41.462 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:31:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:31:41.315+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:31:41.462 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:31:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:31:41.315+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:31:41.462 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:31:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:31:41.157+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:31:41.463 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:31:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:31:41.157+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:31:41.463 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:31:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:31:41.157+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:31:41.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:31:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:31:41.460+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:31:41.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:31:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:31:41.460+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:31:41.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:31:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:31:41.460+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:31:42.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:31:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:31:42.284+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:31:42.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:31:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:31:42.284+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:31:42.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:31:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:31:42.284+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:31:42.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:31:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:31:42.284+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:31:42.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:31:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:31:42.148+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:31:42.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:31:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:31:42.148+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:31:42.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:31:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:31:42.148+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:31:42.841 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:31:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:31:42.550+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 27497 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:31:42.841 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:31:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:31:42.498+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:31:42.841 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:31:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:31:42.498+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:31:42.841 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:31:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:31:42.498+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:31:43.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:31:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:31:43.172+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:31:43.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:31:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:31:43.172+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:31:43.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:31:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:31:43.172+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:31:43.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:31:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:31:43.304+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:31:43.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:31:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:31:43.304+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:31:43.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:31:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:31:43.304+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:31:43.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:31:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:31:43.304+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:31:43.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:31:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:31:43.518+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:31:43.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:31:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:31:43.518+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:31:43.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:31:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:31:43.518+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:31:44.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:31:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:31:44.144+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:31:44.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:31:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:31:44.144+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:31:44.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:31:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:31:44.144+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:31:44.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:31:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:31:44.260+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:31:44.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:31:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:31:44.260+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:31:44.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:31:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:31:44.260+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:31:44.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:31:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:31:44.260+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:31:44.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:31:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:31:44.510+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:31:44.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:31:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:31:44.510+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:31:44.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:31:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:31:44.510+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:31:45.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:31:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:31:45.190+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:31:45.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:31:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:31:45.190+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:31:45.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:31:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:31:45.190+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:31:45.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:31:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:31:45.214+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:31:45.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:31:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:31:45.214+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:31:45.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:31:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:31:45.214+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:31:45.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:31:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:31:45.214+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:31:45.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:31:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:31:45.554+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:31:45.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:31:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:31:45.554+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:31:45.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:31:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:31:45.554+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:31:46.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:31:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:31:46.205+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:31:46.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:31:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:31:46.205+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:31:46.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:31:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:31:46.205+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:31:46.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:31:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:31:46.251+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:31:46.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:31:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:31:46.251+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:31:46.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:31:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:31:46.251+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:31:46.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:31:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:31:46.251+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:31:46.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:31:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:31:46.552+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:31:46.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:31:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:31:46.552+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:31:46.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:31:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:31:46.552+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:31:47.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:31:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:31:47.169+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:31:47.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:31:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:31:47.169+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:31:47.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:31:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:31:47.169+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:31:47.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:31:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:31:47.275+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:31:47.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:31:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:31:47.275+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:31:47.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:31:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:31:47.275+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:31:47.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:31:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:31:47.275+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:31:47.992 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:31:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:31:47.554+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 27568 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:31:47.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:31:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:31:47.508+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:31:47.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:31:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:31:47.508+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:31:47.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:31:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:31:47.508+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:31:48.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:31:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:31:48.324+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:31:48.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:31:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:31:48.324+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:31:48.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:31:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:31:48.324+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:31:48.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:31:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:31:48.324+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:31:48.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:31:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:31:48.211+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:31:48.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:31:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:31:48.211+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:31:48.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:31:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:31:48.211+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:31:48.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:31:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:31:48.515+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:31:48.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:31:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:31:48.515+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:31:48.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:31:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:31:48.515+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:31:49.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:31:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:31:49.277+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:31:49.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:31:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:31:49.277+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:31:49.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:31:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:31:49.277+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:31:49.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:31:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:31:49.277+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:31:49.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:31:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:31:49.179+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:31:49.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:31:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:31:49.179+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:31:49.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:31:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:31:49.179+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:31:49.886 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:31:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:31:49.492+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:31:49.886 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:31:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:31:49.492+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:31:49.886 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:31:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:31:49.492+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:31:50.487 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:31:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:31:50.169+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:31:50.488 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:31:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:31:50.169+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:31:50.488 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:31:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:31:50.169+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:31:50.488 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:31:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:31:50.254+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:31:50.488 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:31:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:31:50.254+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:31:50.488 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:31:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:31:50.254+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:31:50.488 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:31:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:31:50.254+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:31:50.742 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:31:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:31:50.484+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:31:50.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:31:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:31:50.484+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:31:50.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:31:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:31:50.484+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:31:51.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:31:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:31:51.137+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:31:51.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:31:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:31:51.137+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:31:51.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:31:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:31:51.137+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:31:51.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:31:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:31:51.248+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:31:51.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:31:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:31:51.248+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:31:51.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:31:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:31:51.248+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:31:51.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:31:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:31:51.248+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:31:51.773 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:31:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:31:51.515+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:31:51.773 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:31:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:31:51.515+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:31:51.773 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:31:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:31:51.515+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:31:52.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:31:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:31:52.115+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:31:52.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:31:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:31:52.115+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:31:52.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:31:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:31:52.115+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:31:52.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:31:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:31:52.221+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:31:52.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:31:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:31:52.221+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:31:52.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:31:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:31:52.221+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:31:52.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:31:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:31:52.221+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:31:52.993 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:31:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:31:52.558+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 27639 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:31:52.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:31:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:31:52.516+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:31:52.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:31:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:31:52.516+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:31:52.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:31:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:31:52.516+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:31:53.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:31:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:31:53.159+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:31:53.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:31:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:31:53.159+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:31:53.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:31:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:31:53.159+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:31:53.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:31:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:31:53.210+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:31:53.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:31:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:31:53.210+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:31:53.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:31:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:31:53.210+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:31:53.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:31:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:31:53.210+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:31:53.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:31:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:31:53.500+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:31:53.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:31:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:31:53.500+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:31:53.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:31:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:31:53.500+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:31:54.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:31:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:31:54.143+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:31:54.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:31:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:31:54.143+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:31:54.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:31:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:31:54.143+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:31:54.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:31:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:31:54.255+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:31:54.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:31:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:31:54.255+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:31:54.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:31:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:31:54.255+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:31:54.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:31:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:31:54.255+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:31:54.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:31:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:31:54.519+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:31:54.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:31:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:31:54.519+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:31:54.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:31:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:31:54.519+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:31:55.479 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:31:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:31:55.246+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:31:55.480 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:31:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:31:55.246+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:31:55.480 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:31:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:31:55.246+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:31:55.480 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:31:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:31:55.246+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:31:55.480 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:31:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:31:55.122+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:31:55.480 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:31:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:31:55.122+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:31:55.480 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:31:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:31:55.122+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:31:55.742 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:31:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:31:55.477+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:31:55.742 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:31:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:31:55.477+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:31:55.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:31:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:31:55.477+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:31:56.466 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:31:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:31:56.200+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:31:56.466 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:31:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:31:56.200+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:31:56.466 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:31:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:31:56.200+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:31:56.467 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:31:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:31:56.200+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:31:56.467 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:31:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:31:56.167+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:31:56.467 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:31:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:31:56.167+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:31:56.467 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:31:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:31:56.167+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:31:56.742 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:31:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:31:56.464+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:31:56.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:31:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:31:56.464+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:31:56.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:31:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:31:56.464+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:31:57.432 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:31:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:31:57.167+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:31:57.432 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:31:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:31:57.167+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:31:57.432 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:31:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:31:57.167+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:31:57.432 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:31:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:31:57.167+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:31:57.433 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:31:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:31:57.201+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:31:57.433 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:31:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:31:57.201+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:31:57.433 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:31:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:31:57.201+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:31:57.689 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:31:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:31:57.561+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 27710 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:31:57.690 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:31:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:31:57.428+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:31:57.690 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:31:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:31:57.429+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:31:57.690 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:31:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:31:57.429+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:31:58.432 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:31:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:31:58.224+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:31:58.433 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:31:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:31:58.224+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:31:58.433 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:31:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:31:58.224+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:31:58.433 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:31:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:31:58.126+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:31:58.433 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:31:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:31:58.126+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:31:58.433 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:31:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:31:58.126+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:31:58.433 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:31:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:31:58.126+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:31:58.742 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:31:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:31:58.430+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:31:58.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:31:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:31:58.430+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:31:58.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:31:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:31:58.430+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:31:59.418 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:31:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:31:59.204+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:31:59.418 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:31:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:31:59.204+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:31:59.419 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:31:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:31:59.204+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:31:59.419 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:31:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:31:59.126+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:31:59.419 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:31:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:31:59.126+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:31:59.419 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:31:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:31:59.126+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:31:59.419 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:31:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:31:59.126+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:31:59.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:31:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:31:59.415+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:31:59.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:31:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:31:59.415+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:31:59.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:31:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:31:59.415+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:32:00.446 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:32:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:32:00.155+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:32:00.446 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:32:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:32:00.155+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:32:00.446 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:32:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:32:00.155+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:32:00.446 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:32:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:32:00.155+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:32:00.446 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:32:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:32:00.202+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:32:00.447 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:32:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:32:00.202+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:32:00.447 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:32:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:32:00.202+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:32:00.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:32:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:32:00.443+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:32:00.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:32:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:32:00.443+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:32:00.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:32:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:32:00.443+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:32:01.421 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:32:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:32:01.181+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:32:01.422 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:32:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:32:01.181+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:32:01.422 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:32:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:32:01.181+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:32:01.422 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:32:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:32:01.134+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:32:01.422 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:32:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:32:01.134+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:32:01.423 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:32:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:32:01.134+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:32:01.423 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:32:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:32:01.134+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:32:01.742 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:32:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:32:01.419+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:32:01.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:32:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:32:01.419+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:32:01.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:32:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:32:01.419+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:32:02.386 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:32:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:32:02.164+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:32:02.387 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:32:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:32:02.164+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:32:02.387 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:32:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:32:02.164+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:32:02.387 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:32:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:32:02.115+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:32:02.387 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:32:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:32:02.115+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:32:02.387 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:32:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:32:02.115+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:32:02.387 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:32:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:32:02.115+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:32:02.692 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:32:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:32:02.565+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 27775 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:32:02.692 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:32:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:32:02.384+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:32:02.692 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:32:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:32:02.384+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:32:02.693 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:32:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:32:02.384+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:32:03.405 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:32:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:32:03.206+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:32:03.406 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:32:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:32:03.206+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:32:03.406 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:32:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:32:03.206+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:32:03.406 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:32:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:32:03.123+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:32:03.406 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:32:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:32:03.123+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:32:03.406 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:32:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:32:03.123+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:32:03.406 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:32:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:32:03.123+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:32:03.742 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:32:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:32:03.402+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:32:03.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:32:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:32:03.402+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:32:03.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:32:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:32:03.402+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:32:04.454 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:32:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:32:04.251+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:32:04.454 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:32:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:32:04.251+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:32:04.454 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:32:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:32:04.251+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:32:04.454 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:32:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:32:04.094+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:32:04.455 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:32:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:32:04.094+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:32:04.455 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:32:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:32:04.094+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:32:04.455 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:32:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:32:04.094+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:32:04.742 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:32:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:32:04.451+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:32:04.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:32:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:32:04.451+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:32:04.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:32:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:32:04.451+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:32:05.469 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:32:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:32:05.276+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:32:05.469 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:32:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:32:05.276+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:32:05.469 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:32:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:32:05.276+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:32:05.470 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:32:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:32:05.122+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:32:05.470 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:32:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:32:05.122+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:32:05.470 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:32:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:32:05.122+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:32:05.470 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:32:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:32:05.122+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:32:05.742 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:32:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:32:05.466+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:32:05.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:32:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:32:05.466+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:32:05.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:32:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:32:05.466+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:32:06.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:32:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:32:06.257+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:32:06.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:32:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:32:06.257+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:32:06.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:32:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:32:06.257+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:32:06.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:32:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:32:06.120+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:32:06.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:32:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:32:06.120+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:32:06.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:32:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:32:06.120+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:32:06.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:32:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:32:06.120+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:32:06.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:32:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:32:06.505+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:32:06.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:32:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:32:06.505+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:32:06.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:32:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:32:06.505+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:32:07.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:32:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:32:07.260+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:32:07.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:32:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:32:07.260+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:32:07.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:32:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:32:07.260+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:32:07.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:32:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:32:07.113+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:32:07.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:32:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:32:07.113+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:32:07.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:32:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:32:07.113+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:32:07.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:32:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:32:07.113+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:32:07.993 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:32:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:32:07.569+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 27846 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:32:07.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:32:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:32:07.547+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:32:07.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:32:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:32:07.547+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:32:07.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:32:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:32:07.547+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:32:08.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:32:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:32:08.238+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:32:08.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:32:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:32:08.238+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:32:08.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:32:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:32:08.238+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:32:08.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:32:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:32:08.093+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:32:08.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:32:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:32:08.093+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:32:08.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:32:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:32:08.093+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:32:08.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:32:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:32:08.093+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:32:08.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:32:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:32:08.524+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:32:08.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:32:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:32:08.524+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:32:08.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:32:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:32:08.524+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:32:09.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:32:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:32:09.277+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:32:09.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:32:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:32:09.277+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:32:09.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:32:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:32:09.277+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:32:09.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:32:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:32:09.071+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:32:09.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:32:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:32:09.071+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:32:09.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:32:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:32:09.071+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:32:09.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:32:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:32:09.071+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:32:09.886 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:32:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:32:09.496+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:32:09.887 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:32:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:32:09.496+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:32:09.887 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:32:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:32:09.496+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:32:10.240 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:32:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:32:10.083+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:32:10.240 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:32:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:32:10.083+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:32:10.240 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:32:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:32:10.083+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:32:10.240 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:32:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:32:10.083+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:32:10.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:32:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:32:10.455+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:32:10.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:32:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:32:10.455+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:32:10.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:32:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:32:10.455+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:32:10.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:32:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:32:10.302+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:32:10.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:32:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:32:10.302+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:32:10.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:32:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:32:10.302+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:32:11.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:32:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:32:11.291+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:32:11.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:32:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:32:11.291+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:32:11.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:32:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:32:11.291+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:32:11.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:32:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:32:11.102+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:32:11.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:32:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:32:11.102+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:32:11.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:32:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:32:11.102+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:32:11.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:32:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:32:11.102+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:32:11.774 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:32:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:32:11.496+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:32:11.775 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:32:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:32:11.496+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:32:11.775 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:32:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:32:11.496+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:32:12.327 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:32:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:32:12.324+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:32:12.328 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:32:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:32:12.324+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:32:12.328 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:32:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:32:12.324+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:32:12.328 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:32:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:32:12.073+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:32:12.328 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:32:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:32:12.073+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:32:12.328 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:32:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:32:12.073+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:32:12.328 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:32:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:32:12.074+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:32:12.586 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:32:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:32:12.573+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 27917 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:32:12.587 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:32:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:32:12.504+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:32:12.587 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:32:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:32:12.504+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:32:12.587 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:32:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:32:12.504+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:32:13.364 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:32:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:32:13.109+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:32:13.365 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:32:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:32:13.109+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:32:13.365 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:32:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:32:13.109+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:32:13.365 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:32:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:32:13.109+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:32:13.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:32:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:32:13.486+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:32:13.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:32:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:32:13.486+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:32:13.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:32:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:32:13.486+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:32:13.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:32:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:32:13.362+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:32:13.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:32:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:32:13.362+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:32:13.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:32:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:32:13.362+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:32:14.376 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:32:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:32:14.065+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:32:14.377 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:32:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:32:14.065+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:32:14.377 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:32:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:32:14.065+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:32:14.377 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:32:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:32:14.066+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:32:14.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:32:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:32:14.495+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:32:14.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:32:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:32:14.495+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:32:14.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:32:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:32:14.495+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:32:14.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:32:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:32:14.374+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:32:14.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:32:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:32:14.374+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:32:14.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:32:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:32:14.374+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:32:15.450 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:32:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:32:15.344+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:32:15.450 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:32:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:32:15.344+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:32:15.450 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:32:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:32:15.344+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:32:15.451 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:32:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:32:15.102+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:32:15.451 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:32:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:32:15.102+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:32:15.451 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:32:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:32:15.102+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:32:15.451 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:32:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:32:15.102+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:32:15.742 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:32:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:32:15.448+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:32:15.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:32:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:32:15.448+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:32:15.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:32:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:32:15.448+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:32:16.388 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:32:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:32:16.119+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:32:16.389 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:32:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:32:16.119+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:32:16.389 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:32:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:32:16.119+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:32:16.389 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:32:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:32:16.120+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:32:16.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:32:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:32:16.443+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:32:16.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:32:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:32:16.443+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:32:16.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:32:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:32:16.443+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:32:16.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:32:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:32:16.386+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:32:16.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:32:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:32:16.386+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:32:16.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:32:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:32:16.386+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:32:17.380 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:32:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:32:17.101+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:32:17.380 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:32:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:32:17.101+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:32:17.380 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:32:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:32:17.101+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:32:17.380 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:32:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:32:17.101+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:32:17.701 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:32:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:32:17.578+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 27988 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:32:17.702 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:32:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:32:17.378+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:32:17.702 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:32:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:32:17.378+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:32:17.702 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:32:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:32:17.378+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:32:17.702 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:32:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:32:17.419+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:32:17.702 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:32:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:32:17.419+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:32:17.702 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:32:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:32:17.419+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:32:18.384 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:32:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:32:18.063+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:32:18.384 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:32:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:32:18.063+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:32:18.384 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:32:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:32:18.063+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:32:18.384 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:32:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:32:18.063+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:32:18.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:32:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:32:18.382+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:32:18.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:32:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:32:18.382+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:32:18.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:32:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:32:18.382+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:32:18.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:32:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:32:18.393+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:32:18.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:32:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:32:18.393+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:32:18.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:32:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:32:18.393+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:32:19.344 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:32:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:32:19.075+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:32:19.344 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:32:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:32:19.075+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:32:19.344 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:32:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:32:19.075+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:32:19.344 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:32:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:32:19.075+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:32:19.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:32:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:32:19.341+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:32:19.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:32:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:32:19.341+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:32:19.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:32:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:32:19.341+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:32:19.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:32:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:32:19.364+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:32:19.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:32:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:32:19.364+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:32:19.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:32:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:32:19.364+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:32:20.158 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:32:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:32:20.093+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:32:20.159 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:32:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:32:20.093+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:32:20.159 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:32:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:32:20.093+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:32:20.159 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:32:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:32:20.093+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:32:20.492 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:32:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:32:20.308+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:32:20.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:32:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:32:20.308+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:32:20.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:32:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:32:20.308+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:32:20.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:32:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:32:20.332+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:32:20.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:32:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:32:20.332+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:32:20.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:32:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:32:20.332+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:32:21.384 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:32:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:32:21.302+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:32:21.384 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:32:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:32:21.303+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:32:21.384 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:32:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:32:21.303+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:32:21.384 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:32:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:32:21.060+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:32:21.385 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:32:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:32:21.060+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:32:21.385 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:32:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:32:21.060+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:32:21.385 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:32:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:32:21.060+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:32:21.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:32:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:32:21.382+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:32:21.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:32:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:32:21.382+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:32:21.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:32:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:32:21.382+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:32:22.317 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:32:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:32:22.029+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:32:22.318 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:32:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:32:22.029+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:32:22.318 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:32:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:32:22.029+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:32:22.318 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:32:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:32:22.029+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:32:22.584 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:32:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:32:22.316+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:32:22.585 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:32:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:32:22.316+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:32:22.585 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:32:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:32:22.316+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:32:22.585 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:32:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:32:22.406+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:32:22.585 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:32:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:32:22.406+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:32:22.585 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:32:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:32:22.406+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:32:22.986 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:32:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:32:22.582+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 28061 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:32:23.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:32:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:32:22.984+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:32:23.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:32:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:32:22.984+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:32:23.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:32:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:32:22.984+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:32:23.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:32:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:32:22.984+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:32:23.742 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:32:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:32:23.338+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:32:23.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:32:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:32:23.338+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:32:23.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:32:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:32:23.338+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:32:23.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:32:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:32:23.405+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:32:23.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:32:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:32:23.405+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:32:23.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:32:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:32:23.405+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:32:24.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:32:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:32:23.981+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:32:24.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:32:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:32:23.981+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:32:24.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:32:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:32:23.981+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:32:24.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:32:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:32:23.981+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:32:24.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:32:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:32:24.333+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:32:24.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:32:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:32:24.333+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:32:24.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:32:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:32:24.333+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:32:24.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:32:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:32:24.369+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:32:24.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:32:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:32:24.369+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:32:24.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:32:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:32:24.369+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:32:25.328 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:32:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:32:25.006+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:32:25.329 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:32:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:32:25.006+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:32:25.329 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:32:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:32:25.006+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:32:25.329 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:32:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:32:25.006+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:32:25.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:32:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:32:25.326+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:32:25.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:32:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:32:25.326+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:32:25.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:32:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:32:25.326+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:32:25.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:32:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:32:25.364+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:32:25.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:32:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:32:25.364+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:32:25.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:32:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:32:25.364+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:32:26.326 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:32:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:32:26.014+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:32:26.326 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:32:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:32:26.014+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:32:26.326 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:32:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:32:26.014+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:32:26.326 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:32:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:32:26.014+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:32:26.742 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:32:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:32:26.324+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:32:26.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:32:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:32:26.324+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:32:26.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:32:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:32:26.324+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:32:26.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:32:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:32:26.368+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:32:26.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:32:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:32:26.368+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:32:26.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:32:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:32:26.368+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:32:27.367 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:32:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:32:27.288+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:32:27.367 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:32:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:32:27.288+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:32:27.367 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:32:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:32:27.288+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:32:27.368 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:32:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:32:27.046+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:32:27.368 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:32:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:32:27.046+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:32:27.368 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:32:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:32:27.046+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:32:27.368 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:32:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:32:27.046+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:32:27.706 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:32:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:32:27.586+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 28132 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:32:27.706 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:32:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:32:27.364+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:32:27.707 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:32:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:32:27.364+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:32:27.707 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:32:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:32:27.365+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:32:28.331 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:32:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:32:28.278+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:32:28.331 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:32:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:32:28.278+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:32:28.331 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:32:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:32:28.278+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:32:28.331 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:32:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:32:28.068+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:32:28.332 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:32:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:32:28.068+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:32:28.332 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:32:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:32:28.068+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:32:28.332 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:32:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:32:28.068+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:32:28.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:32:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:32:28.329+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:32:28.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:32:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:32:28.329+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:32:28.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:32:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:32:28.329+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:32:29.325 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:32:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:32:29.039+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:32:29.325 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:32:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:32:29.039+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:32:29.326 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:32:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:32:29.039+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:32:29.326 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:32:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:32:29.039+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:32:29.326 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:32:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:32:29.277+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:32:29.326 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:32:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:32:29.277+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:32:29.326 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:32:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:32:29.277+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:32:29.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:32:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:32:29.323+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:32:29.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:32:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:32:29.323+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:32:29.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:32:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:32:29.323+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:32:30.241 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:32:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:32:30.040+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:32:30.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:32:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:32:30.040+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:32:30.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:32:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:32:30.040+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:32:30.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:32:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:32:30.040+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:32:30.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:32:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:32:30.281+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:32:30.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:32:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:32:30.281+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:32:30.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:32:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:32:30.281+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:32:30.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:32:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:32:30.300+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:32:30.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:32:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:32:30.300+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:32:30.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:32:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:32:30.300+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:32:31.263 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:32:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:32:31.066+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:32:31.264 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:32:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:32:31.066+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:32:31.264 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:32:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:32:31.066+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:32:31.264 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:32:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:32:31.066+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:32:31.545 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:32:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:32:31.343+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:32:31.545 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:32:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:32:31.343+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:32:31.545 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:32:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:32:31.343+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:32:31.545 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:32:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:32:31.261+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:32:31.545 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:32:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:32:31.261+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:32:31.546 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:32:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:32:31.261+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:32:32.259 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:32:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:32:32.021+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:32:32.260 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:32:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:32:32.021+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:32:32.260 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:32:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:32:32.021+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:32:32.260 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:32:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:32:32.021+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:32:32.592 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:32:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:32:32.334+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:32:32.592 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:32:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:32:32.334+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:32:32.593 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:32:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:32:32.334+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:32:32.593 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:32:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:32:32.258+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:32:32.593 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:32:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:32:32.258+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:32:32.593 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:32:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:32:32.258+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:32:32.992 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:32:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:32:32.590+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 28205 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:32:33.344 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:32:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:32:33.233+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:32:33.345 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:32:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:32:33.233+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:32:33.345 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:32:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:32:33.233+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:32:33.345 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:32:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:32:33.060+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:32:33.345 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:32:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:32:33.060+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:32:33.345 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:32:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:32:33.060+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:32:33.346 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:32:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:32:33.060+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:32:33.742 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:32:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:32:33.342+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:32:33.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:32:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:32:33.342+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:32:33.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:32:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:32:33.342+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:32:34.347 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:32:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:32:34.095+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:32:34.347 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:32:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:32:34.095+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:32:34.347 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:32:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:32:34.095+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:32:34.347 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:32:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:32:34.095+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:32:34.348 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:32:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:32:34.204+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:32:34.348 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:32:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:32:34.204+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:32:34.348 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:32:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:32:34.204+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:32:34.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:32:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:32:34.344+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:32:34.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:32:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:32:34.344+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:32:34.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:32:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:32:34.344+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:32:35.355 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:32:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:32:35.249+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:32:35.356 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:32:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:32:35.249+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:32:35.356 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:32:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:32:35.249+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:32:35.356 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:32:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:32:35.044+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:32:35.356 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:32:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:32:35.044+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:32:35.356 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:32:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:32:35.044+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:32:35.357 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:32:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:32:35.044+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:32:35.742 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:32:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:32:35.353+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:32:35.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:32:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:32:35.353+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:32:35.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:32:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:32:35.353+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:32:36.345 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:32:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:32:36.224+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:32:36.345 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:32:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:32:36.224+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:32:36.345 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:32:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:32:36.224+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:32:36.346 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:32:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:32:36.025+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:32:36.346 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:32:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:32:36.025+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:32:36.346 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:32:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:32:36.025+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:32:36.346 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:32:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:32:36.025+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:32:36.742 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:32:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:32:36.342+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:32:36.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:32:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:32:36.342+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:32:36.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:32:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:32:36.342+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:32:37.360 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:32:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:32:37.017+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:32:37.360 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:32:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:32:37.017+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:32:37.360 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:32:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:32:37.017+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:32:37.360 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:32:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:32:37.017+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:32:37.361 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:32:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:32:37.230+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:32:37.361 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:32:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:32:37.230+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:32:37.361 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:32:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:32:37.230+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:32:37.619 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:32:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:32:37.593+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 28276 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:32:37.620 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:32:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:32:37.357+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:32:37.620 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:32:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:32:37.357+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:32:37.620 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:32:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:32:37.357+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:32:38.392 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:32:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:32:38.206+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:32:38.393 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:32:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:32:38.206+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:32:38.393 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:32:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:32:38.206+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:32:38.393 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:32:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:32:38.056+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:32:38.393 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:32:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:32:38.056+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:32:38.393 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:32:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:32:38.056+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:32:38.394 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:32:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:32:38.056+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:32:38.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:32:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:32:38.389+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:32:38.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:32:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:32:38.389+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:32:38.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:32:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:32:38.389+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:32:39.363 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:32:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:32:39.181+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:32:39.363 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:32:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:32:39.181+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:32:39.363 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:32:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:32:39.181+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:32:39.363 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:32:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:32:39.038+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:32:39.364 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:32:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:32:39.038+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:32:39.364 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:32:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:32:39.038+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:32:39.364 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:32:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:32:39.038+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:32:39.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:32:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:32:39.360+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:32:39.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:32:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:32:39.360+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:32:39.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:32:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:32:39.360+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:32:40.196 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:32:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:32:40.081+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:32:40.196 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:32:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:32:40.081+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:32:40.196 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:32:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:32:40.081+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:32:40.196 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:32:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:32:40.081+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:32:40.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:32:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:32:40.341+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:32:40.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:32:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:32:40.341+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:32:40.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:32:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:32:40.341+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:32:40.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:32:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:32:40.192+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:32:40.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:32:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:32:40.192+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:32:40.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:32:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:32:40.192+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:32:41.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:32:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:32:41.291+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:32:41.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:32:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:32:41.291+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:32:41.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:32:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:32:41.291+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:32:41.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:32:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:32:41.185+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:32:41.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:32:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:32:41.185+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:32:41.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:32:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:32:41.185+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:32:41.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:32:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:32:41.059+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:32:41.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:32:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:32:41.059+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:32:41.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:32:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:32:41.059+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:32:41.495 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:32:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:32:41.060+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:32:42.321 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:32:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:32:42.159+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:32:42.321 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:32:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:32:42.159+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:32:42.321 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:32:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:32:42.159+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:32:42.321 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:32:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:32:42.046+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:32:42.322 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:32:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:32:42.046+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:32:42.322 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:32:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:32:42.046+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:32:42.322 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:32:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:32:42.046+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:32:42.601 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:32:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:32:42.337+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:32:42.601 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:32:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:32:42.337+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:32:42.601 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:32:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:32:42.337+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:32:42.992 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:32:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:32:42.597+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 28349 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:32:43.328 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:32:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:32:43.014+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:32:43.328 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:32:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:32:43.014+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:32:43.328 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:32:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:32:43.014+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:32:43.328 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:32:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:32:43.014+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:32:43.329 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:32:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:32:43.110+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:32:43.329 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:32:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:32:43.110+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:32:43.329 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:32:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:32:43.110+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:32:43.742 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:32:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:32:43.325+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:32:43.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:32:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:32:43.325+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:32:43.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:32:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:32:43.325+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:32:44.281 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:32:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:32:44.069+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:32:44.281 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:32:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:32:44.069+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:32:44.281 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:32:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:32:44.069+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:32:44.282 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:32:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:32:44.007+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:32:44.282 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:32:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:32:44.007+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:32:44.282 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:32:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:32:44.007+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:32:44.282 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:32:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:32:44.007+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:32:44.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:32:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:32:44.290+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:32:44.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:32:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:32:44.290+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:32:44.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:32:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:32:44.290+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:32:45.298 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:32:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:32:45.030+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:32:45.298 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:32:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:32:45.030+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:32:45.298 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:32:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:32:45.030+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:32:45.299 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:32:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:32:45.016+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:32:45.299 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:32:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:32:45.016+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:32:45.299 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:32:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:32:45.016+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:32:45.299 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:32:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:32:45.016+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:32:45.742 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:32:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:32:45.295+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:32:45.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:32:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:32:45.295+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:32:45.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:32:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:32:45.295+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:32:46.312 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:32:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:32:46.036+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:32:46.312 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:32:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:32:46.036+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:32:46.312 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:32:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:32:46.036+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:32:46.313 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:32:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:32:46.033+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:32:46.313 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:32:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:32:46.033+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:32:46.313 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:32:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:32:46.033+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:32:46.313 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:32:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:32:46.033+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:32:46.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:32:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:32:46.309+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:32:46.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:32:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:32:46.309+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:32:46.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:32:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:32:46.309+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:32:47.288 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:32:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:32:47.024+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:32:47.288 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:32:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:32:47.024+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:32:47.288 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:32:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:32:47.024+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:32:47.288 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:32:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:32:46.996+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:32:47.288 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:32:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:32:46.996+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:32:47.289 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:32:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:32:46.996+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:32:47.289 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:32:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:32:46.996+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:32:47.604 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:32:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:32:47.285+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:32:47.604 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:32:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:32:47.285+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:32:47.605 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:32:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:32:47.285+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:32:47.980 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:32:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:32:47.601+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 28420 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:32:48.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:32:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:32:47.976+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:32:48.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:32:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:32:47.976+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:32:48.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:32:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:32:47.976+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:32:48.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:32:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:32:47.977+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:32:48.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:32:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:32:48.067+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:32:48.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:32:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:32:48.067+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:32:48.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:32:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:32:48.067+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:32:48.742 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:32:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:32:48.321+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:32:48.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:32:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:32:48.321+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:32:48.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:32:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:32:48.321+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:32:49.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:32:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:32:48.959+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:32:49.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:32:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:32:48.959+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:32:49.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:32:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:32:48.959+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:32:49.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:32:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:32:48.959+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:32:49.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:32:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:32:49.091+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:32:49.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:32:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:32:49.091+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:32:49.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:32:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:32:49.091+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:32:49.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:32:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:32:49.318+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:32:49.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:32:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:32:49.318+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:32:49.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:32:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:32:49.318+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:32:50.138 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:32:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:32:49.982+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:32:50.138 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:32:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:32:49.982+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:32:50.138 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:32:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:32:49.982+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:32:50.138 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:32:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:32:49.982+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:32:50.425 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:32:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:32:50.301+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:32:50.426 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:32:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:32:50.301+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:32:50.426 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:32:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:32:50.301+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:32:50.426 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:32:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:32:50.135+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:32:50.426 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:32:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:32:50.135+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:32:50.427 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:32:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:32:50.135+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:32:51.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:32:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:32:51.097+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:32:51.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:32:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:32:51.097+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:32:51.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:32:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:32:51.097+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:32:51.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:32:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:32:50.958+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:32:51.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:32:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:32:50.958+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:32:51.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:32:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:32:50.958+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:32:51.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:32:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:32:50.958+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:32:51.742 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:32:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:32:51.339+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:32:51.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:32:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:32:51.339+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:32:51.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:32:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:32:51.339+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:32:52.121 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:32:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:32:51.982+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:32:52.122 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:32:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:32:51.982+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:32:52.122 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:32:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:32:51.982+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:32:52.122 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:32:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:32:51.982+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:32:52.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:32:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:32:52.364+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:32:52.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:32:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:32:52.364+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:32:52.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:32:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:32:52.364+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:32:52.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:32:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:32:52.119+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:32:52.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:32:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:32:52.119+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:32:52.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:32:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:32:52.119+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:32:52.993 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:32:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:32:52.606+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 28493 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:32:53.357 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:32:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:32:53.154+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:32:53.358 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:32:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:32:53.154+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:32:53.358 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:32:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:32:53.154+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:32:53.358 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:32:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:32:53.032+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:32:53.359 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:32:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:32:53.032+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:32:53.359 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:32:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:32:53.032+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:32:53.359 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:32:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:32:53.032+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:32:53.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:32:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:32:53.354+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:32:53.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:32:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:32:53.354+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:32:53.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:32:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:32:53.354+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:32:54.357 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:32:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:32:54.123+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:32:54.357 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:32:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:32:54.123+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:32:54.357 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:32:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:32:54.123+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:32:54.358 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:32:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:32:53.991+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:32:54.358 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:32:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:32:53.991+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:32:54.358 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:32:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:32:53.991+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:32:54.358 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:32:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:32:53.991+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:32:54.742 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:32:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:32:54.354+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:32:54.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:32:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:32:54.354+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:32:54.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:32:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:32:54.354+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:32:55.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:32:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:32:55.089+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:32:55.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:32:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:32:55.089+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:32:55.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:32:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:32:55.089+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:32:55.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:32:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:32:54.970+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:32:55.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:32:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:32:54.970+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:32:55.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:32:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:32:54.970+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:32:55.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:32:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:32:54.970+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:32:55.742 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:32:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:32:55.365+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:32:55.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:32:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:32:55.365+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:32:55.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:32:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:32:55.365+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:32:56.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:32:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:32:56.108+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:32:56.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:32:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:32:56.108+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:32:56.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:32:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:32:56.108+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:32:56.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:32:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:32:55.948+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:32:56.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:32:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:32:55.948+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:32:56.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:32:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:32:55.948+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:32:56.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:32:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:32:55.948+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:32:56.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:32:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:32:56.339+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:32:56.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:32:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:32:56.339+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:32:56.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:32:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:32:56.339+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:32:57.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:32:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:32:57.087+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:32:57.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:32:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:32:57.087+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:32:57.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:32:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:32:57.087+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:32:57.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:32:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:32:56.900+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:32:57.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:32:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:32:56.900+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:32:57.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:32:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:32:56.900+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:32:57.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:32:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:32:56.900+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:32:57.613 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:32:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:32:57.358+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:32:57.613 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:32:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:32:57.358+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:32:57.614 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:32:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:32:57.358+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:32:57.893 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:32:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:32:57.610+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 28564 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:32:58.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:32:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:32:58.083+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:32:58.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:32:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:32:58.083+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:32:58.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:32:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:32:58.083+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:32:58.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:32:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:32:57.891+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:32:58.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:32:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:32:57.891+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:32:58.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:32:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:32:57.891+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:32:58.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:32:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:32:57.891+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:32:58.742 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:32:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:32:58.395+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:32:58.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:32:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:32:58.395+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:32:58.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:32:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:32:58.395+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:32:59.128 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:32:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:32:58.870+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:32:59.128 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:32:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:32:58.870+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:32:59.128 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:32:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:32:58.870+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:32:59.128 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:32:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:32:58.870+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:32:59.398 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:32:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:32:59.126+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:32:59.399 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:32:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:32:59.126+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:32:59.399 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:32:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:32:59.126+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:32:59.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:32:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:32:59.396+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:32:59.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:32:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:32:59.396+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:32:59.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:32:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:32:59.396+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:33:00.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:32:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:32:59.889+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:33:00.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:32:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:32:59.889+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:33:00.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:32:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:32:59.889+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:33:00.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:32:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:32:59.890+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:33:00.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:33:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:33:00.108+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:33:00.245 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:33:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:33:00.108+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:33:00.245 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:33:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:33:00.108+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:33:00.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:33:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:33:00.413+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:33:00.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:33:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:33:00.413+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:33:00.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:33:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:33:00.413+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:33:01.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:33:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:33:01.088+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:33:01.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:33:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:33:01.088+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:33:01.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:33:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:33:01.088+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:33:01.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:33:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:33:00.846+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:33:01.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:33:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:33:00.846+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:33:01.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:33:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:33:00.846+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:33:01.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:33:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:33:00.846+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:33:01.742 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:33:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:33:01.414+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:33:01.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:33:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:33:01.414+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:33:01.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:33:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:33:01.414+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:33:02.055 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:33:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:33:01.821+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:33:02.056 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:33:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:33:01.821+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:33:02.056 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:33:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:33:01.821+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:33:02.056 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:33:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:33:01.821+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:33:02.382 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:33:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:33:02.053+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:33:02.382 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:33:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:33:02.053+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:33:02.383 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:33:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:33:02.053+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:33:02.637 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:33:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:33:02.614+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 28637 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:33:02.637 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:33:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:33:02.380+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:33:02.638 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:33:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:33:02.380+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:33:02.638 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:33:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:33:02.380+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:33:02.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:33:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:33:02.836+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:33:02.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:33:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:33:02.836+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:33:02.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:33:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:33:02.836+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:33:02.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:33:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:33:02.836+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:33:03.408 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:33:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:33:03.077+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:33:03.408 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:33:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:33:03.077+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:33:03.409 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:33:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:33:03.077+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:33:03.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:33:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:33:03.406+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:33:03.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:33:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:33:03.406+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:33:03.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:33:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:33:03.406+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:33:04.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:33:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:33:03.850+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:33:04.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:33:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:33:03.850+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:33:04.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:33:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:33:03.850+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:33:04.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:33:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:33:03.850+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:33:04.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:33:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:33:04.029+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:33:04.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:33:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:33:04.029+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:33:04.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:33:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:33:04.029+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:33:04.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:33:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:33:04.374+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:33:04.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:33:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:33:04.374+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:33:04.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:33:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:33:04.374+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:33:05.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:33:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:33:04.890+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:33:05.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:33:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:33:04.890+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:33:05.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:33:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:33:04.890+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:33:05.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:33:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:33:04.890+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:33:05.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:33:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:33:05.041+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:33:05.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:33:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:33:05.041+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:33:05.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:33:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:33:05.041+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:33:05.742 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:33:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:33:05.377+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:33:05.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:33:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:33:05.377+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:33:05.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:33:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:33:05.377+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:33:06.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:33:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:33:05.874+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:33:06.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:33:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:33:05.874+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:33:06.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:33:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:33:05.874+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:33:06.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:33:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:33:05.874+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:33:06.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:33:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:33:06.010+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:33:06.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:33:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:33:06.010+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:33:06.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:33:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:33:06.010+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:33:06.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:33:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:33:06.403+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:33:06.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:33:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:33:06.403+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:33:06.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:33:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:33:06.403+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:33:07.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:33:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:33:06.828+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:33:07.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:33:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:33:06.828+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:33:07.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:33:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:33:06.828+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:33:07.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:33:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:33:06.828+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:33:07.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:33:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:33:06.991+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:33:07.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:33:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:33:06.991+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:33:07.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:33:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:33:06.991+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:33:07.742 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:33:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:33:07.620+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 28708 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:33:07.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:33:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:33:07.431+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:33:07.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:33:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:33:07.431+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:33:07.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:33:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:33:07.431+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:33:08.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:33:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:33:07.846+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:33:08.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:33:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:33:07.846+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:33:08.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:33:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:33:07.846+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:33:08.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:33:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:33:07.846+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:33:08.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:33:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:33:07.946+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:33:08.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:33:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:33:07.946+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:33:08.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:33:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:33:07.946+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:33:08.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:33:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:33:08.430+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:33:08.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:33:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:33:08.430+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:33:08.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:33:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:33:08.430+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:33:09.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:33:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:33:08.838+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:33:09.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:33:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:33:08.838+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:33:09.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:33:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:33:08.838+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:33:09.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:33:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:33:08.838+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:33:09.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:33:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:33:08.969+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:33:09.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:33:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:33:08.969+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:33:09.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:33:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:33:08.969+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:33:09.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:33:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:33:09.403+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:33:09.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:33:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:33:09.403+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:33:09.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:33:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:33:09.403+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:33:10.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:33:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:33:09.839+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:33:10.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:33:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:33:09.839+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:33:10.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:33:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:33:09.839+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:33:10.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:33:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:33:09.839+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:33:10.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:33:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:33:09.934+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:33:10.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:33:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:33:09.934+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:33:10.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:33:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:33:09.934+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:33:10.742 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:33:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:33:10.413+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:33:10.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:33:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:33:10.413+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:33:10.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:33:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:33:10.413+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:33:11.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:33:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:33:10.832+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:33:11.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:33:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:33:10.832+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:33:11.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:33:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:33:10.832+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:33:11.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:33:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:33:10.832+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:33:11.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:33:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:33:10.910+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:33:11.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:33:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:33:10.910+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:33:11.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:33:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:33:10.910+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:33:11.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:33:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:33:11.392+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:33:11.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:33:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:33:11.392+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:33:11.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:33:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:33:11.392+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:33:12.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:33:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:33:11.852+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:33:12.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:33:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:33:11.852+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:33:12.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:33:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:33:11.852+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:33:12.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:33:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:33:11.852+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:33:12.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:33:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:33:11.932+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:33:12.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:33:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:33:11.932+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:33:12.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:33:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:33:11.932+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:33:12.626 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:33:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:33:12.344+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:33:12.626 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:33:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:33:12.344+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:33:12.626 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:33:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:33:12.344+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:33:12.879 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:33:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:33:12.623+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 28781 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:33:13.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:33:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:33:12.877+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:33:13.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:33:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:33:12.877+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:33:13.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:33:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:33:12.877+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:33:13.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:33:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:33:12.877+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:33:13.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:33:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:33:12.971+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:33:13.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:33:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:33:12.971+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:33:13.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:33:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:33:12.971+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:33:13.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:33:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:33:13.302+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:33:13.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:33:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:33:13.302+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:33:13.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:33:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:33:13.302+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:33:14.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:33:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:33:13.934+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:33:14.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:33:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:33:13.934+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:33:14.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:33:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:33:13.934+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:33:14.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:33:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:33:13.855+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:33:14.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:33:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:33:13.855+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:33:14.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:33:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:33:13.855+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:33:14.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:33:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:33:13.855+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:33:14.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:33:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:33:14.262+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:33:14.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:33:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:33:14.262+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:33:14.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:33:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:33:14.262+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:33:15.219 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:33:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:33:14.974+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:33:15.219 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:33:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:33:14.974+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:33:15.219 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:33:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:33:14.974+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:33:15.220 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:33:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:33:14.880+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:33:15.220 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:33:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:33:14.880+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:33:15.220 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:33:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:33:14.880+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:33:15.220 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:33:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:33:14.880+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:33:15.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:33:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:33:15.217+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:33:15.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:33:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:33:15.217+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:33:15.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:33:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:33:15.217+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:33:16.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:33:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:33:15.939+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:33:16.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:33:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:33:15.939+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:33:16.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:33:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:33:15.939+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:33:16.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:33:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:33:15.835+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:33:16.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:33:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:33:15.835+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:33:16.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:33:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:33:15.835+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:33:16.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:33:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:33:15.835+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:33:16.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:33:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:33:16.246+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:33:16.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:33:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:33:16.246+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:33:16.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:33:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:33:16.246+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:33:17.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:33:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:33:16.986+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:33:17.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:33:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:33:16.986+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:33:17.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:33:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:33:16.986+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:33:17.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:33:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:33:16.865+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:33:17.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:33:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:33:16.865+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:33:17.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:33:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:33:16.865+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:33:17.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:33:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:33:16.865+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:33:17.631 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:33:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:33:17.265+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:33:17.631 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:33:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:33:17.265+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:33:17.631 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:33:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:33:17.265+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:33:17.963 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:33:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:33:17.627+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 28852 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:33:17.964 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:33:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:33:17.877+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:33:17.964 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:33:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:33:17.877+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:33:17.964 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:33:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:33:17.877+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:33:17.964 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:33:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:33:17.877+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:33:18.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:33:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:33:17.961+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:33:18.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:33:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:33:17.961+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:33:18.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:33:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:33:17.961+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:33:18.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:33:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:33:18.314+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:33:18.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:33:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:33:18.314+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:33:18.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:33:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:33:18.314+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:33:19.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:33:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:33:18.927+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:33:19.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:33:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:33:18.927+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:33:19.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:33:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:33:18.927+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:33:19.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:33:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:33:18.927+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:33:19.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:33:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:33:18.926+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:33:19.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:33:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:33:18.926+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:33:19.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:33:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:33:18.926+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:33:19.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:33:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:33:19.299+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:33:19.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:33:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:33:19.299+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:33:19.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:33:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:33:19.299+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:33:20.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:33:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:33:19.945+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:33:20.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:33:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:33:19.945+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:33:20.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:33:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:33:19.945+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:33:20.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:33:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:33:19.915+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:33:20.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:33:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:33:19.915+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:33:20.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:33:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:33:19.915+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:33:20.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:33:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:33:19.916+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:33:20.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:33:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:33:20.251+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:33:20.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:33:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:33:20.251+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:33:20.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:33:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:33:20.251+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:33:21.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:33:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:33:20.908+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:33:21.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:33:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:33:20.908+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:33:21.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:33:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:33:20.908+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:33:21.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:33:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:33:20.905+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:33:21.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:33:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:33:20.905+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:33:21.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:33:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:33:20.905+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:33:21.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:33:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:33:20.905+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:33:21.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:33:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:33:21.261+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:33:21.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:33:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:33:21.261+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:33:21.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:33:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:33:21.261+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:33:22.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:33:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:33:21.899+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:33:22.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:33:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:33:21.899+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:33:22.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:33:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:33:21.899+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:33:22.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:33:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:33:21.899+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:33:22.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:33:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:33:21.940+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:33:22.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:33:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:33:21.940+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:33:22.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:33:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:33:21.940+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:33:22.634 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:33:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:33:22.284+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:33:22.634 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:33:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:33:22.284+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:33:22.635 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:33:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:33:22.284+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:33:22.918 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:33:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:33:22.632+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 28925 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:33:23.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:33:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:33:22.990+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:33:23.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:33:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:33:22.990+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:33:23.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:33:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:33:22.990+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:33:23.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:33:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:33:22.916+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:33:23.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:33:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:33:22.916+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:33:23.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:33:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:33:22.916+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:33:23.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:33:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:33:22.916+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:33:23.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:33:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:33:23.333+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:33:23.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:33:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:33:23.333+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:33:23.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:33:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:33:23.333+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:33:24.217 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:33:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:33:23.978+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:33:24.218 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:33:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:33:23.978+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:33:24.218 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:33:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:33:23.978+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:33:24.218 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:33:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:33:23.880+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:33:24.218 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:33:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:33:23.880+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:33:24.218 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:33:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:33:23.880+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:33:24.218 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:33:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:33:23.880+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:33:24.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:33:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:33:24.345+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:33:24.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:33:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:33:24.345+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:33:24.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:33:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:33:24.345+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:33:25.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:33:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:33:24.951+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:33:25.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:33:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:33:24.951+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:33:25.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:33:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:33:24.951+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:33:25.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:33:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:33:24.908+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:33:25.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:33:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:33:24.908+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:33:25.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:33:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:33:24.908+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:33:25.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:33:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:33:24.909+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:33:25.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:33:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:33:25.343+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:33:25.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:33:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:33:25.343+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:33:25.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:33:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:33:25.343+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:33:26.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:33:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:33:25.932+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:33:26.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:33:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:33:25.932+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:33:26.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:33:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:33:25.932+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:33:26.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:33:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:33:25.946+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:33:26.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:33:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:33:25.946+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:33:26.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:33:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:33:25.946+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:33:26.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:33:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:33:25.946+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:33:26.742 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:33:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:33:26.325+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:33:26.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:33:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:33:26.325+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:33:26.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:33:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:33:26.325+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:33:27.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:33:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:33:26.915+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:33:27.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:33:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:33:26.915+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:33:27.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:33:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:33:26.915+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:33:27.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:33:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:33:26.982+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:33:27.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:33:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:33:26.982+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:33:27.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:33:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:33:26.982+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:33:27.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:33:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:33:26.982+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:33:27.638 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:33:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:33:27.280+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:33:27.638 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:33:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:33:27.280+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:33:27.639 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:33:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:33:27.280+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:33:27.993 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:33:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:33:27.635+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 28997 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:33:27.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:33:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:33:27.877+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:33:27.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:33:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:33:27.877+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:33:27.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:33:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:33:27.877+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:33:28.277 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:33:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:33:28.004+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:33:28.277 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:33:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:33:28.004+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:33:28.277 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:33:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:33:28.004+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:33:28.277 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:33:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:33:28.004+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:33:28.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:33:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:33:28.273+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:33:28.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:33:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:33:28.273+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:33:28.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:33:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:33:28.273+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:33:29.235 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:33:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:33:28.884+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:33:29.236 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:33:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:33:28.884+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:33:29.236 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:33:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:33:28.884+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:33:29.236 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:33:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:33:29.023+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:33:29.236 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:33:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:33:29.023+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:33:29.236 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:33:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:33:29.023+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:33:29.236 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:33:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:33:29.023+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:33:29.492 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:33:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:33:29.232+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:33:29.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:33:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:33:29.232+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:33:29.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:33:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:33:29.232+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:33:30.211 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:33:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:33:29.894+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:33:30.212 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:33:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:33:29.894+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:33:30.212 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:33:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:33:29.894+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:33:30.213 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:33:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:33:30.034+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:33:30.213 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:33:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:33:30.034+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:33:30.213 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:33:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:33:30.034+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:33:30.213 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:33:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:33:30.035+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:33:30.492 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:33:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:33:30.208+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:33:30.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:33:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:33:30.208+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:33:30.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:33:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:33:30.208+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:33:31.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:33:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:33:30.879+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:33:31.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:33:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:33:30.879+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:33:31.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:33:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:33:30.879+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:33:31.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:33:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:33:31.051+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:33:31.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:33:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:33:31.051+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:33:31.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:33:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:33:31.051+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:33:31.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:33:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:33:31.052+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:33:31.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:33:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:33:31.248+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:33:31.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:33:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:33:31.248+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:33:31.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:33:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:33:31.248+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:33:32.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:33:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:33:31.926+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:33:32.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:33:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:33:31.926+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:33:32.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:33:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:33:31.926+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:33:32.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:33:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:33:32.023+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:33:32.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:33:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:33:32.023+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:33:32.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:33:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:33:32.023+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:33:32.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:33:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:33:32.023+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:33:32.642 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:33:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:33:32.267+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:33:32.642 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:33:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:33:32.267+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:33:32.643 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:33:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:33:32.267+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:33:32.908 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:33:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:33:32.639+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 29070 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:33:33.234 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:33:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:33:33.066+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:33:33.234 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:33:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:33:33.066+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:33:33.234 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:33:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:33:33.066+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:33:33.234 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:33:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:33:33.066+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:33:33.235 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:33:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:33:32.905+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:33:33.235 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:33:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:33:32.905+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:33:33.235 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:33:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:33:32.905+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:33:33.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:33:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:33:33.230+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:33:33.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:33:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:33:33.230+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:33:33.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:33:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:33:33.230+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:33:34.205 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:33:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:33:34.051+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:33:34.205 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:33:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:33:34.051+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:33:34.205 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:33:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:33:34.051+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:33:34.205 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:33:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:33:34.051+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:33:34.206 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:33:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:33:33.927+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:33:34.206 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:33:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:33:33.927+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:33:34.206 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:33:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:33:33.927+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:33:34.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:33:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:33:34.202+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:33:34.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:33:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:33:34.202+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:33:34.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:33:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:33:34.202+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:33:35.215 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:33:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:33:34.897+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:33:35.215 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:33:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:33:34.897+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:33:35.215 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:33:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:33:34.897+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:33:35.215 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:33:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:33:35.086+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:33:35.216 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:33:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:33:35.086+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:33:35.216 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:33:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:33:35.086+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:33:35.216 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:33:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:33:35.086+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:33:35.492 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:33:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:33:35.212+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:33:35.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:33:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:33:35.212+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:33:35.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:33:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:33:35.212+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:33:36.113 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:33:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:33:35.851+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:33:36.113 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:33:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:33:35.851+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:33:36.114 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:33:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:33:35.851+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:33:36.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:33:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:33:36.225+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:33:36.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:33:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:33:36.225+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:33:36.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:33:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:33:36.225+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:33:36.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:33:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:33:36.111+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:33:36.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:33:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:33:36.111+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:33:36.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:33:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:33:36.111+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:33:36.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:33:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:33:36.111+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:33:37.135 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:33:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:33:36.854+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:33:37.135 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:33:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:33:36.854+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:33:37.136 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:33:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:33:36.854+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:33:37.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:33:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:33:37.224+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:33:37.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:33:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:33:37.224+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:33:37.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:33:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:33:37.224+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:33:37.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:33:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:33:37.133+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:33:37.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:33:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:33:37.133+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:33:37.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:33:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:33:37.133+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:33:37.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:33:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:33:37.133+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:33:37.993 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:33:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:33:37.644+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 29141 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:33:37.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:33:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:33:37.845+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:33:37.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:33:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:33:37.845+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:33:37.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:33:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:33:37.845+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:33:38.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:33:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:33:38.170+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:33:38.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:33:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:33:38.170+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:33:38.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:33:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:33:38.170+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:33:38.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:33:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:33:38.170+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:33:38.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:33:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:33:38.215+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:33:38.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:33:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:33:38.215+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:33:38.494 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:33:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:33:38.215+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:33:39.149 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:33:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:33:38.810+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:33:39.150 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:33:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:33:38.810+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:33:39.150 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:33:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:33:38.810+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:33:39.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:33:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:33:39.264+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:33:39.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:33:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:33:39.264+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:33:39.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:33:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:33:39.264+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:33:39.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:33:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:33:39.146+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:33:39.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:33:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:33:39.146+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:33:39.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:33:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:33:39.146+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:33:39.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:33:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:33:39.146+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:33:40.153 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:33:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:33:39.843+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:33:40.153 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:33:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:33:39.843+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:33:40.153 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:33:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:33:39.843+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:33:40.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:33:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:33:40.149+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:33:40.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:33:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:33:40.149+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:33:40.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:33:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:33:40.149+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:33:40.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:33:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:33:40.150+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:33:40.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:33:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:33:40.214+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:33:40.494 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:33:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:33:40.214+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:33:40.494 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:33:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:33:40.214+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:33:41.172 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:33:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:33:40.868+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:33:41.172 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:33:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:33:40.868+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:33:41.173 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:33:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:33:40.868+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:33:41.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:33:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:33:41.169+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:33:41.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:33:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:33:41.169+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:33:41.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:33:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:33:41.169+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:33:41.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:33:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:33:41.169+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:33:41.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:33:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:33:41.214+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:33:41.494 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:33:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:33:41.214+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:33:41.494 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:33:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:33:41.214+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:33:42.141 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:33:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:33:41.893+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:33:42.141 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:33:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:33:41.893+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:33:42.142 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:33:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:33:41.893+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:33:42.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:33:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:33:42.256+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:33:42.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:33:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:33:42.256+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:33:42.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:33:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:33:42.256+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:33:42.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:33:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:33:42.139+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:33:42.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:33:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:33:42.139+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:33:42.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:33:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:33:42.139+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:33:42.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:33:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:33:42.139+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:33:42.993 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:33:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:33:42.649+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 29214 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:33:42.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:33:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:33:42.847+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:33:42.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:33:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:33:42.847+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:33:42.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:33:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:33:42.847+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:33:43.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:33:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:33:43.242+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:33:43.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:33:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:33:43.242+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:33:43.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:33:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:33:43.242+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:33:43.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:33:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:33:43.097+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:33:43.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:33:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:33:43.097+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:33:43.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:33:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:33:43.097+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:33:43.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:33:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:33:43.097+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:33:44.076 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:33:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:33:43.818+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:33:44.076 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:33:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:33:43.818+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:33:44.076 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:33:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:33:43.818+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:33:44.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:33:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:33:44.204+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:33:44.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:33:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:33:44.204+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:33:44.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:33:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:33:44.204+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:33:44.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:33:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:33:44.073+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:33:44.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:33:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:33:44.073+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:33:44.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:33:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:33:44.073+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:33:44.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:33:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:33:44.073+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:33:45.180 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:33:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:33:44.803+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:33:45.180 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:33:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:33:44.803+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:33:45.180 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:33:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:33:44.803+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:33:45.181 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:33:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:33:45.044+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:33:45.181 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:33:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:33:45.044+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:33:45.181 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:33:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:33:45.044+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:33:45.181 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:33:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:33:45.044+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:33:45.492 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:33:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:33:45.176+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:33:45.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:33:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:33:45.176+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:33:45.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:33:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:33:45.176+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:33:46.048 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:33:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:33:45.756+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:33:46.049 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:33:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:33:45.756+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:33:46.049 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:33:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:33:45.756+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:33:46.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:33:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:33:46.139+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:33:46.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:33:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:33:46.139+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:33:46.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:33:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:33:46.139+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:33:46.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:33:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:33:46.046+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:33:46.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:33:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:33:46.046+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:33:46.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:33:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:33:46.046+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:33:46.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:33:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:33:46.046+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:33:46.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:33:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:33:46.707+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:33:46.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:33:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:33:46.707+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:33:46.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:33:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:33:46.707+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:33:47.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:33:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:33:47.101+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:33:47.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:33:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:33:47.101+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:33:47.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:33:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:33:47.101+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:33:47.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:33:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:33:47.000+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:33:47.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:33:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:33:47.000+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:33:47.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:33:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:33:47.000+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:33:47.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:33:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:33:47.000+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:33:47.975 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:33:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:33:47.653+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 29285 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:33:47.975 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:33:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:33:47.681+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:33:47.976 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:33:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:33:47.681+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:33:47.976 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:33:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:33:47.681+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:33:48.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:33:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:33:48.108+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:33:48.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:33:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:33:48.108+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:33:48.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:33:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:33:48.108+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:33:48.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:33:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:33:47.973+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:33:48.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:33:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:33:47.973+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:33:48.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:33:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:33:47.973+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:33:48.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:33:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:33:47.973+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:33:48.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:33:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:33:48.689+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:33:48.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:33:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:33:48.689+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:33:48.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:33:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:33:48.689+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:33:49.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:33:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:33:49.073+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:33:49.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:33:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:33:49.073+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:33:49.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:33:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:33:49.073+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:33:49.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:33:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:33:49.008+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:33:49.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:33:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:33:49.008+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:33:49.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:33:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:33:49.008+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:33:49.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:33:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:33:49.008+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:33:49.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:33:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:33:49.725+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:33:49.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:33:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:33:49.725+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:33:49.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:33:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:33:49.725+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:33:50.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:33:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:33:50.083+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:33:50.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:33:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:33:50.083+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:33:50.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:33:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:33:50.083+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:33:50.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:33:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:33:50.058+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:33:50.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:33:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:33:50.058+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:33:50.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:33:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:33:50.058+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:33:50.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:33:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:33:50.058+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:33:51.065 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:33:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:33:50.758+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:33:51.066 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:33:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:33:50.758+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:33:51.066 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:33:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:33:50.758+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:33:51.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:33:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:33:51.062+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:33:51.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:33:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:33:51.062+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:33:51.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:33:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:33:51.062+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:33:51.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:33:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:33:51.086+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:33:51.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:33:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:33:51.086+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:33:51.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:33:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:33:51.086+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:33:51.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:33:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:33:51.086+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:33:51.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:33:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:33:51.718+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:33:51.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:33:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:33:51.718+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:33:51.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:33:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:33:51.718+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:33:52.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:33:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:33:52.016+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:33:52.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:33:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:33:52.016+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:33:52.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:33:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:33:52.016+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:33:52.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:33:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:33:52.102+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:33:52.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:33:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:33:52.102+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:33:52.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:33:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:33:52.102+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:33:52.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:33:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:33:52.102+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:33:52.993 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:33:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:33:52.657+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 29358 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:33:52.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:33:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:33:52.683+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:33:52.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:33:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:33:52.683+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:33:52.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:33:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:33:52.683+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:33:53.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:33:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:33:53.060+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:33:53.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:33:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:33:53.060+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:33:53.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:33:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:33:53.060+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:33:53.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:33:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:33:53.132+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:33:53.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:33:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:33:53.132+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:33:53.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:33:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:33:53.132+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:33:53.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:33:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:33:53.132+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:33:53.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:33:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:33:53.641+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:33:53.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:33:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:33:53.641+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:33:53.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:33:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:33:53.641+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:33:54.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:33:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:33:54.125+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:33:54.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:33:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:33:54.125+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:33:54.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:33:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:33:54.125+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:33:54.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:33:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:33:54.125+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:33:54.494 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:33:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:33:54.038+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:33:54.494 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:33:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:33:54.038+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:33:54.494 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:33:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:33:54.038+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:33:54.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:33:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:33:54.686+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:33:54.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:33:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:33:54.686+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:33:54.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:33:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:33:54.686+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:33:55.495 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:33:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:33:55.065+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:33:55.496 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:33:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:33:55.065+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:33:55.496 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:33:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:33:55.065+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:33:55.496 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:33:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:33:55.119+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:33:55.496 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:33:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:33:55.119+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:33:55.496 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:33:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:33:55.119+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:33:55.496 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:33:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:33:55.119+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:33:55.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:33:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:33:55.731+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:33:55.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:33:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:33:55.731+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:33:55.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:33:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:33:55.731+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:33:56.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:33:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:33:56.093+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:33:56.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:33:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:33:56.093+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:33:56.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:33:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:33:56.093+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:33:56.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:33:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:33:56.141+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:33:56.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:33:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:33:56.141+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:33:56.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:33:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:33:56.141+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:33:56.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:33:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:33:56.141+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:33:57.145 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:33:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:33:56.781+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:33:57.145 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:33:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:33:56.781+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:33:57.145 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:33:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:33:56.781+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:33:57.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:33:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:33:57.143+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:33:57.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:33:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:33:57.143+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:33:57.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:33:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:33:57.143+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:33:57.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:33:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:33:57.186+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:33:57.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:33:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:33:57.186+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:33:57.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:33:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:33:57.186+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:33:57.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:33:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:33:57.186+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:33:57.993 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:33:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:33:57.663+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 29429 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:33:57.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:33:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:33:57.764+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:33:57.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:33:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:33:57.764+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:33:57.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:33:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:33:57.764+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:33:58.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:33:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:33:58.105+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:33:58.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:33:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:33:58.105+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:33:58.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:33:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:33:58.105+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:33:58.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:33:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:33:58.233+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:33:58.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:33:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:33:58.233+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:33:58.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:33:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:33:58.233+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:33:58.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:33:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:33:58.234+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:33:59.147 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:33:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:33:58.756+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:33:59.147 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:33:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:33:58.756+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:33:59.148 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:33:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:33:58.756+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:33:59.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:33:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:33:59.145+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:33:59.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:33:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:33:59.145+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:33:59.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:33:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:33:59.145+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:33:59.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:33:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:33:59.241+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:33:59.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:33:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:33:59.241+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:33:59.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:33:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:33:59.241+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:33:59.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:33:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:33:59.242+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:34:00.103 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:33:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:33:59.750+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:34:00.103 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:33:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:33:59.750+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:34:00.103 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:33:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:33:59.750+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:34:00.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:34:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:34:00.100+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:34:00.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:34:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:34:00.100+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:34:00.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:34:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:34:00.100+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:34:00.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:34:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:34:00.232+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:34:00.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:34:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:34:00.232+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:34:00.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:34:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:34:00.233+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:34:00.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:34:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:34:00.233+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:34:01.081 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:34:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:34:00.758+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:34:01.081 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:34:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:34:00.758+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:34:01.081 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:34:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:34:00.758+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:34:01.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:34:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:34:01.090+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:34:01.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:34:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:34:01.090+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:34:01.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:34:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:34:01.090+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:34:01.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:34:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:34:01.259+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:34:01.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:34:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:34:01.259+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:34:01.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:34:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:34:01.259+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:34:01.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:34:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:34:01.259+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:34:02.099 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:34:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:34:01.778+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:34:02.099 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:34:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:34:01.778+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:34:02.099 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:34:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:34:01.778+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:34:02.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:34:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:34:02.275+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:34:02.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:34:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:34:02.275+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:34:02.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:34:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:34:02.275+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:34:02.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:34:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:34:02.275+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:34:02.494 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:34:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:34:02.097+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:34:02.494 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:34:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:34:02.097+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:34:02.494 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:34:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:34:02.097+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:34:02.993 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:34:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:34:02.667+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 29502 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:34:02.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:34:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:34:02.782+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:34:02.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:34:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:34:02.782+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:34:02.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:34:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:34:02.782+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:34:03.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:34:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:34:03.117+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:34:03.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:34:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:34:03.117+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:34:03.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:34:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:34:03.118+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:34:03.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:34:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:34:03.235+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:34:03.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:34:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:34:03.235+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:34:03.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:34:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:34:03.235+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:34:03.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:34:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:34:03.235+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:34:04.141 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:34:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:34:03.765+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:34:04.141 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:34:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:34:03.765+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:34:04.141 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:34:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:34:03.765+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:34:04.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:34:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:34:04.139+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:34:04.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:34:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:34:04.139+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:34:04.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:34:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:34:04.139+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:34:04.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:34:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:34:04.210+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:34:04.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:34:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:34:04.210+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:34:04.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:34:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:34:04.210+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:34:04.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:34:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:34:04.211+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:34:05.169 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:34:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:34:04.746+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:34:05.170 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:34:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:34:04.746+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:34:05.170 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:34:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:34:04.746+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:34:05.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:34:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:34:05.176+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:34:05.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:34:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:34:05.176+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:34:05.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:34:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:34:05.176+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:34:05.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:34:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:34:05.167+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:34:05.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:34:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:34:05.167+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:34:05.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:34:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:34:05.167+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:34:05.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:34:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:34:05.167+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:34:05.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:34:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:34:05.712+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:34:05.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:34:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:34:05.712+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:34:05.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:34:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:34:05.712+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:34:06.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:34:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:34:06.127+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:34:06.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:34:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:34:06.127+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:34:06.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:34:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:34:06.127+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:34:06.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:34:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:34:06.155+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:34:06.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:34:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:34:06.155+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:34:06.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:34:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:34:06.155+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:34:06.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:34:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:34:06.155+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:34:07.101 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:34:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:34:06.758+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:34:07.101 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:34:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:34:06.758+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:34:07.102 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:34:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:34:06.758+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:34:07.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:34:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:34:07.099+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:34:07.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:34:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:34:07.099+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:34:07.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:34:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:34:07.099+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:34:07.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:34:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:34:07.174+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:34:07.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:34:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:34:07.174+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:34:07.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:34:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:34:07.174+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:34:07.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:34:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:34:07.174+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:34:07.993 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:34:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:34:07.670+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 29573 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:34:07.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:34:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:34:07.750+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:34:07.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:34:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:34:07.750+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:34:07.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:34:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:34:07.750+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:34:08.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:34:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:34:08.057+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:34:08.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:34:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:34:08.057+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:34:08.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:34:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:34:08.057+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:34:08.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:34:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:34:08.219+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:34:08.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:34:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:34:08.219+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:34:08.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:34:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:34:08.219+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:34:08.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:34:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:34:08.219+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:34:09.056 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:34:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:34:08.795+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:34:09.056 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:34:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:34:08.795+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:34:09.056 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:34:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:34:08.795+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:34:09.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:34:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:34:09.054+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:34:09.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:34:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:34:09.054+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:34:09.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:34:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:34:09.054+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:34:09.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:34:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:34:09.265+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:34:09.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:34:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:34:09.265+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:34:09.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:34:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:34:09.265+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:34:09.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:34:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:34:09.265+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:34:10.099 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:34:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:34:09.767+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:34:10.100 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:34:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:34:09.767+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:34:10.100 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:34:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:34:09.767+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:34:10.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:34:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:34:10.097+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:34:10.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:34:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:34:10.097+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:34:10.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:34:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:34:10.097+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:34:10.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:34:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:34:10.289+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:34:10.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:34:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:34:10.289+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:34:10.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:34:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:34:10.289+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:34:10.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:34:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:34:10.290+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:34:10.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:34:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:34:10.724+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:34:10.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:34:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:34:10.724+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:34:10.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:34:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:34:10.724+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:34:11.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:34:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:34:11.062+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:34:11.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:34:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:34:11.062+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:34:11.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:34:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:34:11.062+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:34:11.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:34:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:34:11.271+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:34:11.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:34:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:34:11.271+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:34:11.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:34:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:34:11.271+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:34:11.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:34:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:34:11.271+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:34:11.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:34:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:34:11.737+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:34:11.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:34:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:34:11.737+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:34:11.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:34:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:34:11.737+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:34:12.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:34:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:34:12.057+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:34:12.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:34:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:34:12.057+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:34:12.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:34:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:34:12.057+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:34:12.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:34:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:34:12.272+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:34:12.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:34:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:34:12.272+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:34:12.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:34:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:34:12.272+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:34:12.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:34:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:34:12.272+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:34:12.993 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:34:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:34:12.675+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 29646 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:34:12.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:34:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:34:12.689+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:34:12.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:34:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:34:12.689+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:34:12.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:34:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:34:12.689+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:34:13.369 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:34:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:34:13.032+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:34:13.369 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:34:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:34:13.032+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:34:13.370 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:34:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:34:13.032+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:34:13.370 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:34:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:34:13.224+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:34:13.370 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:34:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:34:13.224+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:34:13.370 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:34:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:34:13.224+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:34:13.370 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:34:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:34:13.224+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:34:13.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:34:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:34:13.663+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:34:13.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:34:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:34:13.663+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:34:13.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:34:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:34:13.663+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:34:14.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:34:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:34:14.031+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:34:14.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:34:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:34:14.031+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:34:14.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:34:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:34:14.031+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:34:14.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:34:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:34:14.254+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:34:14.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:34:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:34:14.254+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:34:14.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:34:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:34:14.254+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:34:14.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:34:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:34:14.254+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:34:14.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:34:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:34:14.697+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:34:14.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:34:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:34:14.697+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:34:14.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:34:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:34:14.697+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:34:15.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:34:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:34:15.065+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:34:15.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:34:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:34:15.065+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:34:15.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:34:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:34:15.065+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:34:15.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:34:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:34:15.276+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:34:15.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:34:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:34:15.276+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:34:15.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:34:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:34:15.276+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:34:15.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:34:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:34:15.276+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:34:15.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:34:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:34:15.686+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:34:15.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:34:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:34:15.686+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:34:15.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:34:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:34:15.686+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:34:16.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:34:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:34:16.048+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:34:16.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:34:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:34:16.048+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:34:16.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:34:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:34:16.048+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:34:16.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:34:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:34:16.274+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:34:16.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:34:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:34:16.274+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:34:16.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:34:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:34:16.274+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:34:16.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:34:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:34:16.274+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:34:16.992 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:34:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:34:16.682+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:34:16.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:34:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:34:16.682+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:34:16.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:34:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:34:16.682+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:34:17.293 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:34:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:34:16.999+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:34:17.294 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:34:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:34:16.999+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:34:17.294 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:34:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:34:16.999+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:34:17.677 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:34:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:34:17.292+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:34:17.677 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:34:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:34:17.292+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:34:17.677 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:34:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:34:17.292+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:34:17.677 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:34:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:34:17.292+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:34:17.982 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:34:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:34:17.679+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 29717 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:34:17.982 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:34:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:34:17.675+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:34:17.982 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:34:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:34:17.675+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:34:17.982 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:34:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:34:17.675+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:34:18.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:34:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:34:17.979+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:34:18.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:34:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:34:17.979+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:34:18.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:34:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:34:17.979+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:34:18.638 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:34:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:34:18.262+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:34:18.638 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:34:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:34:18.262+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:34:18.638 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:34:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:34:18.262+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:34:18.638 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:34:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:34:18.262+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:34:18.961 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:34:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:34:18.635+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:34:18.961 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:34:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:34:18.635+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:34:18.962 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:34:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:34:18.635+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:34:19.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:34:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:34:18.958+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:34:19.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:34:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:34:18.958+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:34:19.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:34:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:34:18.958+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:34:19.622 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:34:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:34:19.305+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:34:19.623 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:34:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:34:19.305+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:34:19.623 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:34:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:34:19.305+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:34:19.623 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:34:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:34:19.305+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:34:19.887 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:34:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:34:19.619+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:34:19.887 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:34:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:34:19.619+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:34:19.887 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:34:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:34:19.619+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:34:20.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:34:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:34:19.993+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:34:20.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:34:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:34:19.993+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:34:20.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:34:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:34:19.993+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:34:20.587 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:34:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:34:20.339+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:34:20.587 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:34:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:34:20.339+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:34:20.587 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:34:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:34:20.339+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:34:20.587 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:34:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:34:20.339+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:34:20.951 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:34:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:34:20.584+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:34:20.951 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:34:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:34:20.584+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:34:20.951 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:34:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:34:20.584+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:34:21.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:34:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:34:20.947+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:34:21.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:34:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:34:20.948+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:34:21.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:34:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:34:20.948+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:34:21.610 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:34:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:34:21.606+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:34:21.610 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:34:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:34:21.606+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:34:21.610 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:34:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:34:21.606+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:34:21.610 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:34:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:34:21.357+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:34:21.611 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:34:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:34:21.357+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:34:21.611 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:34:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:34:21.357+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:34:21.611 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:34:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:34:21.357+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:34:22.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:34:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:34:21.969+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:34:22.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:34:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:34:21.969+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:34:22.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:34:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:34:21.969+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:34:22.686 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:34:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:34:22.599+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:34:22.686 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:34:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:34:22.599+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:34:22.687 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:34:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:34:22.599+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:34:22.687 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:34:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:34:22.376+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:34:22.687 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:34:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:34:22.376+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:34:22.687 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:34:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:34:22.376+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:34:22.687 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:34:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:34:22.376+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:34:22.946 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:34:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:34:22.683+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 29790 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:34:23.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:34:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:34:22.943+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:34:23.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:34:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:34:22.943+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:34:23.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:34:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:34:22.943+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:34:23.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:34:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:34:23.576+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:34:23.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:34:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:34:23.576+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:34:23.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:34:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:34:23.576+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:34:23.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:34:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:34:23.425+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:34:23.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:34:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:34:23.425+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:34:23.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:34:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:34:23.425+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:34:23.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:34:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:34:23.425+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:34:24.428 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:34:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:34:23.992+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:34:24.428 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:34:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:34:23.992+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:34:24.428 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:34:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:34:23.992+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:34:24.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:34:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:34:24.581+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:34:24.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:34:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:34:24.581+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:34:24.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:34:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:34:24.581+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:34:24.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:34:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:34:24.425+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:34:24.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:34:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:34:24.425+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:34:24.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:34:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:34:24.425+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:34:24.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:34:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:34:24.425+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:34:25.389 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:34:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:34:25.017+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:34:25.390 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:34:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:34:25.017+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:34:25.390 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:34:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:34:25.017+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:34:25.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:34:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:34:25.607+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:34:25.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:34:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:34:25.607+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:34:25.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:34:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:34:25.607+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:34:25.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:34:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:34:25.387+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:34:25.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:34:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:34:25.387+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:34:25.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:34:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:34:25.387+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:34:25.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:34:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:34:25.387+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:34:26.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:34:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:34:25.988+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:34:26.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:34:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:34:25.988+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:34:26.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:34:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:34:25.988+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:34:26.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:34:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:34:26.576+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:34:26.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:34:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:34:26.576+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:34:26.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:34:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:34:26.576+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:34:26.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:34:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:34:26.398+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:34:26.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:34:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:34:26.398+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:34:26.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:34:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:34:26.398+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:34:26.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:34:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:34:26.398+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:34:27.403 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:34:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:34:27.014+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:34:27.403 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:34:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:34:27.014+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:34:27.403 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:34:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:34:27.014+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:34:27.690 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:34:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:34:27.612+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:34:27.690 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:34:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:34:27.612+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:34:27.691 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:34:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:34:27.612+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:34:27.691 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:34:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:34:27.399+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:34:27.691 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:34:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:34:27.399+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:34:27.691 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:34:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:34:27.399+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:34:27.691 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:34:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:34:27.400+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:34:27.992 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:34:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:34:27.687+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 29861 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:34:28.382 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:34:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:34:28.020+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:34:28.382 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:34:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:34:28.020+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:34:28.382 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:34:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:34:28.020+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:34:28.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:34:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:34:28.628+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:34:28.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:34:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:34:28.628+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:34:28.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:34:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:34:28.628+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:34:28.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:34:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:34:28.379+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:34:28.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:34:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:34:28.379+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:34:28.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:34:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:34:28.379+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:34:28.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:34:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:34:28.379+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:34:29.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:34:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:34:28.973+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:34:29.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:34:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:34:28.973+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:34:29.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:34:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:34:28.973+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:34:29.668 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:34:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:34:29.377+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:34:29.668 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:34:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:34:29.377+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:34:29.668 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:34:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:34:29.377+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:34:29.669 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:34:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:34:29.377+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:34:29.955 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:34:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:34:29.665+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:34:29.955 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:34:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:34:29.665+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:34:29.955 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:34:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:34:29.665+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:34:30.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:34:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:34:29.952+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:34:30.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:34:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:34:29.952+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:34:30.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:34:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:34:29.952+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:34:30.682 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:34:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:34:30.357+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:34:30.682 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:34:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:34:30.357+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:34:30.683 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:34:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:34:30.357+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:34:30.683 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:34:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:34:30.357+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:34:30.951 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:34:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:34:30.679+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:34:30.952 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:34:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:34:30.679+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:34:30.952 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:34:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:34:30.679+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:34:31.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:34:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:34:30.948+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:34:31.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:34:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:34:30.948+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:34:31.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:34:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:34:30.948+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:34:31.664 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:34:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:34:31.361+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:34:31.664 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:34:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:34:31.361+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:34:31.665 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:34:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:34:31.361+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:34:31.665 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:34:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:34:31.361+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:34:31.929 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:34:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:34:31.661+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:34:31.930 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:34:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:34:31.661+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:34:31.930 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:34:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:34:31.661+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:34:32.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:34:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:34:31.927+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:34:32.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:34:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:34:31.927+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:34:32.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:34:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:34:31.927+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:34:32.682 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:34:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:34:32.365+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:34:32.682 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:34:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:34:32.365+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:34:32.682 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:34:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:34:32.365+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:34:32.682 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:34:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:34:32.365+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:34:32.976 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:34:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:34:32.692+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 29934 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:34:32.976 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:34:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:34:32.679+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:34:32.976 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:34:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:34:32.679+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:34:32.976 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:34:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:34:32.679+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:34:33.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:34:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:34:32.973+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:34:33.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:34:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:34:32.973+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:34:33.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:34:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:34:32.973+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:34:33.677 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:34:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:34:33.387+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:34:33.677 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:34:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:34:33.387+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:34:33.677 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:34:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:34:33.387+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:34:33.678 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:34:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:34:33.387+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:34:33.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:34:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:34:33.674+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:34:33.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:34:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:34:33.674+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:34:33.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:34:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:34:33.674+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:34:34.397 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:34:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:34:34.005+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:34:34.397 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:34:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:34:34.005+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:34:34.397 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:34:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:34:34.005+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:34:34.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:34:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:34:34.630+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:34:34.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:34:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:34:34.630+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:34:34.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:34:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:34:34.630+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:34:34.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:34:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:34:34.394+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:34:34.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:34:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:34:34.394+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:34:34.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:34:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:34:34.394+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:34:34.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:34:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:34:34.394+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:34:35.407 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:34:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:34:34.997+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:34:35.408 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:34:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:34:34.997+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:34:35.408 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:34:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:34:34.997+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:34:35.677 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:34:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:34:35.405+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:34:35.677 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:34:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:34:35.405+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:34:35.677 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:34:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:34:35.405+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:34:35.677 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:34:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:34:35.405+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:34:35.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:34:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:34:35.674+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:34:35.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:34:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:34:35.674+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:34:35.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:34:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:34:35.674+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:34:36.454 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:34:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:34:35.992+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:34:36.454 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:34:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:34:35.992+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:34:36.455 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:34:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:34:35.992+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:34:36.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:34:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:34:36.657+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:34:36.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:34:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:34:36.657+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:34:36.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:34:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:34:36.657+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:34:36.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:34:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:34:36.451+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:34:36.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:34:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:34:36.451+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:34:36.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:34:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:34:36.451+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:34:36.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:34:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:34:36.451+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:34:37.443 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:34:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:34:37.011+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:34:37.443 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:34:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:34:37.011+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:34:37.443 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:34:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:34:37.011+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:34:37.699 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:34:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:34:37.665+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:34:37.700 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:34:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:34:37.665+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:34:37.700 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:34:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:34:37.665+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:34:37.700 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:34:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:34:37.440+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:34:37.700 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:34:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:34:37.440+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:34:37.700 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:34:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:34:37.440+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:34:37.700 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:34:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:34:37.440+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:34:37.993 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:34:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:34:37.696+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 30005 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:34:38.477 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:34:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:34:38.019+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:34:38.477 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:34:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:34:38.019+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:34:38.477 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:34:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:34:38.019+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:34:38.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:34:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:34:38.639+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:34:38.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:34:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:34:38.639+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:34:38.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:34:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:34:38.639+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:34:38.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:34:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:34:38.474+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:34:38.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:34:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:34:38.474+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:34:38.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:34:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:34:38.474+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:34:38.745 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:34:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:34:38.474+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:34:39.438 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:34:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:34:39.054+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:34:39.439 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:34:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:34:39.054+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:34:39.439 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:34:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:34:39.054+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:34:39.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:34:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:34:39.436+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:34:39.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:34:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:34:39.436+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:34:39.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:34:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:34:39.436+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:34:39.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:34:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:34:39.436+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:34:39.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:34:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:34:39.635+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:34:39.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:34:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:34:39.635+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:34:39.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:34:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:34:39.635+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:34:40.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:34:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:34:40.006+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:34:40.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:34:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:34:40.006+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:34:40.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:34:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:34:40.006+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:34:40.648 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:34:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:34:40.386+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:34:40.648 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:34:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:34:40.386+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:34:40.648 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:34:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:34:40.386+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:34:40.648 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:34:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:34:40.386+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:34:40.963 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:34:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:34:40.645+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:34:40.964 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:34:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:34:40.645+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:34:40.964 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:34:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:34:40.645+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:34:41.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:34:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:34:40.960+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:34:41.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:34:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:34:40.960+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:34:41.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:34:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:34:40.960+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:34:41.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:34:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:34:41.657+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:34:41.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:34:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:34:41.657+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:34:41.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:34:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:34:41.657+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:34:41.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:34:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:34:41.410+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:34:41.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:34:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:34:41.410+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:34:41.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:34:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:34:41.410+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:34:41.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:34:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:34:41.410+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:34:42.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:34:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:34:41.971+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:34:42.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:34:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:34:41.971+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:34:42.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:34:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:34:41.971+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:34:42.703 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:34:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:34:42.435+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:34:42.704 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:34:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:34:42.435+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:34:42.704 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:34:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:34:42.435+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:34:42.704 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:34:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:34:42.435+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:34:42.993 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:34:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:34:42.700+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 30078 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:34:42.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:34:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:34:42.705+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:34:42.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:34:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:34:42.705+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:34:42.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:34:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:34:42.705+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:34:43.392 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:34:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:34:43.016+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:34:43.392 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:34:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:34:43.016+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:34:43.392 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:34:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:34:43.016+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:34:43.704 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:34:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:34:43.390+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:34:43.704 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:34:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:34:43.390+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:34:43.704 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:34:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:34:43.390+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:34:43.704 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:34:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:34:43.390+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:34:43.992 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:34:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:34:43.701+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:34:43.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:34:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:34:43.701+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:34:43.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:34:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:34:43.701+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:34:44.410 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:34:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:34:44.026+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:34:44.410 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:34:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:34:44.026+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:34:44.410 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:34:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:34:44.026+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:34:44.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:34:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:34:44.407+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:34:44.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:34:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:34:44.407+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:34:44.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:34:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:34:44.407+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:34:44.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:34:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:34:44.407+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:34:45.018 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:34:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:34:44.744+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:34:45.019 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:34:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:34:44.744+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:34:45.019 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:34:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:34:44.744+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:34:45.395 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:34:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:34:45.016+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:34:45.395 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:34:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:34:45.016+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:34:45.395 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:34:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:34:45.016+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:34:45.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:34:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:34:45.391+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:34:45.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:34:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:34:45.391+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:34:45.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:34:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:34:45.391+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:34:45.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:34:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:34:45.392+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:34:46.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:34:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:34:46.037+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:34:46.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:34:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:34:46.037+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:34:46.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:34:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:34:46.037+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:34:46.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:34:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:34:45.792+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:34:46.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:34:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:34:45.792+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:34:46.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:34:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:34:45.792+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:34:46.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:34:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:34:46.385+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:34:46.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:34:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:34:46.385+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:34:46.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:34:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:34:46.385+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:34:46.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:34:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:34:46.385+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:34:47.071 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:34:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:34:46.792+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:34:47.072 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:34:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:34:46.792+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:34:47.072 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:34:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:34:46.792+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:34:47.349 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:34:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:34:47.069+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:34:47.349 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:34:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:34:47.069+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:34:47.349 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:34:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:34:47.069+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:34:47.708 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:34:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:34:47.346+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:34:47.708 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:34:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:34:47.346+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:34:47.708 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:34:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:34:47.346+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:34:47.708 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:34:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:34:47.346+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:34:47.993 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:34:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:34:47.704+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 30149 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:34:47.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:34:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:34:47.749+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:34:47.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:34:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:34:47.749+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:34:47.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:34:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:34:47.749+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:34:48.331 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:34:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:34:48.030+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:34:48.331 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:34:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:34:48.030+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:34:48.331 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:34:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:34:48.030+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:34:48.725 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:34:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:34:48.329+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:34:48.725 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:34:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:34:48.329+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:34:48.725 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:34:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:34:48.329+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:34:48.725 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:34:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:34:48.329+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:34:48.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:34:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:34:48.722+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:34:48.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:34:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:34:48.722+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:34:48.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:34:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:34:48.722+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:34:49.358 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:34:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:34:49.069+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:34:49.358 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:34:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:34:49.069+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:34:49.358 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:34:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:34:49.069+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:34:49.697 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:34:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:34:49.356+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:34:49.698 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:34:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:34:49.356+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:34:49.698 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:34:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:34:49.356+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:34:49.699 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:34:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:34:49.356+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:34:49.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:34:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:34:49.695+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:34:49.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:34:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:34:49.695+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:34:49.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:34:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:34:49.695+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:34:50.378 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:34:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:34:50.055+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:34:50.379 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:34:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:34:50.055+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:34:50.379 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:34:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:34:50.055+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:34:50.662 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:34:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:34:50.376+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:34:50.662 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:34:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:34:50.376+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:34:50.663 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:34:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:34:50.376+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:34:50.663 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:34:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:34:50.376+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:34:50.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:34:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:34:50.660+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:34:50.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:34:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:34:50.660+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:34:50.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:34:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:34:50.660+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:34:51.412 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:34:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:34:51.066+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:34:51.412 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:34:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:34:51.066+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:34:51.412 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:34:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:34:51.066+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:34:51.713 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:34:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:34:51.409+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:34:51.713 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:34:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:34:51.409+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:34:51.713 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:34:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:34:51.409+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:34:51.713 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:34:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:34:51.410+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:34:51.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:34:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:34:51.710+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:34:51.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:34:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:34:51.710+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:34:51.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:34:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:34:51.710+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:34:52.435 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:34:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:34:52.040+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:34:52.435 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:34:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:34:52.040+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:34:52.435 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:34:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:34:52.040+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:34:52.713 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:34:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:34:52.432+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:34:52.713 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:34:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:34:52.432+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:34:52.713 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:34:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:34:52.432+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:34:52.713 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:34:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:34:52.432+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:34:52.993 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:34:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:34:52.710+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 30222 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:34:52.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:34:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:34:52.756+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:34:52.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:34:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:34:52.756+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:34:52.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:34:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:34:52.756+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:34:53.305 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:34:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:34:53.013+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:34:53.305 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:34:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:34:53.013+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:34:53.306 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:34:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:34:53.013+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:34:53.723 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:34:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:34:53.444+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:34:53.724 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:34:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:34:53.444+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:34:53.724 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:34:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:34:53.444+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:34:53.724 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:34:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:34:53.444+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:34:53.992 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:34:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:34:53.721+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:34:53.992 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:34:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:34:53.721+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:34:53.992 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:34:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:34:53.721+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:34:54.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:34:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:34:53.990+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:34:54.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:34:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:34:53.990+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:34:54.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:34:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:34:53.990+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:34:54.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:34:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:34:54.722+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:34:54.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:34:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:34:54.722+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:34:54.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:34:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:34:54.722+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:34:54.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:34:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:34:54.478+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:34:54.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:34:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:34:54.478+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:34:54.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:34:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:34:54.478+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:34:54.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:34:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:34:54.478+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:34:55.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:34:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:34:54.942+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:34:55.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:34:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:34:54.942+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:34:55.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:34:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:34:54.942+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:34:55.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:34:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:34:55.483+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:34:55.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:34:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:34:55.483+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:34:55.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:34:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:34:55.483+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:34:55.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:34:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:34:55.484+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:34:56.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:34:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:34:55.770+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:34:56.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:34:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:34:55.770+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:34:56.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:34:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:34:55.770+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:34:56.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:34:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:34:55.930+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:34:56.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:34:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:34:55.930+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:34:56.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:34:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:34:55.930+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:34:56.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:34:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:34:56.465+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:34:56.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:34:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:34:56.465+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:34:56.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:34:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:34:56.465+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:34:56.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:34:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:34:56.465+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:34:57.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:34:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:34:56.791+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:34:57.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:34:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:34:56.791+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:34:57.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:34:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:34:56.791+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:34:57.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:34:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:34:56.892+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:34:57.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:34:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:34:56.892+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:34:57.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:34:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:34:56.892+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:34:57.739 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:34:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:34:57.715+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 30293 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:34:57.739 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:34:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:34:57.476+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:34:57.739 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:34:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:34:57.476+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:34:57.739 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:34:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:34:57.476+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:34:57.739 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:34:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:34:57.476+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:34:57.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:34:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:34:57.837+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:34:57.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:34:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:34:57.837+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:34:57.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:34:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:34:57.837+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:34:57.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:34:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:34:57.934+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:34:57.995 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:34:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:34:57.934+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:34:57.995 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:34:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:34:57.934+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:34:58.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:34:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:34:58.428+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:34:58.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:34:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:34:58.428+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:34:58.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:34:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:34:58.428+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:34:58.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:34:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:34:58.428+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:34:59.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:34:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:34:58.963+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:34:59.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:34:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:34:58.963+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:34:59.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:34:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:34:58.963+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:34:59.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:34:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:34:58.859+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:34:59.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:34:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:34:58.859+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:34:59.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:34:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:34:58.859+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:34:59.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:34:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:34:59.402+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:34:59.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:34:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:34:59.402+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:34:59.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:34:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:34:59.402+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:34:59.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:34:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:34:59.402+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:35:00.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:34:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:34:59.984+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:35:00.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:34:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:34:59.984+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:35:00.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:34:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:34:59.984+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:35:00.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:34:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:34:59.838+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:35:00.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:34:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:34:59.838+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:35:00.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:34:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:34:59.838+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:35:00.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:35:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:35:00.384+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:35:00.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:35:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:35:00.384+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:35:00.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:35:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:35:00.384+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:35:00.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:35:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:35:00.384+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:35:01.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:35:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:35:00.868+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:35:01.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:35:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:35:00.868+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:35:01.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:35:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:35:00.868+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:35:01.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:35:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:35:00.973+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:35:01.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:35:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:35:00.973+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:35:01.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:35:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:35:00.973+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:35:01.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:35:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:35:01.341+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:35:01.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:35:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:35:01.341+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:35:01.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:35:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:35:01.341+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:35:01.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:35:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:35:01.341+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:35:02.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:35:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:35:02.003+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:35:02.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:35:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:35:02.003+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:35:02.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:35:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:35:02.003+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:35:02.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:35:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:35:01.902+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:35:02.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:35:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:35:01.902+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:35:02.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:35:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:35:01.902+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:35:02.722 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:35:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:35:02.374+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:35:02.722 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:35:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:35:02.374+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:35:02.722 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:35:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:35:02.374+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:35:02.722 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:35:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:35:02.374+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:35:02.993 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:35:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:35:02.719+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 30366 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:35:02.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:35:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:35:02.962+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:35:02.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:35:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:35:02.962+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:35:02.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:35:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:35:02.962+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:35:02.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:35:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:35:02.942+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:35:02.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:35:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:35:02.942+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:35:02.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:35:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:35:02.942+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:35:03.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:35:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:35:03.396+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:35:03.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:35:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:35:03.396+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:35:03.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:35:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:35:03.396+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:35:03.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:35:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:35:03.396+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:35:04.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:35:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:35:03.976+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:35:04.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:35:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:35:03.976+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:35:04.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:35:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:35:03.976+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:35:04.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:35:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:35:03.991+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:35:04.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:35:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:35:03.991+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:35:04.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:35:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:35:03.991+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:35:04.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:35:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:35:04.390+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:35:04.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:35:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:35:04.390+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:35:04.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:35:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:35:04.390+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:35:04.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:35:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:35:04.390+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:35:05.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:35:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:35:04.948+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:35:05.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:35:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:35:04.948+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:35:05.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:35:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:35:04.948+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:35:05.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:35:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:35:05.025+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:35:05.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:35:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:35:05.025+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:35:05.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:35:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:35:05.025+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:35:05.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:35:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:35:05.418+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:35:05.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:35:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:35:05.418+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:35:05.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:35:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:35:05.418+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:35:05.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:35:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:35:05.418+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:35:06.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:35:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:35:05.940+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:35:06.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:35:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:35:05.940+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:35:06.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:35:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:35:05.940+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:35:06.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:35:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:35:06.036+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:35:06.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:35:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:35:06.036+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:35:06.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:35:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:35:06.036+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:35:06.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:35:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:35:06.410+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:35:06.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:35:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:35:06.410+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:35:06.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:35:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:35:06.410+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:35:06.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:35:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:35:06.410+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:35:07.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:35:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:35:06.971+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:35:07.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:35:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:35:06.971+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:35:07.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:35:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:35:06.971+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:35:07.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:35:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:35:07.030+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:35:07.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:35:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:35:07.030+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:35:07.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:35:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:35:07.030+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:35:07.727 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:35:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:35:07.453+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:35:07.727 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:35:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:35:07.453+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:35:07.727 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:35:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:35:07.453+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:35:07.727 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:35:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:35:07.453+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:35:07.982 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:35:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:35:07.724+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 30437 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:35:08.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:35:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:35:08.064+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:35:08.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:35:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:35:08.064+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:35:08.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:35:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:35:08.064+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:35:08.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:35:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:35:07.980+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:35:08.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:35:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:35:07.980+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:35:08.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:35:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:35:07.980+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:35:08.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:35:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:35:08.452+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:35:08.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:35:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:35:08.452+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:35:08.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:35:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:35:08.452+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:35:08.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:35:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:35:08.452+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:35:09.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:35:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:35:08.959+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:35:09.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:35:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:35:08.959+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:35:09.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:35:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:35:08.959+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:35:09.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:35:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:35:09.018+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:35:09.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:35:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:35:09.018+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:35:09.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:35:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:35:09.018+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:35:09.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:35:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:35:09.433+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:35:09.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:35:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:35:09.433+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:35:09.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:35:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:35:09.433+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:35:09.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:35:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:35:09.433+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:35:10.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:35:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:35:09.993+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:35:10.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:35:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:35:09.993+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:35:10.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:35:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:35:09.993+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:35:10.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:35:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:35:10.029+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:35:10.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:35:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:35:10.029+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:35:10.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:35:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:35:10.029+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:35:10.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:35:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:35:10.414+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:35:10.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:35:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:35:10.414+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:35:10.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:35:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:35:10.414+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:35:10.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:35:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:35:10.414+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:35:11.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:35:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:35:10.972+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:35:11.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:35:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:35:10.972+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:35:11.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:35:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:35:10.972+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:35:11.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:35:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:35:10.983+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:35:11.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:35:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:35:10.983+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:35:11.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:35:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:35:10.983+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:35:11.737 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:35:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:35:11.388+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:35:11.738 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:35:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:35:11.388+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:35:11.738 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:35:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:35:11.388+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:35:11.738 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:35:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:35:11.388+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:35:11.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:35:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:35:11.952+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:35:11.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:35:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:35:11.952+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:35:11.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:35:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:35:11.952+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:35:12.369 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:35:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:35:12.023+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:35:12.369 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:35:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:35:12.023+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:35:12.369 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:35:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:35:12.023+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:35:12.731 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:35:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:35:12.365+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:35:12.731 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:35:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:35:12.365+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:35:12.731 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:35:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:35:12.365+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:35:12.731 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:35:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:35:12.366+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:35:12.993 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:35:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:35:12.727+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 30510 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:35:12.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:35:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:35:12.916+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:35:12.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:35:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:35:12.916+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:35:12.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:35:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:35:12.916+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:35:13.377 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:35:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:35:13.011+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:35:13.377 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:35:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:35:13.011+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:35:13.377 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:35:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:35:13.011+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:35:13.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:35:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:35:13.373+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:35:13.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:35:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:35:13.373+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:35:13.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:35:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:35:13.373+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:35:13.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:35:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:35:13.373+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:35:14.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:35:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:35:13.962+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:35:14.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:35:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:35:13.962+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:35:14.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:35:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:35:13.962+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:35:14.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:35:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:35:13.965+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:35:14.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:35:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:35:13.965+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:35:14.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:35:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:35:13.965+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:35:14.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:35:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:35:14.378+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:35:14.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:35:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:35:14.378+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:35:14.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:35:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:35:14.378+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:35:14.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:35:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:35:14.378+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:35:15.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:35:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:35:14.927+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:35:15.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:35:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:35:14.927+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:35:15.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:35:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:35:14.927+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:35:15.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:35:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:35:14.986+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:35:15.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:35:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:35:14.986+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:35:15.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:35:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:35:14.986+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:35:15.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:35:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:35:15.345+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:35:15.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:35:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:35:15.345+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:35:15.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:35:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:35:15.345+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:35:15.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:35:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:35:15.345+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:35:16.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:35:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:35:15.915+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:35:16.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:35:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:35:15.915+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:35:16.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:35:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:35:15.915+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:35:16.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:35:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:35:16.022+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:35:16.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:35:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:35:16.022+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:35:16.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:35:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:35:16.022+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:35:16.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:35:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:35:16.318+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:35:16.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:35:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:35:16.318+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:35:16.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:35:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:35:16.318+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:35:16.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:35:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:35:16.318+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:35:17.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:35:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:35:16.957+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:35:17.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:35:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:35:16.957+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:35:17.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:35:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:35:16.957+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:35:17.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:35:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:35:17.026+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:35:17.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:35:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:35:17.026+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:35:17.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:35:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:35:17.026+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:35:17.735 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:35:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:35:17.354+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:35:17.736 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:35:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:35:17.354+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:35:17.736 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:35:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:35:17.354+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:35:17.736 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:35:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:35:17.354+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:35:17.993 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:35:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:35:17.732+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 30581 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:35:17.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:35:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:35:17.913+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:35:17.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:35:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:35:17.913+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:35:17.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:35:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:35:17.913+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:35:18.345 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:35:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:35:18.033+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:35:18.345 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:35:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:35:18.033+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:35:18.345 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:35:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:35:18.033+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:35:18.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:35:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:35:18.342+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:35:18.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:35:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:35:18.342+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:35:18.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:35:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:35:18.342+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:35:18.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:35:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:35:18.342+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:35:19.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:35:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:35:18.958+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:35:19.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:35:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:35:18.958+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:35:19.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:35:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:35:18.958+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:35:19.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:35:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:35:19.057+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:35:19.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:35:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:35:19.057+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:35:19.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:35:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:35:19.057+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:35:19.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:35:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:35:19.360+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:35:19.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:35:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:35:19.360+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:35:19.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:35:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:35:19.360+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:35:19.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:35:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:35:19.360+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:35:20.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:35:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:35:19.933+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:35:20.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:35:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:35:19.933+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:35:20.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:35:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:35:19.933+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:35:20.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:35:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:35:20.012+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:35:20.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:35:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:35:20.012+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:35:20.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:35:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:35:20.012+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:35:20.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:35:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:35:20.399+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:35:20.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:35:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:35:20.399+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:35:20.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:35:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:35:20.399+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:35:20.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:35:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:35:20.399+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:35:21.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:35:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:35:20.977+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:35:21.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:35:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:35:20.977+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:35:21.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:35:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:35:20.977+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:35:21.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:35:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:35:21.029+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:35:21.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:35:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:35:21.029+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:35:21.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:35:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:35:21.029+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:35:21.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:35:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:35:21.431+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:35:21.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:35:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:35:21.431+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:35:21.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:35:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:35:21.431+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:35:21.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:35:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:35:21.431+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:35:22.067 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:35:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:35:22.004+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:35:22.067 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:35:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:35:22.004+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:35:22.067 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:35:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:35:22.004+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:35:22.466 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:35:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:35:22.064+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:35:22.467 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:35:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:35:22.064+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:35:22.467 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:35:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:35:22.064+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:35:22.740 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:35:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:35:22.463+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:35:22.740 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:35:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:35:22.463+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:35:22.740 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:35:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:35:22.463+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:35:22.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:35:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:35:22.463+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:35:22.993 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:35:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:35:22.737+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 30655 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:35:23.465 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:35:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:35:23.077+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:35:23.465 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:35:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:35:23.077+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:35:23.465 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:35:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:35:23.077+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:35:23.465 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:35:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:35:23.006+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:35:23.465 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:35:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:35:23.006+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:35:23.465 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:35:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:35:23.006+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:35:23.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:35:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:35:23.461+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:35:23.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:35:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:35:23.461+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:35:23.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:35:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:35:23.461+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:35:23.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:35:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:35:23.461+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:35:24.470 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:35:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:35:24.052+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:35:24.470 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:35:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:35:24.052+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:35:24.470 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:35:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:35:24.052+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:35:24.471 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:35:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:35:24.010+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:35:24.471 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:35:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:35:24.010+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:35:24.471 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:35:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:35:24.010+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:35:24.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:35:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:35:24.466+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:35:24.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:35:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:35:24.466+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:35:24.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:35:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:35:24.466+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:35:24.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:35:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:35:24.467+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:35:25.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:35:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:35:25.038+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:35:25.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:35:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:35:25.038+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:35:25.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:35:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:35:25.038+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:35:25.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:35:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:35:24.985+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:35:25.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:35:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:35:24.985+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:35:25.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:35:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:35:24.985+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:35:25.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:35:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:35:25.447+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:35:25.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:35:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:35:25.447+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:35:25.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:35:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:35:25.447+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:35:25.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:35:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:35:25.447+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:35:26.468 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:35:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:35:26.001+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:35:26.469 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:35:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:35:26.001+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:35:26.469 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:35:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:35:26.001+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:35:26.469 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:35:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:35:26.009+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:35:26.469 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:35:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:35:26.009+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:35:26.469 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:35:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:35:26.009+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:35:26.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:35:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:35:26.466+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:35:26.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:35:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:35:26.466+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:35:26.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:35:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:35:26.466+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:35:26.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:35:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:35:26.466+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:35:27.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:35:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:35:26.969+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:35:27.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:35:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:35:26.969+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:35:27.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:35:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:35:26.969+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:35:27.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:35:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:35:26.980+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:35:27.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:35:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:35:26.980+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:35:27.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:35:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:35:26.980+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:35:27.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:35:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:35:27.480+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:35:27.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:35:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:35:27.480+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:35:27.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:35:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:35:27.480+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:35:27.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:35:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:35:27.481+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:35:28.014 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:35:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:35:27.741+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 30726 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:35:28.014 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:35:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:35:27.972+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:35:28.014 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:35:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:35:27.972+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:35:28.014 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:35:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:35:27.972+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:35:28.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:35:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:35:28.010+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:35:28.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:35:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:35:28.010+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:35:28.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:35:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:35:28.010+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:35:28.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:35:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:35:28.498+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:35:28.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:35:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:35:28.498+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:35:28.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:35:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:35:28.498+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:35:28.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:35:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:35:28.498+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:35:29.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:35:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:35:28.994+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:35:29.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:35:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:35:28.994+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:35:29.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:35:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:35:28.994+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:35:29.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:35:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:35:28.999+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:35:29.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:35:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:35:28.999+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:35:29.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:35:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:35:28.999+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:35:29.886 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:35:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:35:29.499+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:35:29.886 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:35:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:35:29.499+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:35:29.886 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:35:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:35:29.499+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:35:29.887 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:35:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:35:29.500+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:35:30.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:35:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:35:29.963+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:35:30.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:35:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:35:29.963+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:35:30.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:35:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:35:29.963+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:35:30.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:35:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:35:29.968+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:35:30.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:35:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:35:29.968+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:35:30.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:35:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:35:29.968+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:35:30.972 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:35:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:35:30.505+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:35:30.972 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:35:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:35:30.505+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:35:30.972 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:35:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:35:30.505+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:35:30.973 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:35:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:35:30.505+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:35:31.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:35:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:35:30.969+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:35:31.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:35:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:35:30.969+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:35:31.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:35:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:35:30.969+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:35:31.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:35:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:35:30.970+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:35:31.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:35:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:35:30.970+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:35:31.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:35:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:35:30.970+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:35:31.789 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:35:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:35:31.524+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:35:31.790 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:35:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:35:31.524+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:35:31.790 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:35:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:35:31.524+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:35:31.790 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:35:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:35:31.524+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:35:32.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:35:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:35:32.013+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:35:32.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:35:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:35:32.013+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:35:32.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:35:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:35:32.013+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:35:32.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:35:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:35:31.950+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:35:32.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:35:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:35:31.950+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:35:32.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:35:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:35:31.950+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:35:32.872 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:35:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:35:32.746+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 30799 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:35:32.872 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:35:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:35:32.551+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:35:32.873 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:35:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:35:32.551+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:35:32.873 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:35:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:35:32.551+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:35:32.873 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:35:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:35:32.551+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:35:33.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:35:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:35:32.923+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:35:33.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:35:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:35:32.923+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:35:33.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:35:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:35:32.923+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:35:33.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:35:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:35:33.026+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:35:33.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:35:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:35:33.026+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:35:33.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:35:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:35:33.026+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:35:33.970 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:35:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:35:33.555+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:35:33.970 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:35:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:35:33.555+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:35:33.971 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:35:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:35:33.555+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:35:33.971 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:35:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:35:33.555+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:35:34.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:35:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:35:34.072+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:35:34.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:35:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:35:34.072+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:35:34.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:35:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:35:34.072+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:35:34.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:35:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:35:33.967+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:35:34.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:35:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:35:33.967+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:35:34.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:35:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:35:33.967+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:35:34.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:35:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:35:34.529+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:35:34.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:35:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:35:34.529+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:35:34.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:35:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:35:34.529+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:35:34.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:35:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:35:34.529+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:35:35.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:35:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:35:35.099+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:35:35.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:35:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:35:35.099+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:35:35.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:35:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:35:35.099+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:35:35.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:35:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:35:34.994+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:35:35.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:35:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:35:34.994+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:35:35.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:35:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:35:34.994+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:35:35.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:35:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:35:35.564+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:35:35.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:35:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:35:35.564+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:35:35.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:35:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:35:35.564+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:35:35.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:35:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:35:35.564+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:35:36.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:35:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:35:36.073+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:35:36.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:35:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:35:36.073+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:35:36.494 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:35:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:35:36.073+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:35:36.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:35:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:35:36.032+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:35:36.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:35:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:35:36.032+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:35:36.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:35:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:35:36.032+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:35:36.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:35:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:35:36.577+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:35:36.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:35:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:35:36.577+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:35:36.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:35:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:35:36.577+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:35:36.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:35:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:35:36.577+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:35:37.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:35:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:35:37.045+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:35:37.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:35:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:35:37.045+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:35:37.494 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:35:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:35:37.045+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:35:37.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:35:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:35:37.065+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:35:37.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:35:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:35:37.065+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:35:37.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:35:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:35:37.065+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:35:37.878 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:35:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:35:37.750+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 30870 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:35:37.878 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:35:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:35:37.531+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:35:37.878 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:35:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:35:37.531+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:35:37.878 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:35:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:35:37.531+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:35:37.878 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:35:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:35:37.531+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:35:38.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:35:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:35:38.019+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:35:38.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:35:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:35:38.019+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:35:38.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:35:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:35:38.019+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:35:38.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:35:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:35:38.052+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:35:38.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:35:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:35:38.052+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:35:38.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:35:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:35:38.052+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:35:38.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:35:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:35:38.488+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:35:38.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:35:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:35:38.488+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:35:38.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:35:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:35:38.488+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:35:38.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:35:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:35:38.488+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:35:39.453 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:35:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:35:39.018+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:35:39.453 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:35:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:35:39.018+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:35:39.453 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:35:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:35:39.018+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:35:39.453 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:35:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:35:39.058+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:35:39.454 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:35:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:35:39.058+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:35:39.454 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:35:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:35:39.058+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:35:39.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:35:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:35:39.449+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:35:39.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:35:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:35:39.449+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:35:39.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:35:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:35:39.449+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:35:39.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:35:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:35:39.450+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:35:40.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:35:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:35:40.032+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:35:40.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:35:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:35:40.032+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:35:40.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:35:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:35:40.032+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:35:40.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:35:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:35:40.049+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:35:40.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:35:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:35:40.049+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:35:40.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:35:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:35:40.049+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:35:40.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:35:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:35:40.423+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:35:40.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:35:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:35:40.423+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:35:40.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:35:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:35:40.423+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:35:40.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:35:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:35:40.423+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:35:41.452 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:35:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:35:41.027+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:35:41.453 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:35:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:35:41.027+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:35:41.453 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:35:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:35:41.027+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:35:41.453 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:35:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:35:41.000+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:35:41.453 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:35:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:35:41.000+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:35:41.453 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:35:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:35:41.000+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:35:41.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:35:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:35:41.450+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:35:41.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:35:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:35:41.450+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:35:41.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:35:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:35:41.450+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:35:41.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:35:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:35:41.450+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:35:42.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:35:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:35:42.035+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:35:42.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:35:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:35:42.035+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:35:42.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:35:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:35:42.035+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:35:42.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:35:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:35:42.000+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:35:42.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:35:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:35:42.000+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:35:42.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:35:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:35:42.000+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:35:42.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:35:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:35:42.482+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:35:42.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:35:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:35:42.482+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:35:42.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:35:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:35:42.482+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:35:42.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:35:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:35:42.482+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:35:43.243 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:35:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:35:42.756+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 30943 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:35:43.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:35:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:35:42.987+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:35:43.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:35:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:35:42.987+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:35:43.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:35:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:35:42.987+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:35:43.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:35:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:35:42.997+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:35:43.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:35:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:35:42.997+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:35:43.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:35:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:35:42.997+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:35:43.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:35:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:35:43.465+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:35:43.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:35:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:35:43.465+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:35:43.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:35:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:35:43.465+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:35:43.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:35:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:35:43.465+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:35:44.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:35:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:35:44.034+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:35:44.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:35:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:35:44.034+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:35:44.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:35:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:35:44.034+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:35:44.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:35:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:35:43.970+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:35:44.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:35:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:35:43.970+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:35:44.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:35:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:35:43.970+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:35:44.990 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:35:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:35:44.491+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:35:44.990 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:35:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:35:44.491+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:35:44.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:35:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:35:44.491+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:35:44.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:35:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:35:44.492+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:35:45.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:35:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:35:44.988+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:35:45.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:35:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:35:44.988+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:35:45.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:35:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:35:44.988+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:35:45.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:35:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:35:45.071+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:35:45.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:35:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:35:45.071+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:35:45.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:35:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:35:45.071+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:35:45.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:35:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:35:45.463+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:35:45.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:35:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:35:45.463+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:35:45.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:35:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:35:45.463+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:35:45.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:35:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:35:45.463+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:35:46.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:35:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:35:46.092+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:35:46.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:35:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:35:46.092+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:35:46.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:35:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:35:46.092+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:35:46.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:35:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:35:45.942+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:35:46.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:35:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:35:45.942+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:35:46.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:35:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:35:45.942+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:35:46.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:35:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:35:46.485+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:35:46.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:35:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:35:46.485+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:35:46.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:35:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:35:46.485+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:35:46.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:35:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:35:46.485+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:35:47.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:35:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:35:47.136+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:35:47.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:35:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:35:47.136+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:35:47.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:35:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:35:47.136+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:35:47.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:35:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:35:46.946+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:35:47.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:35:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:35:46.946+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:35:47.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:35:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:35:46.946+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:35:47.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:35:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:35:47.461+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:35:47.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:35:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:35:47.461+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:35:47.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:35:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:35:47.461+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:35:47.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:35:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:35:47.461+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:35:48.098 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:35:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:35:47.760+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 31014 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:35:48.099 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:35:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:35:47.946+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:35:48.099 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:35:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:35:47.946+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:35:48.099 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:35:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:35:47.946+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:35:48.441 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:35:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:35:48.096+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:35:48.441 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:35:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:35:48.096+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:35:48.441 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:35:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:35:48.096+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:35:48.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:35:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:35:48.438+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:35:48.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:35:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:35:48.438+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:35:48.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:35:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:35:48.438+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:35:48.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:35:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:35:48.438+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:35:49.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:35:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:35:49.049+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:35:49.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:35:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:35:49.049+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:35:49.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:35:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:35:49.049+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:35:49.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:35:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:35:48.917+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:35:49.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:35:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:35:48.917+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:35:49.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:35:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:35:48.917+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:35:49.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:35:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:35:49.437+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:35:49.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:35:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:35:49.437+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:35:49.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:35:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:35:49.437+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:35:49.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:35:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:35:49.437+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:35:50.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:35:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:35:49.880+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:35:50.355 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:35:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:35:49.880+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:35:50.355 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:35:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:35:49.880+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:35:50.355 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:35:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:35:50.064+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:35:50.355 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:35:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:35:50.064+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:35:50.355 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:35:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:35:50.064+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:35:50.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:35:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:35:50.434+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:35:50.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:35:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:35:50.434+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:35:50.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:35:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:35:50.434+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:35:50.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:35:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:35:50.434+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:35:51.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:35:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:35:50.921+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:35:51.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:35:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:35:50.921+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:35:51.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:35:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:35:50.921+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:35:51.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:35:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:35:51.054+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:35:51.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:35:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:35:51.054+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:35:51.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:35:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:35:51.054+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:35:51.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:35:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:35:51.471+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:35:51.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:35:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:35:51.471+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:35:51.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:35:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:35:51.471+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:35:51.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:35:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:35:51.471+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:35:52.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:35:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:35:52.026+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:35:52.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:35:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:35:52.026+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:35:52.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:35:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:35:52.026+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:35:52.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:35:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:35:51.965+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:35:52.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:35:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:35:51.965+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:35:52.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:35:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:35:51.965+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:35:52.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:35:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:35:52.443+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:35:52.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:35:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:35:52.443+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:35:52.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:35:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:35:52.443+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:35:52.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:35:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:35:52.443+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:35:53.035 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:35:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:35:52.764+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 31086 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:35:53.036 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:35:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:35:53.014+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:35:53.036 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:35:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:35:53.014+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:35:53.036 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:35:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:35:53.014+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:35:53.443 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:35:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:35:53.033+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:35:53.443 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:35:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:35:53.033+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:35:53.443 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:35:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:35:53.033+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:35:53.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:35:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:35:53.441+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:35:53.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:35:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:35:53.441+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:35:53.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:35:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:35:53.441+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:35:53.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:35:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:35:53.441+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:35:54.395 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:35:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:35:54.019+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:35:54.396 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:35:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:35:54.019+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:35:54.396 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:35:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:35:54.019+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:35:54.396 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:35:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:35:54.014+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:35:54.396 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:35:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:35:54.014+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:35:54.396 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:35:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:35:54.014+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:35:54.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:35:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:35:54.393+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:35:54.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:35:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:35:54.393+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:35:54.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:35:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:35:54.393+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:35:54.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:35:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:35:54.393+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:35:55.362 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:35:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:35:55.001+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:35:55.362 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:35:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:35:55.001+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:35:55.362 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:35:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:35:55.001+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:35:55.363 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:35:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:35:55.012+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:35:55.363 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:35:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:35:55.012+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:35:55.363 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:35:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:35:55.012+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:35:55.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:35:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:35:55.360+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:35:55.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:35:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:35:55.360+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:35:55.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:35:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:35:55.360+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:35:55.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:35:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:35:55.360+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:35:56.378 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:35:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:35:56.047+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:35:56.378 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:35:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:35:56.047+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:35:56.378 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:35:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:35:56.047+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:35:56.379 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:35:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:35:56.008+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:35:56.379 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:35:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:35:56.008+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:35:56.379 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:35:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:35:56.008+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:35:56.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:35:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:35:56.375+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:35:56.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:35:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:35:56.375+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:35:56.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:35:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:35:56.375+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:35:56.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:35:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:35:56.376+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:35:57.416 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:35:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:35:57.058+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:35:57.416 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:35:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:35:57.058+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:35:57.416 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:35:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:35:57.058+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:35:57.416 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:35:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:35:57.043+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:35:57.417 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:35:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:35:57.043+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:35:57.417 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:35:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:35:57.043+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:35:57.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:35:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:35:57.413+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:35:57.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:35:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:35:57.413+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:35:57.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:35:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:35:57.413+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:35:57.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:35:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:35:57.413+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:35:58.072 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:35:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:35:57.770+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 31158 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:35:58.072 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:35:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:35:58.000+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:35:58.072 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:35:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:35:58.000+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:35:58.072 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:35:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:35:58.000+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:35:58.378 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:35:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:35:58.069+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:35:58.378 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:35:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:35:58.069+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:35:58.379 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:35:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:35:58.069+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:35:58.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:35:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:35:58.376+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:35:58.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:35:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:35:58.376+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:35:58.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:35:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:35:58.376+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:35:58.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:35:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:35:58.376+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:35:59.339 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:35:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:35:59.064+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:35:59.339 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:35:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:35:59.064+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:35:59.339 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:35:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:35:59.064+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:35:59.340 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:35:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:35:59.045+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:35:59.340 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:35:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:35:59.045+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:35:59.340 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:35:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:35:59.045+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:35:59.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:35:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:35:59.337+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:35:59.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:35:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:35:59.337+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:35:59.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:35:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:35:59.337+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:35:59.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:35:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:35:59.337+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:36:00.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:36:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:36:00.088+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:36:00.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:36:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:36:00.088+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:36:00.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:36:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:36:00.088+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:36:00.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:36:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:36:00.006+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:36:00.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:36:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:36:00.006+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:36:00.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:36:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:36:00.006+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:36:00.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:36:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:36:00.344+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:36:00.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:36:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:36:00.344+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:36:00.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:36:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:36:00.344+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:36:00.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:36:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:36:00.344+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:36:01.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:36:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:36:01.051+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:36:01.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:36:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:36:01.051+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:36:01.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:36:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:36:01.051+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:36:01.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:36:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:36:00.989+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:36:01.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:36:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:36:00.989+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:36:01.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:36:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:36:00.989+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:36:01.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:36:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:36:01.338+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:36:01.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:36:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:36:01.338+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:36:01.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:36:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:36:01.338+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:36:01.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:36:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:36:01.339+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:36:02.072 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:36:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:36:02.023+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:36:02.072 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:36:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:36:02.023+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:36:02.072 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:36:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:36:02.023+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:36:02.366 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:36:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:36:02.070+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:36:02.366 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:36:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:36:02.070+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:36:02.366 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:36:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:36:02.070+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:36:02.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:36:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:36:02.364+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:36:02.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:36:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:36:02.364+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:36:02.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:36:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:36:02.364+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:36:02.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:36:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:36:02.364+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:36:03.037 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:36:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:36:02.773+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 31230 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:36:03.334 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:36:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:36:03.040+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:36:03.335 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:36:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:36:03.040+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:36:03.335 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:36:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:36:03.040+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:36:03.335 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:36:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:36:03.034+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:36:03.335 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:36:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:36:03.034+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:36:03.335 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:36:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:36:03.034+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:36:03.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:36:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:36:03.331+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:36:03.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:36:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:36:03.331+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:36:03.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:36:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:36:03.331+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:36:03.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:36:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:36:03.331+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:36:04.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:36:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:36:04.000+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:36:04.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:36:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:36:04.000+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:36:04.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:36:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:36:04.000+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:36:04.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:36:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:36:04.078+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:36:04.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:36:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:36:04.078+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:36:04.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:36:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:36:04.078+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:36:04.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:36:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:36:04.343+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:36:04.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:36:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:36:04.343+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:36:04.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:36:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:36:04.343+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:36:04.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:36:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:36:04.343+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:36:05.365 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:36:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:36:05.093+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:36:05.365 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:36:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:36:05.093+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:36:05.365 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:36:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:36:05.093+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:36:05.366 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:36:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:36:05.043+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:36:05.366 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:36:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:36:05.043+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:36:05.366 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:36:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:36:05.043+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:36:05.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:36:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:36:05.362+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:36:05.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:36:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:36:05.362+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:36:05.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:36:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:36:05.362+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:36:05.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:36:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:36:05.362+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:36:06.408 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:36:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:36:06.079+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:36:06.409 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:36:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:36:06.079+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:36:06.409 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:36:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:36:06.079+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:36:06.409 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:36:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:36:06.026+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:36:06.409 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:36:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:36:06.026+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:36:06.409 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:36:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:36:06.026+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:36:06.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:36:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:36:06.405+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:36:06.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:36:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:36:06.405+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:36:06.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:36:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:36:06.405+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:36:06.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:36:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:36:06.405+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:36:07.381 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:36:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:36:07.121+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:36:07.381 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:36:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:36:07.121+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:36:07.381 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:36:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:36:07.121+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:36:07.382 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:36:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:36:07.026+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:36:07.382 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:36:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:36:07.026+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:36:07.382 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:36:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:36:07.026+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:36:07.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:36:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:36:07.378+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:36:07.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:36:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:36:07.378+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:36:07.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:36:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:36:07.378+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:36:07.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:36:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:36:07.378+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:36:08.034 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:36:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:36:07.778+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 31302 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:36:08.340 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:36:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:36:08.031+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:36:08.340 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:36:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:36:08.031+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:36:08.340 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:36:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:36:08.031+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:36:08.341 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:36:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:36:08.086+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:36:08.341 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:36:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:36:08.086+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:36:08.341 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:36:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:36:08.086+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:36:08.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:36:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:36:08.337+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:36:08.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:36:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:36:08.337+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:36:08.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:36:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:36:08.337+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:36:08.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:36:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:36:08.337+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:36:09.365 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:36:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:36:09.066+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:36:09.365 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:36:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:36:09.066+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:36:09.365 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:36:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:36:09.066+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:36:09.365 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:36:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:36:09.064+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:36:09.365 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:36:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:36:09.064+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:36:09.366 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:36:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:36:09.064+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:36:09.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:36:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:36:09.361+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:36:09.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:36:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:36:09.361+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:36:09.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:36:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:36:09.361+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:36:09.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:36:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:36:09.361+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:36:10.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:36:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:36:10.017+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:36:10.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:36:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:36:10.017+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:36:10.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:36:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:36:10.017+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:36:10.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:36:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:36:10.033+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:36:10.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:36:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:36:10.033+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:36:10.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:36:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:36:10.033+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:36:10.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:36:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:36:10.404+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:36:10.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:36:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:36:10.404+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:36:10.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:36:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:36:10.404+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:36:10.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:36:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:36:10.404+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:36:11.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:36:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:36:10.980+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:36:11.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:36:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:36:10.980+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:36:11.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:36:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:36:10.980+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:36:11.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:36:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:36:11.051+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:36:11.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:36:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:36:11.051+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:36:11.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:36:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:36:11.051+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:36:11.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:36:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:36:11.355+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:36:11.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:36:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:36:11.355+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:36:11.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:36:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:36:11.355+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:36:11.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:36:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:36:11.355+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:36:12.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:36:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:36:11.977+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:36:12.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:36:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:36:11.977+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:36:12.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:36:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:36:11.977+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:36:12.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:36:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:36:12.024+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:36:12.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:36:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:36:12.024+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:36:12.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:36:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:36:12.024+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:36:12.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:36:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:36:12.394+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:36:12.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:36:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:36:12.394+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:36:12.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:36:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:36:12.394+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:36:12.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:36:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:36:12.394+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:36:13.177 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:36:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:36:12.783+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 31374 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:36:13.178 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:36:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:36:12.992+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:36:13.178 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:36:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:36:12.992+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:36:13.178 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:36:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:36:12.992+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:36:13.178 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:36:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:36:12.961+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:36:13.179 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:36:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:36:12.962+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:36:13.179 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:36:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:36:12.962+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:36:13.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:36:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:36:13.429+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:36:13.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:36:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:36:13.429+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:36:13.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:36:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:36:13.429+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:36:13.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:36:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:36:13.429+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:36:14.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:36:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:36:13.988+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:36:14.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:36:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:36:13.988+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:36:14.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:36:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:36:13.988+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:36:14.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:36:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:36:13.990+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:36:14.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:36:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:36:13.990+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:36:14.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:36:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:36:13.990+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:36:14.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:36:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:36:14.389+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:36:14.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:36:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:36:14.389+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:36:14.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:36:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:36:14.389+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:36:14.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:36:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:36:14.389+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:36:15.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:36:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:36:14.982+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:36:15.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:36:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:36:14.982+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:36:15.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:36:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:36:14.982+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:36:15.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:36:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:36:15.032+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:36:15.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:36:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:36:15.032+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:36:15.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:36:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:36:15.032+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:36:15.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:36:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:36:15.348+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:36:15.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:36:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:36:15.348+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:36:15.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:36:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:36:15.348+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:36:15.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:36:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:36:15.348+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:36:16.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:36:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:36:15.941+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:36:16.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:36:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:36:15.941+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:36:16.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:36:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:36:15.941+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:36:16.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:36:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:36:16.078+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:36:16.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:36:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:36:16.078+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:36:16.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:36:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:36:16.079+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:36:16.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:36:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:36:16.299+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:36:16.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:36:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:36:16.299+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:36:16.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:36:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:36:16.299+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:36:16.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:36:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:36:16.300+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:36:17.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:36:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:36:16.907+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:36:17.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:36:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:36:16.907+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:36:17.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:36:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:36:16.907+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:36:17.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:36:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:36:17.100+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:36:17.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:36:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:36:17.100+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:36:17.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:36:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:36:17.100+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:36:17.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:36:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:36:17.303+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:36:17.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:36:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:36:17.303+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:36:17.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:36:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:36:17.303+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:36:17.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:36:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:36:17.303+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:36:18.083 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:36:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:36:17.788+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 31447 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:36:18.083 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:36:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:36:17.877+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:36:18.083 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:36:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:36:17.877+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:36:18.083 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:36:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:36:17.877+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:36:18.349 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:36:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:36:18.080+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:36:18.349 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:36:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:36:18.080+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:36:18.349 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:36:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:36:18.080+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:36:18.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:36:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:36:18.346+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:36:18.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:36:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:36:18.346+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:36:18.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:36:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:36:18.346+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:36:18.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:36:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:36:18.346+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:36:19.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:36:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:36:18.860+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:36:19.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:36:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:36:18.860+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:36:19.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:36:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:36:18.860+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:36:19.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:36:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:36:19.054+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:36:19.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:36:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:36:19.054+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:36:19.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:36:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:36:19.054+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:36:19.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:36:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:36:19.309+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:36:19.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:36:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:36:19.309+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:36:19.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:36:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:36:19.309+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:36:19.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:36:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:36:19.309+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:36:20.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:36:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:36:19.882+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:36:20.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:36:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:36:19.882+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:36:20.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:36:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:36:19.882+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:36:20.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:36:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:36:20.054+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:36:20.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:36:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:36:20.054+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:36:20.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:36:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:36:20.054+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:36:20.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:36:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:36:20.349+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:36:20.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:36:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:36:20.349+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:36:20.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:36:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:36:20.349+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:36:20.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:36:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:36:20.349+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:36:21.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:36:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:36:21.050+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:36:21.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:36:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:36:21.050+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:36:21.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:36:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:36:21.050+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:36:21.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:36:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:36:20.921+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:36:21.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:36:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:36:20.921+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:36:21.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:36:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:36:20.921+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:36:21.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:36:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:36:21.335+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:36:21.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:36:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:36:21.335+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:36:21.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:36:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:36:21.335+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:36:21.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:36:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:36:21.336+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:36:22.097 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:36:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:36:21.902+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:36:22.097 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:36:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:36:21.902+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:36:22.097 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:36:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:36:21.902+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:36:22.367 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:36:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:36:22.094+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:36:22.367 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:36:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:36:22.094+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:36:22.367 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:36:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:36:22.094+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:36:22.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:36:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:36:22.364+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:36:22.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:36:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:36:22.364+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:36:22.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:36:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:36:22.364+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:36:22.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:36:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:36:22.364+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:36:23.085 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:36:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:36:22.793+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 31519 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:36:23.086 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:36:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:36:22.917+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:36:23.086 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:36:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:36:22.917+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:36:23.086 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:36:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:36:22.917+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:36:23.346 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:36:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:36:23.083+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:36:23.346 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:36:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:36:23.083+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:36:23.346 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:36:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:36:23.083+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:36:23.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:36:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:36:23.343+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:36:23.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:36:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:36:23.343+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:36:23.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:36:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:36:23.343+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:36:23.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:36:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:36:23.343+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:36:24.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:36:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:36:24.090+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:36:24.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:36:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:36:24.090+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:36:24.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:36:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:36:24.090+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:36:24.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:36:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:36:23.959+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:36:24.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:36:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:36:23.959+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:36:24.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:36:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:36:23.959+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:36:24.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:36:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:36:24.385+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:36:24.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:36:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:36:24.385+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:36:24.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:36:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:36:24.385+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:36:24.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:36:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:36:24.385+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:36:25.428 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:36:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:36:24.992+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:36:25.428 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:36:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:36:24.992+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:36:25.429 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:36:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:36:24.992+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:36:25.429 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:36:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:36:25.078+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:36:25.429 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:36:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:36:25.078+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:36:25.429 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:36:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:36:25.078+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:36:25.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:36:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:36:25.425+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:36:25.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:36:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:36:25.425+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:36:25.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:36:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:36:25.425+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:36:25.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:36:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:36:25.425+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:36:26.477 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:36:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:36:26.036+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:36:26.478 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:36:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:36:26.036+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:36:26.478 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:36:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:36:26.036+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:36:26.478 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:36:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:36:26.051+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:36:26.478 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:36:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:36:26.051+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:36:26.478 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:36:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:36:26.051+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:36:26.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:36:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:36:26.474+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:36:26.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:36:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:36:26.474+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:36:26.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:36:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:36:26.474+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:36:26.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:36:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:36:26.474+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:36:27.439 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:36:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:36:27.007+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:36:27.439 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:36:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:36:27.007+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:36:27.439 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:36:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:36:27.007+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:36:27.440 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:36:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:36:27.032+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:36:27.440 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:36:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:36:27.032+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:36:27.440 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:36:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:36:27.032+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:36:27.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:36:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:36:27.436+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:36:27.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:36:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:36:27.436+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:36:27.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:36:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:36:27.436+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:36:27.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:36:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:36:27.436+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:36:28.070 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:36:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:36:27.798+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 31591 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:36:28.071 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:36:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:36:27.962+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:36:28.071 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:36:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:36:27.962+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:36:28.071 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:36:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:36:27.962+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:36:28.417 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:36:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:36:28.067+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:36:28.417 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:36:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:36:28.067+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:36:28.417 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:36:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:36:28.067+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:36:28.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:36:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:36:28.414+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:36:28.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:36:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:36:28.414+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:36:28.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:36:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:36:28.414+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:36:28.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:36:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:36:28.414+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:36:29.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:36:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:36:28.972+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:36:29.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:36:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:36:28.972+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:36:29.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:36:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:36:28.972+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:36:29.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:36:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:36:29.036+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:36:29.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:36:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:36:29.036+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:36:29.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:36:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:36:29.036+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:36:29.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:36:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:36:29.379+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:36:29.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:36:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:36:29.379+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:36:29.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:36:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:36:29.379+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:36:29.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:36:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:36:29.379+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:36:30.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:36:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:36:29.952+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:36:30.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:36:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:36:29.952+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:36:30.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:36:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:36:29.952+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:36:30.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:36:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:36:30.072+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:36:30.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:36:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:36:30.072+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:36:30.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:36:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:36:30.072+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:36:30.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:36:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:36:30.418+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:36:30.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:36:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:36:30.418+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:36:30.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:36:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:36:30.418+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:36:30.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:36:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:36:30.418+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:36:31.436 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:36:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:36:31.003+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:36:31.436 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:36:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:36:31.003+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:36:31.436 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:36:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:36:31.003+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:36:31.437 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:36:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:36:31.115+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:36:31.437 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:36:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:36:31.115+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:36:31.437 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:36:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:36:31.115+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:36:31.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:36:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:36:31.432+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:36:31.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:36:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:36:31.432+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:36:31.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:36:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:36:31.432+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:36:31.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:36:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:36:31.432+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:36:32.128 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:36:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:36:31.987+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:36:32.129 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:36:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:36:31.987+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:36:32.130 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:36:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:36:31.987+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:36:32.436 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:36:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:36:32.117+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:36:32.436 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:36:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:36:32.117+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:36:32.437 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:36:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:36:32.117+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:36:32.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:36:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:36:32.433+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:36:32.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:36:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:36:32.433+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:36:32.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:36:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:36:32.433+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:36:32.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:36:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:36:32.433+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:36:33.158 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:36:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:36:32.803+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 31663 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:36:33.158 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:36:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:36:32.952+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:36:33.158 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:36:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:36:32.952+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:36:33.158 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:36:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:36:32.952+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:36:33.419 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:36:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:36:33.154+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:36:33.419 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:36:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:36:33.154+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:36:33.420 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:36:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:36:33.154+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:36:33.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:36:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:36:33.417+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:36:33.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:36:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:36:33.417+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:36:33.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:36:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:36:33.417+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:36:33.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:36:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:36:33.417+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:36:34.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:36:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:36:33.961+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:36:34.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:36:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:36:33.961+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:36:34.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:36:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:36:33.961+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:36:34.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:36:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:36:34.186+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:36:34.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:36:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:36:34.186+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:36:34.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:36:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:36:34.186+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:36:34.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:36:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:36:34.452+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:36:34.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:36:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:36:34.452+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:36:34.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:36:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:36:34.452+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:36:34.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:36:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:36:34.452+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:36:35.473 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:36:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:36:35.006+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:36:35.473 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:36:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:36:35.006+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:36:35.473 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:36:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:36:35.006+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:36:35.473 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:36:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:36:35.189+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:36:35.473 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:36:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:36:35.189+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:36:35.473 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:36:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:36:35.189+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:36:35.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:36:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:36:35.470+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:36:35.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:36:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:36:35.470+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:36:35.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:36:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:36:35.470+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:36:35.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:36:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:36:35.470+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:36:36.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:36:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:36:35.973+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:36:36.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:36:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:36:35.973+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:36:36.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:36:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:36:35.973+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:36:36.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:36:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:36:36.142+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:36:36.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:36:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:36:36.142+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:36:36.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:36:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:36:36.142+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:36:36.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:36:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:36:36.436+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:36:36.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:36:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:36:36.436+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:36:36.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:36:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:36:36.436+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:36:36.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:36:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:36:36.436+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:36:37.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:36:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:36:36.936+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:36:37.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:36:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:36:36.936+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:36:37.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:36:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:36:36.936+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:36:37.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:36:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:36:37.164+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:36:37.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:36:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:36:37.164+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:36:37.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:36:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:36:37.164+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:36:37.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:36:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:36:37.401+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:36:37.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:36:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:36:37.401+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:36:37.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:36:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:36:37.401+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:36:37.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:36:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:36:37.401+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:36:38.208 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:36:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:36:37.808+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 31735 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:36:38.208 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:36:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:36:37.935+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:36:38.208 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:36:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:36:37.935+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:36:38.209 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:36:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:36:37.935+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:36:38.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:36:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:36:38.401+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:36:38.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:36:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:36:38.402+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:36:38.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:36:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:36:38.402+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:36:38.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:36:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:36:38.402+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:36:38.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:36:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:36:38.206+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:36:38.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:36:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:36:38.206+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:36:38.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:36:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:36:38.206+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:36:39.174 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:36:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:36:38.891+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:36:39.174 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:36:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:36:38.891+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:36:39.174 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:36:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:36:38.891+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:36:39.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:36:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:36:39.172+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:36:39.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:36:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:36:39.172+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:36:39.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:36:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:36:39.172+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:36:39.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:36:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:36:39.366+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:36:39.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:36:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:36:39.366+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:36:39.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:36:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:36:39.366+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:36:39.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:36:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:36:39.366+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:36:40.207 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:36:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:36:39.915+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:36:40.208 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:36:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:36:39.915+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:36:40.208 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:36:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:36:39.915+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:36:40.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:36:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:36:40.204+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:36:40.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:36:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:36:40.205+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:36:40.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:36:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:36:40.205+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:36:40.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:36:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:36:40.377+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:36:40.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:36:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:36:40.377+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:36:40.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:36:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:36:40.377+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:36:40.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:36:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:36:40.377+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:36:41.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:36:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:36:40.890+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:36:41.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:36:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:36:40.890+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:36:41.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:36:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:36:40.890+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:36:41.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:36:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:36:41.248+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:36:41.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:36:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:36:41.248+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:36:41.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:36:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:36:41.248+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:36:41.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:36:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:36:41.402+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:36:41.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:36:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:36:41.402+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:36:41.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:36:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:36:41.402+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:36:41.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:36:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:36:41.402+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:36:42.238 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:36:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:36:41.858+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:36:42.239 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:36:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:36:41.858+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:36:42.239 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:36:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:36:41.858+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:36:42.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:36:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:36:42.236+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:36:42.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:36:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:36:42.236+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:36:42.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:36:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:36:42.236+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:36:42.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:36:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:36:42.362+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:36:42.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:36:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:36:42.362+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:36:42.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:36:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:36:42.362+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:36:42.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:36:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:36:42.362+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:36:43.243 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:36:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:36:42.813+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 31807 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:36:43.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:36:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:36:42.901+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:36:43.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:36:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:36:42.901+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:36:43.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:36:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:36:42.901+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:36:43.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:36:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:36:43.267+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:36:43.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:36:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:36:43.267+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:36:43.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:36:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:36:43.267+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:36:43.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:36:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:36:43.360+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:36:43.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:36:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:36:43.360+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:36:43.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:36:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:36:43.360+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:36:43.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:36:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:36:43.360+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:36:44.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:36:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:36:43.893+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:36:44.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:36:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:36:43.893+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:36:44.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:36:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:36:43.893+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:36:44.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:36:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:36:44.240+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:36:44.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:36:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:36:44.240+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:36:44.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:36:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:36:44.240+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:36:44.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:36:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:36:44.386+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:36:44.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:36:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:36:44.386+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:36:44.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:36:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:36:44.386+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:36:44.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:36:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:36:44.386+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:36:45.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:36:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:36:44.872+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:36:45.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:36:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:36:44.872+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:36:45.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:36:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:36:44.872+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:36:45.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:36:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:36:45.257+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:36:45.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:36:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:36:45.257+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:36:45.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:36:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:36:45.257+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:36:45.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:36:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:36:45.408+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:36:45.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:36:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:36:45.408+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:36:45.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:36:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:36:45.408+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:36:45.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:36:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:36:45.409+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:36:46.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:36:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:36:45.859+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:36:46.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:36:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:36:45.859+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:36:46.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:36:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:36:45.859+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:36:46.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:36:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:36:46.266+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:36:46.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:36:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:36:46.266+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:36:46.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:36:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:36:46.266+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:36:46.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:36:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:36:46.388+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:36:46.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:36:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:36:46.388+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:36:46.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:36:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:36:46.388+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:36:46.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:36:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:36:46.388+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:36:47.226 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:36:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:36:46.907+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:36:47.227 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:36:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:36:46.907+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:36:47.227 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:36:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:36:46.907+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:36:47.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:36:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:36:47.223+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:36:47.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:36:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:36:47.223+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:36:47.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:36:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:36:47.223+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:36:47.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:36:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:36:47.427+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:36:47.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:36:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:36:47.427+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:36:47.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:36:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:36:47.427+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:36:47.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:36:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:36:47.427+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:36:48.220 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:36:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:36:47.818+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 31879 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:36:48.220 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:36:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:36:47.907+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:36:48.221 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:36:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:36:47.907+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:36:48.221 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:36:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:36:47.907+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:36:48.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:36:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:36:48.217+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:36:48.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:36:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:36:48.217+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:36:48.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:36:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:36:48.217+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:36:48.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:36:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:36:48.431+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:36:48.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:36:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:36:48.431+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:36:48.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:36:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:36:48.431+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:36:48.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:36:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:36:48.431+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:36:49.237 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:36:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:36:48.922+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:36:49.238 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:36:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:36:48.922+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:36:49.238 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:36:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:36:48.922+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:36:49.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:36:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:36:49.235+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:36:49.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:36:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:36:49.235+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:36:49.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:36:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:36:49.235+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:36:49.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:36:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:36:49.412+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:36:49.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:36:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:36:49.412+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:36:49.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:36:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:36:49.412+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:36:49.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:36:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:36:49.412+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:36:50.191 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:36:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:36:49.916+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:36:50.192 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:36:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:36:49.916+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:36:50.192 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:36:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:36:49.916+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:36:50.450 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:36:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:36:50.189+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:36:50.450 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:36:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:36:50.189+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:36:50.450 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:36:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:36:50.189+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:36:50.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:36:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:36:50.448+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:36:50.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:36:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:36:50.448+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:36:50.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:36:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:36:50.448+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:36:50.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:36:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:36:50.448+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:36:51.226 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:36:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:36:50.893+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:36:51.226 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:36:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:36:50.893+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:36:51.226 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:36:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:36:50.893+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:36:51.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:36:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:36:51.223+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:36:51.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:36:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:36:51.223+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:36:51.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:36:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:36:51.223+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:36:51.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:36:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:36:51.443+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:36:51.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:36:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:36:51.443+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:36:51.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:36:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:36:51.443+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:36:51.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:36:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:36:51.444+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:36:52.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:36:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:36:51.870+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:36:52.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:36:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:36:51.870+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:36:52.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:36:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:36:51.870+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:36:52.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:36:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:36:52.243+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:36:52.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:36:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:36:52.243+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:36:52.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:36:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:36:52.243+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:36:52.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:36:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:36:52.492+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:36:52.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:36:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:36:52.492+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:36:52.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:36:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:36:52.492+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:36:52.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:36:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:36:52.492+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:36:53.113 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:36:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:36:52.823+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 31951 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:36:53.114 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:36:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:36:52.890+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:36:53.114 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:36:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:36:52.890+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:36:53.114 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:36:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:36:52.890+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:36:53.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:36:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:36:53.267+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:36:53.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:36:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:36:53.267+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:36:53.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:36:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:36:53.267+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:36:53.920 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:36:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:36:53.518+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:36:53.921 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:36:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:36:53.518+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:36:53.921 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:36:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:36:53.518+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:36:53.921 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:36:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:36:53.518+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:36:54.245 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:36:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:36:53.918+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:36:54.245 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:36:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:36:53.918+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:36:54.245 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:36:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:36:53.918+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:36:54.561 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:36:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:36:54.305+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:36:54.561 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:36:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:36:54.305+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:36:54.561 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:36:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:36:54.305+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:36:54.938 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:36:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:36:54.558+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:36:54.938 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:36:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:36:54.558+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:36:54.938 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:36:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:36:54.558+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:36:54.938 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:36:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:36:54.558+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:36:55.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:36:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:36:54.935+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:36:55.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:36:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:36:54.935+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:36:55.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:36:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:36:54.935+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:36:55.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:36:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:36:55.348+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:36:55.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:36:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:36:55.348+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:36:55.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:36:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:36:55.348+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:36:55.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:36:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:36:55.563+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:36:55.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:36:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:36:55.563+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:36:55.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:36:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:36:55.563+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:36:55.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:36:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:36:55.563+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:36:56.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:36:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:36:55.889+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:36:56.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:36:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:36:55.889+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:36:56.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:36:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:36:55.889+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:36:56.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:36:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:36:56.398+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:36:56.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:36:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:36:56.398+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:36:56.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:36:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:36:56.398+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:36:56.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:36:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:36:56.522+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:36:56.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:36:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:36:56.522+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:36:56.745 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:36:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:36:56.522+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:36:56.745 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:36:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:36:56.522+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:36:57.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:36:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:36:56.857+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:36:57.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:36:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:36:56.857+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:36:57.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:36:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:36:56.857+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:36:57.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:36:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:36:57.434+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:36:57.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:36:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:36:57.434+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:36:57.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:36:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:36:57.434+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:36:57.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:36:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:36:57.540+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:36:57.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:36:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:36:57.540+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:36:57.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:36:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:36:57.540+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:36:57.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:36:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:36:57.540+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:36:58.243 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:36:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:36:57.827+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 32023 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:36:58.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:36:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:36:57.874+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:36:58.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:36:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:36:57.874+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:36:58.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:36:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:36:57.874+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:36:58.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:36:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:36:58.420+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:36:58.745 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:36:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:36:58.420+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:36:58.745 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:36:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:36:58.420+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:36:58.745 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:36:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:36:58.497+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:36:58.745 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:36:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:36:58.497+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:36:58.745 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:36:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:36:58.497+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:36:58.746 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:36:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:36:58.497+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:36:59.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:36:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:36:58.866+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:36:59.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:36:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:36:58.866+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:36:59.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:36:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:36:58.866+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:36:59.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:36:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:36:59.433+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:36:59.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:36:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:36:59.433+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:36:59.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:36:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:36:59.433+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:36:59.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:36:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:36:59.536+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:36:59.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:36:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:36:59.536+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:36:59.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:36:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:36:59.536+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:36:59.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:36:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:36:59.536+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:37:00.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:36:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:36:59.868+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:37:00.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:36:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:36:59.868+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:37:00.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:36:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:36:59.868+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:37:00.515 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:37:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:37:00.445+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:37:00.515 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:37:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:37:00.445+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:37:00.516 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:37:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:37:00.445+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:37:00.912 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:37:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:37:00.512+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:37:00.912 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:37:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:37:00.512+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:37:00.912 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:37:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:37:00.512+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:37:00.912 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:37:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:37:00.512+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:37:01.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:37:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:37:00.909+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:37:01.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:37:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:37:00.909+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:37:01.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:37:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:37:00.909+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:37:01.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:37:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:37:01.467+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:37:01.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:37:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:37:01.467+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:37:01.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:37:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:37:01.467+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:37:01.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:37:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:37:01.558+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:37:01.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:37:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:37:01.558+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:37:01.745 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:37:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:37:01.558+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:37:01.745 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:37:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:37:01.558+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:37:02.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:37:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:37:01.945+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:37:02.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:37:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:37:01.945+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:37:02.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:37:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:37:01.945+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:37:02.745 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:37:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:37:02.505+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:37:02.745 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:37:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:37:02.505+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:37:02.745 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:37:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:37:02.505+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:37:02.745 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:37:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:37:02.524+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:37:02.746 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:37:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:37:02.524+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:37:02.746 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:37:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:37:02.524+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:37:02.746 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:37:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:37:02.525+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:37:03.243 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:37:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:37:02.832+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 32095 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:37:03.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:37:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:37:02.993+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:37:03.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:37:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:37:02.993+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:37:03.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:37:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:37:02.993+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:37:03.960 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:37:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:37:03.560+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:37:03.961 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:37:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:37:03.560+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:37:03.961 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:37:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:37:03.560+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:37:03.961 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:37:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:37:03.560+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:37:03.961 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:37:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:37:03.526+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:37:03.961 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:37:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:37:03.526+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:37:03.961 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:37:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:37:03.526+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:37:04.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:37:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:37:03.957+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:37:04.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:37:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:37:03.957+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:37:04.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:37:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:37:03.957+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:37:04.935 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:37:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:37:04.534+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:37:04.936 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:37:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:37:04.534+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:37:04.936 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:37:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:37:04.534+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:37:04.936 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:37:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:37:04.534+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:37:04.936 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:37:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:37:04.553+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:37:04.936 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:37:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:37:04.553+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:37:04.937 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:37:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:37:04.553+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:37:05.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:37:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:37:04.932+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:37:05.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:37:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:37:04.932+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:37:05.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:37:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:37:04.932+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:37:05.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:37:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:37:05.488+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:37:05.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:37:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:37:05.488+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:37:05.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:37:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:37:05.488+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:37:05.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:37:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:37:05.488+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:37:05.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:37:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:37:05.589+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:37:05.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:37:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:37:05.589+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:37:05.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:37:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:37:05.589+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:37:06.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:37:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:37:05.904+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:37:06.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:37:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:37:05.904+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:37:06.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:37:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:37:05.904+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:37:06.878 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:37:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:37:06.587+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:37:06.878 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:37:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:37:06.587+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:37:06.878 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:37:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:37:06.587+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:37:06.878 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:37:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:37:06.509+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:37:06.878 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:37:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:37:06.509+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:37:06.878 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:37:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:37:06.509+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:37:06.879 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:37:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:37:06.509+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:37:07.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:37:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:37:06.875+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:37:07.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:37:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:37:06.875+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:37:07.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:37:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:37:06.875+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:37:07.841 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:37:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:37:07.550+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:37:07.841 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:37:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:37:07.550+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:37:07.841 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:37:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:37:07.550+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:37:07.841 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:37:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:37:07.559+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:37:07.842 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:37:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:37:07.559+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:37:07.842 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:37:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:37:07.559+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:37:07.842 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:37:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:37:07.559+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:37:08.244 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:37:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:37:07.837+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 32167 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:37:08.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:37:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:37:07.841+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:37:08.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:37:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:37:07.841+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:37:08.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:37:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:37:07.841+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:37:08.861 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:37:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:37:08.506+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:37:08.861 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:37:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:37:08.506+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:37:08.862 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:37:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:37:08.506+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:37:08.862 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:37:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:37:08.580+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:37:08.862 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:37:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:37:08.580+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:37:08.862 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:37:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:37:08.580+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:37:08.862 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:37:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:37:08.580+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:37:09.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:37:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:37:08.858+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:37:09.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:37:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:37:08.858+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:37:09.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:37:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:37:08.858+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:37:09.831 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:37:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:37:09.496+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:37:09.831 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:37:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:37:09.496+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:37:09.832 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:37:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:37:09.496+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:37:09.832 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:37:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:37:09.597+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:37:09.832 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:37:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:37:09.597+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:37:09.832 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:37:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:37:09.597+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:37:09.832 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:37:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:37:09.597+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:37:10.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:37:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:37:09.828+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:37:10.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:37:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:37:09.828+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:37:10.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:37:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:37:09.828+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:37:10.581 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:37:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:37:10.483+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:37:10.581 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:37:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:37:10.483+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:37:10.581 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:37:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:37:10.483+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:37:10.832 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:37:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:37:10.578+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:37:10.832 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:37:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:37:10.578+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:37:10.832 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:37:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:37:10.578+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:37:10.832 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:37:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:37:10.578+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:37:11.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:37:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:37:10.829+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:37:11.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:37:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:37:10.829+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:37:11.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:37:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:37:10.829+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:37:11.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:37:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:37:11.453+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:37:11.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:37:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:37:11.453+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:37:11.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:37:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:37:11.453+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:37:11.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:37:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:37:11.538+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:37:11.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:37:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:37:11.538+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:37:11.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:37:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:37:11.538+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:37:11.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:37:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:37:11.538+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:37:12.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:37:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:37:11.818+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:37:12.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:37:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:37:11.818+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:37:12.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:37:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:37:11.818+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:37:12.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:37:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:37:12.452+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:37:12.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:37:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:37:12.452+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:37:12.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:37:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:37:12.452+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:37:12.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:37:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:37:12.538+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:37:12.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:37:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:37:12.538+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:37:12.745 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:37:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:37:12.538+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:37:12.745 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:37:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:37:12.538+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:37:13.243 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:37:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:37:12.842+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 32239 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:37:13.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:37:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:37:12.842+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:37:13.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:37:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:37:12.842+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:37:13.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:37:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:37:12.842+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:37:13.831 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:37:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:37:13.581+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:37:13.831 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:37:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:37:13.581+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:37:13.832 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:37:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:37:13.581+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:37:13.832 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:37:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:37:13.581+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:37:13.832 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:37:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:37:13.499+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:37:13.832 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:37:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:37:13.499+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:37:13.832 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:37:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:37:13.499+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:37:14.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:37:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:37:13.828+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:37:14.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:37:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:37:13.828+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:37:14.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:37:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:37:13.828+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:37:14.836 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:37:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:37:14.540+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:37:14.836 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:37:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:37:14.540+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:37:14.836 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:37:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:37:14.540+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:37:14.837 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:37:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:37:14.563+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:37:14.837 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:37:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:37:14.563+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:37:14.837 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:37:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:37:14.563+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:37:14.837 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:37:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:37:14.563+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:37:15.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:37:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:37:14.833+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:37:15.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:37:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:37:14.833+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:37:15.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:37:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:37:14.833+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:37:15.860 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:37:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:37:15.584+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:37:15.860 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:37:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:37:15.584+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:37:15.860 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:37:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:37:15.584+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:37:15.861 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:37:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:37:15.571+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:37:15.861 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:37:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:37:15.571+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:37:15.861 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:37:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:37:15.571+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:37:15.861 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:37:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:37:15.571+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:37:16.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:37:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:37:15.857+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:37:16.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:37:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:37:15.857+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:37:16.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:37:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:37:15.857+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:37:16.867 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:37:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:37:16.573+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:37:16.867 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:37:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:37:16.573+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:37:16.867 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:37:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:37:16.573+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:37:16.868 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:37:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:37:16.602+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:37:16.868 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:37:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:37:16.602+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:37:16.868 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:37:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:37:16.602+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:37:16.868 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:37:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:37:16.602+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:37:17.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:37:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:37:16.864+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:37:17.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:37:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:37:16.864+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:37:17.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:37:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:37:16.864+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:37:17.851 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:37:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:37:17.590+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:37:17.851 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:37:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:37:17.590+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:37:17.851 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:37:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:37:17.590+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:37:17.851 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:37:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:37:17.652+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:37:17.852 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:37:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:37:17.652+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:37:17.852 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:37:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:37:17.652+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:37:17.852 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:37:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:37:17.652+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:37:18.243 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:37:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:37:17.847+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 32311 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:37:18.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:37:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:37:17.873+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:37:18.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:37:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:37:17.873+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:37:18.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:37:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:37:17.873+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:37:18.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:37:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:37:18.832+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:37:18.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:37:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:37:18.832+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:37:18.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:37:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:37:18.832+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:37:18.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:37:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:37:18.616+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:37:18.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:37:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:37:18.616+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:37:18.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:37:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:37:18.616+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:37:18.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:37:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:37:18.616+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:37:18.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:37:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:37:18.634+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:37:18.995 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:37:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:37:18.634+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:37:18.995 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:37:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:37:18.634+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:37:19.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:37:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:37:19.844+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:37:19.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:37:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:37:19.844+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:37:19.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:37:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:37:19.844+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:37:19.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:37:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:37:19.641+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:37:19.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:37:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:37:19.641+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:37:19.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:37:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:37:19.641+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:37:19.995 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:37:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:37:19.641+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:37:19.995 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:37:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:37:19.683+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:37:19.995 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:37:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:37:19.683+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:37:19.995 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:37:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:37:19.683+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:37:20.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:37:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:37:20.667+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:37:20.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:37:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:37:20.667+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:37:20.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:37:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:37:20.667+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:37:20.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:37:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:37:20.667+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:37:20.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:37:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:37:20.878+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:37:20.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:37:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:37:20.878+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:37:20.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:37:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:37:20.878+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:37:20.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:37:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:37:20.642+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:37:20.995 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:37:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:37:20.642+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:37:20.995 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:37:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:37:20.642+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:37:21.860 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:37:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:37:21.604+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:37:21.860 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:37:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:37:21.604+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:37:21.860 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:37:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:37:21.604+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:37:21.861 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:37:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:37:21.703+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:37:21.861 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:37:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:37:21.703+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:37:21.861 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:37:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:37:21.704+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:37:21.861 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:37:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:37:21.704+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:37:22.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:37:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:37:21.857+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:37:22.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:37:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:37:21.857+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:37:22.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:37:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:37:21.857+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:37:22.876 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:37:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:37:22.852+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 32383 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:37:22.877 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:37:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:37:22.621+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:37:22.877 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:37:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:37:22.621+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:37:22.877 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:37:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:37:22.621+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:37:22.877 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:37:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:37:22.848+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:37:22.877 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:37:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:37:22.848+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:37:22.878 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:37:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:37:22.848+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:37:22.878 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:37:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:37:22.704+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:37:22.878 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:37:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:37:22.704+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:37:22.878 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:37:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:37:22.704+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:37:22.878 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:37:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:37:22.704+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:37:23.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:37:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:37:23.609+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:37:23.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:37:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:37:23.609+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:37:23.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:37:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:37:23.609+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:37:23.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:37:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:37:23.822+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:37:23.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:37:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:37:23.822+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:37:23.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:37:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:37:23.822+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:37:23.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:37:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:37:23.709+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:37:23.995 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:37:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:37:23.709+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:37:23.995 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:37:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:37:23.709+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:37:23.995 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:37:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:37:23.709+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:37:24.865 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:37:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:37:24.582+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:37:24.865 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:37:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:37:24.582+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:37:24.866 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:37:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:37:24.582+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:37:24.866 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:37:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:37:24.734+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:37:24.866 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:37:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:37:24.734+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:37:24.866 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:37:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:37:24.734+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:37:24.866 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:37:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:37:24.734+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:37:25.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:37:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:37:24.861+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:37:25.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:37:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:37:24.862+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:37:25.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:37:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:37:24.862+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:37:25.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:37:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:37:25.741+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:37:25.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:37:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:37:25.741+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:37:25.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:37:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:37:25.741+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:37:25.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:37:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:37:25.741+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:37:25.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:37:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:37:25.619+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:37:25.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:37:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:37:25.619+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:37:25.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:37:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:37:25.619+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:37:25.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:37:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:37:25.845+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:37:25.995 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:37:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:37:25.845+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:37:25.995 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:37:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:37:25.845+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:37:26.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:37:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:37:26.824+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:37:26.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:37:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:37:26.824+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:37:26.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:37:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:37:26.824+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:37:26.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:37:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:37:26.642+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:37:26.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:37:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:37:26.642+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:37:26.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:37:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:37:26.642+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:37:26.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:37:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:37:26.785+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:37:26.995 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:37:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:37:26.785+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:37:26.995 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:37:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:37:26.785+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:37:26.995 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:37:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:37:26.785+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:37:27.860 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:37:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:37:27.812+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:37:27.860 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:37:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:37:27.812+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:37:27.860 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:37:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:37:27.812+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:37:27.861 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:37:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:37:27.812+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:37:27.861 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:37:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:37:27.601+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:37:27.861 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:37:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:37:27.601+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:37:27.861 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:37:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:37:27.601+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:37:27.861 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:37:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:37:27.841+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:37:27.862 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:37:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:37:27.841+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:37:27.862 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:37:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:37:27.841+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:37:28.243 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:37:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:37:27.857+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 32455 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:37:28.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:37:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:37:28.614+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:37:28.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:37:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:37:28.614+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:37:28.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:37:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:37:28.614+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:37:28.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:37:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:37:28.826+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:37:28.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:37:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:37:28.826+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:37:28.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:37:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:37:28.826+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:37:28.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:37:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:37:28.820+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:37:28.995 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:37:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:37:28.820+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:37:28.995 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:37:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:37:28.820+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:37:28.995 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:37:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:37:28.820+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:37:29.887 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:37:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:37:29.797+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:37:29.888 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:37:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:37:29.797+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:37:29.888 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:37:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:37:29.797+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:37:29.888 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:37:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:37:29.820+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:37:29.888 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:37:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:37:29.820+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:37:29.888 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:37:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:37:29.820+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:37:29.888 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:37:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:37:29.821+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:37:29.889 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:37:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:37:29.624+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:37:29.889 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:37:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:37:29.624+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:37:29.889 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:37:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:37:29.624+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:37:30.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:37:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:37:30.659+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:37:30.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:37:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:37:30.659+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:37:30.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:37:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:37:30.659+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:37:30.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:37:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:37:30.804+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:37:30.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:37:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:37:30.804+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:37:30.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:37:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:37:30.804+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:37:30.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:37:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:37:30.790+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:37:30.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:37:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:37:30.790+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:37:30.995 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:37:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:37:30.790+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:37:30.995 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:37:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:37:30.790+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:37:31.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:37:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:37:31.610+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:37:31.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:37:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:37:31.610+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:37:31.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:37:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:37:31.610+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:37:32.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:37:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:37:31.832+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:37:32.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:37:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:37:31.832+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:37:32.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:37:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:37:31.832+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:37:32.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:37:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:37:31.804+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:37:32.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:37:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:37:31.804+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:37:32.245 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:37:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:37:31.804+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:37:32.245 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:37:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:37:31.804+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:37:32.865 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:37:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:37:32.590+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:37:32.865 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:37:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:37:32.590+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:37:32.865 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:37:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:37:32.590+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:37:32.866 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:37:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:37:32.807+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:37:32.866 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:37:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:37:32.807+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:37:32.866 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:37:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:37:32.807+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:37:32.866 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:37:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:37:32.807+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:37:32.866 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:37:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:37:32.791+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:37:32.867 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:37:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:37:32.792+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:37:32.867 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:37:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:37:32.792+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:37:33.243 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:37:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:37:32.862+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 32527 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:37:33.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:37:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:37:33.777+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:37:33.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:37:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:37:33.777+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:37:33.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:37:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:37:33.777+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:37:33.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:37:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:37:33.805+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:37:33.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:37:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:37:33.805+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:37:33.995 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:37:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:37:33.806+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:37:33.995 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:37:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:37:33.806+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:37:33.995 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:37:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:37:33.636+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:37:33.995 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:37:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:37:33.636+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:37:33.995 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:37:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:37:33.636+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:37:34.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:37:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:37:34.732+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:37:34.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:37:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:37:34.732+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:37:34.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:37:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:37:34.732+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:37:34.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:37:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:37:34.672+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:37:34.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:37:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:37:34.672+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:37:34.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:37:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:37:34.672+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:37:34.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:37:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:37:34.776+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:37:34.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:37:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:37:34.776+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:37:34.995 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:37:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:37:34.776+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:37:34.995 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:37:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:37:34.776+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:37:35.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:37:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:37:35.734+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:37:35.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:37:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:37:35.734+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:37:35.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:37:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:37:35.734+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:37:35.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:37:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:37:35.636+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:37:35.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:37:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:37:35.636+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:37:35.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:37:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:37:35.636+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:37:35.995 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:37:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:37:35.775+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:37:35.995 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:37:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:37:35.775+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:37:35.995 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:37:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:37:35.775+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:37:35.995 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:37:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:37:35.775+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:37:36.894 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:37:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:37:36.694+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:37:36.895 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:37:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:37:36.694+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:37:36.895 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:37:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:37:36.694+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:37:36.895 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:37:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:37:36.768+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:37:36.895 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:37:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:37:36.768+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:37:36.895 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:37:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:37:36.768+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:37:36.895 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:37:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:37:36.768+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:37:36.896 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:37:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:37:36.627+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:37:36.896 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:37:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:37:36.627+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:37:36.896 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:37:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:37:36.627+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:37:37.993 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:37:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:37:37.866+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 32599 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:37:37.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:37:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:37:37.652+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:37:37.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:37:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:37:37.652+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:37:37.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:37:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:37:37.652+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:37:37.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:37:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:37:37.733+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:37:37.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:37:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:37:37.733+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:37:37.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:37:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:37:37.733+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:37:37.995 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:37:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:37:37.767+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:37:37.995 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:37:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:37:37.767+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:37:37.995 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:37:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:37:37.767+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:37:37.995 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:37:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:37:37.767+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:37:38.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:37:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:37:38.611+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:37:38.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:37:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:37:38.611+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:37:38.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:37:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:37:38.611+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:37:38.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:37:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:37:38.749+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:37:38.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:37:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:37:38.749+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:37:38.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:37:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:37:38.749+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:37:38.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:37:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:37:38.727+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:37:38.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:37:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:37:38.727+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:37:38.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:37:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:37:38.727+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:37:38.995 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:37:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:37:38.728+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:37:39.887 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:37:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:37:39.727+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:37:39.887 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:37:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:37:39.727+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:37:39.887 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:37:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:37:39.727+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:37:39.888 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:37:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:37:39.621+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:37:39.888 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:37:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:37:39.621+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:37:39.888 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:37:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:37:39.621+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:37:39.888 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:37:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:37:39.745+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:37:39.888 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:37:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:37:39.745+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:37:39.888 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:37:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:37:39.745+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:37:39.889 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:37:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:37:39.745+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:37:40.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:37:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:37:40.773+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:37:40.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:37:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:37:40.773+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:37:40.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:37:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:37:40.773+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:37:40.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:37:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:37:40.776+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:37:40.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:37:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:37:40.776+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:37:40.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:37:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:37:40.776+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:37:40.995 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:37:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:37:40.776+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:37:40.995 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:37:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:37:40.657+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:37:40.995 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:37:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:37:40.657+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:37:40.995 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:37:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:37:40.657+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:37:41.983 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:37:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:37:41.819+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:37:41.983 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:37:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:37:41.819+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:37:41.983 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:37:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:37:41.819+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:37:41.984 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:37:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:37:41.707+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:37:41.984 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:37:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:37:41.707+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:37:41.984 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:37:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:37:41.707+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:37:41.984 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:37:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:37:41.756+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:37:41.984 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:37:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:37:41.756+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:37:41.985 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:37:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:37:41.756+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:37:41.985 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:37:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:37:41.756+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:37:42.993 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:37:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:37:42.870+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 32671 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:37:42.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:37:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:37:42.845+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:37:42.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:37:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:37:42.845+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:37:42.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:37:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:37:42.845+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:37:42.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:37:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:37:42.683+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:37:42.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:37:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:37:42.683+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:37:42.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:37:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:37:42.683+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:37:42.995 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:37:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:37:42.799+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:37:42.995 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:37:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:37:42.799+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:37:42.995 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:37:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:37:42.799+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:37:42.995 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:37:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:37:42.799+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:37:43.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:37:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:37:43.801+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:37:43.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:37:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:37:43.801+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:37:43.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:37:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:37:43.801+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:37:43.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:37:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:37:43.731+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:37:43.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:37:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:37:43.731+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:37:43.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:37:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:37:43.731+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:37:43.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:37:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:37:43.841+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:37:43.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:37:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:37:43.841+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:37:43.995 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:37:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:37:43.841+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:37:43.995 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:37:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:37:43.841+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:37:45.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:37:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:37:44.839+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:37:45.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:37:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:37:44.839+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:37:45.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:37:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:37:44.839+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:37:45.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:37:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:37:44.760+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:37:45.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:37:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:37:44.760+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:37:45.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:37:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:37:44.760+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:37:45.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:37:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:37:44.804+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:37:45.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:37:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:37:44.804+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:37:45.245 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:37:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:37:44.804+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:37:45.245 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:37:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:37:44.804+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:37:46.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:37:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:37:45.823+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:37:46.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:37:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:37:45.823+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:37:46.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:37:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:37:45.823+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:37:46.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:37:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:37:45.847+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:37:46.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:37:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:37:45.847+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:37:46.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:37:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:37:45.847+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:37:46.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:37:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:37:45.848+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:37:46.245 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:37:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:37:45.798+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:37:46.245 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:37:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:37:45.798+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:37:46.245 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:37:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:37:45.798+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:37:47.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:37:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:37:46.789+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:37:47.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:37:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:37:46.789+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:37:47.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:37:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:37:46.789+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:37:47.245 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:37:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:37:46.893+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:37:47.245 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:37:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:37:46.893+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:37:47.246 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:37:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:37:46.893+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:37:47.246 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:37:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:37:46.893+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:37:47.247 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:37:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:37:46.848+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:37:47.247 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:37:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:37:46.848+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:37:47.247 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:37:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:37:46.848+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:37:48.243 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:37:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:37:47.874+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 32739 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:37:48.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:37:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:37:47.755+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:37:48.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:37:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:37:47.755+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:37:48.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:37:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:37:47.755+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:37:48.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:37:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:37:47.912+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:37:48.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:37:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:37:47.912+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:37:48.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:37:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:37:47.912+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:37:48.245 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:37:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:37:47.912+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:37:48.245 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:37:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:37:47.820+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:37:48.245 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:37:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:37:47.820+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:37:48.245 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:37:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:37:47.820+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:37:49.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:37:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:37:48.801+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:37:49.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:37:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:37:48.801+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:37:49.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:37:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:37:48.801+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:37:49.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:37:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:37:48.945+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:37:49.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:37:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:37:48.945+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:37:49.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:37:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:37:48.945+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:37:49.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:37:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:37:48.945+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:37:49.245 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:37:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:37:48.831+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:37:49.245 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:37:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:37:48.831+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:37:49.245 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:37:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:37:48.831+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:37:50.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:37:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:37:49.777+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:37:50.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:37:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:37:49.777+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:37:50.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:37:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:37:49.777+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:37:50.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:37:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:37:49.825+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:37:50.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:37:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:37:49.825+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:37:50.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:37:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:37:49.825+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:37:50.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:37:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:37:49.928+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:37:50.245 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:37:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:37:49.928+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:37:50.245 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:37:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:37:49.928+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:37:50.245 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:37:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:37:49.928+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:37:51.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:37:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:37:50.834+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:37:51.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:37:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:37:50.834+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:37:51.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:37:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:37:50.834+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:37:51.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:37:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:37:50.753+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:37:51.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:37:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:37:50.753+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:37:51.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:37:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:37:50.753+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:37:51.245 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:37:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:37:50.907+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:37:51.245 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:37:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:37:50.908+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:37:51.245 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:37:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:37:50.908+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:37:51.245 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:37:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:37:50.908+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:37:52.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:37:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:37:51.941+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:37:52.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:37:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:37:51.941+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:37:52.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:37:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:37:51.941+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:37:52.245 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:37:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:37:51.941+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:37:52.245 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:37:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:37:51.835+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:37:52.245 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:37:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:37:51.835+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:37:52.245 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:37:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:37:51.835+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:37:52.246 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:37:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:37:51.793+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:37:52.246 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:37:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:37:51.793+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:37:52.246 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:37:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:37:51.793+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:37:53.243 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:37:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:37:52.880+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 32811 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:37:53.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:37:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:37:52.789+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:37:53.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:37:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:37:52.789+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:37:53.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:37:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:37:52.789+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:37:53.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:37:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:37:52.877+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:37:53.245 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:37:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:37:52.877+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:37:53.245 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:37:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:37:52.877+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:37:53.245 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:37:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:37:52.973+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:37:53.245 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:37:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:37:52.973+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:37:53.245 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:37:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:37:52.973+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:37:53.245 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:37:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:37:52.973+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:37:54.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:37:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:37:53.878+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:37:54.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:37:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:37:53.878+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:37:54.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:37:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:37:53.878+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:37:54.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:37:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:37:53.760+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:37:54.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:37:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:37:53.760+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:37:54.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:37:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:37:53.760+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:37:54.245 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:37:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:37:53.951+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:37:54.245 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:37:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:37:53.951+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:37:54.245 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:37:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:37:53.951+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:37:54.245 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:37:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:37:53.951+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:37:55.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:37:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:37:54.805+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:37:55.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:37:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:37:54.805+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:37:55.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:37:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:37:54.805+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:37:55.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:37:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:37:54.886+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:37:55.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:37:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:37:54.886+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:37:55.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:37:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:37:54.886+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:37:55.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:37:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:37:54.916+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:37:55.245 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:37:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:37:54.916+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:37:55.245 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:37:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:37:54.916+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:37:55.245 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:37:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:37:54.916+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:37:56.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:37:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:37:55.779+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:37:56.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:37:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:37:55.779+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:37:56.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:37:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:37:55.779+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:37:56.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:37:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:37:55.900+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:37:56.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:37:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:37:55.900+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:37:56.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:37:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:37:55.900+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:37:56.245 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:37:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:37:55.945+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:37:56.245 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:37:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:37:55.945+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:37:56.245 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:37:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:37:55.945+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:37:56.245 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:37:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:37:55.945+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:37:57.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:37:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:37:56.824+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:37:57.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:37:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:37:56.824+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:37:57.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:37:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:37:56.824+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:37:57.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:37:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:37:56.904+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:37:57.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:37:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:37:56.904+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:37:57.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:37:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:37:56.904+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:37:57.245 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:37:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:37:56.956+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:37:57.245 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:37:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:37:56.956+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:37:57.245 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:37:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:37:56.956+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:37:57.245 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:37:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:37:56.956+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:37:58.243 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:37:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:37:57.885+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 32883 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:37:58.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:37:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:37:57.855+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:37:58.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:37:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:37:57.855+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:37:58.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:37:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:37:57.855+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:37:58.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:37:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:37:57.914+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:37:58.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:37:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:37:57.914+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:37:58.245 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:37:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:37:57.914+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:37:58.245 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:37:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:37:57.995+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:37:58.245 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:37:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:37:57.995+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:37:58.245 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:37:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:37:57.995+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:37:58.245 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:37:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:37:57.996+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:37:59.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:37:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:37:58.875+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:37:59.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:37:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:37:58.875+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:37:59.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:37:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:37:58.875+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:37:59.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:37:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:37:58.947+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:37:59.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:37:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:37:58.947+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:37:59.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:37:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:37:58.947+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:37:59.245 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:37:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:37:58.949+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:37:59.245 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:37:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:37:58.949+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:37:59.245 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:37:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:37:58.949+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:37:59.245 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:37:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:37:58.949+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:38:00.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:37:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:37:59.861+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:38:00.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:37:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:37:59.861+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:38:00.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:37:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:37:59.861+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:38:00.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:37:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:37:59.964+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:38:00.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:37:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:37:59.964+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:38:00.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:37:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:37:59.964+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:38:00.245 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:37:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:37:59.941+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:38:00.245 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:37:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:37:59.941+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:38:00.245 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:37:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:37:59.941+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:38:00.245 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:37:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:37:59.941+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:38:00.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:38:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:38:00.942+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:38:00.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:38:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:38:00.942+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:38:00.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:38:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:38:00.942+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:38:00.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:38:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:38:00.826+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:38:00.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:38:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:38:00.826+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:38:00.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:38:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:38:00.826+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:38:00.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:38:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:38:00.898+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:38:00.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:38:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:38:00.898+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:38:00.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:38:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:38:00.898+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:38:00.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:38:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:38:00.898+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:38:02.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:38:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:38:01.933+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:38:02.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:38:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:38:01.933+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:38:02.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:38:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:38:01.933+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:38:02.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:38:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:38:01.782+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:38:02.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:38:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:38:01.782+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:38:02.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:38:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:38:01.782+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:38:02.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:38:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:38:01.896+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:38:02.245 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:38:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:38:01.896+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:38:02.245 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:38:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:38:01.896+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:38:02.245 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:38:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:38:01.896+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:38:03.243 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:38:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:38:02.890+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 32955 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:38:03.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:38:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:38:02.902+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:38:03.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:38:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:38:02.902+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:38:03.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:38:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:38:02.902+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:38:03.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:38:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:38:02.776+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:38:03.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:38:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:38:02.776+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:38:03.245 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:38:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:38:02.776+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:38:03.245 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:38:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:38:02.923+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:38:03.245 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:38:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:38:02.923+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:38:03.245 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:38:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:38:02.923+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:38:03.245 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:38:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:38:02.923+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:38:04.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:38:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:38:03.854+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:38:04.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:38:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:38:03.854+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:38:04.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:38:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:38:03.854+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:38:04.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:38:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:38:03.793+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:38:04.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:38:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:38:03.793+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:38:04.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:38:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:38:03.793+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:38:04.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:38:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:38:03.947+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:38:04.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:38:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:38:03.947+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:38:04.245 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:38:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:38:03.947+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:38:04.245 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:38:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:38:03.947+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:38:05.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:38:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:38:04.865+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:38:05.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:38:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:38:04.865+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:38:05.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:38:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:38:04.865+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:38:05.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:38:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:38:04.801+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:38:05.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:38:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:38:04.801+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:38:05.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:38:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:38:04.801+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:38:05.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:38:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:38:04.967+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:38:05.245 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:38:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:38:04.967+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:38:05.245 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:38:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:38:04.967+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:38:05.245 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:38:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:38:04.967+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:38:06.144 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:38:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:38:05.996+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:38:06.145 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:38:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:38:05.996+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:38:06.145 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:38:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:38:05.996+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:38:06.145 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:38:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:38:05.996+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:38:06.146 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:38:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:38:05.819+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:38:06.146 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:38:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:38:05.819+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:38:06.146 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:38:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:38:05.819+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:38:06.146 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:38:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:38:05.854+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:38:06.146 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:38:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:38:05.854+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:38:06.146 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:38:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:38:05.854+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:38:07.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:38:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:38:06.891+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:38:07.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:38:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:38:06.891+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:38:07.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:38:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:38:06.891+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:38:07.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:38:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:38:07.034+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:38:07.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:38:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:38:07.034+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:38:07.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:38:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:38:07.034+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:38:07.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:38:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:38:07.034+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:38:07.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:38:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:38:06.835+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:38:07.245 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:38:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:38:06.835+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:38:07.245 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:38:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:38:06.835+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:38:08.243 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:38:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:38:07.895+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 33027 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:38:08.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:38:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:38:07.854+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:38:08.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:38:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:38:07.854+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:38:08.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:38:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:38:07.854+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:38:08.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:38:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:38:07.864+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:38:08.245 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:38:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:38:07.864+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:38:08.245 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:38:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:38:07.864+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:38:08.245 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:38:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:38:08.080+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:38:08.245 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:38:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:38:08.080+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:38:08.245 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:38:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:38:08.080+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:38:08.245 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:38:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:38:08.080+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:38:09.102 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:38:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:38:08.818+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:38:09.102 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:38:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:38:08.818+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:38:09.102 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:38:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:38:08.818+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:38:09.103 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:38:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:38:08.858+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:38:09.103 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:38:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:38:08.858+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:38:09.103 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:38:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:38:08.858+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:38:09.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:38:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:38:09.099+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:38:09.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:38:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:38:09.099+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:38:09.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:38:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:38:09.099+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:38:09.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:38:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:38:09.099+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:38:10.086 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:38:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:38:09.788+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:38:10.087 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:38:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:38:09.788+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:38:10.087 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:38:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:38:09.788+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:38:10.087 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:38:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:38:09.814+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:38:10.087 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:38:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:38:09.814+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:38:10.087 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:38:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:38:09.814+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:38:10.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:38:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:38:10.083+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:38:10.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:38:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:38:10.083+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:38:10.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:38:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:38:10.083+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:38:10.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:38:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:38:10.083+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:38:11.107 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:38:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:38:10.794+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:38:11.107 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:38:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:38:10.794+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:38:11.107 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:38:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:38:10.794+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:38:11.107 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:38:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:38:10.831+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:38:11.107 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:38:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:38:10.831+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:38:11.107 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:38:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:38:10.831+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:38:11.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:38:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:38:11.104+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:38:11.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:38:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:38:11.104+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:38:11.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:38:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:38:11.104+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:38:11.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:38:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:38:11.104+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:38:12.142 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:38:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:38:11.832+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:38:12.142 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:38:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:38:11.832+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:38:12.142 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:38:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:38:11.832+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:38:12.142 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:38:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:38:11.784+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:38:12.143 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:38:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:38:11.784+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:38:12.143 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:38:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:38:11.784+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:38:12.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:38:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:38:12.139+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:38:12.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:38:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:38:12.139+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:38:12.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:38:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:38:12.139+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:38:12.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:38:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:38:12.139+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:38:13.093 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:38:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:38:12.900+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 33099 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:38:13.094 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:38:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:38:12.808+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:38:13.094 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:38:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:38:12.808+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:38:13.094 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:38:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:38:12.808+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:38:13.094 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:38:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:38:12.777+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:38:13.094 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:38:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:38:12.777+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:38:13.095 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:38:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:38:12.777+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:38:13.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:38:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:38:13.090+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:38:13.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:38:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:38:13.090+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:38:13.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:38:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:38:13.090+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:38:13.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:38:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:38:13.090+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:38:14.096 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:38:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:38:13.841+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:38:14.096 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:38:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:38:13.841+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:38:14.097 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:38:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:38:13.841+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:38:14.097 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:38:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:38:13.779+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:38:14.097 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:38:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:38:13.779+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:38:14.097 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:38:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:38:13.779+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:38:14.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:38:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:38:14.093+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:38:14.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:38:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:38:14.093+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:38:14.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:38:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:38:14.093+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:38:14.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:38:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:38:14.093+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:38:15.121 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:38:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:38:14.878+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:38:15.122 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:38:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:38:14.878+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:38:15.122 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:38:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:38:14.878+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:38:15.122 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:38:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:38:14.812+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:38:15.122 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:38:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:38:14.812+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:38:15.122 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:38:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:38:14.812+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:38:15.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:38:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:38:15.118+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:38:15.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:38:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:38:15.118+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:38:15.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:38:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:38:15.118+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:38:15.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:38:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:38:15.118+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:38:16.170 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:38:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:38:15.838+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:38:16.170 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:38:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:38:15.838+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:38:16.170 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:38:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:38:15.838+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:38:16.170 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:38:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:38:15.767+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:38:16.171 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:38:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:38:15.767+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:38:16.171 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:38:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:38:15.767+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:38:16.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:38:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:38:16.166+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:38:16.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:38:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:38:16.166+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:38:16.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:38:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:38:16.166+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:38:16.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:38:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:38:16.166+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:38:17.199 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:38:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:38:16.808+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:38:17.199 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:38:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:38:16.808+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:38:17.199 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:38:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:38:16.808+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:38:17.200 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:38:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:38:16.804+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:38:17.200 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:38:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:38:16.804+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:38:17.200 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:38:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:38:16.804+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:38:17.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:38:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:38:17.195+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:38:17.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:38:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:38:17.195+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:38:17.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:38:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:38:17.195+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:38:17.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:38:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:38:17.196+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:38:18.029 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:38:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:38:17.905+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 33171 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:38:18.030 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:38:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:38:17.774+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:38:18.031 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:38:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:38:17.774+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:38:18.031 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:38:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:38:17.774+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:38:18.031 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:38:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:38:17.810+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:38:18.031 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:38:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:38:17.810+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:38:18.031 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:38:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:38:17.810+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:38:18.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:38:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:38:18.167+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:38:18.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:38:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:38:18.167+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:38:18.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:38:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:38:18.167+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:38:18.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:38:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:38:18.167+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:38:19.191 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:38:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:38:18.780+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:38:19.191 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:38:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:38:18.780+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:38:19.191 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:38:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:38:18.780+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:38:19.192 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:38:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:38:18.786+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:38:19.192 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:38:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:38:18.786+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:38:19.192 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:38:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:38:18.786+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:38:19.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:38:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:38:19.188+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:38:19.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:38:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:38:19.188+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:38:19.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:38:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:38:19.188+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:38:19.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:38:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:38:19.188+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:38:20.146 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:38:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:38:19.763+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:38:20.147 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:38:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:38:19.763+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:38:20.147 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:38:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:38:19.763+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:38:20.147 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:38:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:38:19.743+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:38:20.147 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:38:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:38:19.743+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:38:20.147 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:38:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:38:19.743+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:38:20.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:38:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:38:20.144+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:38:20.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:38:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:38:20.144+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:38:20.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:38:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:38:20.144+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:38:20.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:38:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:38:20.144+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:38:21.105 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:38:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:38:20.787+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:38:21.105 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:38:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:38:20.787+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:38:21.105 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:38:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:38:20.787+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:38:21.106 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:38:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:38:20.776+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:38:21.106 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:38:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:38:20.776+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:38:21.106 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:38:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:38:20.776+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:38:21.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:38:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:38:21.102+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:38:21.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:38:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:38:21.102+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:38:21.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:38:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:38:21.102+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:38:21.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:38:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:38:21.103+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:38:21.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:38:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:38:21.731+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:38:21.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:38:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:38:21.731+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:38:21.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:38:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:38:21.731+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:38:21.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:38:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:38:21.825+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:38:21.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:38:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:38:21.825+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:38:21.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:38:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:38:21.825+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:38:22.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:38:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:38:22.142+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:38:22.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:38:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:38:22.143+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:38:22.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:38:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:38:22.143+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:38:22.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:38:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:38:22.143+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:38:22.993 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:38:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:38:22.911+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 33243 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:38:22.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:38:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:38:22.794+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:38:22.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:38:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:38:22.794+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:38:22.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:38:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:38:22.794+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:38:22.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:38:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:38:22.686+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:38:22.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:38:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:38:22.686+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:38:22.995 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:38:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:38:22.686+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:38:23.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:38:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:38:23.105+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:38:23.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:38:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:38:23.105+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:38:23.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:38:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:38:23.105+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:38:23.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:38:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:38:23.105+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:38:23.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:38:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:38:23.784+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:38:23.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:38:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:38:23.784+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:38:23.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:38:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:38:23.784+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:38:23.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:38:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:38:23.705+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:38:23.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:38:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:38:23.705+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:38:23.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:38:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:38:23.705+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:38:24.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:38:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:38:24.093+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:38:24.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:38:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:38:24.093+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:38:24.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:38:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:38:24.093+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:38:24.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:38:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:38:24.093+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:38:24.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:38:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:38:24.740+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:38:24.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:38:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:38:24.740+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:38:24.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:38:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:38:24.740+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:38:24.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:38:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:38:24.755+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:38:24.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:38:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:38:24.755+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:38:24.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:38:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:38:24.755+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:38:25.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:38:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:38:25.120+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:38:25.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:38:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:38:25.120+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:38:25.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:38:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:38:25.120+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:38:25.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:38:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:38:25.120+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:38:25.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:38:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:38:25.692+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:38:25.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:38:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:38:25.692+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:38:25.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:38:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:38:25.692+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:38:25.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:38:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:38:25.724+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:38:25.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:38:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:38:25.724+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:38:25.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:38:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:38:25.724+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:38:26.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:38:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:38:26.124+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:38:26.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:38:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:38:26.124+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:38:26.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:38:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:38:26.124+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:38:26.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:38:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:38:26.124+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:38:26.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:38:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:38:26.742+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:38:26.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:38:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:38:26.742+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:38:26.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:38:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:38:26.742+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:38:26.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:38:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:38:26.735+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:38:26.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:38:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:38:26.735+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:38:26.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:38:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:38:26.735+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:38:27.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:38:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:38:27.107+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:38:27.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:38:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:38:27.107+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:38:27.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:38:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:38:27.107+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:38:27.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:38:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:38:27.107+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:38:27.993 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:38:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:38:27.917+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 33315 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:38:27.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:38:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:38:27.746+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:38:27.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:38:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:38:27.746+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:38:27.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:38:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:38:27.746+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:38:27.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:38:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:38:27.717+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:38:27.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:38:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:38:27.717+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:38:27.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:38:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:38:27.717+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:38:28.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:38:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:38:28.119+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:38:28.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:38:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:38:28.119+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:38:28.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:38:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:38:28.119+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:38:28.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:38:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:38:28.119+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:38:28.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:38:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:38:28.744+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:38:28.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:38:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:38:28.744+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:38:28.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:38:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:38:28.744+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:38:28.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:38:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:38:28.721+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:38:28.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:38:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:38:28.721+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:38:28.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:38:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:38:28.721+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:38:29.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:38:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:38:29.074+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:38:29.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:38:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:38:29.074+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:38:29.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:38:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:38:29.074+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:38:29.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:38:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:38:29.074+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:38:29.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:38:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:38:29.728+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:38:29.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:38:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:38:29.728+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:38:29.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:38:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:38:29.728+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:38:29.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:38:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:38:29.702+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:38:29.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:38:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:38:29.702+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:38:29.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:38:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:38:29.702+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:38:30.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:38:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:38:30.057+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:38:30.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:38:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:38:30.057+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:38:30.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:38:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:38:30.057+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:38:30.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:38:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:38:30.058+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:38:30.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:38:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:38:30.682+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:38:30.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:38:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:38:30.682+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:38:30.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:38:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:38:30.682+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:38:30.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:38:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:38:30.680+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:38:30.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:38:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:38:30.680+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:38:30.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:38:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:38:30.680+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:38:31.418 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:38:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:38:31.070+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:38:31.418 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:38:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:38:31.070+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:38:31.418 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:38:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:38:31.070+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:38:31.418 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:38:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:38:31.070+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:38:31.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:38:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:38:31.667+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:38:31.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:38:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:38:31.667+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:38:31.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:38:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:38:31.667+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:38:31.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:38:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:38:31.698+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:38:31.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:38:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:38:31.698+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:38:31.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:38:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:38:31.698+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:38:32.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:38:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:38:32.070+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:38:32.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:38:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:38:32.070+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:38:32.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:38:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:38:32.070+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:38:32.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:38:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:38:32.070+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:38:32.925 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:38:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:38:32.663+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:38:32.925 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:38:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:38:32.663+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:38:32.925 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:38:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:38:32.663+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:38:32.926 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:38:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:38:32.679+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:38:32.926 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:38:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:38:32.679+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:38:32.926 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:38:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:38:32.679+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:38:33.243 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:38:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:38:32.922+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 33387 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:38:33.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:38:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:38:33.113+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:38:33.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:38:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:38:33.113+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:38:33.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:38:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:38:33.113+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:38:33.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:38:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:38:33.113+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:38:33.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:38:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:38:33.690+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:38:33.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:38:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:38:33.690+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:38:33.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:38:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:38:33.690+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:38:33.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:38:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:38:33.671+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:38:33.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:38:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:38:33.671+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:38:33.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:38:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:38:33.671+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:38:34.490 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:38:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:38:34.063+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:38:34.490 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:38:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:38:34.063+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:38:34.490 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:38:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:38:34.063+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:38:34.490 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:38:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:38:34.064+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:38:34.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:38:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:38:34.685+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:38:34.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:38:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:38:34.685+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:38:34.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:38:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:38:34.685+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:38:34.744 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:38:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:38:34.673+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:38:34.744 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:38:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:38:34.673+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:38:34.744 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:38:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:38:34.673+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:38:35.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:38:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:38:35.037+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:38:35.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:38:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:38:35.037+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:38:35.495 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:38:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:38:35.037+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:38:35.495 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:38:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:38:35.037+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:38:35.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:38:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:38:35.633+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:38:35.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:38:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:38:35.633+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:38:35.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:38:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:38:35.633+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:38:35.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:38:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:38:35.694+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:38:35.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:38:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:38:35.694+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:38:35.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:38:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:38:35.694+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:38:36.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:38:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:38:36.044+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:38:36.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:38:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:38:36.044+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:38:36.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:38:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:38:36.044+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:38:36.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:38:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:38:36.044+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:38:36.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:38:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:38:36.675+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:38:36.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:38:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:38:36.675+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:38:36.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:38:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:38:36.675+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:38:36.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:38:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:38:36.696+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:38:36.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:38:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:38:36.696+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:38:36.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:38:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:38:36.696+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:38:37.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:38:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:38:37.071+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:38:37.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:38:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:38:37.071+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:38:37.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:38:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:38:37.071+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:38:37.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:38:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:38:37.071+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:38:37.930 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:38:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:38:37.666+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:38:37.930 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:38:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:38:37.666+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:38:37.930 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:38:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:38:37.666+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:38:37.931 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:38:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:38:37.670+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:38:37.931 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:38:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:38:37.670+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:38:37.931 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:38:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:38:37.670+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:38:38.243 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:38:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:38:37.927+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 33459 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:38:38.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:38:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:38:38.050+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:38:38.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:38:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:38:38.050+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:38:38.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:38:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:38:38.050+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:38:38.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:38:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:38:38.050+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:38:38.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:38:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:38:38.628+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:38:38.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:38:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:38:38.628+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:38:38.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:38:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:38:38.628+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:38:38.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:38:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:38:38.672+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:38:38.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:38:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:38:38.672+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:38:38.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:38:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:38:38.672+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:38:39.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:38:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:38:39.018+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:38:39.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:38:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:38:39.018+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:38:39.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:38:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:38:39.018+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:38:39.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:38:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:38:39.018+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:38:39.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:38:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:38:39.656+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:38:39.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:38:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:38:39.656+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:38:39.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:38:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:38:39.656+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:38:39.995 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:38:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:38:39.714+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:38:39.995 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:38:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:38:39.714+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:38:39.995 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:38:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:38:39.714+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:38:40.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:38:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:38:40.031+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:38:40.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:38:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:38:40.031+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:38:40.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:38:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:38:40.031+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:38:40.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:38:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:38:40.031+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:38:40.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:38:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:38:40.678+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:38:40.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:38:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:38:40.678+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:38:40.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:38:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:38:40.678+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:38:40.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:38:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:38:40.614+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:38:40.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:38:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:38:40.614+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:38:40.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:38:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:38:40.614+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:38:41.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:38:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:38:41.070+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:38:41.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:38:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:38:41.070+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:38:41.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:38:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:38:41.070+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:38:41.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:38:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:38:41.070+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:38:41.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:38:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:38:41.634+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:38:41.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:38:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:38:41.634+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:38:41.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:38:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:38:41.634+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:38:41.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:38:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:38:41.658+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:38:41.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:38:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:38:41.658+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:38:41.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:38:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:38:41.658+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:38:42.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:38:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:38:42.050+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:38:42.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:38:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:38:42.050+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:38:42.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:38:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:38:42.050+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:38:42.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:38:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:38:42.050+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:38:42.935 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:38:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:38:42.590+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:38:42.935 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:38:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:38:42.590+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:38:42.935 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:38:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:38:42.590+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:38:42.936 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:38:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:38:42.689+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:38:42.936 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:38:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:38:42.689+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:38:42.936 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:38:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:38:42.689+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:38:43.243 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:38:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:38:42.931+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 33531 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:38:43.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:38:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:38:43.067+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:38:43.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:38:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:38:43.067+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:38:43.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:38:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:38:43.067+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:38:43.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:38:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:38:43.067+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:38:43.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:38:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:38:43.641+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:38:43.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:38:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:38:43.641+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:38:43.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:38:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:38:43.641+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:38:43.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:38:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:38:43.632+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:38:43.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:38:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:38:43.632+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:38:43.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:38:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:38:43.632+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:38:44.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:38:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:38:44.090+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:38:44.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:38:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:38:44.090+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:38:44.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:38:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:38:44.090+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:38:44.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:38:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:38:44.090+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:38:44.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:38:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:38:44.682+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:38:44.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:38:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:38:44.682+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:38:44.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:38:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:38:44.682+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:38:44.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:38:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:38:44.615+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:38:44.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:38:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:38:44.615+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:38:44.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:38:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:38:44.615+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:38:45.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:38:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:38:45.119+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:38:45.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:38:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:38:45.119+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:38:45.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:38:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:38:45.119+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:38:45.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:38:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:38:45.119+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:38:45.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:38:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:38:45.692+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:38:45.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:38:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:38:45.692+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:38:45.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:38:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:38:45.692+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:38:45.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:38:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:38:45.620+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:38:45.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:38:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:38:45.620+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:38:45.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:38:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:38:45.620+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:38:46.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:38:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:38:46.128+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:38:46.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:38:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:38:46.128+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:38:46.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:38:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:38:46.128+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:38:46.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:38:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:38:46.128+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:38:46.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:38:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:38:46.732+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:38:46.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:38:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:38:46.732+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:38:46.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:38:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:38:46.732+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:38:46.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:38:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:38:46.626+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:38:46.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:38:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:38:46.626+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:38:46.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:38:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:38:46.626+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:38:47.495 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:38:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:38:47.112+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:38:47.495 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:38:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:38:47.112+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:38:47.495 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:38:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:38:47.112+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:38:47.495 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:38:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:38:47.112+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:38:47.940 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:38:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:38:47.719+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:38:47.941 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:38:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:38:47.719+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:38:47.941 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:38:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:38:47.719+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:38:47.941 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:38:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:38:47.630+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:38:47.941 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:38:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:38:47.630+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:38:47.941 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:38:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:38:47.630+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:38:48.243 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:38:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:38:47.937+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 33603 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:38:48.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:38:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:38:48.092+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:38:48.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:38:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:38:48.092+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:38:48.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:38:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:38:48.092+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:38:48.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:38:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:38:48.092+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:38:48.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:38:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:38:48.726+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:38:48.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:38:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:38:48.726+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:38:48.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:38:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:38:48.726+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:38:48.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:38:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:38:48.599+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:38:48.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:38:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:38:48.599+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:38:48.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:38:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:38:48.599+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:38:49.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:38:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:38:49.100+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:38:49.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:38:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:38:49.100+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:38:49.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:38:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:38:49.100+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:38:49.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:38:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:38:49.100+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:38:49.886 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:38:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:38:49.691+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:38:49.887 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:38:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:38:49.691+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:38:49.887 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:38:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:38:49.691+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:38:49.887 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:38:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:38:49.562+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:38:49.887 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:38:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:38:49.562+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:38:49.887 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:38:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:38:49.562+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:38:50.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:38:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:38:50.080+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:38:50.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:38:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:38:50.080+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:38:50.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:38:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:38:50.080+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:38:50.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:38:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:38:50.080+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:38:50.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:38:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:38:50.738+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:38:50.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:38:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:38:50.738+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:38:50.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:38:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:38:50.738+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:19.431352+0000 (oldest deadline 2024-02-20T08:28:39.431352+0000) 2024-02-20T08:38:50.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:38:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:38:50.587+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:38:50.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:38:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:38:50.587+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:38:50.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:38:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:38:50.587+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:38:51.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:38:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:38:51.046+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:38:51.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:38:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:38:51.046+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:38:51.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:38:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:38:51.046+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:38:51.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:38:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:38:51.046+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:38:51.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:38:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:38:51.549+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:38:51.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:38:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:38:51.549+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:38:51.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:38:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:38:51.549+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:38:52.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:38:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:38:52.043+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:38:52.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:38:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:38:52.043+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:38:52.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:38:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:38:52.043+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:38:52.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:38:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:38:52.043+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:38:52.921 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:38:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:38:52.544+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:38:52.922 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:38:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:38:52.544+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:38:52.922 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:38:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:38:52.544+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:38:53.243 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:38:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:38:52.943+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 33675 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:38:53.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:38:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:38:53.067+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:38:53.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:38:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:38:53.067+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:38:53.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:38:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:38:53.067+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:38:53.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:38:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:38:53.067+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:38:53.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:38:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:38:53.588+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:38:53.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:38:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:38:53.588+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:38:53.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:38:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:38:53.588+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:38:54.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:38:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:38:54.080+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:38:54.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:38:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:38:54.080+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:38:54.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:38:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:38:54.080+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:38:54.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:38:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:38:54.080+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:38:54.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:38:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:38:54.582+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:38:54.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:38:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:38:54.582+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:38:54.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:38:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:38:54.582+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:38:55.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:38:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:38:55.058+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:38:55.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:38:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:38:55.058+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:38:55.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:38:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:38:55.058+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:38:55.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:38:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:38:55.058+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:38:55.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:38:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:38:55.580+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:38:55.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:38:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:38:55.580+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:38:55.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:38:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:38:55.580+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:38:56.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:38:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:38:56.020+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:38:56.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:38:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:38:56.020+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:38:56.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:38:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:38:56.020+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:38:56.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:38:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:38:56.020+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:38:56.978 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:38:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:38:56.579+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:38:56.978 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:38:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:38:56.579+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:38:56.978 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:38:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:38:56.579+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:38:57.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:38:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:38:56.975+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:38:57.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:38:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:38:56.975+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:38:57.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:38:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:38:56.975+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:38:57.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:38:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:38:56.975+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:38:57.952 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:38:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:38:57.566+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:38:57.952 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:38:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:38:57.566+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:38:57.952 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:38:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:38:57.566+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:38:58.243 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:38:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:38:57.948+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 33747 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:38:58.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:38:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:38:57.989+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:38:58.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:38:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:38:57.989+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:38:58.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:38:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:38:57.989+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:38:58.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:38:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:38:57.989+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:38:58.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:38:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:38:58.552+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:38:58.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:38:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:38:58.552+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:38:58.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:38:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:38:58.552+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:38:59.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:38:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:38:59.026+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:38:59.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:38:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:38:59.026+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:38:59.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:38:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:38:59.026+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:38:59.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:38:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:38:59.026+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:38:59.887 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:38:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:38:59.573+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:38:59.887 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:38:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:38:59.573+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:38:59.888 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:38:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:38:59.573+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:39:00.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:39:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:39:00.061+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:39:00.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:39:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:39:00.061+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:39:00.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:39:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:39:00.061+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:39:00.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:39:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:39:00.061+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:39:00.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:39:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:39:00.528+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:39:00.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:39:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:39:00.528+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:39:00.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:39:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:39:00.528+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:39:01.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:39:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:39:01.052+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:39:01.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:39:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:39:01.052+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:39:01.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:39:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:39:01.052+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:39:01.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:39:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:39:01.052+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:39:01.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:39:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:39:01.574+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:39:01.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:39:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:39:01.574+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:39:01.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:39:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:39:01.574+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:39:02.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:39:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:39:02.071+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:39:02.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:39:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:39:02.071+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:39:02.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:39:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:39:02.071+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:39:02.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:39:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:39:02.071+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:39:02.957 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:39:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:39:02.607+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:39:02.957 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:39:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:39:02.607+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:39:02.957 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:39:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:39:02.607+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:39:03.243 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:39:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:39:02.953+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 33819 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:39:03.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:39:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:39:03.088+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:39:03.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:39:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:39:03.088+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:39:03.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:39:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:39:03.088+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:39:03.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:39:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:39:03.088+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:39:03.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:39:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:39:03.614+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:39:03.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:39:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:39:03.614+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:39:03.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:39:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:39:03.614+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:39:04.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:39:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:39:04.053+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:39:04.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:39:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:39:04.053+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:39:04.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:39:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:39:04.053+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:39:04.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:39:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:39:04.053+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:39:04.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:39:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:39:04.649+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:39:04.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:39:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:39:04.649+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:39:04.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:39:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:39:04.649+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:27.452536+0000 (oldest deadline 2024-02-20T08:28:47.452536+0000) 2024-02-20T08:39:05.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:39:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:39:05.017+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:39:05.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:39:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:39:05.017+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:39:05.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:39:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:39:05.017+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:39:05.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:39:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:39:05.017+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:39:06.396 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:39:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:39:06.003+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:39:06.396 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:39:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:39:06.003+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:39:06.396 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:39:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:39:06.003+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:39:06.396 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:39:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:39:06.003+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:39:07.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:39:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:39:07.029+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:39:07.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:39:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:39:07.029+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:39:07.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:39:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:39:07.029+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:39:07.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:39:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:39:07.029+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:39:08.243 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:39:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:39:07.959+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 33890 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:39:08.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:39:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:39:08.019+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:39:08.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:39:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:39:08.019+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:39:08.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:39:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:39:08.019+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:39:08.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:39:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:39:08.019+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:39:09.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:39:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:39:08.997+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:39:09.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:39:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:39:08.997+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:39:09.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:39:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:39:08.997+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:39:09.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:39:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:39:08.997+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:39:10.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:39:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:39:10.021+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:39:10.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:39:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:39:10.021+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:39:10.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:39:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:39:10.021+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:39:10.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:39:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:39:10.021+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:39:11.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:39:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:39:10.974+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:39:11.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:39:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:39:10.974+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:39:11.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:39:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:39:10.974+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:39:11.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:39:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:39:10.974+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:39:12.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:39:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:39:11.995+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:39:12.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:39:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:39:11.995+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:39:12.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:39:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:39:11.995+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:39:12.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:39:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:39:11.995+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:39:13.243 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:39:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:39:12.964+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 33963 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:39:13.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:39:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:39:13.020+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:39:13.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:39:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:39:13.020+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:39:13.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:39:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:39:13.020+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:28:44.538659+0000 (oldest deadline 2024-02-20T08:29:04.538659+0000) 2024-02-20T08:39:13.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:39:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:39:13.020+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:39:13.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:39:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:39:13.698+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:39:13.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:39:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:39:13.699+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:39:13.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:39:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:39:13.699+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:39:14.426 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:39:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:39:14.014+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:39:14.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:39:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:39:14.672+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:39:14.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:39:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:39:14.672+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:39:14.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:39:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:39:14.672+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:39:15.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:39:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:39:15.014+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:39:15.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:39:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:39:15.648+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:39:15.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:39:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:39:15.648+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:39:15.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:39:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:39:15.648+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:39:16.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:39:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:39:16.010+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:39:16.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:39:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:39:16.644+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:39:16.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:39:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:39:16.644+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:39:16.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:39:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:39:16.644+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:39:17.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:39:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:39:17.058+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:39:17.974 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:39:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:39:17.620+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:39:17.974 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:39:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:39:17.620+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:39:17.974 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:39:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:39:17.620+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:39:18.243 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:39:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:39:17.971+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 34034 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:39:18.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:39:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:39:18.054+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:39:18.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:39:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:39:18.571+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:39:18.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:39:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:39:18.571+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:39:18.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:39:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:39:18.571+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:39:19.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:39:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:39:19.086+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:39:19.887 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:39:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:39:19.571+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:39:19.887 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:39:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:39:19.571+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:39:19.887 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:39:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:39:19.571+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:39:20.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:39:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:39:20.081+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:39:20.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:39:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:39:20.554+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:39:20.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:39:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:39:20.554+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:39:20.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:39:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:39:20.554+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:39:21.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:39:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:39:21.115+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:39:21.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:39:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:39:21.573+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:39:21.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:39:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:39:21.573+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:39:21.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:39:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:39:21.573+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:39:22.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:39:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:39:22.116+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:39:22.979 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:39:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:39:22.580+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:39:22.979 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:39:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:39:22.580+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:39:22.979 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:39:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:39:22.580+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:39:23.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:39:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:39:23.080+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:39:23.243 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:39:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:39:22.976+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 34107 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:39:23.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:39:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:39:23.581+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:39:23.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:39:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:39:23.581+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:39:23.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:39:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:39:23.581+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:39:24.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:39:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:39:24.039+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:39:24.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:39:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:39:24.618+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:39:24.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:39:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:39:24.618+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:39:24.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:39:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:39:24.618+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:39:25.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:39:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:39:25.084+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:39:25.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:39:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:39:25.585+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:39:25.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:39:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:39:25.585+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:39:25.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:39:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:39:25.585+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:39:26.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:39:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:39:26.094+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:39:26.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:39:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:39:26.611+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:39:26.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:39:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:39:26.611+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:39:26.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:39:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:39:26.611+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:39:27.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:39:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:39:27.101+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:39:27.984 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:39:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:39:27.602+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:39:27.984 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:39:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:39:27.602+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:39:27.985 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:39:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:39:27.602+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:39:28.243 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:39:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:39:27.981+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 34178 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:39:28.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:39:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:39:28.118+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:39:28.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:39:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:39:28.678+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:39:28.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:39:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:39:28.678+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:39:28.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:39:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:39:28.678+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:39:28.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:39:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:39:28.629+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:39:28.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:39:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:39:28.629+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:39:28.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:39:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:39:28.629+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:39:29.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:39:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:39:29.149+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:39:29.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:39:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:39:29.645+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:39:29.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:39:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:39:29.645+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:39:29.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:39:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:39:29.645+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:39:29.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:39:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:39:29.676+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:39:29.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:39:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:39:29.676+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:39:29.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:39:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:39:29.676+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:39:30.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:39:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:39:30.193+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:39:30.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:39:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:39:30.694+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:39:30.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:39:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:39:30.694+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:39:30.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:39:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:39:30.694+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:39:30.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:39:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:39:30.648+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:39:30.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:39:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:39:30.648+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:39:30.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:39:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:39:30.648+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:39:31.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:39:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:39:31.231+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:39:31.995 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:39:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:39:31.691+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:39:31.995 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:39:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:39:31.691+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:39:31.995 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:39:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:39:31.691+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:39:31.995 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:39:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:39:31.713+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:39:31.996 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:39:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:39:31.713+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:39:31.996 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:39:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:39:31.713+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:39:32.652 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:39:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:39:32.255+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:39:32.989 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:39:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:39:32.650+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:39:32.989 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:39:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:39:32.650+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:39:32.989 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:39:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:39:32.650+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:39:32.990 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:39:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:39:32.695+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:39:32.990 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:39:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:39:32.695+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:39:32.990 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:39:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:39:32.695+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:39:33.243 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:39:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:39:32.986+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 34251 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:39:33.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:39:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:39:33.207+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:39:33.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:39:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:39:33.664+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:39:33.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:39:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:39:33.664+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:39:33.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:39:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:39:33.664+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:39:33.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:39:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:39:33.666+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:39:33.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:39:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:39:33.666+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:39:33.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:39:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:39:33.666+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:39:34.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:39:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:39:34.170+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:39:34.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:39:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:39:34.644+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:39:34.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:39:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:39:34.644+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:39:34.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:39:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:39:34.644+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:39:34.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:39:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:39:34.637+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:39:34.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:39:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:39:34.637+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:39:34.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:39:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:39:34.637+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:39:35.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:39:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:39:35.194+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:39:35.995 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:39:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:39:35.647+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:39:35.995 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:39:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:39:35.647+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:39:35.995 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:39:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:39:35.647+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:39:35.996 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:39:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:39:35.666+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:39:35.996 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:39:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:39:35.666+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:39:35.996 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:39:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:39:35.666+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:39:36.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:39:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:39:36.159+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:39:36.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:39:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:39:36.695+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:39:36.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:39:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:39:36.695+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:39:36.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:39:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:39:36.695+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:39:36.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:39:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:39:36.629+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:39:36.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:39:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:39:36.629+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:39:36.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:39:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:39:36.629+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:39:37.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:39:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:39:37.185+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:39:37.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:39:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:39:37.668+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:39:37.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:39:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:39:37.668+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:39:37.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:39:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:39:37.668+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:39:37.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:39:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:39:37.702+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:39:37.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:39:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:39:37.702+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:39:37.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:39:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:39:37.702+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:39:38.495 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:39:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:39:37.990+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 34322 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:39:38.495 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:39:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:39:38.234+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:39:38.495 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:39:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:39:38.234+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:39:38.496 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:39:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:39:38.234+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:39:38.496 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:39:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:39:38.234+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:39:38.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:39:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:39:38.671+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:39:38.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:39:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:39:38.671+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:39:38.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:39:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:39:38.671+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:39:38.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:39:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:39:38.656+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:39:38.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:39:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:39:38.656+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:39:38.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:39:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:39:38.656+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:39:39.635 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:39:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:39:39.275+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:39:39.636 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:39:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:39:39.275+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:39:39.636 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:39:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:39:39.275+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:39:39.636 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:39:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:39:39.275+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:39:39.888 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:39:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:39:39.717+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:39:39.888 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:39:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:39:39.717+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:39:39.888 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:39:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:39:39.717+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:39:39.888 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:39:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:39:39.632+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:39:39.889 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:39:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:39:39.632+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:39:39.889 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:39:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:39:39.632+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:39:40.650 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:39:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:39:40.283+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:39:40.651 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:39:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:39:40.283+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:39:40.651 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:39:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:39:40.283+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:39:40.651 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:39:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:39:40.283+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:39:40.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:39:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:39:40.647+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:39:40.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:39:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:39:40.647+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:39:40.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:39:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:39:40.647+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:39:40.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:39:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:39:40.716+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:39:40.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:39:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:39:40.716+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:39:40.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:39:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:39:40.716+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:39:41.607 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:39:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:39:41.325+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:39:41.607 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:39:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:39:41.325+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:39:41.607 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:39:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:39:41.325+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:39:41.607 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:39:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:39:41.325+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:39:41.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:39:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:39:41.701+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:39:41.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:39:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:39:41.701+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:39:41.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:39:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:39:41.701+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:39:41.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:39:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:39:41.604+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:39:41.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:39:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:39:41.604+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:39:41.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:39:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:39:41.604+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:39:42.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:39:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:39:42.318+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:39:42.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:39:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:39:42.318+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:39:42.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:39:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:39:42.318+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:39:42.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:39:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:39:42.318+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:39:42.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:39:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:39:42.689+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:39:42.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:39:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:39:42.689+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:39:42.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:39:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:39:42.689+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:39:42.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:39:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:39:42.595+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:39:42.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:39:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:39:42.595+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:39:42.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:39:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:39:42.595+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:39:43.316 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:39:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:39:42.995+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 34395 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:39:43.643 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:39:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:39:43.313+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:39:43.643 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:39:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:39:43.313+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:39:43.644 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:39:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:39:43.313+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:39:43.644 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:39:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:39:43.313+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:39:43.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:39:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:39:43.640+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:39:43.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:39:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:39:43.640+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:39:43.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:39:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:39:43.640+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:39:43.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:39:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:39:43.700+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:39:43.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:39:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:39:43.700+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:39:43.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:39:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:39:43.700+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:39:44.676 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:39:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:39:44.307+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:39:44.677 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:39:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:39:44.307+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:39:44.677 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:39:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:39:44.307+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:39:44.677 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:39:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:39:44.307+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:39:44.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:39:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:39:44.695+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:39:44.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:39:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:39:44.695+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:39:44.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:39:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:39:44.695+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:39:44.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:39:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:39:44.673+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:39:44.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:39:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:39:44.673+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:39:44.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:39:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:39:44.673+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:39:45.653 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:39:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:39:45.340+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:39:45.653 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:39:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:39:45.340+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:39:45.653 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:39:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:39:45.340+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:39:45.653 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:39:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:39:45.340+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:39:45.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:39:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:39:45.650+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:39:45.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:39:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:39:45.650+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:39:45.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:39:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:39:45.650+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:39:45.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:39:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:39:45.667+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:39:45.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:39:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:39:45.667+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:39:45.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:39:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:39:45.667+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:39:46.651 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:39:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:39:46.318+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:39:46.651 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:39:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:39:46.318+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:39:46.652 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:39:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:39:46.318+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:39:46.652 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:39:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:39:46.318+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:39:46.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:39:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:39:46.679+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:39:46.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:39:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:39:46.679+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:39:46.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:39:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:39:46.679+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:39:46.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:39:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:39:46.648+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:39:46.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:39:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:39:46.648+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:39:46.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:39:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:39:46.648+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:39:47.619 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:39:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:39:47.283+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:39:47.619 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:39:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:39:47.283+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:39:47.619 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:39:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:39:47.283+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:39:47.619 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:39:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:39:47.283+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:39:47.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:39:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:39:47.685+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:39:47.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:39:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:39:47.685+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:39:47.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:39:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:39:47.685+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:39:47.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:39:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:39:47.616+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:39:47.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:39:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:39:47.616+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:39:47.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:39:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:39:47.616+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:39:48.311 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:39:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:39:48.001+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 34466 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:39:48.654 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:39:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:39:48.307+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:39:48.655 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:39:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:39:48.307+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:39:48.655 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:39:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:39:48.307+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:39:48.655 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:39:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:39:48.307+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:39:48.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:39:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:39:48.690+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:39:48.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:39:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:39:48.690+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:39:48.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:39:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:39:48.690+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:39:48.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:39:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:39:48.651+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:39:48.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:39:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:39:48.651+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:39:48.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:39:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:39:48.651+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:39:49.673 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:39:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:39:49.274+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:39:49.673 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:39:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:39:49.274+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:39:49.673 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:39:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:39:49.274+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:39:49.674 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:39:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:39:49.274+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:39:49.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:39:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:39:49.669+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:39:49.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:39:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:39:49.669+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:39:49.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:39:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:39:49.669+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:39:49.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:39:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:39:49.697+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:39:49.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:39:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:39:49.697+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:39:49.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:39:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:39:49.697+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:39:50.696 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:39:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:39:50.292+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:39:50.696 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:39:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:39:50.292+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:39:50.696 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:39:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:39:50.292+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:39:50.696 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:39:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:39:50.292+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:39:50.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:39:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:39:50.696+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:39:50.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:39:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:39:50.696+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:39:50.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:39:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:39:50.696+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:39:50.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:39:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:39:50.692+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:39:50.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:39:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:39:50.692+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:39:50.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:39:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:39:50.692+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:39:51.691 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:39:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:39:51.332+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:39:51.691 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:39:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:39:51.332+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:39:51.691 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:39:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:39:51.332+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:39:51.691 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:39:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:39:51.332+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:39:51.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:39:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:39:51.688+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:39:51.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:39:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:39:51.688+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:39:51.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:39:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:39:51.688+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:39:51.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:39:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:39:51.731+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:39:51.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:39:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:39:51.731+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:39:51.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:39:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:39:51.731+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:39:52.689 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:39:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:39:52.328+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:39:52.689 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:39:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:39:52.328+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:39:52.689 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:39:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:39:52.328+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:39:52.690 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:39:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:39:52.328+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:39:52.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:39:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:39:52.685+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:39:52.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:39:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:39:52.685+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:39:52.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:39:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:39:52.685+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:39:52.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:39:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:39:52.773+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:39:52.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:39:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:39:52.773+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:39:52.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:39:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:39:52.773+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:39:53.350 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:39:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:39:53.005+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 34538 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:39:53.658 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:39:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:39:53.347+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:39:53.658 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:39:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:39:53.347+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:39:53.658 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:39:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:39:53.347+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:39:53.658 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:39:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:39:53.347+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:39:53.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:39:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:39:53.655+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:39:53.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:39:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:39:53.655+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:39:53.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:39:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:39:53.655+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:39:53.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:39:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:39:53.765+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:39:53.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:39:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:39:53.765+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:39:53.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:39:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:39:53.765+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:39:54.706 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:39:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:39:54.329+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:39:54.706 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:39:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:39:54.329+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:39:54.706 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:39:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:39:54.329+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:39:54.706 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:39:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:39:54.329+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:39:54.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:39:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:39:54.703+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:39:54.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:39:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:39:54.703+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:39:54.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:39:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:39:54.703+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:39:54.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:39:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:39:54.770+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:39:54.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:39:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:39:54.770+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:39:54.995 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:39:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:39:54.770+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:39:55.735 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:39:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:39:55.354+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:39:55.736 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:39:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:39:55.354+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:39:55.736 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:39:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:39:55.354+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:39:55.736 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:39:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:39:55.354+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:39:55.995 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:39:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:39:55.732+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:39:55.996 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:39:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:39:55.732+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:39:55.996 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:39:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:39:55.732+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:39:55.997 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:39:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:39:55.780+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:39:55.997 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:39:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:39:55.780+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:39:55.997 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:39:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:39:55.780+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:39:56.712 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:39:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:39:56.336+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:39:56.712 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:39:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:39:56.336+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:39:56.712 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:39:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:39:56.336+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:39:56.712 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:39:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:39:56.336+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:39:56.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:39:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:39:56.709+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:39:56.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:39:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:39:56.709+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:39:56.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:39:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:39:56.709+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:39:56.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:39:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:39:56.755+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:39:56.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:39:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:39:56.755+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:39:56.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:39:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:39:56.755+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:39:57.715 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:39:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:39:57.382+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:39:57.716 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:39:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:39:57.382+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:39:57.716 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:39:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:39:57.382+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:39:57.716 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:39:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:39:57.382+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:39:57.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:39:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:39:57.713+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:39:57.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:39:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:39:57.713+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:39:57.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:39:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:39:57.713+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:39:57.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:39:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:39:57.713+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:39:57.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:39:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:39:57.713+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:39:57.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:39:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:39:57.713+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:39:58.367 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:39:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:39:58.010+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 34608 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:39:58.719 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:39:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:39:58.364+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:39:58.719 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:39:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:39:58.364+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:39:58.720 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:39:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:39:58.364+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:39:58.720 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:39:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:39:58.364+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:39:58.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:39:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:39:58.728+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:39:58.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:39:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:39:58.728+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:39:58.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:39:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:39:58.728+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:39:58.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:39:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:39:58.715+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:39:58.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:39:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:39:58.715+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:39:58.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:39:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:39:58.715+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:39:59.688 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:39:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:39:59.400+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:39:59.689 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:39:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:39:59.400+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:39:59.689 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:39:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:39:59.400+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:39:59.689 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:39:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:39:59.400+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:39:59.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:39:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:39:59.773+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:39:59.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:39:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:39:59.773+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:39:59.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:39:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:39:59.773+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:39:59.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:39:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:39:59.685+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:39:59.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:39:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:39:59.685+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:39:59.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:39:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:39:59.685+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:40:00.723 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:40:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:40:00.374+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:40:00.723 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:40:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:40:00.374+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:40:00.723 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:40:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:40:00.374+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:40:00.724 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:40:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:40:00.374+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:40:00.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:40:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:40:00.744+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:40:00.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:40:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:40:00.744+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:40:00.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:40:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:40:00.744+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:40:00.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:40:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:40:00.720+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:40:00.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:40:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:40:00.720+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:40:00.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:40:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:40:00.720+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:40:01.689 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:40:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:40:01.346+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:40:01.689 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:40:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:40:01.346+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:40:01.689 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:40:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:40:01.346+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:40:01.690 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:40:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:40:01.346+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:40:01.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:40:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:40:01.773+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:40:01.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:40:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:40:01.773+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:40:01.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:40:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:40:01.773+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:40:01.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:40:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:40:01.686+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:40:01.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:40:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:40:01.686+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:40:01.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:40:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:40:01.686+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:40:02.654 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:40:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:40:02.298+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:40:02.654 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:40:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:40:02.298+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:40:02.654 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:40:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:40:02.298+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:40:02.654 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:40:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:40:02.298+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:40:02.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:40:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:40:02.650+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:40:02.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:40:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:40:02.650+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:40:02.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:40:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:40:02.650+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:40:02.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:40:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:40:02.820+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:40:02.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:40:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:40:02.820+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:40:02.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:40:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:40:02.820+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:40:03.493 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:40:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:40:03.015+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 34681 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:40:03.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:40:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:40:03.255+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:40:03.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:40:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:40:03.255+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:40:03.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:40:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:40:03.255+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:40:03.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:40:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:40:03.256+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:40:03.840 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:40:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:40:03.642+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:40:03.840 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:40:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:40:03.642+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:40:03.840 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:40:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:40:03.642+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:40:04.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:40:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:40:03.836+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:40:04.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:40:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:40:03.836+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:40:04.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:40:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:40:03.836+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:40:04.677 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:40:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:40:04.261+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:40:04.677 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:40:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:40:04.261+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:40:04.677 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:40:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:40:04.261+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:40:04.677 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:40:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:40:04.261+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:40:04.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:40:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:40:04.674+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:40:04.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:40:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:40:04.674+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:40:04.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:40:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:40:04.674+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:40:04.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:40:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:40:04.881+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:40:04.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:40:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:40:04.881+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:40:04.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:40:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:40:04.881+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:40:05.642 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:40:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:40:05.295+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:40:05.642 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:40:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:40:05.295+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:40:05.642 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:40:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:40:05.295+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:40:05.643 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:40:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:40:05.295+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:40:05.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:40:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:40:05.638+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:40:05.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:40:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:40:05.638+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:40:05.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:40:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:40:05.638+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:40:05.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:40:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:40:05.845+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:40:05.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:40:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:40:05.845+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:40:05.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:40:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:40:05.845+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:40:06.688 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:40:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:40:06.253+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:40:06.688 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:40:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:40:06.253+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:40:06.689 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:40:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:40:06.253+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:40:06.689 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:40:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:40:06.253+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:40:06.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:40:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:40:06.685+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:40:06.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:40:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:40:06.685+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:40:06.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:40:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:40:06.685+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:40:06.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:40:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:40:06.820+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:40:06.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:40:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:40:06.820+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:40:06.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:40:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:40:06.820+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:40:07.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:40:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:40:07.219+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:40:07.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:40:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:40:07.219+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:40:07.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:40:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:40:07.219+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:40:07.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:40:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:40:07.220+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:40:07.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:40:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:40:07.696+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:40:07.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:40:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:40:07.696+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:40:07.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:40:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:40:07.696+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:40:07.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:40:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:40:07.817+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:40:07.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:40:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:40:07.817+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:40:07.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:40:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:40:07.817+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:40:08.493 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:40:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:40:08.021+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 34752 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:40:08.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:40:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:40:08.212+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:40:08.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:40:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:40:08.212+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:40:08.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:40:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:40:08.212+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:40:08.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:40:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:40:08.212+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:40:08.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:40:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:40:08.814+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:40:08.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:40:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:40:08.814+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:40:08.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:40:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:40:08.814+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:40:08.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:40:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:40:08.650+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:40:08.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:40:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:40:08.650+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:40:08.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:40:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:40:08.650+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:40:09.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:40:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:40:09.168+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:40:09.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:40:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:40:09.168+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:40:09.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:40:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:40:09.168+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:40:09.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:40:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:40:09.168+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:40:09.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:40:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:40:09.825+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:40:09.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:40:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:40:09.825+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:40:09.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:40:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:40:09.825+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:40:09.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:40:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:40:09.680+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:40:09.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:40:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:40:09.680+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:40:09.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:40:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:40:09.680+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:40:10.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:40:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:40:10.175+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:40:10.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:40:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:40:10.175+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:40:10.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:40:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:40:10.175+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:40:10.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:40:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:40:10.175+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:40:10.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:40:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:40:10.833+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:40:10.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:40:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:40:10.833+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:40:10.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:40:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:40:10.833+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:40:10.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:40:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:40:10.643+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:40:10.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:40:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:40:10.643+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:40:10.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:40:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:40:10.643+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:40:11.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:40:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:40:11.168+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:40:11.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:40:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:40:11.168+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:40:11.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:40:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:40:11.168+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:40:11.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:40:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:40:11.168+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:40:11.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:40:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:40:11.798+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:40:11.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:40:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:40:11.798+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:40:11.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:40:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:40:11.798+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:40:11.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:40:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:40:11.686+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:40:11.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:40:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:40:11.686+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:40:11.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:40:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:40:11.686+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:40:12.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:40:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:40:12.202+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:40:12.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:40:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:40:12.202+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:40:12.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:40:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:40:12.202+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:40:12.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:40:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:40:12.202+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:40:12.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:40:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:40:12.755+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:40:12.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:40:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:40:12.755+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:40:12.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:40:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:40:12.755+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:40:12.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:40:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:40:12.732+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:40:12.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:40:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:40:12.733+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:40:12.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:40:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:40:12.733+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:40:13.493 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:40:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:40:13.028+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 34825 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:40:13.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:40:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:40:13.210+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:40:13.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:40:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:40:13.210+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:40:13.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:40:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:40:13.210+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:40:13.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:40:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:40:13.210+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:40:13.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:40:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:40:13.794+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:40:13.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:40:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:40:13.794+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:40:13.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:40:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:40:13.794+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:40:13.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:40:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:40:13.718+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:40:13.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:40:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:40:13.718+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:40:13.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:40:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:40:13.718+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:40:14.733 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:40:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:40:14.244+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:40:14.733 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:40:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:40:14.244+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:40:14.733 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:40:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:40:14.244+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:40:14.733 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:40:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:40:14.244+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:40:14.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:40:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:40:14.784+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:40:14.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:40:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:40:14.784+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:40:14.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:40:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:40:14.784+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:40:14.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:40:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:40:14.730+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:40:14.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:40:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:40:14.730+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:40:14.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:40:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:40:14.730+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:40:15.737 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:40:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:40:15.282+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:40:15.737 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:40:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:40:15.282+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:40:15.737 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:40:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:40:15.282+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:40:15.737 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:40:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:40:15.282+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:40:15.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:40:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:40:15.734+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:40:15.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:40:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:40:15.734+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:40:15.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:40:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:40:15.734+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:40:15.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:40:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:40:15.762+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:40:15.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:40:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:40:15.762+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:40:15.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:40:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:40:15.762+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:40:16.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:40:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:40:16.314+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:40:16.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:40:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:40:16.314+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:40:16.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:40:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:40:16.314+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:40:16.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:40:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:40:16.315+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:40:17.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:40:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:40:16.763+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:40:17.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:40:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:40:16.763+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:40:17.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:40:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:40:16.763+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:40:17.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:40:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:40:16.799+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:40:17.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:40:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:40:16.799+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:40:17.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:40:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:40:16.799+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:40:17.735 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:40:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:40:17.353+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:40:17.735 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:40:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:40:17.353+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:40:17.735 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:40:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:40:17.353+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:40:17.736 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:40:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:40:17.353+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:40:17.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:40:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:40:17.733+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:40:17.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:40:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:40:17.733+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:40:17.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:40:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:40:17.733+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:40:17.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:40:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:40:17.788+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:40:17.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:40:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:40:17.788+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:40:17.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:40:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:40:17.788+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:40:18.361 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:40:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:40:18.033+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 34896 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:40:18.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:40:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:40:18.359+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:40:18.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:40:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:40:18.359+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:40:18.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:40:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:40:18.359+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:40:18.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:40:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:40:18.359+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:40:19.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:40:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:40:18.748+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:40:19.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:40:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:40:18.748+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:40:19.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:40:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:40:18.748+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:40:19.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:40:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:40:18.766+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:40:19.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:40:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:40:18.766+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:40:19.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:40:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:40:18.766+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:40:19.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:40:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:40:19.351+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:40:19.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:40:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:40:19.351+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:40:19.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:40:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:40:19.351+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:40:19.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:40:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:40:19.351+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:40:19.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:40:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:40:19.794+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:40:19.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:40:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:40:19.794+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:40:19.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:40:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:40:19.794+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:40:19.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:40:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:40:19.740+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:40:19.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:40:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:40:19.740+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:40:19.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:40:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:40:19.740+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:40:20.727 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:40:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:40:20.380+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:40:20.727 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:40:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:40:20.381+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:40:20.728 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:40:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:40:20.381+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:40:20.728 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:40:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:40:20.381+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:40:20.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:40:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:40:20.725+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:40:20.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:40:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:40:20.725+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:40:20.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:40:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:40:20.725+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:40:20.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:40:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:40:20.792+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:40:20.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:40:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:40:20.792+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:40:20.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:40:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:40:20.792+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:40:21.728 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:40:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:40:21.339+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:40:21.729 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:40:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:40:21.339+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:40:21.729 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:40:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:40:21.339+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:40:21.729 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:40:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:40:21.339+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:40:21.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:40:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:40:21.726+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:40:21.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:40:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:40:21.726+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:40:21.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:40:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:40:21.726+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:40:21.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:40:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:40:21.798+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:40:21.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:40:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:40:21.798+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:40:21.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:40:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:40:21.798+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:40:22.713 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:40:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:40:22.315+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:40:22.713 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:40:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:40:22.315+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:40:22.713 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:40:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:40:22.315+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:40:22.713 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:40:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:40:22.315+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:40:22.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:40:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:40:22.820+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:40:22.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:40:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:40:22.820+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:40:22.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:40:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:40:22.820+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:40:22.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:40:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:40:22.710+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:40:22.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:40:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:40:22.710+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:40:22.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:40:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:40:22.710+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:40:23.291 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:40:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:40:23.038+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 34969 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:40:23.710 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:40:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:40:23.288+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:40:23.710 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:40:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:40:23.288+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:40:23.710 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:40:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:40:23.288+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:40:23.711 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:40:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:40:23.288+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:40:23.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:40:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:40:23.863+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:40:23.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:40:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:40:23.863+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:40:23.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:40:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:40:23.863+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:40:23.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:40:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:40:23.707+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:40:23.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:40:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:40:23.707+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:40:23.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:40:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:40:23.707+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:40:24.696 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:40:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:40:24.288+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:40:24.697 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:40:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:40:24.288+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:40:24.697 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:40:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:40:24.288+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:40:24.697 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:40:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:40:24.288+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:40:24.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:40:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:40:24.867+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:40:24.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:40:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:40:24.867+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:40:24.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:40:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:40:24.867+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:40:24.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:40:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:40:24.694+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:40:24.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:40:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:40:24.694+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:40:24.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:40:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:40:24.694+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:40:25.722 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:40:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:40:25.265+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:40:25.722 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:40:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:40:25.265+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:40:25.722 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:40:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:40:25.265+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:40:25.722 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:40:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:40:25.265+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:40:25.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:40:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:40:25.833+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:40:25.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:40:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:40:25.833+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:40:25.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:40:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:40:25.833+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:40:25.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:40:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:40:25.719+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:40:25.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:40:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:40:25.719+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:40:25.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:40:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:40:25.719+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:40:26.699 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:40:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:40:26.265+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:40:26.699 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:40:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:40:26.265+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:40:26.699 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:40:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:40:26.265+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:40:26.700 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:40:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:40:26.265+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:40:26.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:40:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:40:26.809+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:40:26.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:40:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:40:26.809+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:40:26.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:40:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:40:26.809+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:40:26.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:40:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:40:26.697+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:40:26.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:40:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:40:26.697+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:40:26.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:40:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:40:26.697+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:40:27.739 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:40:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:40:27.241+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:40:27.739 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:40:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:40:27.241+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:40:27.739 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:40:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:40:27.241+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:40:27.740 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:40:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:40:27.241+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:40:27.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:40:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:40:27.736+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:40:27.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:40:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:40:27.736+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:40:27.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:40:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:40:27.736+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:40:27.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:40:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:40:27.852+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:40:27.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:40:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:40:27.852+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:40:27.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:40:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:40:27.852+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:40:28.493 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:40:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:40:28.044+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 35040 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:40:28.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:40:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:40:28.195+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:40:28.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:40:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:40:28.195+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:40:28.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:40:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:40:28.195+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:40:28.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:40:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:40:28.195+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:40:28.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:40:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:40:28.720+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:40:28.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:40:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:40:28.720+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:40:28.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:40:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:40:28.720+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:40:28.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:40:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:40:28.818+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:40:28.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:40:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:40:28.818+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:40:28.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:40:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:40:28.818+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:40:29.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:40:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:40:29.174+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:40:29.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:40:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:40:29.174+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:40:29.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:40:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:40:29.174+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:40:29.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:40:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:40:29.174+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:40:29.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:40:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:40:29.823+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:40:29.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:40:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:40:29.823+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:40:29.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:40:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:40:29.823+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:40:29.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:40:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:40:29.721+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:40:29.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:40:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:40:29.721+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:40:29.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:40:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:40:29.721+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:40:30.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:40:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:40:30.158+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:40:30.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:40:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:40:30.158+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:40:30.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:40:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:40:30.158+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:40:30.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:40:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:40:30.158+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:40:31.190 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:40:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:40:30.861+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:40:31.190 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:40:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:40:30.861+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:40:31.191 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:40:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:40:30.861+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:40:31.191 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:40:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:40:30.769+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:40:31.191 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:40:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:40:30.769+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:40:31.191 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:40:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:40:30.769+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:40:31.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:40:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:40:31.187+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:40:31.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:40:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:40:31.187+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:40:31.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:40:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:40:31.187+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:40:31.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:40:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:40:31.187+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:40:32.214 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:40:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:40:31.893+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:40:32.214 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:40:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:40:31.893+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:40:32.215 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:40:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:40:31.893+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:40:32.215 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:40:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:40:31.781+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:40:32.215 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:40:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:40:31.781+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:40:32.215 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:40:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:40:31.781+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:40:32.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:40:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:40:32.211+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:40:32.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:40:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:40:32.211+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:40:32.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:40:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:40:32.211+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:40:32.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:40:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:40:32.211+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:40:33.051 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:40:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:40:32.931+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:40:33.052 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:40:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:40:32.931+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:40:33.052 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:40:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:40:32.931+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:40:33.052 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:40:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:40:32.770+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:40:33.052 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:40:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:40:32.770+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:40:33.052 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:40:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:40:32.770+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:40:33.493 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:40:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:40:33.048+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 35113 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:40:33.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:40:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:40:33.245+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:40:33.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:40:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:40:33.245+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:40:33.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:40:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:40:33.245+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:40:33.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:40:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:40:33.245+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:40:34.201 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:40:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:40:33.980+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:40:34.202 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:40:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:40:33.980+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:40:34.202 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:40:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:40:33.980+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:40:34.202 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:40:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:40:33.792+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:40:34.202 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:40:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:40:33.792+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:40:34.202 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:40:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:40:33.792+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:40:34.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:40:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:40:34.198+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:40:34.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:40:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:40:34.198+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:40:34.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:40:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:40:34.198+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:40:34.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:40:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:40:34.198+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:40:35.227 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:40:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:40:34.830+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:40:35.227 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:40:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:40:34.830+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:40:35.227 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:40:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:40:34.830+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:40:35.227 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:40:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:40:34.985+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:40:35.228 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:40:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:40:34.985+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:40:35.228 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:40:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:40:34.985+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:40:35.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:40:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:40:35.223+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:40:35.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:40:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:40:35.223+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:40:35.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:40:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:40:35.223+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:40:35.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:40:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:40:35.223+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:40:36.237 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:40:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:40:35.783+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:40:36.237 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:40:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:40:35.783+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:40:36.238 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:40:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:40:35.783+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:40:36.238 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:40:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:40:36.022+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:40:36.238 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:40:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:40:36.022+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:40:36.238 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:40:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:40:36.022+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:40:36.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:40:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:40:36.233+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:40:36.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:40:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:40:36.233+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:40:36.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:40:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:40:36.233+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:40:36.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:40:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:40:36.234+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:40:36.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:40:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:40:36.740+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:40:36.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:40:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:40:36.740+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:40:36.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:40:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:40:36.740+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:40:36.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:40:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:40:36.980+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:40:36.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:40:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:40:36.980+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:40:36.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:40:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:40:36.980+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:40:37.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:40:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:40:37.190+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:40:37.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:40:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:40:37.190+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:40:37.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:40:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:40:37.190+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:40:37.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:40:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:40:37.190+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:40:37.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:40:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:40:37.706+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:40:37.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:40:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:40:37.706+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:40:37.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:40:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:40:37.706+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:40:38.493 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:40:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:40:38.053+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 35184 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:40:38.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:40:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:40:38.144+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:40:38.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:40:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:40:38.144+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:40:38.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:40:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:40:38.144+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:40:38.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:40:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:40:38.144+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:40:38.494 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:40:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:40:38.006+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:40:38.494 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:40:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:40:38.006+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:40:38.494 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:40:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:40:38.006+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:40:38.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:40:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:40:38.669+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:40:38.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:40:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:40:38.669+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:40:38.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:40:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:40:38.669+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:40:39.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:40:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:40:39.013+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:40:39.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:40:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:40:39.013+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:40:39.494 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:40:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:40:39.013+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:40:39.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:40:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:40:39.172+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:40:39.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:40:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:40:39.172+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:40:39.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:40:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:40:39.172+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:40:39.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:40:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:40:39.172+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:40:39.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:40:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:40:39.690+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:40:39.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:40:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:40:39.690+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:40:39.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:40:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:40:39.690+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:40:40.441 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:40:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:40:40.057+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:40:40.442 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:40:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:40:40.057+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:40:40.442 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:40:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:40:40.057+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:40:40.442 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:40:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:40:40.141+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:40:40.442 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:40:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:40:40.141+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:40:40.442 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:40:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:40:40.141+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:40:40.442 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:40:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:40:40.142+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:40:40.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:40:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:40:40.700+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:40:40.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:40:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:40:40.700+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:40:40.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:40:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:40:40.700+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:40:41.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:40:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:40:41.061+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:40:41.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:40:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:40:41.061+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:40:41.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:40:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:40:41.061+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:40:41.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:40:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:40:41.163+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:40:41.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:40:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:40:41.163+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:40:41.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:40:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:40:41.163+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:40:41.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:40:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:40:41.163+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:40:41.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:40:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:40:41.683+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:40:41.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:40:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:40:41.683+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:40:41.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:40:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:40:41.683+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:40:42.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:40:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:40:42.043+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:40:42.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:40:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:40:42.043+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:40:42.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:40:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:40:42.043+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:40:42.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:40:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:40:42.167+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:40:42.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:40:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:40:42.167+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:40:42.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:40:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:40:42.167+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:40:42.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:40:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:40:42.168+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:40:42.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:40:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:40:42.678+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:40:42.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:40:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:40:42.678+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:40:42.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:40:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:40:42.678+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:40:43.493 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:40:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:40:43.059+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 35257 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:40:43.494 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:40:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:40:43.063+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:40:43.494 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:40:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:40:43.063+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:40:43.494 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:40:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:40:43.063+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:40:43.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:40:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:40:43.119+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:40:43.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:40:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:40:43.119+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:40:43.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:40:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:40:43.119+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:40:43.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:40:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:40:43.119+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:40:43.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:40:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:40:43.698+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:40:43.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:40:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:40:43.698+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:40:43.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:40:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:40:43.698+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:40:44.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:40:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:40:44.094+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:40:44.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:40:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:40:44.094+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:40:44.494 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:40:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:40:44.094+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:40:44.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:40:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:40:44.135+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:40:44.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:40:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:40:44.135+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:40:44.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:40:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:40:44.135+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:40:44.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:40:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:40:44.135+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:40:44.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:40:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:40:44.716+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:40:44.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:40:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:40:44.716+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:40:44.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:40:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:40:44.716+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:40:45.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:40:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:40:45.115+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:40:45.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:40:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:40:45.115+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:40:45.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:40:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:40:45.115+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:40:45.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:40:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:40:45.089+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:40:45.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:40:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:40:45.089+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:40:45.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:40:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:40:45.089+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:40:45.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:40:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:40:45.089+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:40:45.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:40:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:40:45.696+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:40:45.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:40:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:40:45.696+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:40:45.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:40:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:40:45.696+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:40:46.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:40:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:40:46.065+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:40:46.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:40:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:40:46.065+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:40:46.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:40:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:40:46.065+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:40:46.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:40:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:40:46.113+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:40:46.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:40:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:40:46.113+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:40:46.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:40:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:40:46.113+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:40:46.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:40:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:40:46.113+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:40:47.053 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:40:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:40:46.743+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:40:47.053 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:40:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:40:46.743+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:40:47.054 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:40:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:40:46.743+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:40:47.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:40:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:40:47.050+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:40:47.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:40:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:40:47.050+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:40:47.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:40:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:40:47.050+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:40:47.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:40:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:40:47.161+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:40:47.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:40:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:40:47.161+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:40:47.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:40:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:40:47.161+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:40:47.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:40:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:40:47.161+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:40:48.068 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:40:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:40:47.778+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:40:48.068 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:40:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:40:47.778+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:40:48.069 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:40:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:40:47.778+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:40:48.493 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:40:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:40:48.065+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 35328 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:40:48.494 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:40:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:40:48.082+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:40:48.494 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:40:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:40:48.082+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:40:48.494 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:40:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:40:48.082+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:40:48.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:40:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:40:48.152+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:40:48.495 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:40:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:40:48.152+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:40:48.495 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:40:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:40:48.152+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:40:48.495 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:40:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:40:48.152+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:40:49.143 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:40:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:40:48.792+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:40:49.144 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:40:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:40:48.792+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:40:49.144 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:40:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:40:48.792+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:40:49.144 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:40:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:40:49.038+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:40:49.144 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:40:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:40:49.038+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:40:49.144 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:40:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:40:49.038+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:40:49.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:40:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:40:49.140+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:40:49.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:40:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:40:49.140+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:40:49.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:40:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:40:49.140+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:40:49.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:40:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:40:49.140+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:40:50.088 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:40:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:40:49.788+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:40:50.089 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:40:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:40:49.788+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:40:50.089 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:40:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:40:49.788+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:40:50.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:40:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:40:50.085+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:40:50.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:40:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:40:50.085+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:40:50.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:40:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:40:50.085+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:40:50.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:40:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:40:50.136+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:40:50.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:40:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:40:50.136+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:40:50.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:40:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:40:50.136+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:40:50.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:40:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:40:50.136+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:40:51.137 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:40:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:40:50.818+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:40:51.137 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:40:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:40:50.818+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:40:51.137 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:40:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:40:50.818+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:40:51.137 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:40:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:40:51.064+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:40:51.137 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:40:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:40:51.064+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:40:51.137 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:40:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:40:51.064+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:40:51.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:40:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:40:51.134+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:40:51.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:40:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:40:51.134+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:40:51.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:40:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:40:51.134+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:40:51.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:40:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:40:51.134+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:40:52.165 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:40:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:40:51.793+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:40:52.166 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:40:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:40:51.793+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:40:52.166 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:40:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:40:51.793+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:40:52.166 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:40:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:40:52.017+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:40:52.166 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:40:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:40:52.017+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:40:52.166 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:40:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:40:52.017+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:40:52.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:40:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:40:52.163+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:40:52.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:40:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:40:52.163+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:40:52.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:40:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:40:52.163+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:40:52.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:40:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:40:52.163+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:40:52.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:40:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:40:52.768+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:40:52.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:40:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:40:52.768+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:40:52.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:40:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:40:52.768+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:40:53.493 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:40:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:40:53.070+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 35398 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:40:53.494 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:40:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:40:53.034+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:40:53.494 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:40:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:40:53.034+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:40:53.494 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:40:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:40:53.034+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:40:53.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:40:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:40:53.212+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:40:53.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:40:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:40:53.212+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:40:53.495 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:40:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:40:53.212+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:40:53.495 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:40:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:40:53.212+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:40:54.083 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:40:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:40:53.778+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:40:54.083 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:40:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:40:53.778+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:40:54.083 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:40:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:40:53.778+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:40:54.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:40:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:40:54.080+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:40:54.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:40:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:40:54.080+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:40:54.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:40:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:40:54.080+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:40:54.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:40:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:40:54.242+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:40:54.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:40:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:40:54.242+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:40:54.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:40:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:40:54.242+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:40:54.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:40:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:40:54.242+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:40:54.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:40:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:40:54.733+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:40:54.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:40:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:40:54.733+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:40:54.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:40:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:40:54.733+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:40:55.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:40:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:40:55.075+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:40:55.494 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:40:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:40:55.075+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:40:55.494 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:40:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:40:55.075+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:40:55.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:40:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:40:55.216+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:40:55.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:40:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:40:55.216+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:40:55.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:40:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:40:55.216+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:40:55.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:40:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:40:55.216+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:40:55.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:40:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:40:55.732+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:40:55.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:40:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:40:55.732+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:40:55.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:40:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:40:55.732+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:40:56.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:40:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:40:56.058+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:40:56.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:40:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:40:56.058+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:40:56.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:40:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:40:56.058+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:40:56.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:40:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:40:56.209+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:40:56.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:40:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:40:56.209+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:40:56.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:40:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:40:56.209+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:40:56.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:40:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:40:56.209+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:40:57.095 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:40:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:40:56.777+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:40:57.095 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:40:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:40:56.777+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:40:57.095 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:40:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:40:56.777+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:40:57.494 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:40:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:40:57.091+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:40:57.494 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:40:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:40:57.091+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:40:57.494 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:40:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:40:57.091+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:40:57.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:40:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:40:57.219+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:40:57.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:40:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:40:57.219+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:40:57.495 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:40:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:40:57.219+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:40:57.495 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:40:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:40:57.219+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:40:58.071 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:40:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:40:57.799+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:40:58.071 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:40:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:40:57.799+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:40:58.071 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:40:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:40:57.799+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:40:58.493 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:40:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:40:58.076+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 35466 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:40:58.494 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:40:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:40:58.068+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:40:58.495 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:40:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:40:58.068+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:40:58.495 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:40:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:40:58.068+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:40:58.495 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:40:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:40:58.191+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:40:58.495 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:40:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:40:58.191+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:40:58.495 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:40:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:40:58.191+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:40:58.495 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:40:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:40:58.191+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:40:59.056 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:40:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:40:58.772+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:40:59.056 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:40:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:40:58.772+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:40:59.056 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:40:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:40:58.772+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:40:59.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:40:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:40:59.239+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:40:59.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:40:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:40:59.239+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:40:59.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:40:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:40:59.239+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:40:59.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:40:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:40:59.239+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:40:59.494 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:40:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:40:59.053+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:40:59.494 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:40:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:40:59.053+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:40:59.494 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:40:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:40:59.053+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:41:00.074 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:40:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:40:59.806+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:41:00.074 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:40:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:40:59.806+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:41:00.074 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:40:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:40:59.806+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:41:00.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:41:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:41:00.070+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:41:00.494 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:41:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:41:00.070+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:41:00.494 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:41:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:41:00.070+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:41:00.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:41:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:41:00.216+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:41:00.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:41:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:41:00.216+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:41:00.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:41:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:41:00.216+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:41:00.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:41:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:41:00.216+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:41:01.086 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:41:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:41:00.799+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:41:01.086 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:41:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:41:00.799+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:41:01.087 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:41:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:41:00.799+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:41:01.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:41:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:41:01.083+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:41:01.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:41:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:41:01.083+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:41:01.494 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:41:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:41:01.083+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:41:01.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:41:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:41:01.184+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:41:01.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:41:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:41:01.184+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:41:01.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:41:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:41:01.184+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:41:01.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:41:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:41:01.184+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:41:02.115 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:41:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:41:01.762+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:41:02.116 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:41:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:41:01.762+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:41:02.116 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:41:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:41:01.762+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:41:02.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:41:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:41:02.189+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:41:02.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:41:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:41:02.189+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:41:02.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:41:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:41:02.189+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:41:02.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:41:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:41:02.190+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:41:02.494 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:41:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:41:02.112+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:41:02.494 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:41:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:41:02.112+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:41:02.494 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:41:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:41:02.112+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:41:03.067 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:41:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:41:02.773+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:41:03.067 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:41:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:41:02.773+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:41:03.067 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:41:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:41:02.773+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:41:03.493 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:41:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:41:03.081+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 35536 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:41:03.494 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:41:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:41:03.064+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:41:03.494 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:41:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:41:03.064+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:41:03.494 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:41:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:41:03.064+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:41:03.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:41:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:41:03.183+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:41:03.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:41:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:41:03.183+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:41:03.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:41:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:41:03.183+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:41:03.495 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:41:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:41:03.183+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:41:04.048 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:41:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:41:03.781+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:41:04.048 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:41:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:41:03.781+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:41:04.048 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:41:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:41:03.781+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:41:04.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:41:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:41:04.045+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:41:04.494 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:41:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:41:04.045+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:41:04.494 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:41:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:41:04.045+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:41:04.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:41:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:41:04.141+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:41:04.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:41:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:41:04.141+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:41:04.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:41:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:41:04.141+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:41:04.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:41:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:41:04.141+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:41:05.158 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:41:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:41:05.012+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:41:05.158 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:41:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:41:05.012+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:41:05.158 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:41:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:41:05.012+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:41:05.158 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:41:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:41:04.823+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:41:05.159 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:41:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:41:04.824+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:41:05.159 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:41:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:41:04.824+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:41:05.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:41:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:41:05.155+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:41:05.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:41:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:41:05.155+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:41:05.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:41:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:41:05.155+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:41:05.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:41:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:41:05.155+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:41:06.159 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:41:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:41:05.991+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:41:06.159 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:41:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:41:05.991+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:41:06.160 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:41:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:41:05.991+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:41:06.160 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:41:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:41:05.831+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:41:06.160 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:41:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:41:05.831+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:41:06.160 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:41:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:41:05.831+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:41:06.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:41:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:41:06.156+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:41:06.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:41:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:41:06.157+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:41:06.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:41:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:41:06.157+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:41:06.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:41:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:41:06.157+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:41:07.151 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:41:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:41:06.983+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:41:07.152 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:41:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:41:06.983+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:41:07.152 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:41:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:41:06.983+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:41:07.152 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:41:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:41:06.824+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:41:07.152 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:41:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:41:06.824+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:41:07.152 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:41:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:41:06.824+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:41:07.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:41:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:41:07.149+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:41:07.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:41:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:41:07.149+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:41:07.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:41:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:41:07.149+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:41:07.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:41:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:41:07.149+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:41:08.111 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:41:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:41:08.089+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 35601 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:41:08.112 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:41:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:41:07.857+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:41:08.112 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:41:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:41:07.857+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:41:08.112 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:41:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:41:07.857+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:41:08.112 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:41:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:41:07.994+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:41:08.113 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:41:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:41:07.994+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:41:08.113 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:41:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:41:07.994+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:41:08.113 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:41:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:41:08.100+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:41:08.113 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:41:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:41:08.100+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:41:08.113 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:41:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:41:08.100+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:41:08.113 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:41:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:41:08.100+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:41:09.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:41:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:41:09.009+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:41:09.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:41:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:41:09.009+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:41:09.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:41:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:41:09.009+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:41:09.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:41:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:41:08.831+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:41:09.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:41:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:41:08.831+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:41:09.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:41:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:41:08.831+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:41:09.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:41:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:41:09.058+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:41:09.245 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:41:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:41:09.058+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:41:09.245 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:41:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:41:09.058+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:41:09.245 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:41:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:41:09.058+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:41:10.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:41:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:41:10.052+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:41:10.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:41:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:41:10.052+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:41:10.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:41:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:41:10.052+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:41:10.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:41:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:41:09.843+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:41:10.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:41:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:41:09.843+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:41:10.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:41:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:41:09.843+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:41:10.245 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:41:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:41:10.016+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:41:10.245 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:41:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:41:10.016+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:41:10.245 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:41:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:41:10.016+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:41:10.245 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:41:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:41:10.016+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:41:11.161 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:41:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:41:10.893+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:41:11.162 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:41:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:41:10.893+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:41:11.162 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:41:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:41:10.893+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:41:11.162 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:41:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:41:11.002+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:41:11.162 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:41:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:41:11.002+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:41:11.163 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:41:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:41:11.002+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:41:11.163 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:41:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:41:11.028+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:41:11.163 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:41:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:41:11.028+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:41:11.163 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:41:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:41:11.028+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:41:11.163 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:41:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:41:11.028+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:41:12.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:41:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:41:12.051+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:41:12.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:41:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:41:12.051+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:41:12.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:41:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:41:12.051+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:41:12.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:41:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:41:11.859+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:41:12.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:41:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:41:11.859+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:41:12.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:41:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:41:11.859+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:41:12.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:41:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:41:12.047+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:41:12.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:41:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:41:12.047+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:41:12.245 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:41:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:41:12.047+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:41:12.245 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:41:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:41:12.047+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:41:13.219 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:41:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:41:13.094+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 35667 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:41:13.219 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:41:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:41:13.076+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:41:13.219 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:41:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:41:13.076+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:41:13.219 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:41:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:41:13.076+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:41:13.220 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:41:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:41:12.875+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:41:13.220 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:41:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:41:12.875+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:41:13.220 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:41:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:41:12.875+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:41:13.220 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:41:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:41:13.063+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:41:13.220 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:41:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:41:13.063+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:41:13.221 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:41:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:41:13.063+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:41:13.221 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:41:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:41:13.063+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:41:14.234 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:41:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:41:14.097+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:41:14.234 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:41:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:41:14.097+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:41:14.234 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:41:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:41:14.097+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:41:14.234 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:41:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:41:13.910+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:41:14.234 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:41:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:41:13.910+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:41:14.235 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:41:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:41:13.910+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:41:14.235 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:41:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:41:14.070+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:41:14.235 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:41:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:41:14.070+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:41:14.235 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:41:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:41:14.070+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:41:14.235 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:41:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:41:14.071+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:41:15.129 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:41:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:41:14.873+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:41:15.129 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:41:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:41:14.873+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:41:15.129 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:41:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:41:14.873+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:41:15.129 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:41:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:41:15.054+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:41:15.130 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:41:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:41:15.054+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:41:15.130 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:41:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:41:15.054+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:41:15.130 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:41:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:41:15.054+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:41:15.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:41:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:41:15.126+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:41:15.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:41:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:41:15.126+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:41:15.494 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:41:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:41:15.126+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:41:16.153 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:41:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:41:15.872+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:41:16.154 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:41:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:41:15.872+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:41:16.154 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:41:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:41:15.872+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:41:16.154 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:41:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:41:16.024+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:41:16.154 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:41:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:41:16.024+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:41:16.154 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:41:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:41:16.024+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:41:16.154 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:41:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:41:16.024+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:41:16.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:41:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:41:16.151+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:41:16.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:41:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:41:16.151+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:41:16.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:41:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:41:16.151+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:41:17.190 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:41:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:41:16.922+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:41:17.191 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:41:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:41:16.922+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:41:17.191 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:41:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:41:16.922+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:41:17.191 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:41:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:41:17.045+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:41:17.191 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:41:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:41:17.045+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:41:17.191 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:41:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:41:17.045+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:41:17.191 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:41:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:41:17.045+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:41:17.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:41:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:41:17.188+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:41:17.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:41:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:41:17.188+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:41:17.494 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:41:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:41:17.188+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:41:18.201 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:41:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:41:18.099+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 35732 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:41:18.202 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:41:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:41:17.894+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:41:18.202 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:41:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:41:17.894+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:41:18.202 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:41:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:41:17.894+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:41:18.202 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:41:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:41:18.058+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:41:18.203 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:41:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:41:18.058+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:41:18.203 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:41:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:41:18.058+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:41:18.203 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:41:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:41:18.058+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:41:18.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:41:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:41:18.199+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:41:18.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:41:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:41:18.199+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:41:18.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:41:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:41:18.199+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:41:19.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:41:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:41:19.175+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:41:19.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:41:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:41:19.175+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:41:19.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:41:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:41:19.175+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:41:19.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:41:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:41:18.927+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:41:19.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:41:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:41:18.927+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:41:19.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:41:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:41:18.927+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:41:19.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:41:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:41:19.101+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:41:19.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:41:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:41:19.101+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:41:19.245 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:41:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:41:19.101+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:41:19.245 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:41:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:41:19.101+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:41:20.224 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:41:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:41:19.911+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:41:20.224 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:41:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:41:19.911+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:41:20.224 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:41:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:41:19.911+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:41:20.225 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:41:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:41:20.100+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:41:20.225 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:41:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:41:20.100+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:41:20.225 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:41:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:41:20.100+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:41:20.225 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:41:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:41:20.100+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:41:20.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:41:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:41:20.221+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:41:20.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:41:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:41:20.221+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:41:20.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:41:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:41:20.221+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:41:21.224 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:41:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:41:20.916+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:41:21.225 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:41:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:41:20.916+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:41:21.225 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:41:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:41:20.916+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:41:21.226 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:41:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:41:21.108+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:41:21.226 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:41:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:41:21.108+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:41:21.226 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:41:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:41:21.108+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:41:21.227 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:41:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:41:21.108+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:41:21.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:41:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:41:21.222+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:41:21.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:41:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:41:21.222+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:41:21.494 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:41:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:41:21.222+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:41:22.066 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:41:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:41:21.938+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:41:22.066 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:41:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:41:21.938+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:41:22.066 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:41:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:41:21.938+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:41:22.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:41:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:41:22.264+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:41:22.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:41:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:41:22.264+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:41:22.494 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:41:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:41:22.264+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:41:22.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:41:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:41:22.063+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:41:22.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:41:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:41:22.063+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:41:22.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:41:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:41:22.063+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:41:22.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:41:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:41:22.063+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:41:23.227 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:41:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:41:23.104+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 35799 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:41:23.228 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:41:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:41:22.922+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:41:23.228 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:41:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:41:22.922+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:41:23.228 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:41:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:41:22.922+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:41:23.228 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:41:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:41:23.060+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:41:23.228 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:41:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:41:23.060+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:41:23.228 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:41:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:41:23.060+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:41:23.229 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:41:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:41:23.060+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:41:23.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:41:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:41:23.265+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:41:23.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:41:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:41:23.265+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:41:23.494 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:41:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:41:23.265+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:41:24.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:41:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:41:24.019+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:41:24.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:41:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:41:24.019+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:41:24.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:41:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:41:24.019+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:41:24.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:41:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:41:24.019+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:41:24.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:41:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:41:23.967+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:41:24.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:41:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:41:23.967+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:41:24.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:41:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:41:23.967+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:41:24.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:41:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:41:24.286+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:41:24.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:41:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:41:24.286+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:41:24.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:41:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:41:24.286+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:41:25.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:41:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:41:24.983+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:41:25.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:41:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:41:24.983+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:41:25.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:41:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:41:24.983+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:41:25.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:41:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:41:25.002+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:41:25.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:41:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:41:25.002+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:41:25.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:41:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:41:25.002+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:41:25.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:41:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:41:25.002+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:41:25.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:41:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:41:25.249+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:41:25.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:41:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:41:25.249+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:41:25.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:41:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:41:25.249+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:41:26.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:41:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:41:25.947+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:41:26.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:41:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:41:25.947+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:41:26.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:41:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:41:25.947+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:41:26.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:41:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:41:25.989+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:41:26.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:41:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:41:25.989+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:41:26.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:41:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:41:25.989+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:41:26.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:41:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:41:25.989+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:41:26.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:41:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:41:26.279+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:41:26.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:41:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:41:26.279+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:41:26.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:41:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:41:26.279+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:41:27.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:41:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:41:26.976+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:41:27.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:41:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:41:26.976+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:41:27.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:41:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:41:26.976+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:41:27.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:41:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:41:27.029+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:41:27.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:41:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:41:27.029+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:41:27.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:41:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:41:27.029+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:41:27.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:41:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:41:27.029+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:41:27.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:41:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:41:27.299+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:41:27.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:41:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:41:27.299+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:41:27.744 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:41:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:41:27.299+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:41:28.231 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:41:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:41:28.108+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 35864 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:41:28.232 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:41:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:41:27.948+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:41:28.232 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:41:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:41:27.948+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:41:28.232 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:41:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:41:27.948+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:41:28.232 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:41:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:41:28.038+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:41:28.233 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:41:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:41:28.038+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:41:28.233 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:41:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:41:28.038+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:41:28.233 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:41:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:41:28.039+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:41:28.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:41:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:41:28.276+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:41:28.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:41:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:41:28.276+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:41:28.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:41:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:41:28.276+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:41:29.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:41:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:41:28.932+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:41:29.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:41:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:41:28.932+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:41:29.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:41:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:41:28.932+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:41:29.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:41:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:41:29.004+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:41:29.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:41:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:41:29.004+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:41:29.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:41:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:41:29.004+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:41:29.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:41:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:41:29.004+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:41:29.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:41:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:41:29.325+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:41:29.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:41:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:41:29.325+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:41:29.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:41:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:41:29.325+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:41:30.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:41:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:41:29.910+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:41:30.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:41:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:41:29.910+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:41:30.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:41:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:41:29.910+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:41:30.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:41:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:41:29.991+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:41:30.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:41:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:41:29.991+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:41:30.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:41:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:41:29.991+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:41:30.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:41:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:41:29.991+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:41:30.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:41:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:41:30.281+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:41:30.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:41:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:41:30.281+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:41:30.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:41:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:41:30.282+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:41:31.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:41:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:41:30.947+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:41:31.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:41:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:41:30.947+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:41:31.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:41:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:41:30.947+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:41:31.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:41:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:41:31.007+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:41:31.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:41:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:41:31.007+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:41:31.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:41:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:41:31.007+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:41:31.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:41:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:41:31.008+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:41:31.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:41:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:41:31.313+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:41:31.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:41:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:41:31.313+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:41:31.744 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:41:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:41:31.313+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:41:32.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:41:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:41:31.935+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:41:32.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:41:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:41:31.935+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:41:32.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:41:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:41:31.935+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:41:32.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:41:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:41:32.045+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:41:32.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:41:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:41:32.045+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:41:32.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:41:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:41:32.045+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:41:32.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:41:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:41:32.045+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:41:32.665 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:41:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:41:32.336+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:41:32.666 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:41:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:41:32.336+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:41:32.666 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:41:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:41:32.336+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:41:33.241 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:41:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:41:33.114+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 35934 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:41:33.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:41:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:41:32.925+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:41:33.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:41:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:41:32.925+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:41:33.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:41:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:41:32.925+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:41:33.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:41:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:41:33.049+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:41:33.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:41:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:41:33.049+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:41:33.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:41:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:41:33.049+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:41:33.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:41:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:41:33.049+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:41:33.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:41:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:41:33.373+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:41:33.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:41:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:41:33.373+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:41:33.494 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:41:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:41:33.373+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:41:34.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:41:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:41:34.097+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:41:34.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:41:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:41:34.097+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:41:34.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:41:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:41:34.097+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:41:34.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:41:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:41:34.097+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:41:34.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:41:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:41:33.932+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:41:34.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:41:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:41:33.932+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:41:34.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:41:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:41:33.932+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:41:34.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:41:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:41:34.420+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:41:34.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:41:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:41:34.420+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:41:34.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:41:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:41:34.420+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:41:35.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:41:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:41:34.979+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:41:35.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:41:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:41:34.979+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:41:35.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:41:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:41:34.979+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:41:35.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:41:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:41:35.079+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:41:35.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:41:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:41:35.079+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:41:35.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:41:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:41:35.079+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:41:35.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:41:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:41:35.079+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:41:35.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:41:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:41:35.429+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:41:35.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:41:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:41:35.429+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:41:35.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:41:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:41:35.429+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:41:36.392 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:41:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:41:36.022+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:41:36.392 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:41:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:41:36.022+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:41:36.392 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:41:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:41:36.022+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:41:36.392 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:41:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:41:36.037+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:41:36.393 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:41:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:41:36.037+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:41:36.393 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:41:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:41:36.037+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:41:36.393 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:41:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:41:36.037+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:41:36.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:41:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:41:36.388+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:41:36.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:41:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:41:36.388+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:41:36.744 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:41:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:41:36.388+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:41:37.405 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:41:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:41:37.011+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:41:37.405 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:41:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:41:37.011+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:41:37.405 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:41:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:41:37.011+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:41:37.406 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:41:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:41:36.997+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:41:37.406 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:41:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:41:36.997+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:41:37.406 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:41:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:41:36.997+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:41:37.406 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:41:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:41:36.997+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:41:37.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:41:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:41:37.402+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:41:37.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:41:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:41:37.402+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:41:37.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:41:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:41:37.402+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:41:38.243 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:41:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:41:38.120+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 36002 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:41:38.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:41:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:41:37.983+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:41:38.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:41:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:41:37.983+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:41:38.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:41:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:41:37.983+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:41:38.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:41:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:41:38.030+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:41:38.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:41:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:41:38.030+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:41:38.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:41:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:41:38.030+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:41:38.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:41:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:41:38.030+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:41:38.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:41:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:41:38.373+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:41:38.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:41:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:41:38.373+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:41:38.744 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:41:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:41:38.373+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:41:39.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:41:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:41:38.944+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:41:39.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:41:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:41:38.944+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:41:39.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:41:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:41:38.944+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:41:39.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:41:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:41:39.004+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:41:39.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:41:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:41:39.004+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:41:39.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:41:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:41:39.004+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:41:39.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:41:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:41:39.004+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:41:39.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:41:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:41:39.349+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:41:39.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:41:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:41:39.349+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:41:39.744 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:41:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:41:39.349+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:41:40.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:41:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:41:39.903+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:41:40.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:41:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:41:39.903+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:41:40.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:41:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:41:39.903+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:41:40.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:41:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:41:39.968+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:41:40.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:41:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:41:39.968+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:41:40.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:41:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:41:39.968+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:41:40.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:41:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:41:39.968+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:41:40.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:41:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:41:40.319+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:41:40.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:41:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:41:40.319+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:41:40.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:41:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:41:40.319+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:41:41.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:41:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:41:40.949+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:41:41.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:41:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:41:40.949+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:41:41.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:41:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:41:40.949+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:41:41.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:41:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:41:41.014+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:41:41.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:41:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:41:41.014+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:41:41.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:41:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:41:41.014+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:41:41.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:41:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:41:41.014+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:41:41.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:41:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:41:41.330+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:41:41.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:41:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:41:41.330+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:41:41.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:41:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:41:41.330+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:41:42.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:41:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:41:41.959+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:41:42.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:41:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:41:41.959+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:41:42.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:41:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:41:41.959+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:41:42.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:41:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:41:42.010+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:41:42.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:41:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:41:42.010+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:41:42.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:41:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:41:42.010+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:41:42.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:41:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:41:42.011+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:41:42.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:41:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:41:42.366+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:41:42.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:41:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:41:42.366+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:41:42.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:41:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:41:42.366+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:41:43.243 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:41:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:41:43.126+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 36075 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:41:43.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:41:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:41:42.984+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:41:43.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:41:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:41:42.984+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:41:43.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:41:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:41:42.984+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:41:43.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:41:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:41:42.984+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:41:43.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:41:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:41:42.991+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:41:43.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:41:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:41:42.991+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:41:43.245 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:41:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:41:42.991+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:41:43.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:41:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:41:43.370+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:41:43.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:41:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:41:43.370+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:41:43.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:41:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:41:43.370+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:41:44.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:41:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:41:43.990+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:41:44.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:41:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:41:43.990+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:41:44.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:41:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:41:43.990+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:41:44.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:41:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:41:43.973+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:41:44.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:41:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:41:43.973+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:41:44.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:41:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:41:43.973+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:41:44.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:41:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:41:43.973+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:41:44.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:41:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:41:44.341+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:41:44.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:41:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:41:44.341+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:41:44.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:41:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:41:44.341+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:41:45.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:41:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:41:45.028+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:41:45.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:41:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:41:45.028+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:41:45.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:41:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:41:45.028+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:41:45.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:41:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:41:44.962+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:41:45.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:41:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:41:44.962+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:41:45.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:41:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:41:44.962+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:41:45.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:41:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:41:44.963+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:41:45.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:41:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:41:45.390+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:41:45.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:41:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:41:45.390+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:41:45.744 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:41:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:41:45.390+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:41:46.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:41:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:41:46.077+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:41:46.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:41:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:41:46.077+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:41:46.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:41:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:41:46.077+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:41:46.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:41:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:41:45.960+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:41:46.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:41:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:41:45.960+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:41:46.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:41:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:41:45.960+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:41:46.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:41:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:41:45.960+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:41:46.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:41:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:41:46.424+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:41:46.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:41:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:41:46.424+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:41:46.744 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:41:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:41:46.424+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:41:47.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:41:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:41:46.939+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:41:47.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:41:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:41:46.939+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:41:47.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:41:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:41:46.939+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:41:47.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:41:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:41:46.939+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:41:47.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:41:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:41:47.062+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:41:47.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:41:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:41:47.062+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:41:47.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:41:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:41:47.062+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:41:47.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:41:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:41:47.385+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:41:47.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:41:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:41:47.385+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:41:47.744 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:41:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:41:47.385+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:41:48.244 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:41:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:41:48.131+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 36146 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:41:48.245 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:41:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:41:48.051+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:41:48.245 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:41:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:41:48.051+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:41:48.246 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:41:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:41:48.051+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:41:48.246 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:41:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:41:47.958+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:41:48.247 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:41:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:41:47.958+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:41:48.247 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:41:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:41:47.958+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:41:48.247 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:41:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:41:47.958+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:41:48.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:41:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:41:48.351+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:41:48.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:41:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:41:48.351+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:41:48.744 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:41:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:41:48.351+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:41:49.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:41:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:41:49.009+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:41:49.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:41:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:41:49.009+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:41:49.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:41:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:41:49.009+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:41:49.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:41:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:41:48.980+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:41:49.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:41:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:41:48.980+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:41:49.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:41:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:41:48.980+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:41:49.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:41:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:41:48.981+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:41:49.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:41:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:41:49.311+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:41:49.744 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:41:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:41:49.311+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:41:49.744 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:41:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:41:49.311+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:41:50.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:41:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:41:50.056+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:41:50.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:41:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:41:50.056+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:41:50.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:41:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:41:50.056+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:41:50.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:41:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:41:50.005+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:41:50.245 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:41:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:41:50.005+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:41:50.245 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:41:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:41:50.005+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:41:50.245 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:41:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:41:50.005+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:41:50.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:41:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:41:50.345+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:41:50.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:41:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:41:50.345+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:41:50.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:41:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:41:50.345+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:41:51.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:41:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:41:51.048+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:41:51.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:41:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:41:51.048+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:41:51.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:41:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:41:51.048+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:41:51.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:41:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:41:50.969+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:41:51.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:41:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:41:50.969+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:41:51.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:41:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:41:50.969+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:41:51.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:41:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:41:50.969+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:41:51.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:41:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:41:51.354+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:41:51.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:41:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:41:51.354+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:41:51.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:41:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:41:51.354+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:41:52.087 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:41:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:41:51.981+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:41:52.088 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:41:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:41:51.981+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:41:52.088 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:41:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:41:51.981+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:41:52.088 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:41:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:41:51.981+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:41:52.338 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:41:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:41:52.335+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:41:52.338 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:41:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:41:52.335+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:41:52.338 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:41:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:41:52.335+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:41:52.338 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:41:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:41:52.085+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:41:52.338 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:41:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:41:52.085+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:41:52.339 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:41:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:41:52.085+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:41:53.309 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:41:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:41:53.137+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 36219 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:41:53.310 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:41:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:41:53.017+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:41:53.310 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:41:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:41:53.017+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:41:53.310 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:41:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:41:53.017+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:41:53.310 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:41:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:41:53.017+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:41:53.310 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:41:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:41:53.070+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:41:53.311 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:41:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:41:53.070+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:41:53.311 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:41:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:41:53.070+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:41:53.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:41:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:41:53.306+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:41:53.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:41:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:41:53.306+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:41:53.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:41:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:41:53.306+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:41:54.315 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:41:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:41:54.102+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:41:54.315 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:41:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:41:54.102+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:41:54.315 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:41:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:41:54.102+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:41:54.315 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:41:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:41:54.041+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:41:54.315 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:41:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:41:54.041+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:41:54.316 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:41:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:41:54.041+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:41:54.316 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:41:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:41:54.041+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:41:54.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:41:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:41:54.312+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:41:54.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:41:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:41:54.312+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:41:54.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:41:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:41:54.312+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:41:55.357 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:41:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:41:55.118+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:41:55.358 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:41:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:41:55.118+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:41:55.358 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:41:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:41:55.118+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:41:55.358 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:41:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:41:55.086+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:41:55.358 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:41:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:41:55.086+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:41:55.358 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:41:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:41:55.086+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:41:55.358 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:41:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:41:55.086+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:41:55.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:41:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:41:55.355+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:41:55.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:41:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:41:55.355+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:41:55.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:41:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:41:55.355+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:41:56.351 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:41:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:41:56.145+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:41:56.351 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:41:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:41:56.145+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:41:56.351 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:41:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:41:56.145+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:41:56.352 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:41:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:41:56.069+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:41:56.352 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:41:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:41:56.069+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:41:56.352 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:41:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:41:56.069+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:41:56.352 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:41:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:41:56.069+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:41:56.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:41:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:41:56.348+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:41:56.744 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:41:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:41:56.348+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:41:56.744 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:41:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:41:56.348+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:41:57.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:41:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:41:57.303+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:41:57.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:41:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:41:57.303+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:41:57.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:41:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:41:57.303+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:41:57.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:41:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:41:57.159+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:41:57.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:41:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:41:57.159+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:41:57.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:41:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:41:57.159+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:41:57.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:41:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:41:57.057+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:41:57.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:41:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:41:57.057+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:41:57.495 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:41:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:41:57.057+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:41:57.495 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:41:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:41:57.057+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:41:58.493 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:41:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:41:58.143+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 36290 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:41:58.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:41:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:41:58.129+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:41:58.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:41:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:41:58.129+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:41:58.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:41:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:41:58.129+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:41:58.495 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:41:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:41:58.298+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:41:58.495 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:41:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:41:58.298+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:41:58.495 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:41:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:41:58.298+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:41:58.495 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:41:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:41:58.060+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:41:58.495 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:41:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:41:58.060+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:41:58.495 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:41:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:41:58.060+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:41:58.496 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:41:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:41:58.060+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:41:59.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:41:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:41:59.290+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:41:59.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:41:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:41:59.290+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:41:59.494 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:41:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:41:59.290+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:41:59.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:41:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:41:59.144+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:41:59.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:41:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:41:59.144+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:41:59.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:41:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:41:59.144+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:41:59.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:41:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:41:59.098+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:41:59.495 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:41:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:41:59.098+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:41:59.495 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:41:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:41:59.098+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:41:59.495 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:41:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:41:59.098+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:42:00.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:42:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:42:00.241+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:42:00.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:42:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:42:00.241+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:42:00.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:42:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:42:00.241+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:42:00.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:42:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:42:00.136+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:42:00.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:42:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:42:00.136+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:42:00.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:42:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:42:00.136+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:42:00.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:42:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:42:00.136+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:42:00.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:42:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:42:00.145+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:42:00.495 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:42:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:42:00.145+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:42:00.495 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:42:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:42:00.145+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:42:01.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:42:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:42:01.265+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:42:01.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:42:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:42:01.265+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:42:01.494 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:42:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:42:01.265+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:42:01.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:42:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:42:01.152+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:42:01.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:42:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:42:01.152+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:42:01.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:42:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:42:01.152+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:42:01.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:42:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:42:01.152+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:42:01.495 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:42:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:42:01.148+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:42:01.495 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:42:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:42:01.148+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:42:01.495 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:42:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:42:01.148+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:42:02.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:42:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:42:02.124+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:42:02.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:42:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:42:02.124+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:42:02.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:42:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:42:02.124+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:42:02.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:42:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:42:02.124+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:42:02.494 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:42:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:42:02.224+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:42:02.494 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:42:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:42:02.224+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:42:02.494 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:42:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:42:02.224+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:42:02.495 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:42:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:42:02.160+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:42:02.495 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:42:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:42:02.160+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:42:02.495 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:42:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:42:02.160+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:42:03.385 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:42:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:42:03.148+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 36363 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:42:03.386 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:42:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:42:03.151+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:42:03.386 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:42:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:42:03.151+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:42:03.386 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:42:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:42:03.151+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:42:03.387 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:42:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:42:03.196+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:42:03.387 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:42:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:42:03.196+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:42:03.387 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:42:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:42:03.196+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:42:03.387 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:42:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:42:03.099+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:42:03.387 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:42:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:42:03.099+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:42:03.387 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:42:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:42:03.099+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:42:03.388 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:42:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:42:03.099+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:42:04.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:42:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:42:04.181+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:42:04.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:42:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:42:04.181+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:42:04.494 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:42:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:42:04.181+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:42:04.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:42:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:42:04.185+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:42:04.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:42:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:42:04.185+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:42:04.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:42:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:42:04.185+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:42:04.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:42:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:42:04.133+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:42:04.495 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:42:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:42:04.133+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:42:04.495 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:42:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:42:04.133+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:42:04.495 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:42:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:42:04.133+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:42:05.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:42:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:42:05.221+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:42:05.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:42:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:42:05.221+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:42:05.494 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:42:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:42:05.221+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:42:05.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:42:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:42:05.229+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:42:05.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:42:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:42:05.229+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:42:05.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:42:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:42:05.229+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:42:05.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:42:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:42:05.135+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:42:05.495 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:42:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:42:05.135+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:42:05.495 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:42:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:42:05.135+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:42:05.495 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:42:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:42:05.135+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:42:06.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:42:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:42:06.218+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:42:06.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:42:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:42:06.218+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:42:06.494 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:42:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:42:06.218+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:42:06.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:42:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:42:06.239+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:42:06.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:42:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:42:06.239+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:42:06.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:42:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:42:06.239+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:42:06.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:42:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:42:06.181+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:42:06.495 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:42:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:42:06.181+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:42:06.495 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:42:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:42:06.181+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:42:06.495 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:42:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:42:06.182+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:42:07.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:42:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:42:07.243+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:42:07.494 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:42:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:42:07.243+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:42:07.494 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:42:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:42:07.243+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:42:07.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:42:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:42:07.257+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:42:07.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:42:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:42:07.257+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:42:07.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:42:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:42:07.257+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:42:07.495 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:42:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:42:07.187+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:42:07.495 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:42:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:42:07.187+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:42:07.495 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:42:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:42:07.187+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:42:07.495 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:42:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:42:07.187+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:42:08.493 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:42:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:42:08.154+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 36434 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:42:08.494 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:42:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:42:08.229+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:42:08.494 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:42:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:42:08.229+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:42:08.494 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:42:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:42:08.229+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:42:08.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:42:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:42:08.291+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:42:08.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:42:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:42:08.291+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:42:08.495 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:42:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:42:08.291+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:42:08.495 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:42:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:42:08.217+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:42:08.495 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:42:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:42:08.217+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:42:08.495 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:42:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:42:08.217+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:42:08.495 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:42:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:42:08.217+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:42:09.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:42:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:42:09.218+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:42:09.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:42:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:42:09.218+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:42:09.494 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:42:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:42:09.218+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:42:09.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:42:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:42:09.242+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:42:09.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:42:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:42:09.242+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:42:09.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:42:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:42:09.242+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:42:09.495 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:42:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:42:09.246+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:42:09.495 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:42:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:42:09.246+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:42:09.495 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:42:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:42:09.246+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:42:09.495 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:42:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:42:09.246+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:42:10.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:42:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:42:10.228+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:42:10.494 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:42:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:42:10.228+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:42:10.494 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:42:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:42:10.228+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:42:10.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:42:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:42:10.234+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:42:10.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:42:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:42:10.234+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:42:10.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:42:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:42:10.234+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:42:10.495 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:42:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:42:10.234+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:42:10.495 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:42:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:42:10.280+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:42:10.496 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:42:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:42:10.280+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:42:10.496 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:42:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:42:10.280+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:42:11.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:42:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:42:11.242+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:42:11.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:42:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:42:11.242+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:42:11.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:42:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:42:11.242+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:42:11.494 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:42:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:42:11.194+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:42:11.494 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:42:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:42:11.194+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:42:11.495 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:42:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:42:11.194+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:42:11.495 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:42:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:42:11.186+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:42:11.495 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:42:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:42:11.186+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:42:11.495 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:42:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:42:11.186+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:42:11.495 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:42:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:42:11.186+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:42:12.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:42:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:42:12.155+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:42:12.494 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:42:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:42:12.155+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:42:12.494 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:42:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:42:12.155+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:42:12.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:42:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:42:12.191+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:42:12.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:42:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:42:12.191+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:42:12.495 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:42:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:42:12.191+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:42:12.495 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:42:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:42:12.191+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:42:12.495 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:42:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:42:12.207+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:42:12.495 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:42:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:42:12.207+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:42:12.495 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:42:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:42:12.207+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:42:13.493 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:42:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:42:13.159+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 36507 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:42:13.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:42:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:42:13.215+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:42:13.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:42:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:42:13.215+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:42:13.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:42:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:42:13.215+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:42:13.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:42:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:42:13.215+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:42:13.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:42:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:42:13.257+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:42:13.495 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:42:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:42:13.257+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:42:13.495 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:42:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:42:13.257+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:42:13.495 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:42:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:42:13.188+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:42:13.495 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:42:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:42:13.188+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:42:13.495 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:42:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:42:13.188+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:42:14.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:42:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:42:14.213+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:42:14.494 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:42:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:42:14.213+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:42:14.494 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:42:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:42:14.213+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:42:14.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:42:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:42:14.295+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:42:14.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:42:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:42:14.295+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:42:14.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:42:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:42:14.295+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:42:14.495 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:42:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:42:14.194+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:42:14.495 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:42:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:42:14.194+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:42:14.495 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:42:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:42:14.194+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:42:14.495 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:42:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:42:14.194+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:42:15.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:42:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:42:15.197+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:42:15.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:42:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:42:15.197+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:42:15.494 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:42:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:42:15.197+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:42:15.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:42:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:42:15.166+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:42:15.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:42:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:42:15.166+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:42:15.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:42:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:42:15.166+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:42:15.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:42:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:42:15.166+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:42:15.495 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:42:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:42:15.292+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:42:15.495 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:42:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:42:15.292+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:42:15.495 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:42:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:42:15.292+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:42:16.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:42:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:42:16.155+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:42:16.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:42:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:42:16.155+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:42:16.494 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:42:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:42:16.155+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:42:16.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:42:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:42:16.335+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:42:16.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:42:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:42:16.335+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:42:16.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:42:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:42:16.335+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:42:16.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:42:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:42:16.149+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:42:16.495 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:42:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:42:16.149+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:42:16.495 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:42:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:42:16.149+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:42:16.495 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:42:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:42:16.149+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:42:17.379 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:42:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:42:17.126+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:42:17.379 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:42:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:42:17.126+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:42:17.380 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:42:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:42:17.126+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:42:17.380 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:42:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:42:17.126+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:42:17.380 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:42:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:42:17.149+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:42:17.380 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:42:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:42:17.149+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:42:17.380 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:42:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:42:17.149+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:42:17.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:42:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:42:17.376+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:42:17.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:42:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:42:17.376+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:42:17.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:42:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:42:17.376+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:42:18.356 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:42:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:42:18.165+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 36579 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:42:18.357 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:42:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:42:18.080+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:42:18.357 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:42:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:42:18.080+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:42:18.357 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:42:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:42:18.080+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:42:18.357 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:42:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:42:18.080+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:42:18.357 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:42:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:42:18.129+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:42:18.357 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:42:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:42:18.129+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:42:18.358 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:42:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:42:18.129+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:42:18.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:42:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:42:18.354+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:42:18.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:42:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:42:18.354+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:42:18.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:42:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:42:18.354+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:42:19.334 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:42:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:42:19.075+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:42:19.335 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:42:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:42:19.075+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:42:19.335 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:42:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:42:19.075+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:42:19.335 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:42:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:42:19.075+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:42:19.335 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:42:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:42:19.141+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:42:19.335 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:42:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:42:19.141+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:42:19.335 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:42:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:42:19.141+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:42:19.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:42:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:42:19.332+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:42:19.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:42:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:42:19.332+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:42:19.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:42:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:42:19.332+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:42:20.193 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:42:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:42:20.071+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:42:20.193 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:42:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:42:20.071+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:42:20.193 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:42:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:42:20.071+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:42:20.193 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:42:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:42:20.071+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:42:20.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:42:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:42:20.189+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:42:20.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:42:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:42:20.189+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:42:20.494 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:42:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:42:20.189+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:42:20.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:42:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:42:20.376+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:42:20.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:42:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:42:20.376+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:42:20.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:42:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:42:20.376+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:42:21.405 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:42:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:42:21.053+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:42:21.405 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:42:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:42:21.054+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:42:21.405 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:42:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:42:21.054+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:42:21.405 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:42:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:42:21.054+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:42:21.406 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:42:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:42:21.234+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:42:21.406 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:42:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:42:21.234+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:42:21.406 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:42:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:42:21.234+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:42:21.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:42:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:42:21.401+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:42:21.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:42:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:42:21.401+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:42:21.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:42:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:42:21.401+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:42:22.454 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:42:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:42:22.259+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:42:22.455 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:42:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:42:22.259+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:42:22.455 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:42:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:42:22.259+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:42:22.455 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:42:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:42:22.087+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:42:22.455 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:42:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:42:22.087+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:42:22.455 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:42:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:42:22.087+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:42:22.455 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:42:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:42:22.087+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:42:22.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:42:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:42:22.451+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:42:22.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:42:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:42:22.451+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:42:22.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:42:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:42:22.451+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:42:23.493 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:42:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:42:23.170+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 36652 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:42:23.494 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:42:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:42:23.233+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:42:23.494 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:42:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:42:23.233+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:42:23.494 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:42:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:42:23.233+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:42:23.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:42:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:42:23.127+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:42:23.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:42:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:42:23.127+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:42:23.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:42:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:42:23.127+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:42:23.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:42:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:42:23.127+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:42:23.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:42:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:42:23.499+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:42:23.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:42:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:42:23.499+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:42:23.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:42:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:42:23.499+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:42:24.471 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:42:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:42:24.197+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:42:24.471 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:42:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:42:24.197+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:42:24.472 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:42:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:42:24.197+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:42:24.472 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:42:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:42:24.086+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:42:24.472 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:42:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:42:24.086+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:42:24.472 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:42:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:42:24.086+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:42:24.472 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:42:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:42:24.086+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:42:24.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:42:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:42:24.468+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:42:24.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:42:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:42:24.468+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:42:24.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:42:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:42:24.468+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:42:25.227 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:42:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:42:25.089+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:42:25.228 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:42:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:42:25.089+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:42:25.228 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:42:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:42:25.089+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:42:25.228 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:42:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:42:25.089+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:42:25.480 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:42:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:42:25.224+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:42:25.481 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:42:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:42:25.224+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:42:25.481 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:42:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:42:25.224+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:42:25.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:42:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:42:25.477+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:42:25.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:42:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:42:25.477+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:42:25.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:42:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:42:25.477+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:42:26.474 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:42:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:42:26.183+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:42:26.474 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:42:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:42:26.183+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:42:26.474 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:42:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:42:26.183+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:42:26.475 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:42:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:42:26.083+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:42:26.476 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:42:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:42:26.084+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:42:26.476 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:42:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:42:26.084+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:42:26.476 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:42:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:42:26.084+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:42:26.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:42:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:42:26.470+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:42:26.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:42:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:42:26.470+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:42:26.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:42:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:42:26.470+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:42:27.485 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:42:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:42:27.141+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:42:27.485 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:42:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:42:27.141+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:42:27.485 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:42:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:42:27.141+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:42:27.486 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:42:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:42:27.092+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:42:27.486 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:42:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:42:27.092+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:42:27.486 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:42:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:42:27.092+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:42:27.486 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:42:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:42:27.092+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:42:27.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:42:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:42:27.482+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:42:27.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:42:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:42:27.482+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:42:27.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:42:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:42:27.482+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:42:28.480 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:42:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:42:28.176+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 36724 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:42:28.481 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:42:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:42:28.138+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:42:28.481 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:42:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:42:28.138+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:42:28.481 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:42:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:42:28.138+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:42:28.482 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:42:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:42:28.119+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:42:28.482 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:42:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:42:28.119+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:42:28.482 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:42:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:42:28.119+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:42:28.482 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:42:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:42:28.119+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:42:28.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:42:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:42:28.477+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:42:28.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:42:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:42:28.477+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:42:28.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:42:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:42:28.477+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:42:29.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:42:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:42:29.169+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:42:29.494 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:42:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:42:29.169+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:42:29.494 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:42:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:42:29.169+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:42:29.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:42:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:42:29.158+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:42:29.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:42:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:42:29.158+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:42:29.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:42:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:42:29.158+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:42:29.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:42:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:42:29.158+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:42:29.887 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:42:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:42:29.521+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:42:29.888 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:42:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:42:29.521+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:42:29.888 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:42:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:42:29.521+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:42:30.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:42:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:42:30.213+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:42:30.494 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:42:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:42:30.213+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:42:30.494 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:42:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:42:30.213+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:42:30.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:42:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:42:30.145+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:42:30.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:42:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:42:30.145+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:42:30.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:42:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:42:30.145+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:42:30.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:42:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:42:30.145+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:42:30.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:42:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:42:30.513+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:42:30.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:42:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:42:30.513+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:42:30.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:42:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:42:30.513+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:42:31.479 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:42:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:42:31.210+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:42:31.480 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:42:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:42:31.210+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:42:31.480 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:42:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:42:31.210+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:42:31.480 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:42:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:42:31.144+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:42:31.480 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:42:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:42:31.144+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:42:31.481 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:42:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:42:31.144+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:42:31.481 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:42:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:42:31.144+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:42:31.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:42:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:42:31.475+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:42:31.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:42:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:42:31.475+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:42:31.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:42:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:42:31.475+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:42:32.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:42:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:42:32.251+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:42:32.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:42:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:42:32.251+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:42:32.494 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:42:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:42:32.251+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:42:32.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:42:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:42:32.141+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:42:32.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:42:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:42:32.141+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:42:32.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:42:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:42:32.141+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:42:32.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:42:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:42:32.141+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:42:32.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:42:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:42:32.490+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:42:32.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:42:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:42:32.490+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:42:32.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:42:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:42:32.490+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:42:33.493 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:42:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:42:33.182+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 36793 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:42:33.494 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:42:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:42:33.201+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:42:33.494 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:42:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:42:33.201+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:42:33.494 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:42:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:42:33.201+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:42:33.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:42:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:42:33.119+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:42:33.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:42:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:42:33.119+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:42:33.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:42:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:42:33.119+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:42:33.495 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:42:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:42:33.119+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:42:33.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:42:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:42:33.505+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:42:33.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:42:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:42:33.505+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:42:33.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:42:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:42:33.505+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:42:34.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:42:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:42:34.159+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:42:34.494 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:42:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:42:34.159+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:42:34.494 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:42:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:42:34.159+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:42:34.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:42:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:42:34.163+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:42:34.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:42:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:42:34.163+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:42:34.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:42:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:42:34.163+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:42:34.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:42:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:42:34.163+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:42:34.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:42:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:42:34.511+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:42:34.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:42:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:42:34.511+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:42:34.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:42:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:42:34.511+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:42:35.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:42:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:42:35.147+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:42:35.494 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:42:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:42:35.147+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:42:35.494 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:42:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:42:35.147+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:42:35.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:42:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:42:35.115+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:42:35.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:42:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:42:35.115+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:42:35.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:42:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:42:35.115+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:42:35.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:42:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:42:35.115+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:42:35.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:42:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:42:35.520+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:42:35.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:42:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:42:35.520+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:42:35.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:42:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:42:35.520+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:42:36.495 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:42:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:42:36.171+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:42:36.523 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:42:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:42:36.171+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:42:36.523 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:42:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:42:36.171+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:42:36.524 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:42:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:42:36.068+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:42:36.524 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:42:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:42:36.068+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:42:36.524 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:42:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:42:36.068+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:42:36.524 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:42:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:42:36.068+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:42:36.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:42:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:42:36.490+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:42:36.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:42:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:42:36.490+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:42:36.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:42:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:42:36.490+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:42:37.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:42:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:42:37.199+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:42:37.494 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:42:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:42:37.199+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:42:37.494 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:42:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:42:37.199+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:42:37.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:42:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:42:37.068+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:42:37.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:42:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:42:37.068+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:42:37.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:42:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:42:37.068+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:42:37.495 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:42:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:42:37.068+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:42:37.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:42:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:42:37.507+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:42:37.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:42:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:42:37.507+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:42:37.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:42:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:42:37.507+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:42:38.483 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:42:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:42:38.187+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 36864 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:42:38.483 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:42:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:42:38.151+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:42:38.483 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:42:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:42:38.151+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:42:38.483 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:42:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:42:38.151+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:42:38.484 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:42:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:42:38.116+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:42:38.484 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:42:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:42:38.116+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:42:38.484 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:42:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:42:38.116+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:42:38.484 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:42:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:42:38.116+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:42:38.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:42:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:42:38.480+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:42:38.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:42:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:42:38.480+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:42:38.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:42:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:42:38.480+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:42:39.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:42:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:42:39.190+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:42:39.494 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:42:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:42:39.190+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:42:39.494 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:42:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:42:39.190+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:42:39.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:42:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:42:39.077+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:42:39.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:42:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:42:39.077+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:42:39.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:42:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:42:39.077+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:42:39.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:42:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:42:39.077+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:42:39.886 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:42:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:42:39.527+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:42:39.887 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:42:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:42:39.527+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:42:39.887 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:42:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:42:39.527+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:42:40.147 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:42:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:42:40.071+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:42:40.147 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:42:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:42:40.071+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:42:40.147 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:42:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:42:40.071+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:42:40.148 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:42:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:42:40.071+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:42:40.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:42:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:42:40.143+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:42:40.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:42:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:42:40.143+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:42:40.494 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:42:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:42:40.143+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:42:40.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:42:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:42:40.512+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:42:40.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:42:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:42:40.512+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:42:40.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:42:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:42:40.512+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:42:41.471 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:42:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:42:41.181+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:42:41.472 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:42:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:42:41.181+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:42:41.472 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:42:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:42:41.181+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:42:41.472 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:42:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:42:41.096+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:42:41.472 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:42:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:42:41.096+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:42:41.472 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:42:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:42:41.096+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:42:41.472 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:42:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:42:41.096+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:42:41.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:42:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:42:41.468+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:42:41.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:42:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:42:41.468+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:42:41.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:42:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:42:41.468+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:42:42.203 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:42:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:42:42.064+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:42:42.203 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:42:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:42:42.064+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:42:42.203 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:42:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:42:42.064+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:42:42.203 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:42:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:42:42.065+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:42:42.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:42:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:42:42.200+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:42:42.494 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:42:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:42:42.200+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:42:42.494 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:42:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:42:42.200+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:42:42.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:42:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:42:42.434+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:42:42.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:42:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:42:42.434+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:42:42.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:42:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:42:42.434+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:42:43.168 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:42:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:42:43.077+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:42:43.169 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:42:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:42:43.077+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:42:43.169 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:42:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:42:43.077+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:42:43.169 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:42:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:42:43.077+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:42:43.493 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:42:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:42:43.191+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 36936 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:42:43.494 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:42:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:42:43.165+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:42:43.494 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:42:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:42:43.165+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:42:43.494 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:42:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:42:43.165+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:42:43.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:42:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:42:43.385+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:42:43.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:42:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:42:43.385+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:42:43.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:42:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:42:43.385+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:42:44.359 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:42:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:42:44.150+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:42:44.359 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:42:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:42:44.150+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:42:44.359 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:42:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:42:44.150+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:42:44.359 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:42:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:42:44.045+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:42:44.360 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:42:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:42:44.045+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:42:44.360 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:42:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:42:44.045+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:42:44.360 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:42:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:42:44.045+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:42:44.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:42:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:42:44.356+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:42:44.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:42:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:42:44.356+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:42:44.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:42:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:42:44.356+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:42:45.355 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:42:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:42:45.132+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:42:45.355 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:42:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:42:45.132+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:42:45.355 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:42:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:42:45.132+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:42:45.355 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:42:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:42:45.052+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:42:45.355 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:42:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:42:45.052+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:42:45.356 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:42:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:42:45.052+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:42:45.356 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:42:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:42:45.052+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:42:45.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:42:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:42:45.351+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:42:45.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:42:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:42:45.351+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:42:45.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:42:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:42:45.351+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:42:46.354 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:42:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:42:46.105+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:42:46.355 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:42:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:42:46.105+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:42:46.355 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:42:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:42:46.105+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:42:46.355 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:42:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:42:46.004+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:42:46.355 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:42:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:42:46.004+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:42:46.355 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:42:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:42:46.004+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:42:46.356 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:42:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:42:46.004+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:42:46.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:42:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:42:46.350+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:42:46.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:42:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:42:46.350+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:42:46.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:42:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:42:46.350+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:42:47.321 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:42:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:42:47.115+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:42:47.321 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:42:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:42:47.115+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:42:47.322 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:42:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:42:47.115+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:42:47.322 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:42:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:42:47.019+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:42:47.322 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:42:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:42:47.019+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:42:47.322 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:42:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:42:47.019+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:42:47.322 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:42:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:42:47.019+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:42:47.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:42:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:42:47.317+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:42:47.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:42:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:42:47.317+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:42:47.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:42:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:42:47.317+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:42:48.293 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:42:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:42:48.196+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 37007 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:42:48.293 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:42:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:42:48.085+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:42:48.293 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:42:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:42:48.085+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:42:48.293 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:42:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:42:48.085+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:42:48.294 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:42:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:42:48.039+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:42:48.294 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:42:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:42:48.039+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:42:48.294 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:42:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:42:48.039+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:42:48.294 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:42:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:42:48.039+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:42:48.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:42:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:42:48.290+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:42:48.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:42:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:42:48.290+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:42:48.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:42:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:42:48.290+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:42:49.334 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:42:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:42:49.104+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:42:49.335 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:42:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:42:49.104+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:42:49.335 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:42:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:42:49.104+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:42:49.335 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:42:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:42:49.007+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:42:49.335 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:42:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:42:49.007+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:42:49.335 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:42:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:42:49.007+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:42:49.335 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:42:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:42:49.008+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:42:49.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:42:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:42:49.332+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:42:49.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:42:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:42:49.332+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:42:49.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:42:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:42:49.332+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:42:50.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:42:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:42:50.090+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:42:50.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:42:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:42:50.090+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:42:50.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:42:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:42:50.090+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:42:50.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:42:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:42:49.989+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:42:50.245 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:42:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:42:49.989+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:42:50.245 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:42:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:42:49.989+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:42:50.245 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:42:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:42:49.989+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:42:50.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:42:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:42:50.354+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:42:50.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:42:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:42:50.354+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:42:50.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:42:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:42:50.354+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:42:51.360 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:42:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:42:51.039+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:42:51.360 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:42:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:42:51.039+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:42:51.360 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:42:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:42:51.039+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:42:51.360 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:42:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:42:51.039+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:42:51.361 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:42:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:42:51.115+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:42:51.361 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:42:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:42:51.115+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:42:51.361 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:42:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:42:51.115+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:42:51.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:42:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:42:51.357+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:42:51.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:42:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:42:51.357+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:42:51.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:42:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:42:51.357+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:42:52.143 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:42:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:42:52.066+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:42:52.143 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:42:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:42:52.066+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:42:52.143 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:42:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:42:52.066+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:42:52.144 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:42:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:42:52.066+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:42:52.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:42:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:42:52.341+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:42:52.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:42:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:42:52.341+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:42:52.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:42:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:42:52.341+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:42:52.494 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:42:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:42:52.140+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:42:52.494 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:42:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:42:52.140+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:42:52.494 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:42:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:42:52.140+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:42:53.472 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:42:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:42:53.203+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 37079 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:42:53.473 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:42:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:42:53.301+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:42:53.473 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:42:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:42:53.301+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:42:53.473 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:42:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:42:53.301+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:42:53.473 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:42:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:42:53.137+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:42:53.473 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:42:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:42:53.137+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:42:53.473 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:42:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:42:53.137+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:42:53.474 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:42:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:42:53.093+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:42:53.474 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:42:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:42:53.093+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:42:53.474 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:42:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:42:53.093+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:42:53.474 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:42:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:42:53.093+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:42:54.338 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:42:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:42:54.116+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:42:54.339 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:42:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:42:54.116+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:42:54.339 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:42:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:42:54.116+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:42:54.339 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:42:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:42:54.078+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:42:54.339 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:42:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:42:54.078+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:42:54.339 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:42:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:42:54.078+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:42:54.339 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:42:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:42:54.078+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:42:54.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:42:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:42:54.335+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:42:54.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:42:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:42:54.335+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:42:54.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:42:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:42:54.335+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:42:55.353 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:42:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:42:55.157+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:42:55.353 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:42:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:42:55.157+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:42:55.354 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:42:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:42:55.157+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:42:55.354 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:42:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:42:55.067+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:42:55.354 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:42:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:42:55.067+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:42:55.354 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:42:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:42:55.067+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:42:55.354 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:42:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:42:55.067+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:42:55.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:42:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:42:55.351+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:42:55.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:42:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:42:55.351+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:42:55.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:42:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:42:55.351+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:42:56.317 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:42:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:42:56.115+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:42:56.317 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:42:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:42:56.115+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:42:56.317 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:42:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:42:56.115+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:42:56.317 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:42:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:42:56.042+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:42:56.318 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:42:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:42:56.042+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:42:56.318 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:42:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:42:56.042+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:42:56.318 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:42:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:42:56.042+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:42:56.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:42:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:42:56.314+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:42:56.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:42:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:42:56.314+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:42:56.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:42:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:42:56.314+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:42:57.351 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:42:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:42:57.096+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:42:57.351 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:42:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:42:57.096+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:42:57.351 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:42:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:42:57.096+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:42:57.351 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:42:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:42:57.002+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:42:57.351 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:42:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:42:57.002+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:42:57.352 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:42:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:42:57.002+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:42:57.352 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:42:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:42:57.002+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:42:57.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:42:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:42:57.344+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:42:57.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:42:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:42:57.344+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:42:57.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:42:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:42:57.344+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:42:58.243 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:42:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:42:58.208+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 37147 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:42:58.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:42:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:42:58.133+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:42:58.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:42:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:42:58.133+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:42:58.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:42:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:42:58.133+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:42:58.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:42:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:42:57.989+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:42:58.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:42:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:42:57.989+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:42:58.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:42:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:42:57.989+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:42:58.245 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:42:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:42:57.989+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:42:58.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:42:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:42:58.296+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:42:58.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:42:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:42:58.296+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:42:58.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:42:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:42:58.296+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:42:59.313 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:42:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:42:59.162+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:42:59.313 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:42:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:42:59.163+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:42:59.314 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:42:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:42:59.163+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:42:59.314 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:42:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:42:59.015+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:42:59.314 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:42:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:42:59.015+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:42:59.314 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:42:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:42:59.015+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:42:59.314 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:42:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:42:59.016+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:42:59.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:42:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:42:59.310+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:42:59.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:42:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:42:59.310+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:42:59.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:42:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:42:59.310+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:43:00.138 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:43:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:43:00.025+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:43:00.139 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:43:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:43:00.025+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:43:00.139 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:43:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:43:00.025+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:43:00.139 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:43:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:43:00.025+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:43:00.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:43:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:43:00.136+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:43:00.494 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:43:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:43:00.136+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:43:00.494 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:43:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:43:00.136+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:43:00.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:43:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:43:00.297+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:43:00.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:43:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:43:00.297+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:43:00.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:43:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:43:00.297+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:43:01.276 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:43:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:43:01.010+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:43:01.277 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:43:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:43:01.010+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:43:01.277 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:43:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:43:01.010+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:43:01.277 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:43:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:43:01.010+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:43:01.277 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:43:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:43:01.101+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:43:01.278 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:43:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:43:01.101+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:43:01.278 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:43:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:43:01.101+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:43:01.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:43:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:43:01.274+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:43:01.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:43:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:43:01.274+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:43:01.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:43:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:43:01.274+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:43:02.302 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:43:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:43:02.144+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:43:02.302 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:43:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:43:02.144+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:43:02.302 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:43:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:43:02.144+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:43:02.302 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:43:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:43:02.029+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:43:02.302 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:43:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:43:02.029+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:43:02.303 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:43:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:43:02.029+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:43:02.303 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:43:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:43:02.030+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:43:02.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:43:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:43:02.299+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:43:02.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:43:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:43:02.299+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:43:02.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:43:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:43:02.299+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:43:03.339 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:43:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:43:03.213+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 37219 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:43:03.340 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:43:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:43:03.136+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:43:03.340 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:43:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:43:03.136+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:43:03.340 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:43:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:43:03.136+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:43:03.341 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:43:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:43:03.249+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:43:03.341 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:43:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:43:03.249+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:43:03.341 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:43:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:43:03.249+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:43:03.341 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:43:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:43:03.048+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:43:03.341 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:43:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:43:03.048+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:43:03.341 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:43:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:43:03.048+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:43:03.342 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:43:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:43:03.048+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:43:04.294 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:43:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:43:04.120+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:43:04.295 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:43:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:43:04.120+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:43:04.295 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:43:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:43:04.120+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:43:04.295 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:43:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:43:04.033+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:43:04.295 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:43:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:43:04.033+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:43:04.295 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:43:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:43:04.033+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:43:04.295 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:43:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:43:04.033+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:43:04.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:43:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:43:04.292+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:43:04.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:43:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:43:04.292+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:43:04.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:43:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:43:04.292+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:43:05.104 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:43:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:43:04.999+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:43:05.104 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:43:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:43:04.999+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:43:05.105 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:43:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:43:04.999+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:43:05.105 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:43:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:43:04.999+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:43:05.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:43:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:43:05.102+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:43:05.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:43:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:43:05.102+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:43:05.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:43:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:43:05.102+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:43:05.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:43:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:43:05.278+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:43:05.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:43:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:43:05.278+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:43:05.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:43:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:43:05.278+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:43:06.284 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:43:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:43:06.070+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:43:06.285 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:43:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:43:06.070+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:43:06.285 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:43:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:43:06.070+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:43:06.285 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:43:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:43:06.024+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:43:06.285 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:43:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:43:06.024+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:43:06.285 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:43:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:43:06.024+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:43:06.286 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:43:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:43:06.024+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:43:06.556 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:43:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:43:06.282+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:43:06.556 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:43:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:43:06.282+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:43:06.556 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:43:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:43:06.282+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:43:07.317 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:43:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:43:07.025+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:43:07.317 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:43:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:43:07.025+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:43:07.318 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:43:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:43:07.025+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:43:07.318 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:43:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:43:07.025+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:43:07.318 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:43:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:43:07.021+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:43:07.318 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:43:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:43:07.021+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:43:07.318 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:43:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:43:07.021+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:43:07.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:43:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:43:07.315+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:43:07.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:43:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:43:07.315+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:43:07.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:43:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:43:07.315+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:43:08.290 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:43:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:43:08.219+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 37291 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:43:08.290 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:43:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:43:08.003+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:43:08.290 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:43:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:43:08.003+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:43:08.290 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:43:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:43:08.003+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:43:08.291 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:43:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:43:07.998+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:43:08.291 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:43:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:43:07.998+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:43:08.291 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:43:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:43:07.998+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:43:08.291 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:43:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:43:07.998+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:43:08.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:43:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:43:08.287+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:43:08.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:43:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:43:08.287+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:43:08.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:43:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:43:08.287+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:43:09.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:43:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:43:08.967+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:43:09.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:43:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:43:08.967+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:43:09.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:43:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:43:08.967+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:43:09.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:43:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:43:08.949+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:43:09.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:43:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:43:08.949+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:43:09.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:43:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:43:08.949+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:43:09.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:43:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:43:08.949+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:43:09.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:43:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:43:09.257+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:43:09.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:43:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:43:09.257+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:43:09.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:43:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:43:09.257+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:43:10.217 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:43:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:43:09.920+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:43:10.218 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:43:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:43:09.920+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:43:10.218 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:43:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:43:09.920+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:43:10.218 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:43:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:43:09.949+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:43:10.218 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:43:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:43:09.949+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:43:10.218 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:43:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:43:09.949+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:43:10.219 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:43:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:43:09.949+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:43:10.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:43:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:43:10.215+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:43:10.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:43:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:43:10.215+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:43:10.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:43:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:43:10.215+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:43:11.192 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:43:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:43:10.894+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:43:11.192 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:43:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:43:10.894+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:43:11.192 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:43:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:43:10.894+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:43:11.192 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:43:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:43:10.997+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:43:11.192 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:43:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:43:10.997+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:43:11.193 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:43:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:43:10.997+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:43:11.193 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:43:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:43:10.997+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:43:11.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:43:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:43:11.189+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:43:11.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:43:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:43:11.189+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:43:11.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:43:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:43:11.189+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:43:12.194 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:43:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:43:11.938+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:43:12.194 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:43:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:43:11.938+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:43:12.194 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:43:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:43:11.938+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:43:12.194 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:43:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:43:12.045+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:43:12.195 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:43:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:43:12.045+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:43:12.195 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:43:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:43:12.045+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:43:12.195 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:43:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:43:12.045+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:43:12.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:43:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:43:12.191+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:43:12.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:43:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:43:12.191+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:43:12.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:43:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:43:12.191+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:43:13.227 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:43:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:43:13.090+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:43:13.227 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:43:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:43:13.090+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:43:13.228 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:43:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:43:13.090+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:43:13.228 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:43:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:43:13.090+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:43:13.228 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:43:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:43:12.951+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:43:13.228 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:43:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:43:12.951+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:43:13.228 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:43:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:43:12.951+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:43:13.493 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:43:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:43:13.224+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 37363 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:43:13.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:43:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:43:13.235+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:43:13.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:43:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:43:13.235+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:43:13.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:43:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:43:13.235+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:43:14.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:43:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:43:14.051+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:43:14.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:43:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:43:14.052+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:43:14.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:43:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:43:14.052+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:43:14.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:43:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:43:14.052+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:43:14.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:43:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:43:13.906+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:43:14.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:43:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:43:13.906+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:43:14.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:43:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:43:13.906+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:43:14.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:43:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:43:14.263+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:43:14.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:43:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:43:14.263+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:43:14.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:43:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:43:14.263+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:43:15.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:43:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:43:14.899+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:43:15.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:43:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:43:14.899+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:43:15.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:43:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:43:14.899+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:43:15.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:43:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:43:15.087+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:43:15.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:43:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:43:15.087+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:43:15.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:43:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:43:15.087+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:43:15.245 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:43:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:43:15.088+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:43:15.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:43:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:43:15.302+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:43:15.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:43:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:43:15.302+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:43:15.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:43:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:43:15.302+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:43:16.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:43:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:43:15.907+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:43:16.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:43:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:43:15.907+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:43:16.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:43:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:43:15.907+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:43:16.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:43:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:43:16.044+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:43:16.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:43:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:43:16.044+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:43:16.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:43:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:43:16.044+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:43:16.245 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:43:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:43:16.044+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:43:16.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:43:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:43:16.274+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:43:16.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:43:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:43:16.274+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:43:16.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:43:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:43:16.274+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:43:17.241 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:43:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:43:17.044+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:43:17.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:43:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:43:17.044+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:43:17.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:43:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:43:17.044+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:43:17.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:43:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:43:17.044+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:43:17.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:43:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:43:16.866+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:43:17.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:43:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:43:16.866+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:43:17.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:43:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:43:16.866+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:43:17.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:43:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:43:17.238+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:43:17.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:43:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:43:17.238+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:43:17.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:43:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:43:17.238+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:43:18.204 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:43:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:43:17.900+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:43:18.205 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:43:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:43:17.900+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:43:18.205 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:43:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:43:17.900+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:43:18.205 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:43:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:43:18.089+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:43:18.206 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:43:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:43:18.089+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:43:18.206 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:43:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:43:18.089+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:43:18.206 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:43:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:43:18.089+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:43:18.493 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:43:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:43:18.229+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 37435 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:43:18.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:43:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:43:18.201+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:43:18.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:43:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:43:18.201+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:43:18.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:43:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:43:18.201+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:43:19.176 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:43:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:43:19.062+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:43:19.176 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:43:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:43:19.062+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:43:19.176 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:43:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:43:19.062+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:43:19.176 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:43:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:43:19.062+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:43:19.177 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:43:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:43:18.866+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:43:19.177 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:43:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:43:18.866+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:43:19.177 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:43:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:43:18.866+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:43:19.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:43:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:43:19.172+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:43:19.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:43:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:43:19.172+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:43:19.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:43:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:43:19.172+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:43:20.182 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:43:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:43:20.029+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:43:20.183 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:43:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:43:20.029+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:43:20.183 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:43:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:43:20.029+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:43:20.183 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:43:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:43:20.029+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:43:20.183 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:43:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:43:19.834+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:43:20.183 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:43:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:43:19.834+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:43:20.183 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:43:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:43:19.834+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:43:20.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:43:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:43:20.179+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:43:20.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:43:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:43:20.179+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:43:20.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:43:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:43:20.179+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:43:21.204 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:43:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:43:20.993+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:43:21.204 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:43:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:43:20.993+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:43:21.204 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:43:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:43:20.993+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:43:21.204 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:43:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:43:20.993+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:43:21.205 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:43:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:43:20.792+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:43:21.205 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:43:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:43:20.792+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:43:21.205 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:43:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:43:20.792+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:43:21.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:43:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:43:21.200+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:43:21.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:43:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:43:21.200+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:43:21.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:43:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:43:21.200+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:43:22.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:43:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:43:21.990+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:43:22.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:43:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:43:21.990+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:43:22.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:43:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:43:21.990+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:43:22.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:43:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:43:21.990+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:43:22.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:43:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:43:21.765+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:43:22.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:43:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:43:21.765+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:43:22.245 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:43:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:43:21.765+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:43:22.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:43:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:43:22.247+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:43:22.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:43:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:43:22.247+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:43:22.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:43:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:43:22.247+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:43:23.238 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:43:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:43:22.808+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:43:23.238 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:43:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:43:22.808+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:43:23.238 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:43:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:43:22.808+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:43:23.239 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:43:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:43:22.973+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:43:23.239 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:43:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:43:22.973+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:43:23.239 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:43:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:43:22.973+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:43:23.239 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:43:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:43:22.973+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:43:23.493 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:43:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:43:23.234+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 37507 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:43:23.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:43:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:43:23.254+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:43:23.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:43:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:43:23.254+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:43:23.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:43:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:43:23.254+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:43:24.213 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:43:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:43:23.811+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:43:24.213 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:43:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:43:23.811+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:43:24.213 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:43:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:43:23.811+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:43:24.213 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:43:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:43:23.934+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:43:24.213 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:43:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:43:23.934+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:43:24.214 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:43:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:43:23.934+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:43:24.214 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:43:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:43:23.934+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:43:24.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:43:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:43:24.210+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:43:24.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:43:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:43:24.210+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:43:24.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:43:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:43:24.210+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:43:25.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:43:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:43:24.767+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:43:25.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:43:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:43:24.767+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:43:25.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:43:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:43:24.767+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:43:25.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:43:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:43:24.896+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:43:25.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:43:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:43:24.896+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:43:25.245 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:43:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:43:24.896+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:43:25.245 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:43:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:43:24.896+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:43:25.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:43:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:43:25.241+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:43:25.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:43:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:43:25.241+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:43:25.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:43:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:43:25.241+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:43:26.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:43:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:43:25.796+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:43:26.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:43:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:43:25.796+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:43:26.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:43:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:43:25.796+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:43:26.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:43:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:43:25.888+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:43:26.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:43:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:43:25.888+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:43:26.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:43:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:43:25.888+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:43:26.245 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:43:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:43:25.888+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:43:26.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:43:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:43:26.275+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:43:26.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:43:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:43:26.275+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:43:26.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:43:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:43:26.275+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:43:27.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:43:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:43:26.839+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:43:27.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:43:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:43:26.839+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:43:27.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:43:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:43:26.839+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:43:27.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:43:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:43:26.929+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:43:27.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:43:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:43:26.929+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:43:27.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:43:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:43:26.929+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:43:27.245 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:43:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:43:26.929+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:43:27.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:43:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:43:27.248+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:43:27.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:43:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:43:27.248+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:43:27.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:43:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:43:27.248+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:43:28.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:43:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:43:27.856+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:43:28.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:43:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:43:27.856+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:43:28.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:43:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:43:27.856+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:43:28.245 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:43:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:43:27.914+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:43:28.245 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:43:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:43:27.914+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:43:28.245 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:43:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:43:27.914+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:43:28.245 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:43:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:43:27.914+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:43:28.743 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:43:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:43:28.240+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 37579 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:43:28.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:43:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:43:28.255+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:43:28.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:43:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:43:28.255+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:43:28.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:43:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:43:28.255+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:43:29.233 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:43:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:43:28.833+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:43:29.234 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:43:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:43:28.833+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:43:29.234 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:43:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:43:28.833+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:43:29.234 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:43:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:43:28.908+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:43:29.235 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:43:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:43:28.908+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:43:29.235 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:43:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:43:28.908+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:43:29.235 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:43:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:43:28.908+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:43:29.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:43:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:43:29.230+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:43:29.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:43:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:43:29.230+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:43:29.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:43:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:43:29.230+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:43:30.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:43:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:43:29.785+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:43:30.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:43:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:43:29.785+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:43:30.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:43:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:43:29.785+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:43:30.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:43:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:43:29.911+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:43:30.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:43:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:43:29.911+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:43:30.245 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:43:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:43:29.911+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:43:30.245 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:43:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:43:29.911+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:43:30.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:43:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:43:30.267+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:43:30.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:43:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:43:30.267+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:43:30.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:43:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:43:30.267+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:43:31.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:43:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:43:30.794+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:43:31.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:43:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:43:30.794+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:43:31.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:43:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:43:30.794+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:43:31.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:43:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:43:30.953+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:43:31.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:43:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:43:30.953+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:43:31.245 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:43:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:43:30.953+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:43:31.245 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:43:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:43:30.953+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:43:31.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:43:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:43:31.296+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:43:31.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:43:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:43:31.296+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:43:31.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:43:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:43:31.296+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:43:32.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:43:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:43:31.821+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:43:32.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:43:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:43:31.821+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:43:32.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:43:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:43:31.821+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:43:32.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:43:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:43:31.938+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:43:32.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:43:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:43:31.938+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:43:32.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:43:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:43:31.938+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:43:32.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:43:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:43:31.938+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:43:32.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:43:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:43:32.271+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:43:32.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:43:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:43:32.271+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:43:32.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:43:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:43:32.271+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:43:33.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:43:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:43:32.780+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:43:33.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:43:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:43:32.780+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:43:33.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:43:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:43:32.780+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:43:33.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:43:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:43:32.891+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:43:33.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:43:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:43:32.891+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:43:33.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:43:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:43:32.891+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:43:33.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:43:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:43:32.891+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:43:33.743 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:43:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:43:33.245+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 37651 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:43:33.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:43:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:43:33.293+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:43:33.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:43:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:43:33.293+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:43:33.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:43:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:43:33.293+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:43:34.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:43:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:43:33.829+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:43:34.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:43:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:43:33.829+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:43:34.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:43:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:43:33.829+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:43:34.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:43:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:43:33.857+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:43:34.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:43:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:43:33.857+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:43:34.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:43:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:43:33.857+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:43:34.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:43:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:43:33.857+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:43:34.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:43:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:43:34.305+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:43:34.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:43:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:43:34.305+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:43:34.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:43:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:43:34.305+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:43:35.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:43:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:43:34.812+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:43:35.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:43:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:43:34.812+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:43:35.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:43:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:43:34.812+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:43:35.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:43:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:43:34.834+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:43:35.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:43:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:43:34.834+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:43:35.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:43:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:43:34.834+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:43:35.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:43:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:43:34.834+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:43:35.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:43:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:43:35.320+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:43:35.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:43:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:43:35.320+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:43:35.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:43:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:43:35.320+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:43:36.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:43:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:43:35.809+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:43:36.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:43:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:43:35.809+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:43:36.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:43:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:43:35.809+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:43:36.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:43:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:43:35.833+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:43:36.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:43:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:43:35.833+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:43:36.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:43:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:43:35.833+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:43:36.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:43:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:43:35.833+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:43:36.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:43:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:43:36.310+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:43:36.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:43:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:43:36.310+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:43:36.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:43:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:43:36.310+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:43:37.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:43:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:43:36.794+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:43:37.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:43:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:43:36.794+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:43:37.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:43:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:43:36.794+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:43:37.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:43:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:43:36.826+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:43:37.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:43:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:43:36.826+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:43:37.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:43:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:43:36.826+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:43:37.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:43:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:43:36.826+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:43:37.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:43:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:43:37.286+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:43:37.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:43:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:43:37.286+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:43:37.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:43:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:43:37.286+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:43:38.245 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:43:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:43:37.765+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:43:38.253 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:43:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:43:37.765+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:43:38.253 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:43:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:43:37.765+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:43:38.253 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:43:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:43:37.792+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:43:38.254 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:43:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:43:37.792+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:43:38.254 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:43:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:43:37.792+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:43:38.254 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:43:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:43:37.792+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:43:38.726 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:43:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:43:38.251+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 37723 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:43:38.726 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:43:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:43:38.247+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:43:38.727 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:43:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:43:38.247+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:43:38.727 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:43:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:43:38.247+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:43:38.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:43:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:43:38.722+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:43:38.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:43:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:43:38.722+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:43:38.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:43:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:43:38.722+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:43:38.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:43:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:43:38.803+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:43:38.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:43:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:43:38.803+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:43:38.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:43:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:43:38.803+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:43:38.995 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:43:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:43:38.803+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:43:39.677 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:43:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:43:39.265+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:43:39.677 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:43:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:43:39.265+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:43:39.677 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:43:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:43:39.265+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:43:39.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:43:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:43:39.674+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:43:39.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:43:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:43:39.674+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:43:39.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:43:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:43:39.674+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:43:39.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:43:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:43:39.826+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:43:39.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:43:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:43:39.826+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:43:39.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:43:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:43:39.826+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:43:39.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:43:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:43:39.826+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:43:40.639 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:43:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:43:40.263+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:43:40.639 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:43:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:43:40.263+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:43:40.639 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:43:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:43:40.263+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:43:40.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:43:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:43:40.636+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:43:40.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:43:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:43:40.636+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:43:40.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:43:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:43:40.636+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:43:40.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:43:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:43:40.787+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:43:40.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:43:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:43:40.787+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:43:40.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:43:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:43:40.787+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:43:40.995 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:43:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:43:40.787+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:43:41.628 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:43:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:43:41.276+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:43:41.629 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:43:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:43:41.276+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:43:41.629 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:43:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:43:41.276+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:43:41.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:43:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:43:41.625+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:43:41.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:43:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:43:41.625+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:43:41.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:43:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:43:41.625+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:43:41.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:43:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:43:41.825+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:43:41.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:43:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:43:41.825+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:43:41.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:43:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:43:41.825+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:43:41.995 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:43:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:43:41.826+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:43:42.669 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:43:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:43:42.306+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:43:42.669 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:43:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:43:42.306+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:43:42.669 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:43:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:43:42.306+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:43:42.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:43:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:43:42.786+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:43:42.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:43:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:43:42.786+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:43:42.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:43:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:43:42.786+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:43:42.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:43:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:43:42.786+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:43:42.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:43:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:43:42.666+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:43:42.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:43:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:43:42.666+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:43:42.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:43:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:43:42.666+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:43:43.648 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:43:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:43:43.257+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 37795 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:43:43.649 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:43:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:43:43.338+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:43:43.649 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:43:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:43:43.338+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:43:43.649 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:43:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:43:43.338+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:43:43.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:43:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:43:43.645+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:43:43.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:43:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:43:43.645+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:43:43.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:43:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:43:43.645+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:43:43.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:43:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:43:43.787+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:43:43.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:43:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:43:43.787+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:43:43.995 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:43:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:43:43.787+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:43:43.995 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:43:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:43:43.788+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:43:44.681 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:43:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:43:44.299+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:43:44.681 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:43:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:43:44.299+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:43:44.681 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:43:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:43:44.299+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:43:44.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:43:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:43:44.678+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:43:44.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:43:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:43:44.678+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:43:44.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:43:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:43:44.678+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:43:44.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:43:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:43:44.776+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:43:44.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:43:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:43:44.776+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:43:44.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:43:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:43:44.776+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:43:44.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:43:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:43:44.776+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:43:45.719 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:43:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:43:45.252+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:43:45.719 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:43:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:43:45.252+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:43:45.719 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:43:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:43:45.252+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:43:45.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:43:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:43:45.716+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:43:45.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:43:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:43:45.716+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:43:45.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:43:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:43:45.716+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:43:45.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:43:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:43:45.808+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:43:45.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:43:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:43:45.808+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:43:45.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:43:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:43:45.808+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:43:45.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:43:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:43:45.808+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:43:46.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:43:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:43:46.232+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:43:46.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:43:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:43:46.232+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:43:46.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:43:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:43:46.232+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:43:46.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:43:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:43:46.676+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:43:46.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:43:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:43:46.676+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:43:46.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:43:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:43:46.676+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:43:46.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:43:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:43:46.818+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:43:46.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:43:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:43:46.818+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:43:46.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:43:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:43:46.818+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:43:46.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:43:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:43:46.818+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:43:47.700 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:43:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:43:47.280+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:43:47.700 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:43:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:43:47.280+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:43:47.700 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:43:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:43:47.280+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:43:47.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:43:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:43:47.697+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:43:47.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:43:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:43:47.697+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:43:47.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:43:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:43:47.697+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:43:47.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:43:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:43:47.843+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:43:47.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:43:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:43:47.843+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:43:47.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:43:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:43:47.843+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:43:47.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:43:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:43:47.843+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:43:48.706 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:43:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:43:48.264+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 37867 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:43:48.706 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:43:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:43:48.302+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:43:48.706 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:43:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:43:48.302+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:43:48.706 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:43:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:43:48.302+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:43:48.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:43:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:43:48.703+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:43:48.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:43:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:43:48.703+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:43:48.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:43:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:43:48.703+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:43:48.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:43:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:43:48.845+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:43:48.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:43:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:43:48.845+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:43:48.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:43:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:43:48.845+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:43:48.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:43:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:43:48.845+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:43:49.701 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:43:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:43:49.348+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:43:49.701 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:43:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:43:49.348+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:43:49.701 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:43:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:43:49.348+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:43:49.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:43:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:43:49.698+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:43:49.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:43:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:43:49.698+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:43:49.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:43:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:43:49.698+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:43:49.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:43:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:43:49.861+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:43:49.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:43:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:43:49.861+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:43:49.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:43:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:43:49.861+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:43:49.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:43:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:43:49.861+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:43:50.722 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:43:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:43:50.348+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:43:50.722 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:43:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:43:50.348+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:43:50.722 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:43:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:43:50.348+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:43:50.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:43:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:43:50.719+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:43:50.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:43:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:43:50.719+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:43:50.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:43:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:43:50.719+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:43:50.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:43:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:43:50.833+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:43:50.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:43:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:43:50.833+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:43:50.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:43:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:43:50.833+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:43:50.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:43:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:43:50.833+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:43:51.673 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:43:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:43:51.398+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:43:51.673 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:43:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:43:51.398+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:43:51.673 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:43:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:43:51.398+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:43:51.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:43:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:43:51.806+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:43:51.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:43:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:43:51.806+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:43:51.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:43:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:43:51.806+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:43:51.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:43:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:43:51.806+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:43:51.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:43:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:43:51.670+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:43:51.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:43:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:43:51.670+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:43:51.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:43:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:43:51.670+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:43:52.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:43:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:43:52.436+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:43:52.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:43:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:43:52.436+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:43:52.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:43:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:43:52.436+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:43:52.744 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:43:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:43:52.666+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:43:52.744 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:43:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:43:52.666+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:43:52.744 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:43:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:43:52.666+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:43:53.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:43:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:43:52.819+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:43:53.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:43:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:43:52.819+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:43:53.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:43:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:43:52.819+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:43:53.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:43:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:43:52.819+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:43:53.690 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:43:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:43:53.270+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 37939 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:43:53.691 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:43:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:43:53.392+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:43:53.691 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:43:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:43:53.392+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:43:53.691 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:43:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:43:53.392+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:43:53.978 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:43:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:43:53.687+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:43:53.978 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:43:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:43:53.687+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:43:53.978 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:43:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:43:53.687+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:43:53.978 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:43:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:43:53.811+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:43:53.978 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:43:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:43:53.811+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:43:53.979 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:43:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:43:53.811+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:43:53.979 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:43:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:43:53.811+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:43:54.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:43:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:43:54.650+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:43:54.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:43:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:43:54.650+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:43:54.744 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:43:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:43:54.650+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:43:54.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:43:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:43:54.439+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:43:54.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:43:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:43:54.439+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:43:54.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:43:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:43:54.439+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:43:55.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:43:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:43:54.776+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:43:55.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:43:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:43:54.776+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:43:55.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:43:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:43:54.776+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:43:55.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:43:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:43:54.776+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:43:55.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:43:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:43:55.661+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:43:55.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:43:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:43:55.661+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:43:55.744 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:43:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:43:55.661+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:43:55.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:43:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:43:55.427+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:43:55.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:43:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:43:55.427+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:43:55.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:43:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:43:55.427+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:43:56.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:43:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:43:55.775+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:43:56.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:43:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:43:55.775+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:43:56.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:43:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:43:55.775+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:43:56.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:43:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:43:55.775+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:43:56.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:43:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:43:56.707+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:43:56.744 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:43:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:43:56.707+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:43:56.744 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:43:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:43:56.707+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:43:56.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:43:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:43:56.470+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:43:56.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:43:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:43:56.470+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:43:56.745 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:43:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:43:56.470+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:43:57.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:43:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:43:56.754+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:43:57.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:43:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:43:56.754+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:43:57.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:43:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:43:56.754+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:43:57.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:43:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:43:56.754+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:43:57.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:43:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:43:57.701+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:43:57.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:43:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:43:57.701+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:43:57.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:43:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:43:57.701+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:43:57.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:43:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:43:57.457+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:43:57.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:43:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:43:57.457+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:43:57.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:43:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:43:57.457+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:43:58.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:43:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:43:57.798+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:43:58.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:43:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:43:57.798+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:43:58.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:43:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:43:57.798+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:43:58.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:43:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:43:57.798+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:43:58.743 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:43:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:43:58.275+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 38011 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:43:58.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:43:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:43:58.455+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:43:58.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:43:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:43:58.455+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:43:58.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:43:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:43:58.455+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:43:59.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:43:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:43:58.823+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:43:59.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:43:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:43:58.823+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:43:59.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:43:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:43:58.823+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:43:59.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:43:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:43:58.823+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:43:59.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:43:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:43:58.742+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:43:59.245 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:43:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:43:58.742+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:43:59.245 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:43:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:43:58.742+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:43:59.728 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:43:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:43:59.452+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:43:59.728 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:43:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:43:59.452+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:43:59.729 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:43:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:43:59.452+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:43:59.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:43:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:43:59.827+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:43:59.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:43:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:43:59.827+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:43:59.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:43:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:43:59.827+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:43:59.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:43:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:43:59.827+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:43:59.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:43:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:43:59.725+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:43:59.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:43:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:43:59.725+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:43:59.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:43:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:43:59.725+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:44:00.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:44:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:44:00.426+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:44:00.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:44:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:44:00.426+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:44:00.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:44:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:44:00.426+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:44:00.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:44:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:44:00.821+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:44:00.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:44:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:44:00.821+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:44:00.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:44:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:44:00.821+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:44:00.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:44:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:44:00.821+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:44:00.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:44:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:44:00.740+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:44:00.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:44:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:44:00.740+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:44:00.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:44:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:44:00.740+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:44:01.740 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:44:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:44:01.386+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:44:01.740 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:44:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:44:01.386+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:44:01.740 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:44:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:44:01.386+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:44:01.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:44:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:44:01.813+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:44:01.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:44:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:44:01.813+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:44:01.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:44:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:44:01.813+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:44:01.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:44:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:44:01.813+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:44:01.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:44:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:44:01.737+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:44:01.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:44:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:44:01.737+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:44:01.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:44:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:44:01.737+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:44:02.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:44:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:44:02.434+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:44:02.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:44:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:44:02.434+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:44:02.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:44:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:44:02.434+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:44:03.194 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:44:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:44:02.784+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:44:03.194 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:44:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:44:02.784+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:44:03.194 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:44:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:44:02.784+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:44:03.194 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:44:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:44:02.813+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:44:03.195 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:44:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:44:02.813+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:44:03.195 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:44:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:44:02.813+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:44:03.195 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:44:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:44:02.813+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:44:03.464 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:44:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:44:03.281+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 38083 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:44:03.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:44:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:44:03.462+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:44:03.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:44:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:44:03.462+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:44:03.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:44:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:44:03.462+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:44:04.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:44:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:44:03.782+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:44:04.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:44:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:44:03.782+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:44:04.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:44:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:44:03.782+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:44:04.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:44:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:44:03.782+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:44:04.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:44:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:44:03.833+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:44:04.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:44:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:44:03.833+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:44:04.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:44:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:44:03.833+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:44:04.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:44:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:44:04.456+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:44:04.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:44:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:44:04.456+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:44:04.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:44:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:44:04.456+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:44:05.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:44:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:44:04.840+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:44:05.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:44:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:44:04.840+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:44:05.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:44:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:44:04.840+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:44:05.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:44:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:44:04.819+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:44:05.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:44:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:44:04.819+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:44:05.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:44:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:44:04.819+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:44:05.245 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:44:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:44:04.820+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:44:05.813 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:44:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:44:05.492+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:44:05.813 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:44:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:44:05.492+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:44:05.813 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:44:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:44:05.492+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:44:06.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:44:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:44:05.831+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:44:06.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:44:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:44:05.831+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:44:06.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:44:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:44:05.831+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:44:06.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:44:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:44:05.810+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:44:06.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:44:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:44:05.810+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:44:06.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:44:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:44:05.810+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:44:06.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:44:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:44:05.810+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:44:06.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:44:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:44:06.489+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:44:06.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:44:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:44:06.489+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:44:06.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:44:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:44:06.489+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:44:07.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:44:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:44:06.879+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:44:07.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:44:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:44:06.879+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:44:07.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:44:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:44:06.879+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:44:07.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:44:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:44:06.841+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:44:07.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:44:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:44:06.841+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:44:07.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:44:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:44:06.841+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:44:07.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:44:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:44:06.841+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:44:07.848 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:44:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:44:07.509+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:44:07.848 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:44:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:44:07.509+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:44:07.848 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:44:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:44:07.509+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:44:08.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:44:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:44:07.861+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:44:08.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:44:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:44:07.861+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:44:08.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:44:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:44:07.861+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:44:08.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:44:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:44:07.844+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:44:08.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:44:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:44:07.844+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:44:08.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:44:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:44:07.844+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:44:08.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:44:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:44:07.845+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:44:08.743 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:44:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:44:08.286+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 38154 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:44:08.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:44:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:44:08.511+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:44:08.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:44:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:44:08.511+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:44:08.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:44:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:44:08.511+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:44:09.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:44:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:44:08.850+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:44:09.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:44:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:44:08.850+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:44:09.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:44:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:44:08.850+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:44:09.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:44:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:44:08.874+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:44:09.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:44:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:44:08.874+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:44:09.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:44:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:44:08.874+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:44:09.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:44:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:44:08.874+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:44:09.830 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:44:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:44:09.491+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:44:09.830 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:44:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:44:09.491+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:44:09.830 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:44:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:44:09.491+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:44:10.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:44:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:44:09.826+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:44:10.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:44:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:44:09.826+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:44:10.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:44:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:44:09.826+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:44:10.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:44:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:44:09.834+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:44:10.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:44:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:44:09.834+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:44:10.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:44:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:44:09.834+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:44:10.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:44:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:44:09.834+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:44:10.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:44:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:44:10.521+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:44:10.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:44:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:44:10.521+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:44:10.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:44:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:44:10.521+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:44:11.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:44:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:44:10.873+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:44:11.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:44:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:44:10.873+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:44:11.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:44:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:44:10.873+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:44:11.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:44:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:44:10.814+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:44:11.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:44:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:44:10.814+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:44:11.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:44:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:44:10.814+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:44:11.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:44:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:44:10.814+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:44:11.826 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:44:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:44:11.552+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:44:11.827 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:44:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:44:11.552+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:44:11.827 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:44:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:44:11.552+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:44:11.827 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:44:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:44:11.797+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:44:11.827 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:44:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:44:11.797+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:44:11.827 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:44:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:44:11.797+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:44:11.827 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:44:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:44:11.797+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:44:12.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:44:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:44:11.826+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:44:12.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:44:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:44:11.826+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:44:12.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:44:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:44:11.826+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:44:12.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:44:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:44:12.581+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:44:12.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:44:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:44:12.581+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:44:12.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:44:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:44:12.581+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:44:13.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:44:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:44:12.832+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:44:13.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:44:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:44:12.832+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:44:13.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:44:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:44:12.833+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:44:13.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:44:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:44:12.830+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:44:13.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:44:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:44:12.830+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:44:13.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:44:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:44:12.830+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:44:13.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:44:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:44:12.830+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:44:13.624 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:44:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:44:13.292+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 38226 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:44:13.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:44:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:44:13.860+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:44:13.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:44:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:44:13.860+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:44:13.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:44:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:44:13.860+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:44:13.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:44:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:44:13.621+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:44:13.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:44:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:44:13.621+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:44:13.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:44:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:44:13.621+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:44:13.995 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:44:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:44:13.808+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:44:13.995 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:44:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:44:13.808+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:44:13.995 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:44:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:44:13.808+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:44:13.995 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:44:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:44:13.808+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:44:14.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:44:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:44:14.847+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:44:14.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:44:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:44:14.847+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:44:14.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:44:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:44:14.847+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:44:14.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:44:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:44:14.646+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:44:14.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:44:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:44:14.646+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:44:14.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:44:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:44:14.646+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:44:14.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:44:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:44:14.787+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:44:14.995 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:44:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:44:14.787+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:44:14.995 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:44:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:44:14.787+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:44:14.995 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:44:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:44:14.787+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:44:15.899 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:44:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:44:15.637+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:44:15.899 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:44:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:44:15.637+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:44:15.899 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:44:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:44:15.637+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:44:15.899 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:44:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:44:15.815+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:44:15.900 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:44:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:44:15.815+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:44:15.900 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:44:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:44:15.815+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:44:15.900 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:44:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:44:15.815+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:44:16.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:44:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:44:15.895+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:44:16.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:44:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:44:15.895+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:44:16.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:44:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:44:15.895+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:44:16.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:44:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:44:16.888+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:44:16.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:44:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:44:16.888+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:44:16.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:44:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:44:16.888+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:44:16.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:44:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:44:16.828+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:44:16.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:44:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:44:16.828+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:44:16.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:44:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:44:16.828+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:44:16.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:44:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:44:16.828+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:44:16.995 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:44:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:44:16.668+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:44:16.995 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:44:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:44:16.668+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:44:16.995 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:44:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:44:16.668+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:44:17.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:44:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:44:17.622+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:44:18.186 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:44:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:44:17.622+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:44:18.187 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:44:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:44:17.622+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:44:18.187 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:44:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:44:17.840+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:44:18.187 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:44:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:44:17.840+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:44:18.187 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:44:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:44:17.840+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:44:18.188 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:44:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:44:17.831+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:44:18.188 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:44:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:44:17.831+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:44:18.189 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:44:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:44:17.831+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:44:18.189 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:44:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:44:17.831+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:44:18.637 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:44:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:44:18.298+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 38299 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:44:18.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:44:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:44:18.840+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:44:18.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:44:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:44:18.840+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:44:18.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:44:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:44:18.840+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:44:18.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:44:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:44:18.855+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:44:18.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:44:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:44:18.855+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:44:18.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:44:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:44:18.855+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:44:18.995 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:44:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:44:18.855+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:44:18.995 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:44:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:44:18.634+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:44:18.995 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:44:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:44:18.634+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:44:18.995 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:44:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:44:18.634+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:44:19.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:44:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:44:19.647+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:44:19.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:44:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:44:19.647+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:44:19.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:44:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:44:19.647+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:44:19.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:44:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:44:19.880+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:44:19.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:44:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:44:19.880+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:44:19.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:44:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:44:19.880+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:44:19.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:44:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:44:19.806+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:44:19.995 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:44:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:44:19.806+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:44:19.995 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:44:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:44:19.806+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:44:19.995 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:44:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:44:19.806+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:44:20.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:44:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:44:20.864+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:44:20.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:44:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:44:20.864+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:44:20.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:44:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:44:20.864+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:44:20.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:44:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:44:20.622+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:44:20.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:44:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:44:20.622+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:44:20.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:44:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:44:20.622+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:44:20.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:44:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:44:20.793+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:44:20.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:44:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:44:20.793+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:44:20.995 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:44:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:44:20.793+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:44:20.995 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:44:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:44:20.793+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:44:21.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:44:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:44:21.843+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:44:21.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:44:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:44:21.843+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:44:21.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:44:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:44:21.843+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:44:21.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:44:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:44:21.625+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:44:21.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:44:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:44:21.625+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:44:21.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:44:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:44:21.625+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:44:21.995 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:44:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:44:21.778+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:44:21.995 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:44:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:44:21.778+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:44:21.995 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:44:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:44:21.778+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:44:21.995 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:44:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:44:21.778+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:44:22.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:44:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:44:22.854+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:44:22.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:44:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:44:22.854+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:44:22.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:44:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:44:22.854+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:44:22.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:44:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:44:22.625+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:44:22.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:44:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:44:22.625+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:44:22.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:44:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:44:22.625+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:44:22.995 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:44:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:44:22.814+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:44:22.995 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:44:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:44:22.814+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:44:22.995 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:44:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:44:22.814+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:44:22.995 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:44:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:44:22.814+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:44:23.585 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:44:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:44:23.304+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 38370 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:44:23.840 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:44:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:44:23.579+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:44:23.840 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:44:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:44:23.579+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:44:23.841 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:44:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:44:23.579+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:44:24.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:44:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:44:23.833+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:44:24.248 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:44:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:44:23.833+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:44:24.248 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:44:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:44:23.833+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:44:24.248 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:44:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:44:23.833+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:44:24.248 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:44:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:44:23.875+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:44:24.249 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:44:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:44:23.875+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:44:24.249 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:44:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:44:23.875+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:44:24.838 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:44:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:44:24.557+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:44:24.838 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:44:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:44:24.557+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:44:24.838 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:44:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:44:24.557+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:44:25.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:44:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:44:24.880+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:44:25.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:44:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:44:24.880+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:44:25.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:44:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:44:24.880+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:44:25.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:44:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:44:24.834+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:44:25.245 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:44:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:44:24.834+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:44:25.245 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:44:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:44:24.834+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:44:25.245 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:44:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:44:24.834+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:44:25.914 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:44:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:44:25.573+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:44:25.915 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:44:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:44:25.573+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:44:25.915 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:44:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:44:25.573+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:44:25.915 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:44:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:44:25.800+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:44:25.915 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:44:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:44:25.800+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:44:25.915 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:44:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:44:25.800+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:44:25.915 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:44:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:44:25.800+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:44:26.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:44:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:44:25.911+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:44:26.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:44:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:44:25.911+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:44:26.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:44:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:44:25.911+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:44:26.816 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:44:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:44:26.525+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:44:26.817 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:44:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:44:26.526+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:44:26.817 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:44:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:44:26.526+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:44:27.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:44:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:44:26.862+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:44:27.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:44:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:44:26.862+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:44:27.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:44:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:44:26.862+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:44:27.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:44:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:44:26.813+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:44:27.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:44:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:44:26.813+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:44:27.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:44:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:44:26.813+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:44:27.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:44:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:44:26.813+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:44:27.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:44:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:44:27.480+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:44:27.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:44:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:44:27.480+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:44:27.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:44:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:44:27.480+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:44:28.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:44:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:44:27.844+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:44:28.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:44:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:44:27.844+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:44:28.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:44:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:44:27.844+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:44:28.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:44:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:44:27.823+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:44:28.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:44:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:44:27.823+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:44:28.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:44:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:44:27.823+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:44:28.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:44:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:44:27.823+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:44:28.743 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:44:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:44:28.310+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 38443 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:44:28.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:44:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:44:28.499+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:44:28.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:44:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:44:28.499+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:44:28.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:44:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:44:28.499+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:44:29.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:44:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:44:28.832+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:44:29.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:44:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:44:28.832+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:44:29.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:44:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:44:28.832+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:44:29.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:44:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:44:28.776+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:44:29.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:44:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:44:28.776+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:44:29.245 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:44:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:44:28.776+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:44:29.245 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:44:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:44:28.776+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:44:29.791 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:44:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:44:29.505+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:44:29.791 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:44:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:44:29.505+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:44:29.791 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:44:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:44:29.505+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:44:29.791 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:44:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:44:29.746+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:44:29.791 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:44:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:44:29.746+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:44:29.792 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:44:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:44:29.746+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:44:29.792 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:44:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:44:29.746+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:44:30.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:44:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:44:29.787+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:44:30.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:44:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:44:29.787+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:44:30.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:44:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:44:29.787+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:44:30.733 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:44:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:44:30.521+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:44:30.734 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:44:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:44:30.521+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:44:30.734 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:44:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:44:30.521+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:44:30.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:44:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:44:30.730+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:44:30.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:44:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:44:30.730+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:44:30.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:44:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:44:30.730+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:44:30.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:44:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:44:30.730+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:44:30.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:44:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:44:30.832+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:44:30.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:44:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:44:30.832+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:44:30.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:44:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:44:30.832+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:44:31.827 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:44:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:44:31.692+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:44:31.827 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:44:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:44:31.692+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:44:31.827 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:44:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:44:31.692+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:44:31.827 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:44:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:44:31.692+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:44:31.828 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:44:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:44:31.529+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:44:31.828 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:44:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:44:31.529+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:44:31.828 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:44:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:44:31.529+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:44:32.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:44:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:44:31.863+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:44:32.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:44:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:44:31.863+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:44:32.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:44:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:44:31.863+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:44:32.869 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:44:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:44:32.644+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:44:32.869 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:44:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:44:32.644+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:44:32.869 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:44:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:44:32.644+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:44:32.869 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:44:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:44:32.644+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:44:32.869 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:44:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:44:32.502+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:44:32.870 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:44:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:44:32.502+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:44:32.870 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:44:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:44:32.502+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:44:33.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:44:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:44:32.866+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:44:33.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:44:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:44:32.866+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:44:33.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:44:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:44:32.866+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:44:33.663 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:44:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:44:33.316+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 38514 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:44:33.663 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:44:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:44:33.456+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:44:33.663 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:44:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:44:33.456+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:44:33.664 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:44:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:44:33.456+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:44:33.914 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:44:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:44:33.863+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:44:33.914 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:44:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:44:33.863+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:44:33.914 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:44:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:44:33.863+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:44:33.915 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:44:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:44:33.660+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:44:33.915 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:44:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:44:33.660+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:44:33.915 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:44:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:44:33.660+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:44:33.915 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:44:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:44:33.660+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:44:34.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:44:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:44:34.440+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:44:34.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:44:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:44:34.440+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:44:34.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:44:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:44:34.440+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:44:34.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:44:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:44:34.624+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:44:34.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:44:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:44:34.624+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:44:34.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:44:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:44:34.624+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:44:34.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:44:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:44:34.624+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:44:35.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:44:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:44:34.855+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:44:35.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:44:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:44:34.855+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:44:35.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:44:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:44:34.855+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:44:35.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:44:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:44:35.600+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:44:35.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:44:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:44:35.600+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:44:35.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:44:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:44:35.600+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:44:35.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:44:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:44:35.600+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:44:35.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:44:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:44:35.424+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:44:35.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:44:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:44:35.424+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:44:35.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:44:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:44:35.424+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:44:36.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:44:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:44:35.817+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:44:36.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:44:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:44:35.817+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:44:36.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:44:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:44:35.817+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:44:36.554 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:44:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:44:36.417+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:44:36.554 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:44:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:44:36.417+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:44:36.555 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:44:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:44:36.417+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:44:36.856 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:44:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:44:36.551+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:44:36.856 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:44:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:44:36.551+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:44:36.857 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:44:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:44:36.551+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:44:36.857 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:44:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:44:36.551+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:44:37.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:44:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:44:36.854+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:44:37.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:44:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:44:36.854+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:44:37.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:44:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:44:36.854+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:44:37.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:44:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:44:37.589+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:44:37.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:44:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:44:37.589+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:44:37.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:44:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:44:37.589+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:44:37.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:44:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:44:37.589+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:44:37.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:44:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:44:37.395+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:44:37.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:44:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:44:37.395+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:44:37.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:44:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:44:37.395+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:44:38.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:44:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:44:37.872+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:44:38.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:44:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:44:37.872+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:44:38.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:44:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:44:37.872+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:44:38.637 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:44:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:44:38.322+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 38587 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:44:38.638 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:44:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:44:38.402+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:44:38.638 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:44:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:44:38.402+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:44:38.638 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:44:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:44:38.402+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:44:38.638 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:44:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:44:38.635+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:44:38.639 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:44:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:44:38.635+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:44:38.892 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:44:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:44:38.635+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:44:38.893 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:44:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:44:38.635+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:44:39.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:44:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:44:38.890+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:44:39.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:44:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:44:38.890+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:44:39.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:44:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:44:38.890+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:44:39.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:44:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:44:39.437+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:44:39.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:44:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:44:39.437+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:44:39.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:44:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:44:39.437+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:44:39.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:44:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:44:39.644+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:44:39.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:44:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:44:39.644+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:44:39.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:44:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:44:39.644+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:44:39.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:44:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:44:39.644+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:44:40.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:44:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:44:39.916+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:44:40.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:44:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:44:39.916+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:44:40.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:44:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:44:39.916+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:44:40.622 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:44:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:44:40.396+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:44:40.622 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:44:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:44:40.396+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:44:40.622 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:44:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:44:40.396+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:44:40.956 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:44:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:44:40.619+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:44:40.956 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:44:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:44:40.619+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:44:40.956 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:44:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:44:40.619+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:44:40.956 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:44:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:44:40.619+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:44:41.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:44:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:44:40.953+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:44:41.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:44:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:44:40.953+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:44:41.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:44:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:44:40.953+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:44:41.649 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:44:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:44:41.395+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:44:41.649 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:44:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:44:41.395+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:44:41.649 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:44:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:44:41.395+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:44:41.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:44:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:44:41.646+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:44:41.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:44:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:44:41.646+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:44:41.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:44:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:44:41.646+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:44:41.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:44:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:44:41.646+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:44:42.371 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:44:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:44:41.994+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:44:42.371 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:44:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:44:41.994+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:44:42.372 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:44:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:44:41.994+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:44:42.687 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:44:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:44:42.368+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:44:42.687 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:44:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:44:42.368+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:44:42.688 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:44:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:44:42.368+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:44:42.981 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:44:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:44:42.685+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:44:42.981 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:44:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:44:42.685+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:44:42.981 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:44:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:44:42.685+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:44:42.981 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:44:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:44:42.685+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:44:43.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:44:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:44:42.978+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:44:43.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:44:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:44:42.978+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:44:43.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:44:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:44:42.978+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:44:43.690 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:44:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:44:43.329+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 38658 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:44:43.690 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:44:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:44:43.336+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:44:43.690 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:44:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:44:43.336+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:44:43.690 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:44:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:44:43.336+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:44:43.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:44:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:44:43.933+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:44:43.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:44:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:44:43.933+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:44:43.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:44:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:44:43.933+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:44:43.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:44:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:44:43.687+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:44:43.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:44:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:44:43.687+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:44:43.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:44:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:44:43.687+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:44:43.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:44:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:44:43.687+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:44:44.670 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:44:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:44:44.293+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:44:44.670 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:44:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:44:44.293+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:44:44.670 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:44:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:44:44.293+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:44:44.947 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:44:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:44:44.667+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:44:44.947 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:44:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:44:44.667+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:44:44.948 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:44:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:44:44.667+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:44:44.948 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:44:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:44:44.667+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:44:45.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:44:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:44:44.945+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:44:45.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:44:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:44:44.945+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:44:45.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:44:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:44:44.945+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:44:45.719 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:44:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:44:45.284+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:44:45.719 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:44:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:44:45.284+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:44:45.719 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:44:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:44:45.284+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:44:45.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:44:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:44:45.920+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:44:45.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:44:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:44:45.920+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:44:45.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:44:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:44:45.920+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:44:45.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:44:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:44:45.716+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:44:45.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:44:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:44:45.716+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:44:45.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:44:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:44:45.716+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:44:45.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:44:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:44:45.716+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:44:46.740 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:44:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:44:46.246+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:44:46.740 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:44:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:44:46.246+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:44:46.740 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:44:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:44:46.246+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:44:46.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:44:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:44:46.889+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:44:46.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:44:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:44:46.889+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:44:46.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:44:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:44:46.889+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:44:46.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:44:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:44:46.737+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:44:46.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:44:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:44:46.737+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:44:46.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:44:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:44:46.737+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:44:46.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:44:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:44:46.738+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:44:47.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:44:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:44:47.228+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:44:47.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:44:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:44:47.228+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:44:47.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:44:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:44:47.228+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:44:48.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:44:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:44:47.843+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:44:48.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:44:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:44:47.843+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:44:48.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:44:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:44:47.843+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:44:48.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:44:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:44:47.786+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:44:48.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:44:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:44:47.786+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:44:48.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:44:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:44:47.786+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:44:48.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:44:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:44:47.786+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:44:48.743 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:44:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:44:48.335+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 38731 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:44:48.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:44:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:44:48.270+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:44:48.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:44:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:44:48.270+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:44:48.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:44:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:44:48.270+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:44:49.228 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:44:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:44:48.893+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:44:49.228 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:44:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:44:48.893+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:44:49.228 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:44:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:44:48.893+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:44:49.229 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:44:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:44:48.777+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:44:49.229 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:44:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:44:48.777+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:44:49.229 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:44:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:44:48.777+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:44:49.229 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:44:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:44:48.777+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:44:49.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:44:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:44:49.225+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:44:49.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:44:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:44:49.225+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:44:49.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:44:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:44:49.225+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:44:49.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:44:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:44:49.861+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:44:49.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:44:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:44:49.861+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:44:49.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:44:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:44:49.861+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:44:49.995 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:44:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:44:49.735+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:44:49.995 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:44:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:44:49.735+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:44:49.995 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:44:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:44:49.735+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:44:49.995 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:44:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:44:49.735+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:44:50.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:44:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:44:50.231+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:44:50.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:44:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:44:50.231+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:44:50.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:44:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:44:50.231+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:44:51.207 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:44:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:44:50.864+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:44:51.208 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:44:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:44:50.864+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:44:51.208 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:44:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:44:50.864+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:44:51.208 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:44:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:44:50.755+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:44:51.208 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:44:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:44:50.755+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:44:51.209 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:44:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:44:50.755+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:44:51.209 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:44:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:44:50.755+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:44:51.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:44:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:44:51.204+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:44:51.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:44:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:44:51.204+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:44:51.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:44:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:44:51.204+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:44:51.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:44:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:44:51.851+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:44:51.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:44:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:44:51.851+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:44:51.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:44:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:44:51.851+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:44:51.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:44:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:44:51.738+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:44:51.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:44:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:44:51.738+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:44:51.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:44:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:44:51.738+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:44:51.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:44:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:44:51.738+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:44:52.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:44:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:44:52.180+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:44:52.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:44:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:44:52.180+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:44:52.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:44:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:44:52.180+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:44:53.167 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:44:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:44:52.895+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:44:53.168 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:44:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:44:52.895+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:44:53.168 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:44:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:44:52.895+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:44:53.168 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:44:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:44:52.783+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:44:53.168 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:44:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:44:52.783+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:44:53.168 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:44:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:44:52.783+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:44:53.169 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:44:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:44:52.783+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:44:53.464 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:44:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:44:53.340+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 38802 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:44:53.465 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:44:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:44:53.165+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:44:53.465 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:44:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:44:53.165+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:44:53.465 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:44:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:44:53.165+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:44:54.118 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:44:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:44:53.887+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:44:54.118 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:44:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:44:53.887+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:44:54.118 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:44:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:44:53.887+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:44:54.119 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:44:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:44:53.821+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:44:54.119 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:44:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:44:53.821+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:44:54.119 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:44:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:44:53.821+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:44:54.119 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:44:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:44:53.821+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:44:54.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:44:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:44:54.115+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:44:54.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:44:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:44:54.115+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:44:54.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:44:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:44:54.115+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:44:55.118 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:44:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:44:54.929+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:44:55.118 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:44:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:44:54.929+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:44:55.118 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:44:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:44:54.929+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:44:55.118 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:44:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:44:54.799+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:44:55.119 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:44:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:44:54.799+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:44:55.119 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:44:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:44:54.799+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:44:55.119 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:44:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:44:54.799+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:44:55.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:44:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:44:55.115+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:44:55.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:44:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:44:55.115+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:44:55.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:44:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:44:55.115+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:44:56.163 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:44:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:44:55.891+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:44:56.163 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:44:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:44:55.891+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:44:56.164 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:44:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:44:55.891+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:44:56.164 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:44:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:44:55.827+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:44:56.164 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:44:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:44:55.827+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:44:56.164 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:44:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:44:55.827+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:44:56.164 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:44:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:44:55.827+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:44:56.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:44:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:44:56.160+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:44:56.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:44:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:44:56.160+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:44:56.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:44:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:44:56.160+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:44:57.213 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:44:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:44:56.920+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:44:57.213 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:44:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:44:56.920+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:44:57.214 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:44:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:44:56.920+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:44:57.214 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:44:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:44:56.820+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:44:57.214 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:44:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:44:56.820+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:44:57.214 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:44:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:44:56.820+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:44:57.214 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:44:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:44:56.820+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:44:57.495 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:44:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:44:57.210+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:44:57.495 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:44:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:44:57.210+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:44:57.495 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:44:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:44:57.210+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:44:58.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:44:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:44:57.880+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:44:58.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:44:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:44:57.880+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:44:58.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:44:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:44:57.880+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:44:58.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:44:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:44:57.853+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:44:58.245 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:44:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:44:57.853+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:44:58.245 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:44:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:44:57.853+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:44:58.245 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:44:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:44:57.853+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:44:58.743 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:44:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:44:58.345+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 38875 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:44:58.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:44:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:44:58.249+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:44:58.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:44:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:44:58.249+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:44:58.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:44:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:44:58.249+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:44:59.210 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:44:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:44:58.813+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:44:59.211 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:44:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:44:58.813+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:44:59.211 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:44:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:44:58.813+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:44:59.211 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:44:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:44:58.813+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:44:59.212 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:44:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:44:58.851+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:44:59.212 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:44:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:44:58.851+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:44:59.212 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:44:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:44:58.851+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:44:59.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:44:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:44:59.207+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:44:59.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:44:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:44:59.207+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:44:59.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:44:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:44:59.207+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:45:00.214 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:44:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:44:59.827+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:45:00.214 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:44:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:44:59.828+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:45:00.215 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:44:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:44:59.828+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:45:00.215 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:44:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:44:59.828+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:45:00.215 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:44:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:44:59.881+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:45:00.215 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:44:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:44:59.881+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:45:00.215 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:44:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:44:59.881+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:45:00.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:45:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:45:00.211+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:45:00.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:45:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:45:00.211+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:45:00.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:45:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:45:00.211+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:45:01.221 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:45:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:45:00.874+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:45:01.221 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:45:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:45:00.874+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:45:01.221 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:45:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:45:00.874+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:45:01.222 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:45:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:45:00.813+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:45:01.222 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:45:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:45:00.813+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:45:01.222 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:45:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:45:00.813+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:45:01.222 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:45:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:45:00.813+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:45:01.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:45:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:45:01.217+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:45:01.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:45:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:45:01.217+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:45:01.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:45:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:45:01.217+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:45:02.212 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:45:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:45:01.823+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:45:02.213 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:45:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:45:01.823+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:45:02.213 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:45:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:45:01.823+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:45:02.213 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:45:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:45:01.823+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:45:02.213 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:45:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:45:01.876+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:45:02.213 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:45:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:45:01.876+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:45:02.213 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:45:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:45:01.876+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:45:02.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:45:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:45:02.209+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:45:02.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:45:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:45:02.209+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:45:02.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:45:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:45:02.209+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:45:03.177 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:45:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:45:02.912+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:45:03.177 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:45:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:45:02.912+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:45:03.177 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:45:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:45:02.912+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:45:03.178 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:45:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:45:02.812+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:45:03.178 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:45:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:45:02.812+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:45:03.178 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:45:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:45:02.812+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:45:03.178 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:45:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:45:02.812+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:45:03.477 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:45:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:45:03.351+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 38946 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:45:03.478 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:45:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:45:03.173+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:45:03.478 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:45:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:45:03.173+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:45:03.478 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:45:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:45:03.173+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:45:04.202 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:45:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:45:03.877+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:45:04.202 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:45:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:45:03.877+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:45:04.202 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:45:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:45:03.877+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:45:04.203 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:45:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:45:03.787+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:45:04.203 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:45:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:45:03.787+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:45:04.203 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:45:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:45:03.787+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:45:04.203 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:45:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:45:03.787+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:45:04.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:45:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:45:04.198+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:45:04.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:45:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:45:04.198+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:45:04.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:45:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:45:04.198+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:45:04.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:45:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:45:04.868+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:45:04.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:45:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:45:04.868+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:45:04.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:45:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:45:04.868+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:45:04.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:45:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:45:04.784+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:45:04.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:45:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:45:04.784+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:45:04.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:45:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:45:04.784+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:45:04.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:45:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:45:04.784+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:45:05.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:45:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:45:05.194+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:45:05.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:45:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:45:05.194+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:45:05.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:45:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:45:05.194+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:45:06.214 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:45:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:45:05.825+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:45:06.215 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:45:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:45:05.825+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:45:06.215 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:45:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:45:05.825+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:45:06.215 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:45:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:45:05.758+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:45:06.215 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:45:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:45:05.758+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:45:06.215 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:45:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:45:05.758+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:45:06.215 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:45:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:45:05.758+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:45:06.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:45:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:45:06.210+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:45:06.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:45:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:45:06.211+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:45:06.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:45:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:45:06.211+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:45:06.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:45:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:45:06.842+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:45:06.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:45:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:45:06.842+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:45:06.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:45:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:45:06.842+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:45:06.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:45:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:45:06.724+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:45:06.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:45:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:45:06.724+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:45:06.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:45:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:45:06.724+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:45:06.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:45:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:45:06.724+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:45:07.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:45:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:45:07.172+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:45:07.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:45:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:45:07.172+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:45:07.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:45:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:45:07.172+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:45:08.168 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:45:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:45:07.859+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:45:08.168 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:45:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:45:07.859+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:45:08.168 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:45:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:45:07.859+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:45:08.169 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:45:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:45:07.741+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:45:08.169 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:45:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:45:07.741+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:45:08.169 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:45:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:45:07.741+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:45:08.169 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:45:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:45:07.741+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:45:08.483 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:45:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:45:08.356+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 39019 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:45:08.484 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:45:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:45:08.165+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:45:08.484 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:45:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:45:08.165+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:45:08.484 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:45:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:45:08.165+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:45:08.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:45:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:45:08.695+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:45:08.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:45:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:45:08.695+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:45:08.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:45:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:45:08.695+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:45:08.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:45:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:45:08.695+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:45:09.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:45:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:45:09.142+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:45:09.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:45:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:45:09.142+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:45:09.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:45:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:45:09.142+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:45:09.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:45:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:45:08.900+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:45:09.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:45:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:45:08.900+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:45:09.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:45:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:45:08.900+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:45:09.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:45:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:45:09.856+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:45:09.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:45:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:45:09.856+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:45:09.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:45:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:45:09.856+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:45:09.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:45:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:45:09.714+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:45:09.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:45:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:45:09.714+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:45:09.995 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:45:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:45:09.714+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:45:09.995 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:45:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:45:09.714+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:45:10.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:45:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:45:10.112+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:45:10.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:45:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:45:10.112+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:45:10.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:45:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:45:10.112+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:45:11.164 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:45:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:45:10.822+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:45:11.164 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:45:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:45:10.822+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:45:11.164 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:45:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:45:10.822+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:45:11.164 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:45:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:45:10.752+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:45:11.165 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:45:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:45:10.752+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:45:11.165 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:45:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:45:10.752+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:45:11.165 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:45:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:45:10.752+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:45:11.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:45:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:45:11.160+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:45:11.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:45:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:45:11.160+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:45:11.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:45:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:45:11.160+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:45:11.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:45:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:45:11.842+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:45:11.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:45:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:45:11.842+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:45:11.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:45:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:45:11.842+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:45:11.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:45:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:45:11.715+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:45:11.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:45:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:45:11.715+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:45:11.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:45:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:45:11.715+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:45:11.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:45:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:45:11.715+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:45:12.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:45:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:45:12.169+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:45:12.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:45:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:45:12.169+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:45:12.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:45:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:45:12.169+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:45:13.177 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:45:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:45:12.831+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:45:13.177 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:45:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:45:12.831+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:45:13.177 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:45:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:45:12.831+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:45:13.177 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:45:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:45:12.753+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:45:13.178 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:45:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:45:12.753+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:45:13.178 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:45:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:45:12.753+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:45:13.178 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:45:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:45:12.753+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:45:13.486 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:45:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:45:13.362+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 39090 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:45:13.486 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:45:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:45:13.174+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:45:13.486 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:45:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:45:13.174+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:45:13.486 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:45:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:45:13.174+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:45:13.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:45:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:45:13.717+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:45:13.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:45:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:45:13.717+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:45:13.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:45:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:45:13.717+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:45:13.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:45:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:45:13.717+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:45:14.143 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:45:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:45:13.786+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:45:14.143 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:45:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:45:13.786+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:45:14.143 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:45:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:45:13.786+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:45:14.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:45:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:45:14.140+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:45:14.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:45:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:45:14.140+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:45:14.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:45:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:45:14.140+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:45:14.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:45:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:45:14.780+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:45:14.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:45:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:45:14.780+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:45:14.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:45:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:45:14.780+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:45:14.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:45:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:45:14.681+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:45:14.995 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:45:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:45:14.681+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:45:14.995 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:45:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:45:14.681+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:45:14.995 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:45:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:45:14.681+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:45:15.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:45:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:45:15.177+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:45:15.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:45:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:45:15.177+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:45:15.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:45:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:45:15.177+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:45:15.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:45:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:45:15.827+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:45:15.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:45:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:45:15.827+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:45:15.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:45:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:45:15.827+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:45:15.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:45:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:45:15.721+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:45:15.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:45:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:45:15.721+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:45:15.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:45:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:45:15.721+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:45:15.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:45:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:45:15.721+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:45:16.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:45:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:45:16.195+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:45:16.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:45:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:45:16.195+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:45:16.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:45:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:45:16.195+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:45:16.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:45:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:45:16.844+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:45:16.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:45:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:45:16.844+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:45:16.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:45:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:45:16.844+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:45:16.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:45:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:45:16.702+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:45:16.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:45:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:45:16.702+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:45:16.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:45:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:45:16.702+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:45:16.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:45:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:45:16.702+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:45:17.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:45:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:45:17.191+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:45:17.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:45:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:45:17.191+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:45:17.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:45:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:45:17.191+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:45:17.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:45:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:45:17.853+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:45:17.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:45:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:45:17.853+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:45:17.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:45:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:45:17.853+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:45:17.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:45:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:45:17.652+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:45:17.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:45:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:45:17.652+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:45:17.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:45:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:45:17.652+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:45:17.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:45:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:45:17.652+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:45:18.493 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:45:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:45:18.368+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 39164 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:45:18.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:45:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:45:18.167+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:45:18.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:45:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:45:18.167+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:45:18.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:45:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:45:18.167+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:45:18.875 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:45:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:45:18.643+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:45:18.875 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:45:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:45:18.643+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:45:18.875 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:45:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:45:18.643+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:45:18.875 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:45:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:45:18.643+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:45:19.210 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:45:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:45:18.872+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:45:19.210 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:45:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:45:18.872+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:45:19.210 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:45:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:45:18.872+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:45:19.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:45:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:45:19.206+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:45:19.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:45:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:45:19.206+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:45:19.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:45:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:45:19.206+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:45:19.918 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:45:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:45:19.665+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:45:19.919 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:45:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:45:19.665+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:45:19.919 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:45:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:45:19.665+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:45:19.919 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:45:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:45:19.665+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:45:20.223 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:45:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:45:19.915+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:45:20.224 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:45:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:45:19.915+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:45:20.224 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:45:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:45:19.915+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:45:20.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:45:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:45:20.220+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:45:20.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:45:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:45:20.220+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:45:20.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:45:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:45:20.220+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:45:20.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:45:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:45:20.669+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:45:20.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:45:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:45:20.669+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:45:20.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:45:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:45:20.669+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:45:20.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:45:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:45:20.669+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:45:20.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:45:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:45:20.868+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:45:20.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:45:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:45:20.868+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:45:20.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:45:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:45:20.868+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:45:21.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:45:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:45:21.190+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:45:21.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:45:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:45:21.190+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:45:21.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:45:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:45:21.190+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:45:21.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:45:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:45:21.913+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:45:21.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:45:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:45:21.913+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:45:21.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:45:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:45:21.913+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:45:21.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:45:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:45:21.711+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:45:21.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:45:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:45:21.711+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:45:21.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:45:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:45:21.711+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:45:21.995 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:45:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:45:21.711+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:45:22.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:45:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:45:22.174+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:45:22.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:45:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:45:22.174+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:45:22.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:45:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:45:22.174+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:45:22.945 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:45:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:45:22.691+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:45:22.945 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:45:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:45:22.691+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:45:22.945 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:45:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:45:22.691+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:45:22.945 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:45:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:45:22.691+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:45:23.224 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:45:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:45:22.942+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:45:23.225 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:45:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:45:22.942+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:45:23.225 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:45:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:45:22.942+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:45:23.493 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:45:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:45:23.373+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 39235 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:45:23.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:45:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:45:23.221+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:45:23.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:45:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:45:23.221+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:45:23.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:45:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:45:23.221+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:45:23.915 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:45:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:45:23.704+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:45:23.915 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:45:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:45:23.704+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:45:23.915 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:45:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:45:23.704+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:45:23.915 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:45:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:45:23.704+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:45:24.198 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:45:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:45:23.911+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:45:24.198 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:45:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:45:23.911+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:45:24.198 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:45:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:45:23.911+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:45:24.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:45:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:45:24.195+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:45:24.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:45:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:45:24.195+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:45:24.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:45:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:45:24.195+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:45:24.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:45:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:45:24.667+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:45:24.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:45:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:45:24.667+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:45:24.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:45:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:45:24.667+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:45:24.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:45:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:45:24.667+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:45:24.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:45:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:45:24.867+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:45:24.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:45:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:45:24.868+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:45:24.995 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:45:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:45:24.868+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:45:25.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:45:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:45:25.149+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:45:25.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:45:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:45:25.149+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:45:25.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:45:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:45:25.149+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:45:25.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:45:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:45:25.639+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:45:25.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:45:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:45:25.639+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:45:25.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:45:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:45:25.639+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:45:25.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:45:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:45:25.639+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:45:25.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:45:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:45:25.853+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:45:25.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:45:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:45:25.853+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:45:25.995 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:45:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:45:25.853+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:45:26.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:45:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:45:26.174+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:45:26.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:45:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:45:26.174+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:45:26.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:45:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:45:26.174+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:45:26.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:45:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:45:26.840+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:45:26.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:45:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:45:26.840+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:45:26.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:45:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:45:26.840+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:45:26.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:45:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:45:26.598+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:45:26.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:45:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:45:26.598+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:45:26.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:45:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:45:26.598+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:45:26.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:45:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:45:26.598+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:45:27.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:45:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:45:27.162+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:45:27.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:45:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:45:27.162+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:45:27.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:45:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:45:27.162+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:45:27.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:45:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:45:27.813+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:45:27.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:45:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:45:27.813+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:45:27.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:45:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:45:27.813+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:45:27.995 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:45:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:45:27.619+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:45:27.995 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:45:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:45:27.619+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:45:27.995 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:45:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:45:27.619+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:45:27.995 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:45:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:45:27.619+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:45:28.493 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:45:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:45:28.379+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 39308 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:45:28.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:45:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:45:28.167+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:45:28.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:45:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:45:28.167+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:45:28.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:45:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:45:28.167+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:45:28.812 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:45:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:45:28.621+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:45:28.812 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:45:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:45:28.621+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:45:28.813 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:45:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:45:28.621+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:45:28.813 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:45:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:45:28.621+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:45:29.179 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:45:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:45:28.809+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:45:29.179 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:45:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:45:28.809+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:45:29.179 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:45:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:45:28.809+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:45:29.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:45:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:45:29.176+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:45:29.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:45:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:45:29.176+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:45:29.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:45:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:45:29.176+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:45:29.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:45:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:45:29.806+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:45:29.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:45:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:45:29.806+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:45:29.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:45:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:45:29.806+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:45:29.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:45:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:45:29.636+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:45:29.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:45:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:45:29.636+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:45:29.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:45:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:45:29.636+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:45:29.995 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:45:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:45:29.636+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:45:30.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:45:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:45:30.223+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:45:30.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:45:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:45:30.223+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:45:30.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:45:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:45:30.223+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:45:30.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:45:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:45:30.761+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:45:30.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:45:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:45:30.761+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:45:30.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:45:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:45:30.761+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:45:30.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:45:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:45:30.629+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:45:30.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:45:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:45:30.629+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:45:30.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:45:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:45:30.629+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:45:30.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:45:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:45:30.629+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:45:31.658 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:45:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:45:31.260+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:45:31.658 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:45:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:45:31.260+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:45:31.658 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:45:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:45:31.260+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:45:31.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:45:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:45:31.713+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:45:31.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:45:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:45:31.713+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:45:31.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:45:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:45:31.713+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:45:31.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:45:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:45:31.655+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:45:31.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:45:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:45:31.655+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:45:31.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:45:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:45:31.655+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:45:31.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:45:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:45:31.655+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:45:32.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:45:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:45:32.225+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:45:32.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:45:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:45:32.225+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:45:32.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:45:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:45:32.225+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:45:32.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:45:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:45:32.725+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:45:32.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:45:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:45:32.725+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:45:32.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:45:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:45:32.725+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:45:32.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:45:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:45:32.672+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:45:32.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:45:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:45:32.672+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:45:32.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:45:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:45:32.672+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:45:32.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:45:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:45:32.673+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:45:33.493 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:45:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:45:33.386+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 39379 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:45:33.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:45:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:45:33.204+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:45:33.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:45:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:45:33.204+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:45:33.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:45:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:45:33.204+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:45:33.777 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:45:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:45:33.654+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:45:33.778 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:45:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:45:33.654+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:45:33.778 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:45:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:45:33.654+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:45:33.778 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:45:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:45:33.655+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:45:34.235 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:45:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:45:33.775+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:45:34.235 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:45:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:45:33.775+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:45:34.236 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:45:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:45:33.775+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:45:34.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:45:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:45:34.232+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:45:34.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:45:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:45:34.232+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:45:34.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:45:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:45:34.232+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:45:34.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:45:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:45:34.729+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:45:34.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:45:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:45:34.729+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:45:34.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:45:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:45:34.729+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:45:34.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:45:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:45:34.653+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:45:34.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:45:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:45:34.653+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:45:34.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:45:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:45:34.653+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:45:34.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:45:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:45:34.653+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:45:35.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:45:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:45:35.215+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:45:35.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:45:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:45:35.215+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:45:35.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:45:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:45:35.215+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:45:35.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:45:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:45:35.778+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:45:35.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:45:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:45:35.778+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:45:35.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:45:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:45:35.778+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:45:35.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:45:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:45:35.660+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:45:35.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:45:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:45:35.660+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:45:35.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:45:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:45:35.660+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:45:35.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:45:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:45:35.660+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:45:36.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:45:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:45:36.237+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:45:36.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:45:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:45:36.237+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:45:36.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:45:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:45:36.237+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:45:36.746 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:45:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:45:36.658+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:45:36.747 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:45:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:45:36.658+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:45:36.747 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:45:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:45:36.658+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:45:36.747 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:45:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:45:36.659+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:45:37.210 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:45:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:45:36.743+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:45:37.210 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:45:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:45:36.743+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:45:37.211 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:45:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:45:36.743+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:45:37.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:45:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:45:37.207+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:45:37.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:45:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:45:37.207+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:45:37.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:45:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:45:37.207+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:45:37.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:45:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:45:37.734+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:45:37.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:45:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:45:37.734+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:45:37.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:45:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:45:37.734+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:45:37.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:45:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:45:37.659+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:45:37.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:45:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:45:37.659+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:45:37.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:45:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:45:37.659+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:45:37.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:45:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:45:37.659+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:45:38.493 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:45:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:45:38.392+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 39452 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:45:38.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:45:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:45:38.189+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:45:38.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:45:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:45:38.189+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:45:38.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:45:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:45:38.189+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:45:38.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:45:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:45:38.745+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:45:38.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:45:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:45:38.745+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:45:38.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:45:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:45:38.745+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:45:38.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:45:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:45:38.629+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:45:38.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:45:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:45:38.629+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:45:38.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:45:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:45:38.629+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:45:38.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:45:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:45:38.629+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:45:39.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:45:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:45:39.223+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:45:39.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:45:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:45:39.223+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:45:39.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:45:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:45:39.223+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:45:39.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:45:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:45:39.793+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:45:39.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:45:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:45:39.793+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:45:39.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:45:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:45:39.793+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:45:39.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:45:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:45:39.668+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:45:39.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:45:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:45:39.668+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:45:39.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:45:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:45:39.668+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:45:39.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:45:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:45:39.668+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:45:40.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:45:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:45:40.207+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:45:40.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:45:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:45:40.207+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:45:40.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:45:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:45:40.207+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:45:40.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:45:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:45:40.802+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:45:40.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:45:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:45:40.802+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:45:40.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:45:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:45:40.802+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:45:40.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:45:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:45:40.698+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:45:40.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:45:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:45:40.698+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:45:40.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:45:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:45:40.698+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:45:40.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:45:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:45:40.698+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:45:41.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:45:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:45:41.170+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:45:41.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:45:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:45:41.170+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:45:41.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:45:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:45:41.170+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:45:41.758 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:45:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:45:41.655+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:45:41.759 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:45:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:45:41.655+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:45:41.759 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:45:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:45:41.655+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:45:41.759 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:45:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:45:41.656+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:45:42.196 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:45:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:45:41.756+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:45:42.196 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:45:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:45:41.756+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:45:42.197 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:45:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:45:41.756+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:45:42.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:45:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:45:42.193+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:45:42.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:45:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:45:42.193+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:45:42.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:45:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:45:42.193+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:45:42.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:45:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:45:42.761+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:45:42.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:45:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:45:42.761+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:45:42.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:45:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:45:42.761+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:45:42.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:45:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:45:42.660+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:45:42.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:45:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:45:42.660+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:45:42.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:45:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:45:42.660+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:45:42.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:45:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:45:42.660+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:45:43.493 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:45:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:45:43.398+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 39523 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:45:43.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:45:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:45:43.232+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:45:43.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:45:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:45:43.232+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:45:43.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:45:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:45:43.232+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:45:43.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:45:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:45:43.713+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:45:43.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:45:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:45:43.713+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:45:43.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:45:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:45:43.713+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:45:43.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:45:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:45:43.633+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:45:43.995 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:45:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:45:43.633+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:45:43.995 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:45:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:45:43.633+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:45:43.995 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:45:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:45:43.633+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:45:44.570 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:45:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:45:44.276+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:45:44.570 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:45:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:45:44.276+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:45:44.570 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:45:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:45:44.276+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:45:44.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:45:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:45:44.729+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:45:44.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:45:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:45:44.729+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:45:44.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:45:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:45:44.729+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:45:44.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:45:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:45:44.663+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:45:44.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:45:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:45:44.663+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:45:44.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:45:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:45:44.663+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:45:44.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:45:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:45:44.663+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:45:45.713 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:45:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:45:45.292+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:45:45.714 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:45:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:45:45.292+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:45:45.714 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:45:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:45:45.292+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:45:45.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:45:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:45:45.729+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:45:45.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:45:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:45:45.729+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:45:45.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:45:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:45:45.729+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:45:45.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:45:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:45:45.711+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:45:45.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:45:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:45:45.711+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:45:45.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:45:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:45:45.711+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:45:45.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:45:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:45:45.711+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:45:46.693 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:45:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:45:46.341+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:45:46.694 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:45:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:45:46.341+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:45:46.694 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:45:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:45:46.341+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:45:46.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:45:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:45:46.691+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:45:46.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:45:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:45:46.691+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:45:46.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:45:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:45:46.691+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:45:46.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:45:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:45:46.710+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:45:46.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:45:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:45:46.710+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:45:46.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:45:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:45:46.710+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:45:46.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:45:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:45:46.710+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:45:47.671 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:45:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:45:47.355+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:45:47.672 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:45:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:45:47.355+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:45:47.672 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:45:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:45:47.355+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:45:47.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:45:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:45:47.669+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:45:47.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:45:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:45:47.669+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:45:47.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:45:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:45:47.669+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:45:47.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:45:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:45:47.744+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:45:47.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:45:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:45:47.744+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:45:47.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:45:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:45:47.744+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:45:47.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:45:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:45:47.744+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:45:48.663 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:45:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:45:48.404+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 39596 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:45:48.663 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:45:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:45:48.380+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:45:48.664 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:45:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:45:48.380+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:45:48.664 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:45:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:45:48.380+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:45:48.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:45:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:45:48.660+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:45:48.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:45:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:45:48.660+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:45:48.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:45:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:45:48.660+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:45:48.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:45:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:45:48.746+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:45:48.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:45:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:45:48.746+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:45:48.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:45:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:45:48.746+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:45:48.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:45:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:45:48.746+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:45:49.654 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:45:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:45:49.359+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:45:49.654 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:45:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:45:49.359+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:45:49.655 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:45:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:45:49.359+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:45:49.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:45:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:45:49.652+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:45:49.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:45:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:45:49.652+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:45:49.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:45:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:45:49.652+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:45:49.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:45:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:45:49.741+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:45:49.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:45:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:45:49.741+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:45:49.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:45:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:45:49.741+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:45:49.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:45:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:45:49.741+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:45:50.691 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:45:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:45:50.358+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:45:50.692 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:45:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:45:50.358+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:45:50.692 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:45:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:45:50.358+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:45:50.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:45:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:45:50.689+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:45:50.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:45:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:45:50.689+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:45:50.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:45:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:45:50.689+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:45:50.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:45:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:45:50.762+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:45:50.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:45:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:45:50.762+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:45:50.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:45:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:45:50.762+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:45:50.995 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:45:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:45:50.762+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:45:51.728 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:45:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:45:51.338+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:45:51.728 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:45:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:45:51.338+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:45:51.728 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:45:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:45:51.338+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:45:51.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:45:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:45:51.726+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:45:51.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:45:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:45:51.726+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:45:51.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:45:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:45:51.726+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:45:51.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:45:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:45:51.804+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:45:51.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:45:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:45:51.804+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:45:51.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:45:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:45:51.804+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:45:51.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:45:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:45:51.804+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:45:52.688 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:45:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:45:52.332+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:45:52.688 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:45:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:45:52.332+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:45:52.688 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:45:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:45:52.332+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:45:52.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:45:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:45:52.685+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:45:52.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:45:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:45:52.685+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:45:52.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:45:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:45:52.685+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:45:52.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:45:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:45:52.829+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:45:52.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:45:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:45:52.829+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:45:52.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:45:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:45:52.829+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:45:52.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:45:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:45:52.829+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:45:53.730 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:45:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:45:53.410+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 39667 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:45:53.731 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:45:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:45:53.333+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:45:53.731 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:45:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:45:53.334+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:45:53.731 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:45:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:45:53.334+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:45:53.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:45:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:45:53.726+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:45:53.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:45:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:45:53.726+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:45:53.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:45:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:45:53.726+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:45:53.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:45:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:45:53.875+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:45:53.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:45:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:45:53.875+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:45:53.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:45:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:45:53.875+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:45:53.995 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:45:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:45:53.876+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:45:54.696 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:45:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:45:54.288+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:45:54.696 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:45:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:45:54.288+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:45:54.696 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:45:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:45:54.288+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:45:54.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:45:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:45:54.692+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:45:54.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:45:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:45:54.692+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:45:54.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:45:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:45:54.692+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:45:54.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:45:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:45:54.886+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:45:54.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:45:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:45:54.886+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:45:54.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:45:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:45:54.886+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:45:54.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:45:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:45:54.886+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:45:55.667 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:45:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:45:55.335+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:45:55.667 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:45:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:45:55.335+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:45:55.668 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:45:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:45:55.335+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:45:55.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:45:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:45:55.664+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:45:55.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:45:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:45:55.664+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:45:55.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:45:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:45:55.664+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:45:55.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:45:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:45:55.901+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:45:55.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:45:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:45:55.901+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:45:55.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:45:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:45:55.901+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:45:55.995 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:45:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:45:55.901+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:45:56.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:45:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:45:56.628+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:45:56.744 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:45:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:45:56.628+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:45:56.744 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:45:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:45:56.628+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:45:56.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:45:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:45:56.383+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:45:56.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:45:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:45:56.383+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:45:56.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:45:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:45:56.383+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:45:57.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:45:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:45:56.932+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:45:57.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:45:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:45:56.932+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:45:57.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:45:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:45:56.932+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:45:57.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:45:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:45:56.932+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:45:57.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:45:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:45:57.583+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:45:57.744 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:45:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:45:57.583+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:45:57.744 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:45:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:45:57.583+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:45:57.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:45:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:45:57.422+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:45:57.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:45:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:45:57.422+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:45:57.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:45:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:45:57.422+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:45:58.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:45:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:45:57.914+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:45:58.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:45:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:45:57.914+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:45:58.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:45:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:45:57.914+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:45:58.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:45:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:45:57.914+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:45:58.744 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:45:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:45:58.415+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 39740 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:45:58.744 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:45:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:45:58.536+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:45:58.744 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:45:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:45:58.536+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:45:58.745 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:45:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:45:58.536+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:45:58.745 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:45:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:45:58.430+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:45:58.745 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:45:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:45:58.430+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:45:58.745 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:45:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:45:58.430+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:45:59.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:45:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:45:58.955+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:45:59.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:45:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:45:58.955+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:45:59.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:45:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:45:58.955+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:45:59.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:45:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:45:58.955+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:45:59.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:45:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:45:59.570+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:45:59.744 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:45:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:45:59.570+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:45:59.744 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:45:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:45:59.570+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:45:59.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:45:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:45:59.436+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:45:59.745 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:45:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:45:59.436+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:45:59.745 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:45:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:45:59.436+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:46:00.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:45:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:45:59.983+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:46:00.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:45:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:45:59.983+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:46:00.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:45:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:45:59.983+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:46:00.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:45:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:45:59.983+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:46:00.609 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:46:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:46:00.390+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:46:00.610 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:46:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:46:00.390+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:46:00.610 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:46:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:46:00.390+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:46:00.972 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:46:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:46:00.606+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:46:00.973 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:46:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:46:00.606+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:46:00.973 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:46:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:46:00.606+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:46:01.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:46:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:46:00.969+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:46:01.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:46:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:46:00.969+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:46:01.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:46:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:46:00.969+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:46:01.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:46:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:46:00.969+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:46:01.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:46:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:46:01.644+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:46:01.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:46:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:46:01.644+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:46:01.744 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:46:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:46:01.644+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:46:01.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:46:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:46:01.428+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:46:01.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:46:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:46:01.428+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:46:01.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:46:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:46:01.428+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:46:02.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:46:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:46:01.986+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:46:02.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:46:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:46:01.986+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:46:02.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:46:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:46:01.986+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:46:02.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:46:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:46:01.986+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:46:02.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:46:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:46:02.434+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:46:02.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:46:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:46:02.435+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:46:02.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:46:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:46:02.435+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:46:02.744 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:46:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:46:02.632+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:46:02.745 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:46:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:46:02.632+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:46:02.745 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:46:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:46:02.632+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:46:03.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:46:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:46:02.988+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:46:03.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:46:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:46:02.988+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:46:03.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:46:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:46:02.988+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:46:03.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:46:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:46:02.988+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:46:03.743 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:46:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:46:03.422+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 39811 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:46:03.744 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:46:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:46:03.651+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:46:03.744 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:46:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:46:03.651+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:46:03.744 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:46:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:46:03.651+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:46:03.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:46:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:46:03.453+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:46:03.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:46:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:46:03.454+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:46:03.745 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:46:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:46:03.454+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:46:04.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:46:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:46:03.957+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:46:04.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:46:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:46:03.957+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:46:04.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:46:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:46:03.957+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:46:04.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:46:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:46:03.957+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:46:04.702 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:46:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:46:04.410+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:46:04.703 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:46:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:46:04.410+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:46:04.703 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:46:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:46:04.410+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:46:04.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:46:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:46:04.699+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:46:04.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:46:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:46:04.699+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:46:04.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:46:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:46:04.699+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:46:05.449 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:46:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:46:04.993+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:46:05.449 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:46:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:46:04.993+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:46:05.449 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:46:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:46:04.993+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:46:05.449 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:46:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:46:04.993+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:46:05.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:46:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:46:05.445+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:46:05.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:46:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:46:05.445+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:46:05.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:46:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:46:05.445+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:46:05.744 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:46:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:46:05.694+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:46:05.744 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:46:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:46:05.694+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:46:05.745 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:46:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:46:05.694+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:46:06.421 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:46:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:46:06.017+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:46:06.421 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:46:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:46:06.017+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:46:06.422 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:46:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:46:06.017+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:46:06.422 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:46:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:46:06.017+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:46:06.737 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:46:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:46:06.418+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:46:06.737 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:46:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:46:06.418+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:46:06.737 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:46:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:46:06.418+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:46:06.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:46:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:46:06.734+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:46:06.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:46:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:46:06.734+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:46:06.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:46:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:46:06.734+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:46:07.412 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:46:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:46:07.046+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:46:07.413 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:46:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:46:07.046+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:46:07.413 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:46:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:46:07.046+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:46:07.413 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:46:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:46:07.046+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:46:07.704 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:46:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:46:07.409+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:46:07.705 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:46:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:46:07.409+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:46:07.705 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:46:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:46:07.409+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:46:07.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:46:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:46:07.701+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:46:07.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:46:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:46:07.701+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:46:07.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:46:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:46:07.701+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:46:08.405 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:46:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:46:08.016+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:46:08.406 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:46:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:46:08.016+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:46:08.406 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:46:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:46:08.016+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:46:08.406 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:46:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:46:08.017+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:46:08.735 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:46:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:46:08.427+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 39884 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:46:08.735 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:46:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:46:08.402+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:46:08.736 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:46:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:46:08.402+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:46:08.736 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:46:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:46:08.402+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:46:08.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:46:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:46:08.731+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:46:08.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:46:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:46:08.731+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:46:08.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:46:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:46:08.731+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:46:09.434 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:46:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:46:09.053+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:46:09.435 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:46:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:46:09.053+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:46:09.435 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:46:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:46:09.053+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:46:09.435 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:46:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:46:09.053+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:46:09.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:46:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:46:09.431+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:46:09.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:46:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:46:09.431+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:46:09.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:46:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:46:09.431+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:46:10.082 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:46:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:46:09.749+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:46:10.082 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:46:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:46:09.749+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:46:10.082 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:46:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:46:09.749+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:46:10.415 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:46:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:46:10.079+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:46:10.415 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:46:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:46:10.079+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:46:10.415 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:46:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:46:10.079+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:46:10.415 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:46:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:46:10.079+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:46:10.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:46:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:46:10.412+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:46:10.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:46:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:46:10.412+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:46:10.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:46:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:46:10.412+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:46:11.086 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:46:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:46:10.796+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:46:11.086 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:46:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:46:10.796+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:46:11.086 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:46:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:46:10.796+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:46:11.426 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:46:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:46:11.083+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:46:11.426 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:46:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:46:11.083+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:46:11.426 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:46:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:46:11.083+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:46:11.426 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:46:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:46:11.083+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:46:11.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:46:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:46:11.423+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:46:11.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:46:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:46:11.423+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:46:11.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:46:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:46:11.423+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:46:12.104 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:46:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:46:11.808+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:46:12.104 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:46:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:46:11.808+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:46:12.104 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:46:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:46:11.808+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:46:12.453 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:46:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:46:12.101+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:46:12.453 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:46:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:46:12.101+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:46:12.454 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:46:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:46:12.101+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:46:12.454 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:46:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:46:12.101+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:46:12.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:46:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:46:12.450+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:46:12.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:46:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:46:12.450+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:46:12.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:46:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:46:12.450+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:46:13.145 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:46:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:46:12.809+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:46:13.145 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:46:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:46:12.809+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:46:13.146 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:46:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:46:12.809+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:46:13.437 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:46:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:46:13.141+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:46:13.438 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:46:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:46:13.141+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:46:13.438 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:46:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:46:13.141+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:46:13.438 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:46:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:46:13.141+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:46:13.743 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:46:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:46:13.433+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 39955 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:46:13.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:46:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:46:13.479+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:46:13.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:46:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:46:13.479+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:46:13.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:46:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:46:13.479+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:46:14.125 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:46:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:46:13.764+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:46:14.125 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:46:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:46:13.764+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:46:14.125 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:46:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:46:13.764+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:46:14.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:46:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:46:14.122+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:46:14.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:46:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:46:14.122+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:46:14.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:46:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:46:14.122+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:46:14.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:46:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:46:14.122+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:46:14.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:46:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:46:14.517+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:46:14.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:46:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:46:14.517+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:46:14.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:46:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:46:14.517+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:46:14.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:46:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:46:14.755+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:46:14.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:46:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:46:14.755+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:46:14.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:46:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:46:14.755+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:46:15.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:46:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:46:15.167+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:46:15.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:46:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:46:15.167+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:46:15.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:46:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:46:15.167+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:46:15.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:46:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:46:15.167+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:46:15.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:46:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:46:15.762+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:46:15.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:46:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:46:15.762+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:46:15.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:46:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:46:15.762+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:46:15.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:46:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:46:15.530+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:46:15.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:46:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:46:15.530+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:46:15.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:46:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:46:15.530+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:46:16.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:46:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:46:16.182+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:46:16.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:46:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:46:16.182+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:46:16.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:46:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:46:16.182+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:46:16.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:46:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:46:16.182+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:46:16.776 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:46:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:46:16.513+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:46:16.776 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:46:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:46:16.513+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:46:16.776 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:46:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:46:16.513+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:46:17.168 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:46:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:46:16.773+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:46:17.169 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:46:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:46:16.773+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:46:17.169 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:46:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:46:16.773+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:46:17.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:46:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:46:17.165+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:46:17.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:46:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:46:17.165+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:46:17.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:46:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:46:17.165+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:46:17.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:46:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:46:17.165+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:46:17.813 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:46:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:46:17.554+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:46:17.813 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:46:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:46:17.554+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:46:17.814 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:46:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:46:17.554+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:46:18.216 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:46:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:46:17.810+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:46:18.217 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:46:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:46:17.810+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:46:18.217 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:46:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:46:17.810+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:46:18.493 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:46:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:46:18.439+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 40027 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:46:18.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:46:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:46:18.213+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:46:18.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:46:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:46:18.213+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:46:18.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:46:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:46:18.213+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:46:18.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:46:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:46:18.213+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:46:18.856 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:46:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:46:18.602+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:46:18.856 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:46:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:46:18.602+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:46:18.856 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:46:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:46:18.602+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:46:19.240 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:46:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:46:18.853+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:46:19.240 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:46:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:46:18.853+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:46:19.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:46:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:46:18.853+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:46:19.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:46:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:46:19.237+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:46:19.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:46:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:46:19.237+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:46:19.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:46:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:46:19.237+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:46:19.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:46:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:46:19.237+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:46:19.887 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:46:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:46:19.616+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:46:19.887 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:46:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:46:19.616+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:46:19.887 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:46:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:46:19.616+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:46:19.888 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:46:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:46:19.859+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:46:19.888 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:46:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:46:19.859+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:46:19.888 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:46:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:46:19.859+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:46:20.632 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:46:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:46:20.255+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:46:20.632 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:46:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:46:20.255+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:46:20.633 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:46:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:46:20.255+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:46:20.633 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:46:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:46:20.255+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:46:20.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:46:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:46:20.629+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:46:20.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:46:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:46:20.629+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:46:20.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:46:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:46:20.629+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:46:20.995 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:46:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:46:20.810+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:46:20.995 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:46:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:46:20.810+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:46:20.995 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:46:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:46:20.810+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:46:21.601 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:46:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:46:21.243+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:46:21.601 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:46:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:46:21.243+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:46:21.601 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:46:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:46:21.243+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:46:21.602 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:46:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:46:21.243+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:46:21.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:46:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:46:21.784+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:46:21.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:46:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:46:21.784+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:46:21.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:46:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:46:21.784+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:46:21.995 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:46:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:46:21.598+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:46:21.995 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:46:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:46:21.598+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:46:21.995 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:46:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:46:21.598+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:46:22.635 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:46:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:46:22.290+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:46:22.635 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:46:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:46:22.290+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:46:22.635 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:46:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:46:22.290+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:46:22.636 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:46:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:46:22.290+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:46:22.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:46:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:46:22.766+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:46:22.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:46:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:46:22.766+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:46:22.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:46:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:46:22.766+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:46:22.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:46:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:46:22.631+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:46:22.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:46:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:46:22.631+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:46:22.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:46:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:46:22.631+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:46:23.569 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:46:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:46:23.446+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 40099 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:46:23.570 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:46:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:46:23.312+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:46:23.570 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:46:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:46:23.312+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:46:23.570 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:46:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:46:23.312+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:46:23.570 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:46:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:46:23.312+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:46:23.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:46:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:46:23.814+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:46:23.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:46:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:46:23.814+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:46:23.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:46:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:46:23.814+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:46:23.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:46:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:46:23.676+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:46:23.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:46:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:46:23.676+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:46:23.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:46:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:46:23.676+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:46:24.634 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:46:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:46:24.281+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:46:24.635 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:46:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:46:24.281+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:46:24.635 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:46:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:46:24.281+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:46:24.635 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:46:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:46:24.281+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:46:24.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:46:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:46:24.823+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:46:24.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:46:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:46:24.823+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:46:24.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:46:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:46:24.823+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:46:24.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:46:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:46:24.632+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:46:24.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:46:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:46:24.632+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:46:24.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:46:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:46:24.632+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:46:25.677 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:46:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:46:25.306+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:46:25.677 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:46:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:46:25.306+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:46:25.677 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:46:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:46:25.306+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:46:25.677 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:46:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:46:25.306+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:46:25.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:46:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:46:25.785+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:46:25.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:46:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:46:25.785+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:46:25.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:46:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:46:25.785+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:46:25.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:46:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:46:25.674+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:46:25.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:46:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:46:25.674+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:46:25.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:46:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:46:25.674+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:46:26.650 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:46:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:46:26.281+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:46:26.650 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:46:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:46:26.281+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:46:26.650 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:46:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:46:26.281+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:46:26.650 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:46:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:46:26.281+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:46:26.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:46:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:46:26.735+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:46:26.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:46:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:46:26.735+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:46:26.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:46:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:46:26.735+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:46:26.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:46:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:46:26.647+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:46:26.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:46:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:46:26.647+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:46:26.995 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:46:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:46:26.647+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:46:27.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:46:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:46:27.240+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:46:27.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:46:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:46:27.240+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:46:27.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:46:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:46:27.240+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:46:27.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:46:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:46:27.240+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:46:27.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:46:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:46:27.710+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:46:27.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:46:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:46:27.710+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:46:27.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:46:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:46:27.710+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:46:27.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:46:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:46:27.682+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:46:27.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:46:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:46:27.682+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:46:27.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:46:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:46:27.682+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:46:28.575 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:46:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:46:28.452+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 40171 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:46:28.575 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:46:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:46:28.285+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:46:28.575 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:46:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:46:28.285+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:46:28.575 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:46:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:46:28.285+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:46:28.575 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:46:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:46:28.285+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:46:28.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:46:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:46:28.700+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:46:28.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:46:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:46:28.700+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:46:28.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:46:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:46:28.700+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:46:28.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:46:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:46:28.692+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:46:28.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:46:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:46:28.692+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:46:28.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:46:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:46:28.692+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:46:29.669 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:46:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:46:29.331+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:46:29.669 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:46:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:46:29.331+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:46:29.670 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:46:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:46:29.331+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:46:29.670 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:46:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:46:29.332+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:46:29.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:46:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:46:29.664+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:46:29.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:46:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:46:29.664+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:46:29.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:46:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:46:29.664+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:46:29.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:46:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:46:29.690+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:46:29.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:46:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:46:29.690+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:46:29.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:46:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:46:29.690+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:46:30.650 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:46:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:46:30.294+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:46:30.650 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:46:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:46:30.294+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:46:30.650 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:46:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:46:30.294+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:46:30.651 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:46:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:46:30.294+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:46:30.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:46:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:46:30.693+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:46:30.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:46:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:46:30.693+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:46:30.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:46:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:46:30.693+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:46:30.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:46:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:46:30.704+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:46:30.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:46:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:46:30.704+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:46:30.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:46:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:46:30.704+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:46:31.702 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:46:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:46:31.270+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:46:31.702 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:46:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:46:31.270+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:46:31.702 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:46:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:46:31.270+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:46:31.702 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:46:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:46:31.270+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:46:31.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:46:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:46:31.699+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:46:31.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:46:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:46:31.699+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:46:31.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:46:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:46:31.699+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:46:31.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:46:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:46:31.738+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:46:31.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:46:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:46:31.738+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:46:31.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:46:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:46:31.738+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:46:32.731 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:46:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:46:32.279+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:46:32.731 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:46:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:46:32.279+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:46:32.731 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:46:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:46:32.279+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:46:32.731 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:46:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:46:32.279+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:46:32.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:46:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:46:32.764+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:46:32.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:46:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:46:32.764+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:46:32.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:46:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:46:32.764+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:46:32.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:46:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:46:32.728+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:46:32.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:46:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:46:32.728+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:46:32.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:46:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:46:32.728+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:46:33.582 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:46:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:46:33.458+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 40243 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:46:33.582 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:46:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:46:33.249+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:46:33.583 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:46:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:46:33.249+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:46:33.583 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:46:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:46:33.249+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:46:33.583 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:46:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:46:33.249+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:46:33.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:46:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:46:33.764+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:46:33.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:46:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:46:33.764+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:46:33.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:46:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:46:33.764+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:46:33.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:46:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:46:33.725+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:46:33.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:46:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:46:33.725+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:46:33.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:46:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:46:33.725+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:46:34.723 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:46:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:46:34.298+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:46:34.724 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:46:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:46:34.298+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:46:34.724 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:46:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:46:34.298+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:46:34.724 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:46:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:46:34.298+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:46:34.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:46:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:46:34.801+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:46:34.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:46:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:46:34.801+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:46:34.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:46:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:46:34.801+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:46:34.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:46:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:46:34.721+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:46:34.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:46:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:46:34.721+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:46:34.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:46:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:46:34.721+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:46:35.717 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:46:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:46:35.282+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:46:35.717 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:46:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:46:35.282+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:46:35.718 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:46:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:46:35.282+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:46:35.718 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:46:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:46:35.283+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:46:35.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:46:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:46:35.824+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:46:35.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:46:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:46:35.824+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:46:35.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:46:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:46:35.824+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:46:35.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:46:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:46:35.714+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:46:35.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:46:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:46:35.714+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:46:35.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:46:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:46:35.714+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:46:36.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:46:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:46:36.323+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:46:36.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:46:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:46:36.323+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:46:36.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:46:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:46:36.323+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:46:36.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:46:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:46:36.323+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:46:37.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:46:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:46:36.816+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:46:37.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:46:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:46:36.816+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:46:37.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:46:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:46:36.816+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:46:37.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:46:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:46:36.758+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:46:37.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:46:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:46:36.758+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:46:37.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:46:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:46:36.758+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:46:37.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:46:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:46:37.296+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:46:37.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:46:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:46:37.296+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:46:37.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:46:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:46:37.296+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:46:37.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:46:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:46:37.296+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:46:38.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:46:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:46:37.792+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:46:38.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:46:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:46:37.792+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:46:38.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:46:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:46:37.792+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:46:38.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:46:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:46:37.784+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:46:38.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:46:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:46:37.784+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:46:38.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:46:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:46:37.784+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:46:38.743 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:46:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:46:38.464+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 40315 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:46:38.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:46:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:46:38.345+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:46:38.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:46:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:46:38.345+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:46:38.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:46:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:46:38.345+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:46:38.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:46:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:46:38.345+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:46:39.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:46:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:46:38.782+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:46:39.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:46:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:46:38.782+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:46:39.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:46:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:46:38.782+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:46:39.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:46:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:46:38.819+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:46:39.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:46:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:46:38.819+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:46:39.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:46:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:46:38.819+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:46:39.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:46:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:46:39.354+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:46:39.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:46:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:46:39.354+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:46:39.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:46:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:46:39.354+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:46:39.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:46:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:46:39.354+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:46:40.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:46:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:46:39.799+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:46:40.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:46:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:46:39.799+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:46:40.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:46:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:46:39.799+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:46:40.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:46:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:46:39.834+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:46:40.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:46:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:46:39.834+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:46:40.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:46:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:46:39.834+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:46:40.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:46:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:46:40.321+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:46:40.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:46:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:46:40.321+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:46:40.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:46:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:46:40.321+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:46:40.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:46:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:46:40.321+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:46:41.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:46:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:46:40.790+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:46:41.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:46:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:46:40.790+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:46:41.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:46:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:46:40.790+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:46:41.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:46:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:46:40.804+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:46:41.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:46:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:46:40.804+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:46:41.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:46:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:46:40.804+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:46:41.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:46:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:46:41.359+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:46:41.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:46:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:46:41.359+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:46:41.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:46:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:46:41.359+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:46:41.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:46:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:46:41.359+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:46:42.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:46:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:46:41.835+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:46:42.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:46:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:46:41.835+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:46:42.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:46:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:46:41.835+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:46:42.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:46:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:46:41.759+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:46:42.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:46:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:46:41.759+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:46:42.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:46:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:46:41.759+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:46:42.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:46:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:46:42.340+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:46:42.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:46:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:46:42.340+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:46:42.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:46:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:46:42.340+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:46:42.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:46:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:46:42.340+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:46:43.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:46:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:46:42.876+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:46:43.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:46:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:46:42.876+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:46:43.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:46:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:46:42.876+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:46:43.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:46:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:46:42.780+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:46:43.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:46:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:46:42.780+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:46:43.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:46:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:46:42.780+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:46:43.741 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:46:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:46:43.470+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 40387 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:46:43.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:46:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:46:43.358+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:46:43.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:46:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:46:43.358+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:46:43.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:46:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:46:43.358+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:46:43.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:46:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:46:43.358+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:46:43.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:46:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:46:43.738+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:46:43.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:46:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:46:43.738+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:46:43.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:46:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:46:43.738+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:46:43.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:46:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:46:43.894+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:46:43.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:46:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:46:43.894+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:46:43.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:46:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:46:43.894+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:46:44.715 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:46:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:46:44.408+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:46:44.715 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:46:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:46:44.408+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:46:44.715 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:46:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:46:44.408+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:46:44.716 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:46:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:46:44.408+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:46:44.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:46:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:46:44.713+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:46:44.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:46:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:46:44.713+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:46:44.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:46:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:46:44.713+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:46:44.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:46:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:46:44.923+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:46:44.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:46:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:46:44.923+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:46:44.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:46:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:46:44.923+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:46:45.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:46:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:46:45.409+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:46:45.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:46:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:46:45.409+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:46:45.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:46:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:46:45.409+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:46:45.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:46:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:46:45.409+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:46:46.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:46:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:46:45.759+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:46:46.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:46:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:46:45.759+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:46:46.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:46:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:46:45.759+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:46:46.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:46:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:46:45.916+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:46:46.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:46:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:46:45.916+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:46:46.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:46:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:46:45.916+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:46:46.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:46:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:46:46.406+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:46:46.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:46:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:46:46.406+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:46:46.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:46:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:46:46.406+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:46:46.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:46:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:46:46.406+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:46:47.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:46:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:46:46.790+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:46:47.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:46:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:46:46.790+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:46:47.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:46:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:46:46.790+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:46:47.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:46:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:46:46.930+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:46:47.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:46:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:46:46.930+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:46:47.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:46:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:46:46.930+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:46:47.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:46:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:46:47.372+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:46:47.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:46:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:46:47.372+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:46:47.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:46:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:46:47.372+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:46:47.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:46:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:46:47.372+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:46:48.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:46:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:46:47.904+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:46:48.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:46:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:46:47.904+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:46:48.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:46:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:46:47.904+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:46:48.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:46:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:46:47.820+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:46:48.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:46:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:46:47.820+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:46:48.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:46:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:46:47.820+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:46:48.743 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:46:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:46:48.475+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 40459 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:46:48.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:46:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:46:48.406+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:46:48.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:46:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:46:48.406+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:46:48.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:46:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:46:48.406+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:46:48.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:46:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:46:48.406+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:46:49.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:46:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:46:48.809+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:46:49.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:46:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:46:48.809+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:46:49.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:46:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:46:48.809+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:46:49.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:46:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:46:48.871+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:46:49.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:46:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:46:48.871+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:46:49.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:46:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:46:48.871+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:46:49.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:46:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:46:49.383+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:46:49.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:46:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:46:49.383+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:46:49.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:46:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:46:49.383+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:46:49.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:46:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:46:49.383+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:46:50.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:46:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:46:49.771+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:46:50.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:46:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:46:49.771+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:46:50.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:46:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:46:49.771+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:46:50.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:46:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:46:49.863+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:46:50.245 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:46:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:46:49.863+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:46:50.245 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:46:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:46:49.863+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:46:50.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:46:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:46:50.416+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:46:50.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:46:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:46:50.416+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:46:50.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:46:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:46:50.416+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:46:50.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:46:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:46:50.416+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:46:51.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:46:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:46:50.767+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:46:51.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:46:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:46:50.767+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:46:51.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:46:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:46:50.767+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:46:51.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:46:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:46:50.877+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:46:51.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:46:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:46:50.877+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:46:51.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:46:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:46:50.877+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:46:51.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:46:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:46:51.454+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:46:51.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:46:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:46:51.454+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:46:51.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:46:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:46:51.454+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:46:51.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:46:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:46:51.454+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:46:52.154 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:46:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:46:51.897+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:46:52.154 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:46:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:46:51.897+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:46:52.154 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:46:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:46:51.897+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:46:52.154 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:46:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:46:51.761+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:46:52.155 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:46:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:46:51.761+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:46:52.155 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:46:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:46:51.761+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:46:52.727 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:46:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:46:52.406+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:46:52.727 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:46:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:46:52.406+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:46:52.727 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:46:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:46:52.406+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:46:52.727 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:46:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:46:52.406+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:46:52.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:46:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:46:52.847+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:46:52.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:46:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:46:52.847+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:46:52.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:46:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:46:52.847+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:46:52.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:46:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:46:52.723+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:46:52.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:46:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:46:52.723+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:46:52.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:46:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:46:52.723+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:46:53.740 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:46:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:46:53.482+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 40531 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:46:53.740 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:46:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:46:53.413+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:46:53.740 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:46:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:46:53.413+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:46:53.740 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:46:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:46:53.413+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:46:53.740 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:46:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:46:53.413+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:46:53.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:46:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:46:53.736+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:46:53.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:46:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:46:53.736+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:46:53.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:46:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:46:53.736+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:46:53.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:46:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:46:53.873+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:46:53.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:46:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:46:53.873+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:46:53.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:46:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:46:53.873+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:46:54.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:46:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:46:54.387+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:46:54.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:46:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:46:54.387+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:46:54.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:46:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:46:54.387+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:46:54.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:46:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:46:54.387+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:46:55.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:46:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:46:54.773+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:46:55.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:46:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:46:54.773+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:46:55.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:46:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:46:54.773+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:46:55.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:46:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:46:54.886+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:46:55.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:46:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:46:54.886+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:46:55.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:46:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:46:54.886+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:46:55.730 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:46:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:46:55.416+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:46:55.730 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:46:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:46:55.416+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:46:55.730 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:46:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:46:55.416+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:46:55.731 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:46:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:46:55.416+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:46:55.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:46:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:46:55.727+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:46:55.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:46:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:46:55.727+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:46:55.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:46:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:46:55.727+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:46:55.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:46:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:46:55.844+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:46:55.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:46:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:46:55.844+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:46:55.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:46:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:46:55.844+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:46:56.730 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:46:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:46:56.429+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:46:56.730 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:46:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:46:56.429+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:46:56.730 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:46:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:46:56.429+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:46:56.730 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:46:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:46:56.429+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:46:56.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:46:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:46:56.726+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:46:56.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:46:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:46:56.726+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:46:56.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:46:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:46:56.726+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:46:56.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:46:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:46:56.890+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:46:56.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:46:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:46:56.890+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:46:56.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:46:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:46:56.890+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:46:57.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:46:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:46:57.693+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:46:57.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:46:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:46:57.693+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:46:57.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:46:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:46:57.693+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:46:57.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:46:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:46:57.448+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:46:57.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:46:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:46:57.448+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:46:57.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:46:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:46:57.448+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:46:57.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:46:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:46:57.448+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:46:58.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:46:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:46:57.843+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:46:58.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:46:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:46:57.843+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:46:58.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:46:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:46:57.843+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:46:58.743 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:46:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:46:58.488+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 40603 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:46:58.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:46:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:46:58.665+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:46:58.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:46:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:46:58.665+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:46:58.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:46:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:46:58.665+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:46:58.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:46:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:46:58.459+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:46:58.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:46:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:46:58.459+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:46:58.745 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:46:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:46:58.459+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:46:58.745 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:46:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:46:58.459+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:46:59.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:46:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:46:58.886+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:46:59.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:46:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:46:58.886+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:46:59.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:46:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:46:58.886+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:46:59.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:46:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:46:59.715+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:46:59.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:46:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:46:59.715+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:46:59.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:46:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:46:59.715+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:46:59.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:46:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:46:59.476+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:46:59.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:46:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:46:59.476+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:46:59.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:46:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:46:59.476+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:46:59.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:46:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:46:59.476+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:47:00.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:46:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:46:59.925+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:47:00.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:46:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:46:59.925+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:47:00.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:46:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:46:59.925+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:47:00.729 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:47:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:47:00.430+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:47:00.729 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:47:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:47:00.430+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:47:00.729 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:47:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:47:00.430+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:47:00.729 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:47:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:47:00.430+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:47:00.997 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:47:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:47:00.725+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:47:00.998 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:47:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:47:00.725+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:47:00.998 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:47:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:47:00.725+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:47:00.998 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:47:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:47:00.932+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:47:00.998 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:47:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:47:00.932+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:47:00.998 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:47:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:47:00.932+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:47:01.716 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:47:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:47:01.462+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:47:01.717 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:47:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:47:01.462+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:47:01.717 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:47:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:47:01.462+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:47:01.717 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:47:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:47:01.462+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:47:01.967 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:47:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:47:01.962+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:47:01.967 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:47:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:47:01.962+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:47:01.967 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:47:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:47:01.962+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:47:01.967 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:47:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:47:01.713+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:47:01.967 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:47:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:47:01.713+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:47:01.968 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:47:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:47:01.713+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:47:02.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:47:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:47:02.670+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:47:02.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:47:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:47:02.670+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:47:02.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:47:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:47:02.670+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:47:02.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:47:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:47:02.433+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:47:02.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:47:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:47:02.433+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:47:02.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:47:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:47:02.433+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:47:02.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:47:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:47:02.433+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:47:03.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:47:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:47:02.928+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:47:03.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:47:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:47:02.928+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:47:03.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:47:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:47:02.928+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:47:03.743 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:47:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:47:03.494+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 40675 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:47:03.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:47:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:47:03.478+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:47:03.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:47:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:47:03.478+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:47:03.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:47:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:47:03.478+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:47:03.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:47:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:47:03.478+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:47:03.745 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:47:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:47:03.636+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:47:03.745 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:47:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:47:03.636+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:47:03.745 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:47:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:47:03.636+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:47:04.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:47:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:47:03.902+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:47:04.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:47:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:47:03.902+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:47:04.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:47:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:47:03.902+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:47:04.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:47:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:47:04.593+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:47:04.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:47:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:47:04.593+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:47:04.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:47:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:47:04.593+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:47:04.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:47:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:47:04.518+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:47:04.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:47:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:47:04.518+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:47:04.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:47:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:47:04.518+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:47:04.745 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:47:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:47:04.518+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:47:05.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:47:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:47:04.899+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:47:05.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:47:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:47:04.899+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:47:05.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:47:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:47:04.899+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:47:05.923 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:47:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:47:05.613+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:47:05.923 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:47:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:47:05.613+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:47:05.923 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:47:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:47:05.613+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:47:05.923 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:47:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:47:05.519+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:47:05.923 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:47:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:47:05.519+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:47:05.924 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:47:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:47:05.519+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:47:05.924 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:47:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:47:05.519+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:47:06.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:47:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:47:05.919+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:47:06.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:47:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:47:05.919+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:47:06.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:47:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:47:05.919+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:47:06.898 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:47:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:47:06.649+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:47:06.898 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:47:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:47:06.649+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:47:06.898 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:47:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:47:06.649+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:47:06.899 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:47:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:47:06.541+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:47:06.899 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:47:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:47:06.541+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:47:06.899 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:47:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:47:06.541+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:47:06.899 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:47:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:47:06.541+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:47:07.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:47:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:47:06.895+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:47:07.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:47:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:47:06.895+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:47:07.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:47:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:47:06.895+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:47:07.872 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:47:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:47:07.645+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:47:07.872 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:47:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:47:07.645+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:47:07.872 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:47:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:47:07.645+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:47:07.873 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:47:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:47:07.582+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:47:07.873 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:47:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:47:07.582+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:47:07.873 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:47:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:47:07.582+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:47:07.873 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:47:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:47:07.582+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:47:08.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:47:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:47:07.868+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:47:08.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:47:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:47:07.868+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:47:08.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:47:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:47:07.868+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:47:08.847 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:47:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:47:08.500+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 40747 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:47:08.848 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:47:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:47:08.628+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:47:08.848 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:47:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:47:08.628+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:47:08.848 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:47:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:47:08.628+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:47:08.848 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:47:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:47:08.606+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:47:08.848 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:47:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:47:08.606+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:47:08.849 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:47:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:47:08.606+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:47:08.849 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:47:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:47:08.606+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:47:09.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:47:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:47:08.843+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:47:09.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:47:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:47:08.843+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:47:09.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:47:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:47:08.843+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:47:09.884 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:47:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:47:09.881+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:47:09.884 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:47:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:47:09.881+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:47:09.884 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:47:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:47:09.881+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:47:09.885 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:47:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:47:09.671+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:47:09.885 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:47:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:47:09.671+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:47:09.885 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:47:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:47:09.671+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:47:09.885 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:47:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:47:09.631+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:47:09.885 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:47:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:47:09.631+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:47:09.885 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:47:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:47:09.631+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:47:09.885 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:47:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:47:09.631+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:47:10.861 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:47:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:47:10.616+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:47:10.862 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:47:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:47:10.616+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:47:10.862 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:47:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:47:10.616+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:47:10.862 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:47:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:47:10.616+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:47:10.862 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:47:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:47:10.692+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:47:10.863 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:47:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:47:10.692+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:47:10.863 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:47:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:47:10.692+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:47:11.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:47:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:47:10.858+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:47:11.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:47:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:47:10.858+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:47:11.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:47:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:47:10.858+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:47:11.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:47:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:47:11.810+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:47:11.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:47:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:47:11.810+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:47:11.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:47:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:47:11.810+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:47:11.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:47:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:47:11.714+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:47:11.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:47:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:47:11.714+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:47:11.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:47:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:47:11.714+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:47:11.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:47:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:47:11.630+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:47:11.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:47:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:47:11.630+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:47:11.995 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:47:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:47:11.630+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:47:11.995 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:47:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:47:11.630+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:47:12.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:47:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:47:12.765+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:47:12.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:47:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:47:12.765+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:47:12.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:47:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:47:12.765+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:47:12.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:47:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:47:12.653+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:47:12.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:47:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:47:12.653+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:47:12.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:47:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:47:12.653+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:47:12.995 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:47:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:47:12.653+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:47:12.995 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:47:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:47:12.719+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:47:12.995 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:47:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:47:12.719+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:47:12.995 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:47:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:47:12.719+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:47:13.769 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:47:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:47:13.506+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 40819 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:47:13.770 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:47:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:47:13.710+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:47:13.770 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:47:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:47:13.710+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:47:13.770 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:47:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:47:13.710+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:47:13.770 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:47:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:47:13.665+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:47:13.770 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:47:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:47:13.665+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:47:13.771 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:47:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:47:13.665+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:47:13.771 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:47:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:47:13.665+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:47:14.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:47:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:47:13.766+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:47:14.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:47:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:47:13.766+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:47:14.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:47:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:47:13.766+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:47:14.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:47:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:47:14.766+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:47:14.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:47:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:47:14.766+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:47:14.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:47:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:47:14.766+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:47:14.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:47:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:47:14.676+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:47:14.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:47:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:47:14.676+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:47:14.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:47:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:47:14.676+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:47:14.995 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:47:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:47:14.699+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:47:14.995 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:47:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:47:14.699+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:47:14.995 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:47:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:47:14.699+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:47:14.995 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:47:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:47:14.699+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:47:15.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:47:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:47:15.765+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:47:15.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:47:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:47:15.765+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:47:15.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:47:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:47:15.765+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:47:15.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:47:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:47:15.651+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:47:15.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:47:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:47:15.651+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:47:15.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:47:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:47:15.651+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:47:15.995 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:47:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:47:15.677+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:47:15.995 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:47:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:47:15.677+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:47:15.995 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:47:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:47:15.677+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:47:15.995 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:47:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:47:15.677+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:47:16.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:47:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:47:16.620+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:47:16.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:47:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:47:16.620+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:47:16.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:47:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:47:16.620+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:47:16.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:47:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:47:16.778+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:47:16.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:47:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:47:16.778+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:47:16.995 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:47:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:47:16.778+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:47:16.995 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:47:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:47:16.666+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:47:16.995 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:47:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:47:16.666+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:47:16.995 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:47:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:47:16.666+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:47:16.995 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:47:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:47:16.666+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:47:17.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:47:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:47:17.634+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:47:17.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:47:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:47:17.634+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:47:17.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:47:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:47:17.634+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:47:17.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:47:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:47:17.776+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:47:17.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:47:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:47:17.776+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:47:17.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:47:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:47:17.776+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:47:17.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:47:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:47:17.673+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:47:17.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:47:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:47:17.673+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:47:17.995 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:47:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:47:17.673+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:47:17.995 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:47:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:47:17.673+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:47:18.777 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:47:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:47:18.511+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 40891 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:47:18.777 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:47:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:47:18.605+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:47:18.778 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:47:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:47:18.605+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:47:18.778 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:47:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:47:18.605+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:47:18.778 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:47:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:47:18.661+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:47:18.778 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:47:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:47:18.661+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:47:18.778 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:47:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:47:18.661+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:47:18.778 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:47:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:47:18.661+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:47:19.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:47:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:47:18.774+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:47:19.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:47:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:47:18.774+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:47:19.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:47:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:47:18.774+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:47:19.887 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:47:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:47:19.735+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:47:19.887 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:47:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:47:19.735+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:47:19.887 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:47:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:47:19.735+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:47:19.887 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:47:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:47:19.627+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:47:19.887 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:47:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:47:19.627+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:47:19.888 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:47:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:47:19.627+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:47:19.888 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:47:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:47:19.666+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:47:19.888 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:47:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:47:19.666+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:47:19.888 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:47:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:47:19.666+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:47:19.888 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:47:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:47:19.666+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:47:20.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:47:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:47:20.589+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:47:20.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:47:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:47:20.589+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:47:20.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:47:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:47:20.589+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:47:20.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:47:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:47:20.724+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:47:20.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:47:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:47:20.724+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:47:20.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:47:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:47:20.724+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:47:20.995 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:47:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:47:20.641+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:47:20.995 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:47:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:47:20.641+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:47:20.995 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:47:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:47:20.641+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:47:20.995 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:47:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:47:20.641+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:47:21.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:47:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:47:21.684+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:47:21.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:47:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:47:21.684+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:47:21.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:47:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:47:21.684+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:47:21.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:47:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:47:21.657+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:47:21.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:47:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:47:21.657+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:47:21.995 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:47:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:47:21.657+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:47:21.995 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:47:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:47:21.657+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:47:21.995 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:47:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:47:21.591+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:47:21.995 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:47:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:47:21.591+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:47:21.995 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:47:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:47:21.591+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:47:22.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:47:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:47:22.690+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:47:22.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:47:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:47:22.690+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:47:22.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:47:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:47:22.690+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:47:22.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:47:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:47:22.638+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:47:22.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:47:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:47:22.638+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:47:22.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:47:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:47:22.638+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:47:22.995 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:47:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:47:22.668+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:47:22.995 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:47:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:47:22.668+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:47:22.995 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:47:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:47:22.668+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:47:22.995 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:47:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:47:22.669+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:47:23.993 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:47:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:47:23.519+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 40963 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:47:23.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:47:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:47:23.669+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:47:23.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:47:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:47:23.670+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:47:23.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:47:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:47:23.670+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:47:23.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:47:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:47:23.710+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:47:23.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:47:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:47:23.710+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:47:23.995 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:47:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:47:23.710+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:47:23.995 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:47:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:47:23.710+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:47:23.995 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:47:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:47:23.681+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:47:23.995 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:47:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:47:23.681+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:47:23.995 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:47:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:47:23.681+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:47:24.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:47:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:47:24.706+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:47:24.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:47:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:47:24.706+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:47:24.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:47:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:47:24.706+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:47:24.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:47:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:47:24.640+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:47:24.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:47:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:47:24.640+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:47:24.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:47:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:47:24.640+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:47:24.995 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:47:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:47:24.757+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:47:24.995 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:47:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:47:24.757+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:47:24.995 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:47:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:47:24.757+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:47:24.995 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:47:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:47:24.758+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:47:25.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:47:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:47:25.742+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:47:25.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:47:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:47:25.742+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:47:25.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:47:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:47:25.742+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:47:25.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:47:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:47:25.754+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:47:25.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:47:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:47:25.754+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:47:25.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:47:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:47:25.754+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:47:25.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:47:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:47:25.754+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:47:25.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:47:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:47:25.674+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:47:25.995 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:47:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:47:25.674+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:47:25.995 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:47:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:47:25.674+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:47:26.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:47:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:47:26.689+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:47:26.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:47:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:47:26.689+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:47:26.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:47:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:47:26.689+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:47:26.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:47:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:47:26.740+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:47:26.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:47:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:47:26.740+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:47:26.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:47:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:47:26.740+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:47:26.995 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:47:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:47:26.740+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:47:26.995 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:47:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:47:26.740+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:47:26.995 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:47:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:47:26.740+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:47:26.995 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:47:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:47:26.740+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:47:27.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:47:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:47:27.724+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:47:27.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:47:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:47:27.724+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:47:27.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:47:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:47:27.724+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:47:27.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:47:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:47:27.681+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:47:27.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:47:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:47:27.681+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:47:27.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:47:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:47:27.681+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:47:27.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:47:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:47:27.692+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:47:27.995 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:47:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:47:27.692+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:47:27.995 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:47:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:47:27.692+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:47:27.995 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:47:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:47:27.692+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:47:28.993 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:47:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:47:28.524+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 41035 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:47:28.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:47:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:47:28.676+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:47:28.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:47:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:47:28.676+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:47:28.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:47:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:47:28.676+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:47:28.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:47:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:47:28.676+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:47:28.995 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:47:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:47:28.728+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:47:28.995 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:47:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:47:28.728+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:47:28.995 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:47:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:47:28.728+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:47:28.995 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:47:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:47:28.746+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:47:28.995 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:47:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:47:28.746+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:47:28.995 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:47:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:47:28.746+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:47:29.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:47:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:47:29.763+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:47:29.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:47:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:47:29.763+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:47:29.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:47:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:47:29.763+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:47:29.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:47:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:47:29.722+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:47:29.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:47:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:47:29.722+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:47:29.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:47:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:47:29.722+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:47:29.995 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:47:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:47:29.722+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:47:29.995 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:47:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:47:29.752+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:47:29.995 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:47:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:47:29.752+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:47:29.995 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:47:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:47:29.752+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:47:31.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:47:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:47:30.761+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:47:31.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:47:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:47:30.761+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:47:31.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:47:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:47:30.761+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:47:31.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:47:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:47:30.768+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:47:31.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:47:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:47:30.768+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:47:31.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:47:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:47:30.768+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:47:31.245 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:47:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:47:30.768+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:47:31.245 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:47:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:47:30.782+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:47:31.245 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:47:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:47:30.782+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:47:31.245 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:47:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:47:30.782+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:47:32.090 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:47:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:47:31.792+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:47:32.090 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:47:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:47:31.792+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:47:32.090 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:47:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:47:31.792+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:47:32.090 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:47:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:47:31.768+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:47:32.090 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:47:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:47:31.768+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:47:32.091 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:47:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:47:31.768+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:47:32.091 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:47:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:47:31.768+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:47:32.091 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:47:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:47:31.792+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:47:32.091 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:47:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:47:31.792+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:47:32.091 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:47:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:47:31.792+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:47:32.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:47:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:47:32.778+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:47:32.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:47:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:47:32.778+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:47:32.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:47:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:47:32.778+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:47:32.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:47:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:47:32.743+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:47:32.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:47:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:47:32.743+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:47:32.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:47:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:47:32.743+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:47:32.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:47:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:47:32.725+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:47:32.995 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:47:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:47:32.725+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:47:32.995 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:47:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:47:32.725+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:47:32.995 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:47:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:47:32.725+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:47:33.993 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:47:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:47:33.530+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 41107 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:47:33.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:47:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:47:33.751+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:47:33.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:47:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:47:33.751+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:47:33.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:47:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:47:33.751+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:47:33.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:47:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:47:33.716+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:47:33.995 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:47:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:47:33.716+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:47:33.995 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:47:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:47:33.716+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:47:33.995 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:47:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:47:33.716+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:47:33.995 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:47:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:47:33.699+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:47:33.995 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:47:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:47:33.699+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:47:33.995 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:47:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:47:33.699+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:47:34.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:47:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:47:34.747+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:47:34.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:47:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:47:34.747+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:47:34.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:47:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:47:34.747+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:47:34.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:47:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:47:34.708+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:47:34.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:47:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:47:34.708+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:47:34.995 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:47:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:47:34.708+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:47:34.995 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:47:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:47:34.709+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:47:34.995 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:47:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:47:34.796+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:47:34.995 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:47:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:47:34.796+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:47:34.995 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:47:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:47:34.796+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:47:35.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:47:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:47:35.769+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:47:35.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:47:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:47:35.769+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:47:35.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:47:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:47:35.769+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:47:35.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:47:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:47:35.744+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:47:35.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:47:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:47:35.744+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:47:35.995 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:47:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:47:35.744+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:47:35.995 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:47:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:47:35.666+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:47:35.995 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:47:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:47:35.666+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:47:35.995 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:47:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:47:35.666+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:47:35.995 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:47:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:47:35.666+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:47:36.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:47:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:47:36.706+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:47:36.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:47:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:47:36.706+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:47:36.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:47:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:47:36.706+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:47:36.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:47:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:47:36.751+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:47:36.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:47:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:47:36.751+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:47:36.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:47:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:47:36.751+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:47:36.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:47:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:47:36.670+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:47:36.995 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:47:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:47:36.670+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:47:36.995 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:47:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:47:36.670+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:47:36.995 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:47:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:47:36.670+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:47:37.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:47:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:47:37.677+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:47:37.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:47:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:47:37.677+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:47:37.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:47:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:47:37.677+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:47:37.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:47:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:47:37.796+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:47:37.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:47:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:47:37.796+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:47:37.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:47:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:47:37.796+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:47:37.995 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:47:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:47:37.623+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:47:37.995 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:47:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:47:37.623+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:47:37.995 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:47:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:47:37.623+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:47:37.995 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:47:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:47:37.623+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:47:38.993 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:47:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:47:38.536+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 41179 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:47:38.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:47:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:47:38.757+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:47:38.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:47:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:47:38.757+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:47:38.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:47:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:47:38.757+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:47:38.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:47:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:47:38.646+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:47:38.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:47:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:47:38.646+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:47:38.995 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:47:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:47:38.646+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:47:38.995 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:47:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:47:38.647+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:47:38.995 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:47:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:47:38.689+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:47:38.995 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:47:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:47:38.689+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:47:38.995 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:47:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:47:38.689+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:47:39.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:47:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:47:39.685+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:47:39.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:47:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:47:39.685+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:47:39.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:47:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:47:39.685+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:47:39.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:47:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:47:39.693+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:47:39.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:47:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:47:39.693+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:47:39.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:47:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:47:39.693+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:47:39.995 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:47:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:47:39.693+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:47:39.995 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:47:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:47:39.770+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:47:39.995 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:47:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:47:39.770+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:47:39.995 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:47:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:47:39.770+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:47:40.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:47:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:47:40.745+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:47:40.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:47:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:47:40.745+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:47:40.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:47:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:47:40.745+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:47:40.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:47:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:47:40.712+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:47:40.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:47:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:47:40.712+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:47:40.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:47:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:47:40.712+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:47:40.995 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:47:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:47:40.740+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:47:40.995 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:47:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:47:40.740+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:47:40.995 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:47:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:47:40.740+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:47:40.995 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:47:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:47:40.740+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:47:41.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:47:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:47:41.753+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:47:41.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:47:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:47:41.753+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:47:41.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:47:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:47:41.753+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:47:41.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:47:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:47:41.721+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:47:41.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:47:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:47:41.721+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:47:41.995 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:47:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:47:41.721+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:47:41.995 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:47:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:47:41.733+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:47:41.995 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:47:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:47:41.733+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:47:41.995 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:47:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:47:41.733+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:47:41.995 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:47:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:47:41.734+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:47:42.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:47:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:47:42.767+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:47:42.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:47:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:47:42.767+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:47:42.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:47:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:47:42.767+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:47:42.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:47:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:47:42.754+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:47:42.995 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:47:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:47:42.754+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:47:42.995 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:47:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:47:42.754+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:47:42.995 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:47:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:47:42.684+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:47:42.995 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:47:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:47:42.684+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:47:42.995 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:47:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:47:42.684+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:47:42.995 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:47:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:47:42.684+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:47:43.811 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:47:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:47:43.541+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 41251 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:47:43.812 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:47:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:47:43.706+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:47:43.812 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:47:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:47:43.706+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:47:43.812 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:47:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:47:43.706+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:47:43.812 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:47:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:47:43.642+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:47:43.813 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:47:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:47:43.642+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:47:43.813 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:47:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:47:43.642+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:47:43.813 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:47:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:47:43.643+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:47:44.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:47:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:47:43.808+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:47:44.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:47:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:47:43.808+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:47:44.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:47:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:47:43.808+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:47:44.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:47:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:47:44.612+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:47:44.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:47:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:47:44.612+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:47:44.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:47:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:47:44.612+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:47:44.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:47:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:47:44.612+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:47:44.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:47:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:47:44.739+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:47:44.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:47:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:47:44.739+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:47:44.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:47:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:47:44.739+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:47:44.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:47:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:47:44.833+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:47:44.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:47:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:47:44.833+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:47:44.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:47:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:47:44.833+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:47:45.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:47:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:47:45.745+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:47:45.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:47:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:47:45.745+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:47:45.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:47:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:47:45.745+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:47:45.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:47:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:47:45.660+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:47:45.995 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:47:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:47:45.660+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:47:45.995 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:47:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:47:45.660+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:47:45.995 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:47:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:47:45.660+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:47:45.995 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:47:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:47:45.853+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:47:45.995 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:47:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:47:45.853+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:47:45.995 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:47:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:47:45.853+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:47:46.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:47:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:47:46.773+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:47:46.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:47:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:47:46.773+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:47:46.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:47:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:47:46.773+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:47:46.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:47:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:47:46.866+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:47:46.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:47:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:47:46.866+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:47:46.995 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:47:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:47:46.866+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:47:46.995 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:47:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:47:46.633+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:47:46.995 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:47:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:47:46.633+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:47:46.995 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:47:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:47:46.633+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:47:46.995 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:47:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:47:46.633+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:47:47.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:47:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:47:47.756+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:47:47.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:47:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:47:47.756+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:47:47.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:47:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:47:47.756+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:47:47.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:47:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:47:47.882+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:47:47.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:47:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:47:47.882+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:47:47.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:47:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:47:47.882+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:47:47.995 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:47:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:47:47.664+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:47:47.995 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:47:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:47:47.664+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:47:47.995 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:47:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:47:47.664+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:47:47.995 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:47:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:47:47.664+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:47:48.848 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:47:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:47:48.548+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 41323 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:47:48.848 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:47:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:47:48.722+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:47:48.848 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:47:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:47:48.722+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:47:48.849 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:47:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:47:48.722+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:47:48.849 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:47:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:47:48.684+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:47:48.849 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:47:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:47:48.684+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:47:48.849 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:47:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:47:48.684+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:47:48.849 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:47:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:47:48.684+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:47:49.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:47:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:47:48.844+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:47:49.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:47:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:47:48.844+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:47:49.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:47:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:47:48.844+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:47:49.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:47:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:47:49.709+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:47:49.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:47:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:47:49.709+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:47:49.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:47:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:47:49.709+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:47:49.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:47:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:47:49.877+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:47:49.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:47:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:47:49.877+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:47:49.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:47:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:47:49.877+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:47:49.995 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:47:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:47:49.660+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:47:49.995 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:47:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:47:49.660+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:47:49.995 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:47:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:47:49.660+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:47:49.995 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:47:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:47:49.660+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:47:50.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:47:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:47:50.847+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:47:50.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:47:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:47:50.847+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:47:50.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:47:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:47:50.847+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:47:50.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:47:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:47:50.706+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:47:50.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:47:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:47:50.706+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:47:50.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:47:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:47:50.706+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:47:50.995 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:47:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:47:50.700+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:47:50.995 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:47:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:47:50.700+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:47:50.995 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:47:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:47:50.700+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:47:50.995 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:47:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:47:50.701+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:47:51.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:47:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:47:51.692+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:47:51.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:47:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:47:51.692+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:47:51.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:47:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:47:51.692+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:47:51.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:47:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:47:51.692+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:47:51.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:47:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:47:51.665+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:47:51.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:47:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:47:51.665+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:47:51.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:47:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:47:51.665+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:47:51.995 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:47:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:47:51.846+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:47:51.995 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:47:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:47:51.846+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:47:51.995 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:47:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:47:51.846+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:47:52.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:47:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:47:52.652+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:47:52.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:47:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:47:52.652+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:47:52.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:47:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:47:52.652+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:47:52.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:47:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:47:52.652+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:47:52.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:47:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:47:52.683+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:47:52.995 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:47:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:47:52.683+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:47:52.995 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:47:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:47:52.683+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:47:52.995 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:47:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:47:52.871+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:47:52.995 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:47:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:47:52.871+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:47:52.995 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:47:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:47:52.871+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:47:53.894 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:47:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:47:53.553+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 41395 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:47:53.894 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:47:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:47:53.674+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:47:53.894 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:47:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:47:53.674+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:47:53.895 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:47:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:47:53.674+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:47:53.895 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:47:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:47:53.683+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:47:53.895 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:47:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:47:53.683+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:47:53.895 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:47:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:47:53.683+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:47:53.896 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:47:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:47:53.683+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:47:54.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:47:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:47:53.890+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:47:54.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:47:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:47:53.890+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:47:54.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:47:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:47:53.890+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:47:54.941 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:47:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:47:54.638+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:47:54.941 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:47:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:47:54.638+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:47:54.942 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:47:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:47:54.638+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:47:54.942 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:47:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:47:54.731+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:47:54.942 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:47:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:47:54.731+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:47:54.942 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:47:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:47:54.731+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:47:54.942 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:47:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:47:54.731+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:47:55.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:47:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:47:54.937+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:47:55.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:47:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:47:54.937+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:47:55.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:47:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:47:54.937+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:47:55.938 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:47:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:47:55.604+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:47:55.938 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:47:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:47:55.604+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:47:55.938 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:47:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:47:55.604+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:47:55.938 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:47:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:47:55.691+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:47:55.938 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:47:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:47:55.691+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:47:55.939 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:47:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:47:55.691+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:47:55.939 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:47:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:47:55.691+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:47:56.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:47:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:47:55.934+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:47:56.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:47:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:47:55.934+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:47:56.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:47:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:47:55.934+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:47:56.894 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:47:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:47:56.613+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:47:56.894 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:47:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:47:56.613+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:47:56.894 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:47:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:47:56.613+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:47:56.895 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:47:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:47:56.682+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:47:56.895 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:47:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:47:56.682+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:47:56.895 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:47:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:47:56.682+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:47:56.895 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:47:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:47:56.682+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:47:57.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:47:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:47:56.890+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:47:57.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:47:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:47:56.890+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:47:57.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:47:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:47:56.890+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:47:57.917 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:47:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:47:57.618+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:47:57.918 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:47:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:47:57.618+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:47:57.918 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:47:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:47:57.618+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:47:57.918 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:47:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:47:57.719+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:47:57.919 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:47:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:47:57.719+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:47:57.919 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:47:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:47:57.719+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:47:57.919 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:47:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:47:57.719+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:47:58.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:47:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:47:57.914+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:47:58.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:47:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:47:57.914+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:47:58.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:47:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:47:57.914+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:47:58.949 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:47:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:47:58.559+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 41468 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:47:58.950 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:47:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:47:58.659+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:47:58.950 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:47:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:47:58.659+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:47:58.950 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:47:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:47:58.659+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:47:58.951 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:47:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:47:58.695+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:47:58.951 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:47:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:47:58.695+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:47:58.951 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:47:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:47:58.695+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:47:58.951 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:47:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:47:58.695+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:47:59.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:47:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:47:58.946+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:47:59.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:47:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:47:58.946+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:47:59.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:47:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:47:58.946+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:47:59.987 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:47:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:47:59.693+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:47:59.987 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:47:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:47:59.693+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:47:59.987 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:47:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:47:59.693+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:47:59.988 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:47:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:47:59.696+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:47:59.988 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:47:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:47:59.696+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:47:59.988 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:47:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:47:59.696+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:47:59.988 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:47:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:47:59.696+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:48:00.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:47:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:47:59.984+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:48:00.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:47:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:47:59.984+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:48:00.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:47:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:47:59.984+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:48:00.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:48:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:48:00.696+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:48:00.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:48:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:48:00.696+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:48:00.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:48:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:48:00.696+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:48:00.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:48:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:48:00.716+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:48:00.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:48:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:48:00.716+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:48:00.995 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:48:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:48:00.716+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:48:00.995 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:48:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:48:00.716+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:48:00.995 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:48:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:48:00.937+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:48:00.995 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:48:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:48:00.937+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:48:00.995 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:48:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:48:00.937+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:48:01.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:48:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:48:01.694+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:48:01.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:48:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:48:01.694+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:48:01.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:48:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:48:01.694+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:48:01.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:48:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:48:01.706+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:48:01.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:48:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:48:01.706+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:48:01.995 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:48:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:48:01.706+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:48:01.995 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:48:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:48:01.706+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:48:01.995 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:48:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:48:01.907+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:48:01.995 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:48:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:48:01.907+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:48:01.995 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:48:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:48:01.907+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:48:02.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:48:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:48:02.719+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:48:02.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:48:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:48:02.719+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:48:02.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:48:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:48:02.719+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:48:02.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:48:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:48:02.869+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:48:02.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:48:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:48:02.869+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:48:02.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:48:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:48:02.869+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:48:02.995 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:48:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:48:02.711+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:48:02.995 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:48:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:48:02.711+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:48:02.995 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:48:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:48:02.711+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:48:02.995 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:48:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:48:02.711+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:48:03.844 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:48:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:48:03.566+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 41540 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:48:03.844 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:48:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:48:03.759+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:48:03.844 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:48:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:48:03.759+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:48:03.844 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:48:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:48:03.759+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:48:03.845 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:48:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:48:03.719+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:48:03.845 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:48:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:48:03.719+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:48:03.845 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:48:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:48:03.719+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:48:03.845 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:48:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:48:03.719+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:48:04.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:48:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:48:03.840+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:48:04.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:48:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:48:03.840+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:48:04.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:48:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:48:03.840+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:48:04.979 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:48:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:48:04.743+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:48:04.980 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:48:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:48:04.743+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:48:04.980 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:48:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:48:04.743+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:48:04.980 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:48:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:48:04.719+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:48:04.980 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:48:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:48:04.719+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:48:04.981 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:48:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:48:04.719+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:48:04.981 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:48:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:48:04.719+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:48:04.981 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:48:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:48:04.865+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:48:04.981 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:48:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:48:04.865+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:48:04.981 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:48:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:48:04.865+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:48:05.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:48:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:48:05.752+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:48:05.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:48:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:48:05.752+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:48:05.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:48:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:48:05.752+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:48:05.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:48:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:48:05.873+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:48:05.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:48:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:48:05.873+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:48:05.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:48:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:48:05.873+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:48:05.995 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:48:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:48:05.721+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:48:05.995 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:48:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:48:05.721+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:48:05.995 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:48:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:48:05.721+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:48:05.995 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:48:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:48:05.721+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:48:06.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:48:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:48:06.836+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:48:06.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:48:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:48:06.836+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:48:06.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:48:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:48:06.836+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:48:06.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:48:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:48:06.780+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:48:06.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:48:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:48:06.780+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:48:06.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:48:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:48:06.780+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:48:06.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:48:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:48:06.736+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:48:06.995 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:48:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:48:06.736+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:48:06.995 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:48:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:48:06.736+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:48:06.995 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:48:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:48:06.736+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:48:08.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:48:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:48:07.857+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:48:08.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:48:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:48:07.857+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:48:08.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:48:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:48:07.857+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:48:08.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:48:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:48:07.808+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:48:08.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:48:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:48:07.808+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:48:08.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:48:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:48:07.808+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:48:08.245 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:48:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:48:07.785+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:48:08.245 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:48:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:48:07.785+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:48:08.245 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:48:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:48:07.785+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:48:08.245 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:48:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:48:07.785+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:48:08.834 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:48:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:48:08.572+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 41612 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:48:09.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:48:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:48:08.830+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:48:09.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:48:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:48:08.830+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:48:09.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:48:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:48:08.830+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:48:09.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:48:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:48:08.830+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:48:09.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:48:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:48:08.830+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:48:09.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:48:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:48:08.830+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:48:09.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:48:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:48:08.830+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:48:09.245 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:48:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:48:08.903+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:48:09.245 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:48:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:48:08.903+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:48:09.245 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:48:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:48:08.903+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:48:10.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:48:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:48:09.809+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:48:10.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:48:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:48:09.809+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:48:10.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:48:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:48:09.809+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:48:10.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:48:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:48:09.891+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:48:10.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:48:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:48:09.891+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:48:10.245 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:48:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:48:09.891+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:48:10.245 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:48:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:48:09.881+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:48:10.245 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:48:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:48:09.881+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:48:10.245 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:48:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:48:09.881+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:48:10.245 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:48:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:48:09.881+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:48:11.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:48:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:48:10.851+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:48:11.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:48:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:48:10.851+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:48:11.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:48:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:48:10.851+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:48:11.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:48:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:48:10.806+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:48:11.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:48:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:48:10.806+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:48:11.245 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:48:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:48:10.806+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:48:11.245 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:48:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:48:10.924+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:48:11.245 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:48:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:48:10.924+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:48:11.245 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:48:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:48:10.924+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:48:11.245 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:48:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:48:10.924+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:48:12.028 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:48:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:48:11.837+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:48:12.028 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:48:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:48:11.837+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:48:12.028 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:48:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:48:11.837+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:48:12.029 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:48:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:48:11.826+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:48:12.029 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:48:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:48:11.826+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:48:12.029 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:48:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:48:11.826+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:48:12.032 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:48:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:48:11.886+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:48:12.032 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:48:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:48:11.886+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:48:12.032 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:48:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:48:11.886+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:48:12.032 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:48:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:48:11.886+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:48:13.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:48:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:48:12.913+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:48:13.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:48:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:48:12.913+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:48:13.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:48:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:48:12.913+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:48:13.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:48:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:48:12.913+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:48:13.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:48:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:48:12.836+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:48:13.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:48:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:48:12.836+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:48:13.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:48:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:48:12.836+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:48:13.245 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:48:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:48:12.820+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:48:13.245 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:48:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:48:12.820+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:48:13.245 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:48:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:48:12.820+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:48:13.901 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:48:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:48:13.579+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 41684 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:48:13.901 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:48:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:48:13.793+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:48:13.901 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:48:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:48:13.793+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:48:13.901 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:48:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:48:13.793+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:48:13.902 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:48:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:48:13.793+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:48:13.902 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:48:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:48:13.793+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:48:13.902 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:48:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:48:13.793+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:48:14.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:48:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:48:13.898+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:48:14.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:48:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:48:13.898+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:48:14.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:48:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:48:13.898+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:48:14.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:48:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:48:13.898+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:48:15.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:48:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:48:14.908+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:48:15.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:48:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:48:14.908+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:48:15.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:48:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:48:14.908+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:48:15.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:48:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:48:14.909+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:48:15.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:48:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:48:14.782+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:48:15.245 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:48:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:48:14.782+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:48:15.245 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:48:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:48:14.782+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:48:15.245 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:48:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:48:14.746+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:48:15.245 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:48:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:48:14.746+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:48:15.245 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:48:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:48:14.746+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:48:16.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:48:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:48:15.930+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:48:16.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:48:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:48:15.930+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:48:16.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:48:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:48:15.930+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:48:16.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:48:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:48:15.930+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:48:16.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:48:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:48:15.759+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:48:16.245 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:48:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:48:15.759+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:48:16.245 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:48:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:48:15.759+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:48:16.245 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:48:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:48:15.743+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:48:16.245 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:48:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:48:15.743+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:48:16.245 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:48:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:48:15.743+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:48:16.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:48:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:48:16.953+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:48:16.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:48:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:48:16.953+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:48:16.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:48:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:48:16.953+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:48:16.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:48:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:48:16.953+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:48:16.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:48:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:48:16.704+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:48:16.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:48:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:48:16.704+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:48:16.995 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:48:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:48:16.704+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:48:16.995 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:48:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:48:16.716+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:48:16.995 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:48:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:48:16.716+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:48:16.995 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:48:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:48:16.716+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:48:17.928 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:48:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:48:17.720+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:48:17.929 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:48:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:48:17.720+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:48:17.929 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:48:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:48:17.720+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:48:17.929 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:48:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:48:17.656+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:48:17.929 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:48:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:48:17.656+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:48:17.929 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:48:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:48:17.656+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:48:18.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:48:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:48:17.926+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:48:18.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:48:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:48:17.926+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:48:18.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:48:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:48:17.926+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:48:18.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:48:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:48:17.926+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:48:18.893 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:48:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:48:18.586+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 41756 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:48:18.894 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:48:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:48:18.739+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:48:18.894 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:48:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:48:18.739+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:48:18.894 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:48:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:48:18.739+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:48:18.894 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:48:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:48:18.614+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:48:18.895 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:48:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:48:18.614+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:48:18.895 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:48:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:48:18.614+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:48:19.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:48:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:48:18.890+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:48:19.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:48:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:48:18.890+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:48:19.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:48:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:48:18.891+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:48:19.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:48:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:48:18.891+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:48:19.887 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:48:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:48:19.761+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:48:19.888 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:48:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:48:19.761+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:48:19.888 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:48:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:48:19.761+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:48:19.888 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:48:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:48:19.626+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:48:19.888 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:48:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:48:19.626+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:48:19.889 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:48:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:48:19.626+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:48:19.889 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:48:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:48:19.849+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:48:19.889 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:48:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:48:19.849+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:48:19.889 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:48:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:48:19.849+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:48:19.889 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:48:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:48:19.849+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:48:20.884 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:48:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:48:20.746+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:48:20.884 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:48:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:48:20.746+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:48:20.884 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:48:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:48:20.746+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:48:20.885 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:48:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:48:20.596+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:48:20.885 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:48:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:48:20.596+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:48:20.885 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:48:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:48:20.596+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:48:21.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:48:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:48:20.881+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:48:21.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:48:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:48:20.881+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:48:21.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:48:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:48:20.881+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:48:21.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:48:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:48:20.881+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:48:21.841 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:48:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:48:21.728+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:48:21.841 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:48:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:48:21.728+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:48:21.841 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:48:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:48:21.728+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:48:21.841 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:48:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:48:21.559+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:48:21.842 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:48:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:48:21.559+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:48:21.842 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:48:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:48:21.559+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:48:22.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:48:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:48:21.877+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:48:22.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:48:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:48:21.877+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:48:22.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:48:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:48:21.877+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:48:22.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:48:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:48:21.877+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:48:22.906 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:48:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:48:22.717+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:48:22.906 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:48:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:48:22.717+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:48:22.907 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:48:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:48:22.717+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:48:22.907 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:48:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:48:22.577+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:48:22.907 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:48:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:48:22.577+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:48:22.907 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:48:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:48:22.577+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:48:23.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:48:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:48:22.903+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:48:23.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:48:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:48:22.903+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:48:23.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:48:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:48:22.903+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:48:23.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:48:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:48:22.903+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:48:23.865 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:48:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:48:23.592+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 41828 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:48:23.865 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:48:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:48:23.862+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:48:23.866 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:48:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:48:23.862+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:48:23.866 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:48:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:48:23.862+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:48:23.866 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:48:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:48:23.753+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:48:23.866 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:48:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:48:23.753+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:48:23.866 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:48:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:48:23.753+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:48:23.866 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:48:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:48:23.625+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:48:23.867 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:48:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:48:23.625+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:48:23.867 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:48:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:48:23.625+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:48:24.642 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:48:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:48:23.862+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:48:24.903 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:48:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:48:24.710+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:48:24.904 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:48:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:48:24.710+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:48:24.904 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:48:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:48:24.710+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:48:24.904 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:48:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:48:24.639+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:48:24.904 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:48:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:48:24.639+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:48:24.904 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:48:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:48:24.639+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:48:25.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:48:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:48:24.901+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:48:25.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:48:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:48:24.901+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:48:25.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:48:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:48:24.901+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:48:25.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:48:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:48:24.901+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:48:25.891 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:48:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:48:25.750+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:48:25.891 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:48:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:48:25.750+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:48:25.892 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:48:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:48:25.750+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:48:25.892 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:48:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:48:25.599+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:48:25.892 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:48:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:48:25.599+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:48:25.892 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:48:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:48:25.599+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:48:26.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:48:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:48:25.889+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:48:26.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:48:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:48:25.889+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:48:26.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:48:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:48:25.889+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:48:26.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:48:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:48:25.889+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:48:26.938 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:48:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:48:26.731+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:48:26.938 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:48:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:48:26.731+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:48:26.938 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:48:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:48:26.731+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:48:26.938 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:48:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:48:26.592+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:48:26.939 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:48:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:48:26.592+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:48:26.939 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:48:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:48:26.592+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:48:27.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:48:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:48:26.935+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:48:27.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:48:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:48:26.935+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:48:27.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:48:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:48:26.935+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:48:27.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:48:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:48:26.935+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:48:27.971 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:48:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:48:27.690+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:48:27.972 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:48:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:48:27.690+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:48:27.972 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:48:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:48:27.690+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:48:27.972 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:48:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:48:27.599+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:48:27.972 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:48:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:48:27.599+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:48:27.972 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:48:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:48:27.599+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:48:28.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:48:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:48:27.968+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:48:28.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:48:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:48:27.968+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:48:28.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:48:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:48:27.968+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:48:28.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:48:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:48:27.969+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:48:28.987 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:48:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:48:28.598+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 41900 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:48:28.988 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:48:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:48:28.714+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:48:28.988 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:48:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:48:28.714+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:48:28.988 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:48:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:48:28.714+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:48:28.988 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:48:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:48:28.552+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:48:28.988 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:48:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:48:28.552+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:48:28.988 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:48:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:48:28.552+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:48:29.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:48:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:48:28.984+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:48:29.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:48:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:48:28.984+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:48:29.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:48:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:48:28.984+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:48:29.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:48:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:48:28.984+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:48:29.886 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:48:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:48:29.704+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:48:29.887 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:48:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:48:29.704+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:48:29.887 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:48:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:48:29.704+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:48:29.887 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:48:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:48:29.507+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:48:29.887 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:48:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:48:29.507+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:48:29.887 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:48:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:48:29.507+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:48:30.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:48:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:48:29.967+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:48:30.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:48:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:48:29.967+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:48:30.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:48:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:48:29.967+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:48:30.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:48:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:48:29.967+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:48:30.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:48:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:48:30.459+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:48:30.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:48:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:48:30.459+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:48:30.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:48:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:48:30.459+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:48:31.005 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:48:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:48:30.741+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:48:31.005 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:48:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:48:30.741+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:48:31.005 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:48:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:48:30.741+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:48:31.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:48:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:48:31.002+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:48:31.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:48:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:48:31.002+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:48:31.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:48:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:48:31.002+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:48:31.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:48:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:48:31.002+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:48:31.767 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:48:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:48:31.502+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:48:31.767 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:48:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:48:31.502+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:48:31.767 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:48:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:48:31.502+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:48:32.047 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:48:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:48:31.764+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:48:32.048 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:48:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:48:31.764+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:48:32.048 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:48:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:48:31.764+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:48:32.476 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:48:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:48:32.045+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:48:32.476 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:48:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:48:32.045+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:48:32.476 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:48:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:48:32.045+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:48:32.476 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:48:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:48:32.045+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:48:32.730 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:48:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:48:32.473+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:48:32.730 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:48:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:48:32.473+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:48:32.730 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:48:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:48:32.473+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:48:32.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:48:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:48:32.727+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:48:32.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:48:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:48:32.727+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:48:32.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:48:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:48:32.727+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:48:33.478 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:48:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:48:33.070+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:48:33.479 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:48:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:48:33.070+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:48:33.479 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:48:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:48:33.070+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:48:33.479 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:48:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:48:33.070+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:48:33.730 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:48:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:48:33.605+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 41972 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:48:33.731 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:48:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:48:33.476+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:48:33.731 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:48:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:48:33.476+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:48:33.731 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:48:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:48:33.476+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:48:33.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:48:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:48:33.731+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:48:33.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:48:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:48:33.731+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:48:33.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:48:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:48:33.731+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:48:34.435 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:48:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:48:34.030+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:48:34.436 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:48:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:48:34.030+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:48:34.436 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:48:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:48:34.030+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:48:34.436 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:48:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:48:34.030+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:48:34.696 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:48:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:48:34.433+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:48:34.696 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:48:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:48:34.433+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:48:34.696 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:48:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:48:34.433+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:48:34.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:48:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:48:34.693+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:48:34.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:48:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:48:34.693+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:48:34.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:48:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:48:34.693+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:48:35.437 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:48:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:48:35.037+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:48:35.437 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:48:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:48:35.037+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:48:35.437 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:48:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:48:35.037+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:48:35.437 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:48:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:48:35.037+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:48:35.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:48:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:48:35.653+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:48:35.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:48:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:48:35.653+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:48:35.744 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:48:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:48:35.653+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:48:35.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:48:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:48:35.433+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:48:35.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:48:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:48:35.433+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:48:35.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:48:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:48:35.433+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:48:36.472 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:48:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:48:36.086+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:48:36.473 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:48:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:48:36.086+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:48:36.473 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:48:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:48:36.086+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:48:36.473 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:48:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:48:36.086+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:48:36.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:48:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:48:36.610+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:48:36.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:48:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:48:36.611+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:48:36.744 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:48:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:48:36.611+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:48:36.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:48:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:48:36.469+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:48:36.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:48:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:48:36.469+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:48:36.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:48:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:48:36.469+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:48:37.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:48:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:48:37.101+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:48:37.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:48:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:48:37.101+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:48:37.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:48:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:48:37.101+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:48:37.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:48:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:48:37.101+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:48:37.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:48:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:48:37.623+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:48:37.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:48:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:48:37.623+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:48:37.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:48:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:48:37.623+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:48:37.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:48:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:48:37.505+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:48:37.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:48:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:48:37.505+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:48:37.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:48:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:48:37.505+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:48:38.471 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:48:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:48:38.101+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:48:38.471 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:48:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:48:38.101+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:48:38.471 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:48:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:48:38.101+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:48:38.471 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:48:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:48:38.101+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:48:38.735 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:48:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:48:38.610+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 42044 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:48:38.736 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:48:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:48:38.671+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:48:38.736 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:48:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:48:38.671+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:48:38.736 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:48:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:48:38.671+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:48:38.736 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:48:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:48:38.467+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:48:38.737 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:48:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:48:38.467+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:48:38.737 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:48:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:48:38.467+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:48:39.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:48:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:48:39.098+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:48:39.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:48:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:48:39.098+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:48:39.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:48:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:48:39.098+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:48:39.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:48:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:48:39.098+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:48:39.887 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:48:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:48:39.637+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:48:39.887 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:48:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:48:39.637+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:48:39.887 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:48:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:48:39.637+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:48:39.888 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:48:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:48:39.501+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:48:39.888 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:48:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:48:39.501+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:48:39.888 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:48:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:48:39.501+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:48:40.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:48:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:48:40.064+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:48:40.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:48:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:48:40.064+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:48:40.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:48:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:48:40.064+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:48:40.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:48:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:48:40.064+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:48:40.635 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:48:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:48:40.482+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:48:40.635 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:48:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:48:40.482+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:48:40.635 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:48:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:48:40.482+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:48:40.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:48:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:48:40.631+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:48:40.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:48:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:48:40.631+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:48:40.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:48:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:48:40.631+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:48:41.482 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:48:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:48:41.100+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:48:41.483 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:48:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:48:41.100+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:48:41.483 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:48:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:48:41.100+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:48:41.483 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:48:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:48:41.100+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:48:41.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:48:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:48:41.644+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:48:41.744 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:48:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:48:41.644+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:48:41.744 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:48:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:48:41.644+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:48:41.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:48:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:48:41.478+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:48:41.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:48:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:48:41.478+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:48:41.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:48:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:48:41.478+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:48:42.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:48:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:48:42.052+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:48:42.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:48:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:48:42.052+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:48:42.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:48:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:48:42.052+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:48:42.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:48:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:48:42.052+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:48:42.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:48:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:48:42.599+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:48:42.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:48:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:48:42.599+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:48:42.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:48:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:48:42.599+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:48:42.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:48:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:48:42.502+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:48:42.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:48:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:48:42.502+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:48:42.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:48:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:48:42.502+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:48:43.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:48:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:48:43.100+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:48:43.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:48:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:48:43.100+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:48:43.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:48:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:48:43.100+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:48:43.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:48:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:48:43.100+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:48:43.993 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:48:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:48:43.616+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 42116 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:48:43.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:48:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:48:43.577+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:48:43.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:48:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:48:43.577+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:48:43.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:48:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:48:43.577+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:48:43.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:48:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:48:43.530+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:48:43.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:48:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:48:43.530+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:48:43.995 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:48:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:48:43.530+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:48:44.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:48:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:48:44.135+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:48:44.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:48:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:48:44.135+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:48:44.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:48:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:48:44.135+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:48:44.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:48:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:48:44.135+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:48:44.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:48:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:48:44.621+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:48:44.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:48:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:48:44.621+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:48:44.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:48:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:48:44.621+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:48:44.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:48:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:48:44.530+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:48:44.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:48:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:48:44.530+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:48:44.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:48:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:48:44.530+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:48:45.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:48:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:48:45.096+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:48:45.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:48:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:48:45.096+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:48:45.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:48:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:48:45.096+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:48:45.495 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:48:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:48:45.096+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:48:45.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:48:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:48:45.615+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:48:45.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:48:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:48:45.615+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:48:45.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:48:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:48:45.615+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:48:45.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:48:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:48:45.508+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:48:45.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:48:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:48:45.508+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:48:45.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:48:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:48:45.508+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:48:46.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:48:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:48:46.122+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:48:46.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:48:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:48:46.122+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:48:46.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:48:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:48:46.122+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:48:46.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:48:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:48:46.122+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:48:46.997 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:48:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:48:46.629+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:48:46.997 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:48:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:48:46.629+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:48:46.997 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:48:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:48:46.629+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:48:46.998 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:48:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:48:46.490+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:48:46.998 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:48:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:48:46.490+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:48:46.998 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:48:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:48:46.490+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:48:47.455 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:48:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:48:47.142+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:48:47.455 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:48:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:48:47.142+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:48:47.456 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:48:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:48:47.142+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:48:47.456 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:48:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:48:47.142+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:48:47.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:48:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:48:47.621+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:48:47.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:48:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:48:47.621+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:48:47.744 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:48:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:48:47.621+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:48:47.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:48:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:48:47.452+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:48:47.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:48:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:48:47.452+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:48:47.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:48:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:48:47.452+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:48:48.471 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:48:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:48:48.116+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:48:48.471 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:48:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:48:48.116+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:48:48.471 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:48:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:48:48.117+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:48:48.471 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:48:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:48:48.117+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:48:48.743 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:48:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:48:48.622+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 42188 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:48:48.744 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:48:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:48:48.644+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:48:48.744 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:48:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:48:48.644+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:48:48.744 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:48:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:48:48.644+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:48:48.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:48:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:48:48.467+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:48:48.745 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:48:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:48:48.467+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:48:48.745 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:48:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:48:48.467+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:48:49.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:48:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:48:49.154+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:48:49.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:48:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:48:49.154+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:48:49.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:48:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:48:49.154+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:48:49.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:48:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:48:49.154+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:48:49.886 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:48:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:48:49.502+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:48:49.887 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:48:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:48:49.502+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:48:49.887 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:48:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:48:49.502+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:48:49.887 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:48:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:48:49.639+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:48:49.887 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:48:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:48:49.639+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:48:49.887 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:48:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:48:49.639+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:48:50.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:48:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:48:50.120+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:48:50.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:48:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:48:50.120+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:48:50.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:48:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:48:50.120+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:48:50.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:48:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:48:50.120+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:48:50.630 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:48:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:48:50.495+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:48:50.631 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:48:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:48:50.495+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:48:50.631 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:48:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:48:50.495+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:48:50.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:48:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:48:50.627+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:48:50.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:48:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:48:50.627+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:48:50.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:48:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:48:50.627+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:48:51.487 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:48:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:48:51.129+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:48:51.488 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:48:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:48:51.129+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:48:51.488 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:48:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:48:51.129+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:48:51.488 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:48:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:48:51.129+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:48:51.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:48:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:48:51.629+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:48:51.744 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:48:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:48:51.629+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:48:51.744 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:48:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:48:51.629+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:48:51.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:48:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:48:51.484+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:48:51.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:48:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:48:51.484+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:48:51.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:48:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:48:51.484+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:48:52.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:48:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:48:52.097+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:48:52.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:48:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:48:52.097+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:48:52.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:48:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:48:52.097+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:48:52.495 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:48:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:48:52.097+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:48:52.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:48:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:48:52.585+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:48:52.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:48:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:48:52.585+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:48:52.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:48:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:48:52.585+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:48:52.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:48:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:48:52.493+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:48:52.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:48:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:48:52.493+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:48:52.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:48:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:48:52.493+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:48:53.478 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:48:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:48:53.144+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:48:53.478 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:48:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:48:53.144+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:48:53.478 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:48:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:48:53.144+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:48:53.478 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:48:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:48:53.144+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:48:53.743 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:48:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:48:53.628+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 42260 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:48:53.744 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:48:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:48:53.566+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:48:53.744 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:48:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:48:53.566+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:48:53.744 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:48:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:48:53.566+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:48:53.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:48:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:48:53.474+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:48:53.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:48:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:48:53.474+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:48:53.745 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:48:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:48:53.474+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:48:54.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:48:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:48:54.193+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:48:54.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:48:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:48:54.193+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:48:54.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:48:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:48:54.193+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:48:54.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:48:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:48:54.193+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:48:54.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:48:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:48:54.565+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:48:54.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:48:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:48:54.565+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:48:54.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:48:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:48:54.565+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:48:54.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:48:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:48:54.518+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:48:54.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:48:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:48:54.518+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:48:54.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:48:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:48:54.518+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:48:55.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:48:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:48:55.235+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:48:55.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:48:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:48:55.235+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:48:55.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:48:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:48:55.235+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:48:55.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:48:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:48:55.235+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:48:55.995 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:48:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:48:55.539+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:48:55.996 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:48:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:48:55.539+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:48:55.996 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:48:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:48:55.539+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:48:55.996 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:48:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:48:55.611+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:48:55.996 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:48:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:48:55.611+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:48:55.996 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:48:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:48:55.611+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:48:56.539 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:48:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:48:56.282+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:48:56.539 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:48:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:48:56.282+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:48:56.539 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:48:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:48:56.282+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:48:56.539 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:48:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:48:56.282+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:48:56.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:48:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:48:56.536+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:48:56.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:48:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:48:56.536+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:48:56.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:48:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:48:56.536+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:48:56.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:48:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:48:56.648+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:48:56.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:48:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:48:56.648+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:48:56.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:48:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:48:56.649+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:48:57.651 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:48:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:48:57.548+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:48:57.651 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:48:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:48:57.548+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:48:57.651 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:48:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:48:57.548+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:48:57.652 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:48:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:48:57.324+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:48:57.652 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:48:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:48:57.324+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:48:57.652 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:48:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:48:57.324+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:48:57.652 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:48:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:48:57.324+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:48:57.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:48:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:48:57.647+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:48:57.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:48:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:48:57.647+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:48:57.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:48:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:48:57.647+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:48:58.536 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:48:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:48:58.276+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:48:58.536 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:48:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:48:58.276+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:48:58.536 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:48:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:48:58.276+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:48:58.537 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:48:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:48:58.276+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:48:58.993 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:48:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:48:58.634+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 42332 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:48:58.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:48:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:48:58.655+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:48:58.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:48:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:48:58.655+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:48:58.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:48:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:48:58.655+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:48:58.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:48:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:48:58.533+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:48:58.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:48:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:48:58.533+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:48:58.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:48:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:48:58.533+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:48:59.675 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:48:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:48:59.539+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:48:59.676 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:48:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:48:59.539+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:48:59.676 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:48:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:48:59.539+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:48:59.676 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:48:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:48:59.317+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:48:59.676 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:48:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:48:59.317+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:48:59.676 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:48:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:48:59.317+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:48:59.676 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:48:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:48:59.317+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:48:59.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:48:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:48:59.672+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:48:59.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:48:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:48:59.672+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:48:59.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:48:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:48:59.672+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:49:00.562 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:49:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:49:00.281+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:49:00.562 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:49:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:49:00.281+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:49:00.562 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:49:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:49:00.281+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:49:00.562 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:49:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:49:00.281+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:49:00.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:49:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:49:00.559+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:49:00.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:49:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:49:00.559+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:49:00.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:49:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:49:00.559+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:49:00.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:49:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:49:00.637+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:49:00.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:49:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:49:00.637+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:49:00.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:49:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:49:00.637+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:49:01.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:49:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:49:01.295+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:49:01.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:49:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:49:01.295+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:49:01.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:49:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:49:01.295+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:49:01.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:49:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:49:01.295+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:49:01.843 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:49:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:49:01.598+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:49:01.844 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:49:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:49:01.598+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:49:01.844 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:49:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:49:01.598+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:49:01.844 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:49:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:49:01.587+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:49:01.844 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:49:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:49:01.587+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:49:01.844 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:49:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:49:01.587+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:49:02.644 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:49:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:49:02.570+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:49:02.644 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:49:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:49:02.570+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:49:02.644 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:49:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:49:02.570+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:49:02.645 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:49:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:49:02.325+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:49:02.645 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:49:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:49:02.325+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:49:02.645 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:49:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:49:02.325+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:49:02.645 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:49:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:49:02.325+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:49:02.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:49:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:49:02.641+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:49:02.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:49:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:49:02.641+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:49:02.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:49:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:49:02.641+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:49:03.574 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:49:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:49:03.277+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:49:03.574 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:49:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:49:03.277+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:49:03.574 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:49:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:49:03.277+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:49:03.574 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:49:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:49:03.277+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:49:03.993 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:49:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:49:03.641+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 42404 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:49:03.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:49:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:49:03.601+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:49:03.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:49:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:49:03.601+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:49:03.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:49:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:49:03.601+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:49:03.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:49:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:49:03.571+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:49:03.995 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:49:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:49:03.571+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:49:03.995 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:49:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:49:03.571+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:49:04.623 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:49:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:49:04.303+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:49:04.623 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:49:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:49:04.303+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:49:04.623 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:49:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:49:04.303+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:49:04.623 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:49:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:49:04.303+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:49:04.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:49:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:49:04.619+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:49:04.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:49:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:49:04.619+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:49:04.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:49:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:49:04.619+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:49:04.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:49:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:49:04.619+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:49:04.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:49:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:49:04.619+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:49:04.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:49:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:49:04.619+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:49:05.654 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:49:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:49:05.271+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:49:05.654 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:49:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:49:05.271+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:49:05.654 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:49:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:49:05.271+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:49:05.654 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:49:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:49:05.271+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:49:05.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:49:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:49:05.650+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:49:05.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:49:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:49:05.650+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:49:05.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:49:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:49:05.650+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:49:05.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:49:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:49:05.658+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:49:05.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:49:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:49:05.658+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:49:05.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:49:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:49:05.658+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:49:06.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:49:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:49:06.238+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:49:06.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:49:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:49:06.238+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:49:06.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:49:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:49:06.238+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:49:06.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:49:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:49:06.238+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:49:06.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:49:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:49:06.691+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:49:06.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:49:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:49:06.691+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:49:06.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:49:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:49:06.691+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:49:06.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:49:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:49:06.612+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:49:06.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:49:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:49:06.612+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:49:06.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:49:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:49:06.612+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:49:07.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:49:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:49:07.223+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:49:07.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:49:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:49:07.223+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:49:07.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:49:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:49:07.223+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:49:07.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:49:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:49:07.223+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:49:07.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:49:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:49:07.644+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:49:07.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:49:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:49:07.644+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:49:07.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:49:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:49:07.644+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:49:07.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:49:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:49:07.629+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:49:07.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:49:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:49:07.629+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:49:07.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:49:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:49:07.629+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:49:08.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:49:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:49:08.226+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:49:08.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:49:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:49:08.226+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:49:08.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:49:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:49:08.226+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:49:08.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:49:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:49:08.226+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:49:08.993 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:49:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:49:08.648+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 42476 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:49:08.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:49:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:49:08.635+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:49:08.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:49:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:49:08.635+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:49:08.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:49:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:49:08.635+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:49:08.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:49:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:49:08.671+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:49:08.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:49:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:49:08.671+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:49:08.995 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:49:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:49:08.671+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:49:09.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:49:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:49:09.227+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:49:09.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:49:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:49:09.227+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:49:09.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:49:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:49:09.227+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:49:09.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:49:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:49:09.227+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:49:09.887 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:49:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:49:09.607+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:49:09.887 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:49:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:49:09.607+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:49:09.888 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:49:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:49:09.607+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:49:09.888 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:49:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:49:09.706+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:49:09.888 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:49:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:49:09.706+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:49:09.888 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:49:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:49:09.706+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:49:10.615 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:49:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:49:10.245+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:49:10.616 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:49:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:49:10.245+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:49:10.616 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:49:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:49:10.245+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:49:10.616 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:49:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:49:10.245+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:49:10.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:49:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:49:10.613+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:49:10.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:49:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:49:10.613+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:49:10.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:49:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:49:10.613+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:49:10.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:49:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:49:10.697+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:49:10.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:49:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:49:10.697+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:49:10.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:49:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:49:10.697+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:49:11.620 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:49:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:49:11.246+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:49:11.620 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:49:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:49:11.246+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:49:11.620 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:49:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:49:11.246+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:49:11.621 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:49:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:49:11.246+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:49:11.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:49:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:49:11.617+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:49:11.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:49:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:49:11.617+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:49:11.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:49:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:49:11.617+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:49:11.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:49:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:49:11.649+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:49:11.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:49:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:49:11.649+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:49:11.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:49:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:49:11.649+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:49:12.624 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:49:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:49:12.263+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:49:12.625 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:49:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:49:12.263+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:49:12.625 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:49:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:49:12.263+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:49:12.625 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:49:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:49:12.263+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:49:12.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:49:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:49:12.646+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:49:12.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:49:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:49:12.646+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:49:12.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:49:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:49:12.646+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:49:12.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:49:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:49:12.621+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:49:12.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:49:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:49:12.621+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:49:12.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:49:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:49:12.621+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:49:13.618 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:49:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:49:13.302+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:49:13.618 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:49:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:49:13.302+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:49:13.618 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:49:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:49:13.302+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:49:13.618 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:49:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:49:13.302+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:49:13.993 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:49:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:49:13.655+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 42548 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:49:13.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:49:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:49:13.615+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:49:13.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:49:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:49:13.615+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:49:13.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:49:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:49:13.615+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:49:13.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:49:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:49:13.642+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:49:13.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:49:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:49:13.642+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:49:13.995 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:49:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:49:13.642+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:49:14.569 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:49:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:49:14.301+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:49:14.570 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:49:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:49:14.301+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:49:14.570 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:49:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:49:14.301+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:49:14.570 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:49:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:49:14.301+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:49:14.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:49:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:49:14.567+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:49:14.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:49:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:49:14.567+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:49:14.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:49:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:49:14.567+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:49:14.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:49:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:49:14.674+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:49:14.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:49:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:49:14.674+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:49:14.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:49:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:49:14.674+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:49:15.689 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:49:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:49:15.534+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:49:15.689 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:49:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:49:15.534+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:49:15.689 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:49:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:49:15.534+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:49:15.689 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:49:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:49:15.331+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:49:15.689 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:49:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:49:15.331+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:49:15.689 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:49:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:49:15.331+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:49:15.690 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:49:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:49:15.331+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:49:15.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:49:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:49:15.686+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:49:15.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:49:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:49:15.686+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:49:15.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:49:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:49:15.686+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:49:16.713 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:49:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:49:16.554+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:49:16.714 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:49:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:49:16.554+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:49:16.714 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:49:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:49:16.554+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:49:16.714 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:49:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:49:16.333+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:49:16.714 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:49:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:49:16.333+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:49:16.714 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:49:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:49:16.333+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:49:16.714 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:49:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:49:16.333+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:49:16.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:49:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:49:16.711+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:49:16.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:49:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:49:16.711+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:49:16.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:49:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:49:16.711+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:49:17.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:49:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:49:17.519+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:49:17.744 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:49:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:49:17.519+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:49:17.744 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:49:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:49:17.519+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:49:17.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:49:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:49:17.302+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:49:17.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:49:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:49:17.302+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:49:17.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:49:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:49:17.302+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:49:17.745 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:49:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:49:17.302+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:49:18.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:49:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:49:17.758+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:49:18.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:49:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:49:17.758+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:49:18.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:49:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:49:17.758+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:49:18.664 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:49:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:49:18.516+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:49:18.665 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:49:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:49:18.516+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:49:18.665 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:49:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:49:18.516+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:49:18.665 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:49:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:49:18.310+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:49:18.665 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:49:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:49:18.310+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:49:18.665 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:49:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:49:18.310+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:49:18.665 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:49:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:49:18.310+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:49:18.993 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:49:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:49:18.661+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 42620 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:49:18.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:49:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:49:18.723+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:49:18.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:49:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:49:18.723+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:49:18.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:49:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:49:18.723+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:49:19.559 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:49:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:49:19.290+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:49:19.560 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:49:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:49:19.290+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:49:19.560 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:49:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:49:19.290+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:49:19.560 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:49:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:49:19.290+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:49:19.887 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:49:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:49:19.694+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:49:19.887 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:49:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:49:19.694+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:49:19.888 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:49:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:49:19.694+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:49:19.888 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:49:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:49:19.557+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:49:19.888 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:49:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:49:19.557+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:49:19.888 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:49:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:49:19.557+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:49:20.573 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:49:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:49:20.289+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:49:20.573 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:49:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:49:20.289+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:49:20.573 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:49:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:49:20.289+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:49:20.573 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:49:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:49:20.289+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:49:20.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:49:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:49:20.570+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:49:20.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:49:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:49:20.570+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:49:20.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:49:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:49:20.570+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:49:20.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:49:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:49:20.734+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:49:20.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:49:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:49:20.734+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:49:20.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:49:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:49:20.734+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:49:21.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:49:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:49:21.534+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:49:21.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:49:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:49:21.534+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:49:21.744 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:49:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:49:21.534+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:49:21.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:49:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:49:21.316+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:49:21.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:49:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:49:21.316+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:49:21.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:49:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:49:21.316+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:49:21.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:49:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:49:21.316+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:49:22.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:49:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:49:21.757+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:49:22.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:49:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:49:21.757+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:49:22.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:49:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:49:21.757+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:49:22.682 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:49:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:49:22.499+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:49:22.682 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:49:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:49:22.499+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:49:22.682 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:49:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:49:22.499+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:49:22.683 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:49:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:49:22.356+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:49:22.683 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:49:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:49:22.356+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:49:22.683 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:49:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:49:22.356+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:49:22.683 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:49:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:49:22.356+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:49:22.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:49:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:49:22.751+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:49:22.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:49:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:49:22.751+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:49:22.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:49:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:49:22.751+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:49:23.671 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:49:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:49:23.532+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:49:23.671 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:49:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:49:23.532+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:49:23.672 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:49:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:49:23.532+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:49:23.672 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:49:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:49:23.379+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:49:23.672 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:49:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:49:23.379+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:49:23.672 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:49:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:49:23.379+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:49:23.672 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:49:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:49:23.379+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:49:23.993 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:49:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:49:23.668+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 42692 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:49:23.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:49:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:49:23.788+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:49:23.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:49:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:49:23.788+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:49:23.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:49:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:49:23.788+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:49:24.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:49:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:49:24.577+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:49:24.744 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:49:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:49:24.577+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:49:24.744 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:49:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:49:24.577+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:38:53.514185+0000 (oldest deadline 2024-02-20T08:39:13.514185+0000) 2024-02-20T08:49:24.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:49:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:49:24.356+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:49:24.745 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:49:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:49:24.356+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:49:24.745 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:49:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:49:24.356+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:49:24.745 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:49:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:49:24.356+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:49:25.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:49:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:49:24.830+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:49:25.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:49:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:49:24.830+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:49:25.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:49:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:49:24.830+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:49:25.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:49:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:49:25.339+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:49:25.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:49:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:49:25.339+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:49:25.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:49:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:49:25.339+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:49:25.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:49:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:49:25.339+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:49:26.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:49:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:49:25.817+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:49:26.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:49:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:49:25.817+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:49:26.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:49:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:49:25.817+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:49:26.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:49:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:49:26.307+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:49:26.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:49:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:49:26.307+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:49:26.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:49:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:49:26.307+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:49:26.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:49:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:49:26.308+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:49:27.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:49:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:49:26.789+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:49:27.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:49:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:49:26.789+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:49:27.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:49:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:49:26.789+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:49:27.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:49:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:49:27.346+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:49:27.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:49:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:49:27.346+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:49:27.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:49:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:49:27.346+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:49:27.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:49:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:49:27.346+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:49:28.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:49:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:49:27.772+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:49:28.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:49:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:49:27.772+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:49:28.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:49:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:49:27.772+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:49:28.678 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:49:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:49:28.350+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:49:28.679 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:49:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:49:28.350+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:49:28.679 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:49:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:49:28.350+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:49:28.679 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:49:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:49:28.350+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:49:28.993 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:49:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:49:28.676+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 42764 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:49:28.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:49:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:49:28.749+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:49:28.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:49:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:49:28.749+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:49:28.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:49:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:49:28.749+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:49:29.723 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:49:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:49:29.303+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:49:29.723 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:49:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:49:29.303+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:49:29.723 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:49:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:49:29.303+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:49:29.723 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:49:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:49:29.303+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:49:29.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:49:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:49:29.720+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:49:29.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:49:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:49:29.720+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:49:29.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:49:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:49:29.720+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:49:30.705 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:49:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:49:30.347+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:49:30.705 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:49:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:49:30.347+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:49:30.705 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:49:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:49:30.347+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:49:30.705 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:49:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:49:30.347+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:49:30.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:49:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:49:30.701+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:49:30.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:49:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:49:30.701+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:49:30.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:49:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:49:30.701+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:49:31.685 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:49:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:49:31.315+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:49:31.685 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:49:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:49:31.315+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:49:31.686 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:49:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:49:31.315+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:49:31.686 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:49:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:49:31.315+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:49:31.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:49:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:49:31.681+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:49:31.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:49:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:49:31.681+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:49:31.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:49:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:49:31.681+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:49:32.720 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:49:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:49:32.318+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:49:32.720 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:49:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:49:32.318+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:49:32.720 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:49:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:49:32.318+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:49:32.720 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:49:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:49:32.318+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:49:32.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:49:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:49:32.716+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:49:32.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:49:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:49:32.716+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:49:32.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:49:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:49:32.716+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:49:33.685 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:49:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:49:33.269+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:49:33.685 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:49:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:49:33.269+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:49:33.685 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:49:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:49:33.269+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:49:33.685 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:49:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:49:33.270+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:49:33.993 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:49:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:49:33.681+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 42836 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:49:33.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:49:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:49:33.743+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:49:33.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:49:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:49:33.743+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:49:33.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:49:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:49:33.743+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:49:34.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:49:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:49:34.264+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:49:34.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:49:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:49:34.264+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:49:34.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:49:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:49:34.264+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:49:34.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:49:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:49:34.264+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:49:34.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:49:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:49:34.738+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:49:34.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:49:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:49:34.738+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:49:34.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:49:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:49:34.738+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:49:35.721 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:49:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:49:35.244+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:49:35.721 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:49:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:49:35.244+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:49:35.721 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:49:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:49:35.244+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:49:35.721 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:49:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:49:35.244+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:49:35.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:49:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:49:35.718+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:49:35.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:49:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:49:35.718+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:49:35.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:49:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:49:35.718+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:49:36.680 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:49:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:49:36.278+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:49:36.681 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:49:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:49:36.278+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:49:36.681 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:49:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:49:36.278+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:49:36.681 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:49:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:49:36.278+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:49:36.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:49:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:49:36.677+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:49:36.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:49:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:49:36.677+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:49:36.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:49:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:49:36.677+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:49:37.660 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:49:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:49:37.319+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:49:37.660 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:49:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:49:37.319+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:49:37.661 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:49:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:49:37.319+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:49:37.661 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:49:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:49:37.319+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:49:37.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:49:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:49:37.657+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:49:37.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:49:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:49:37.657+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:49:37.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:49:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:49:37.657+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:49:38.616 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:49:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:49:38.334+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:49:38.616 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:49:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:49:38.334+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:49:38.616 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:49:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:49:38.334+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:49:38.616 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:49:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:49:38.334+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:49:38.993 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:49:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:49:38.688+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 42908 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:49:38.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:49:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:49:38.612+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:49:38.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:49:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:49:38.612+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:49:38.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:49:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:49:38.612+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:49:39.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:49:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:49:39.381+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:49:39.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:49:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:49:39.381+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:49:39.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:49:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:49:39.381+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:49:39.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:49:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:49:39.381+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:49:39.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:49:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:49:39.610+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:49:39.745 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:49:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:49:39.610+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:49:39.745 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:49:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:49:39.610+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:08.531784+0000 (oldest deadline 2024-02-20T08:39:28.531784+0000) 2024-02-20T08:49:40.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:49:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:49:40.339+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:49:40.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:49:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:49:40.339+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:49:40.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:49:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:49:40.339+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:49:40.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:49:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:49:40.339+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:49:41.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:49:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:49:41.305+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:49:41.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:49:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:49:41.305+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:49:41.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:49:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:49:41.305+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:49:41.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:49:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:49:41.305+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:49:42.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:49:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:49:42.312+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:49:42.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:49:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:49:42.312+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:49:42.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:49:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:49:42.312+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:49:42.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:49:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:49:42.312+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:49:43.698 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:49:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:49:43.311+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:49:43.699 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:49:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:49:43.311+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:49:43.699 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:49:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:49:43.311+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:49:43.699 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:49:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:49:43.311+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:49:43.993 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:49:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:49:43.694+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 42979 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:49:44.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:49:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:49:44.307+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:49:44.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:49:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:49:44.307+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:49:44.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:49:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:49:44.307+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:49:44.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:49:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:49:44.307+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:49:45.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:49:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:49:45.356+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:49:45.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:49:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:49:45.356+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:49:45.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:49:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:49:45.356+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:49:45.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:49:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:49:45.356+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:49:46.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:49:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:49:45.781+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:49:46.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:49:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:49:45.781+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:49:46.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:49:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:49:45.781+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:49:46.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:49:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:49:46.367+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:49:46.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:49:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:49:46.367+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:49:46.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:49:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:49:46.367+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:49:46.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:49:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:49:46.367+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:49:47.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:49:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:49:46.785+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:49:47.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:49:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:49:46.785+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:49:47.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:49:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:49:46.785+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:49:47.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:49:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:49:47.409+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:49:47.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:49:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:49:47.409+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:49:47.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:49:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:49:47.409+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:49:47.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:49:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:49:47.409+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:49:48.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:49:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:49:47.789+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:49:48.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:49:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:49:47.789+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:49:48.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:49:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:49:47.789+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:49:48.704 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:49:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:49:48.381+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:49:48.705 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:49:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:49:48.381+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:49:48.705 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:49:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:49:48.381+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:49:48.705 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:49:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:49:48.381+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:49:48.993 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:49:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:49:48.700+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 43052 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:49:48.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:49:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:49:48.833+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:49:48.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:49:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:49:48.833+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:49:48.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:49:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:49:48.833+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:49:49.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:49:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:49:49.377+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:49:49.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:49:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:49:49.377+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:49:49.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:49:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:49:49.377+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:49:49.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:49:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:49:49.377+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:49:50.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:49:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:49:49.846+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:49:50.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:49:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:49:49.846+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:49:50.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:49:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:49:49.846+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:49:50.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:49:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:49:50.379+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:49:50.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:49:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:49:50.379+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:49:50.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:49:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:49:50.379+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:49:50.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:49:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:49:50.379+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:49:51.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:49:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:49:50.888+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:49:51.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:49:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:49:50.888+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:49:51.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:49:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:49:50.888+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:49:51.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:49:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:49:51.363+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:49:51.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:49:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:49:51.363+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:49:51.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:49:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:49:51.363+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:39:17.623232+0000 (oldest deadline 2024-02-20T08:39:37.623232+0000) 2024-02-20T08:49:51.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:49:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:49:51.363+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:49:52.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:49:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:49:51.861+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:49:52.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:49:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:49:51.861+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:49:52.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:49:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:49:51.861+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:49:52.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:49:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:49:52.318+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:49:53.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:49:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:49:52.873+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:49:53.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:49:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:49:52.873+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:49:53.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:49:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:49:52.873+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:49:53.711 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:49:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:49:53.288+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:49:53.993 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:49:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:49:53.707+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 43123 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:49:53.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:49:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:49:53.862+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:49:53.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:49:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:49:53.862+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:49:53.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:49:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:49:53.862+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:49:54.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:49:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:49:54.259+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:49:55.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:49:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:49:54.871+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:49:55.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:49:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:49:54.871+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:49:55.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:49:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:49:54.871+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:49:55.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:49:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:49:55.295+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:49:56.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:49:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:49:55.858+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:49:56.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:49:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:49:55.858+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:49:56.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:49:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:49:55.858+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:49:56.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:49:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:49:56.316+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:49:57.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:49:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:49:56.896+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:49:57.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:49:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:49:56.896+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:49:57.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:49:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:49:56.896+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:49:57.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:49:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:49:57.361+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:49:58.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:49:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:49:57.892+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:49:58.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:49:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:49:57.892+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:49:58.245 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:49:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:49:57.892+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:49:58.718 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:49:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:49:58.335+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:49:58.993 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:49:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:49:58.714+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 43196 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:49:58.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:49:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:49:58.862+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:49:58.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:49:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:49:58.863+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:49:58.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:49:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:49:58.863+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:49:59.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:49:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:49:59.357+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:50:00.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:49:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:49:59.816+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:50:00.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:49:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:49:59.816+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:50:00.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:49:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:49:59.816+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:50:00.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:50:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:50:00.333+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:50:01.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:50:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:50:00.852+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:50:01.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:50:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:50:00.852+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:50:01.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:50:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:50:00.852+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:50:01.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:50:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:50:00.920+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:50:01.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:50:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:50:00.920+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:50:01.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:50:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:50:00.920+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:50:01.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:50:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:50:01.285+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:50:02.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:50:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:50:01.870+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:50:02.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:50:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:50:01.870+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:50:02.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:50:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:50:01.870+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:50:02.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:50:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:50:01.924+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:50:02.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:50:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:50:01.924+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:50:02.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:50:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:50:01.924+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:50:02.618 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:50:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:50:02.260+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:50:03.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:50:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:50:02.920+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:50:03.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:50:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:50:02.920+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:50:03.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:50:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:50:02.920+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:50:03.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:50:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:50:02.903+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:50:03.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:50:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:50:02.903+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:50:03.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:50:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:50:02.903+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:50:03.724 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:50:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:50:03.274+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:50:03.993 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:50:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:50:03.721+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 43267 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:50:03.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:50:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:50:03.927+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:50:03.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:50:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:50:03.927+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:50:03.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:50:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:50:03.927+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:50:03.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:50:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:50:03.897+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:50:03.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:50:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:50:03.897+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:50:03.995 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:50:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:50:03.897+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:50:04.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:50:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:50:04.281+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:50:05.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:50:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:50:04.886+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:50:05.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:50:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:50:04.886+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:50:05.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:50:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:50:04.886+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:50:05.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:50:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:50:04.874+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:50:05.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:50:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:50:04.874+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:50:05.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:50:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:50:04.874+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:50:05.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:50:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:50:05.295+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:50:06.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:50:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:50:05.913+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:50:06.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:50:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:50:05.913+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:50:06.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:50:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:50:05.913+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:50:06.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:50:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:50:05.890+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:50:06.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:50:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:50:05.890+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:50:06.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:50:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:50:05.890+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:50:06.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:50:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:50:06.265+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:50:07.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:50:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:50:06.928+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:50:07.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:50:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:50:06.928+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:50:07.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:50:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:50:06.928+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:50:07.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:50:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:50:06.867+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:50:07.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:50:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:50:06.867+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:50:07.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:50:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:50:06.867+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:50:07.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:50:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:50:07.303+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:50:08.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:50:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:50:07.904+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:50:08.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:50:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:50:07.904+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:50:08.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:50:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:50:07.904+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:50:08.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:50:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:50:07.840+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:50:08.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:50:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:50:07.840+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:50:08.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:50:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:50:07.840+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:50:08.731 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:50:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:50:08.318+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:50:08.993 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:50:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:50:08.728+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 43340 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:50:08.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:50:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:50:08.946+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:50:08.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:50:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:50:08.946+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:50:08.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:50:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:50:08.946+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:50:08.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:50:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:50:08.870+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:50:08.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:50:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:50:08.870+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:50:08.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:50:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:50:08.870+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:50:09.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:50:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:50:09.294+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:50:10.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:50:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:50:09.983+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:50:10.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:50:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:50:09.983+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:50:10.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:50:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:50:09.983+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:50:10.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:50:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:50:09.888+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:50:10.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:50:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:50:09.888+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:50:10.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:50:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:50:09.888+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:50:10.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:50:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:50:10.311+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:50:11.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:50:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:50:10.922+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:50:11.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:50:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:50:10.922+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:50:11.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:50:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:50:10.922+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:50:11.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:50:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:50:11.027+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:50:11.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:50:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:50:11.027+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:50:11.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:50:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:50:11.027+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:50:11.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:50:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:50:11.287+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:50:12.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:50:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:50:11.926+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:50:12.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:50:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:50:11.926+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:50:12.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:50:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:50:11.926+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:50:12.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:50:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:50:12.035+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:50:12.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:50:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:50:12.035+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:50:12.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:50:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:50:12.035+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:50:12.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:50:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:50:12.250+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:50:13.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:50:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:50:12.915+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:50:13.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:50:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:50:12.915+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:50:13.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:50:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:50:12.915+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:50:13.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:50:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:50:13.026+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:50:13.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:50:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:50:13.026+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:50:13.245 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:50:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:50:13.026+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:50:13.738 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:50:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:50:13.248+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:50:13.993 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:50:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:50:13.735+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 43411 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:50:13.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:50:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:50:13.947+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:50:13.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:50:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:50:13.947+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:50:13.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:50:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:50:13.947+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:50:14.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:50:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:50:14.220+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:50:14.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:50:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:50:14.220+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:50:14.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:50:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:50:14.220+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:50:14.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:50:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:50:14.220+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:50:14.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:50:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:50:14.002+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:50:14.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:50:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:50:14.002+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:50:14.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:50:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:50:14.002+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:50:15.203 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:50:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:50:14.958+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:50:15.203 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:50:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:50:14.958+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:50:15.203 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:50:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:50:14.958+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:50:15.203 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:50:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:50:14.961+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:50:15.203 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:50:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:50:14.961+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:50:15.204 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:50:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:50:14.961+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:50:15.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:50:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:50:15.200+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:50:15.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:50:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:50:15.200+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:50:15.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:50:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:50:15.200+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:50:15.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:50:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:50:15.200+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:50:16.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:50:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:50:15.909+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:50:16.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:50:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:50:15.909+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:50:16.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:50:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:50:15.909+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:50:16.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:50:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:50:15.999+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:50:16.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:50:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:50:15.999+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:50:16.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:50:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:50:15.999+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:50:16.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:50:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:50:16.246+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:50:16.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:50:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:50:16.246+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:50:16.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:50:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:50:16.246+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:50:16.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:50:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:50:16.246+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:50:17.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:50:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:50:16.970+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:50:17.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:50:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:50:16.970+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:50:17.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:50:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:50:16.970+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:50:17.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:50:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:50:16.949+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:50:17.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:50:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:50:16.949+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:50:17.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:50:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:50:16.949+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:50:17.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:50:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:50:17.245+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:50:17.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:50:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:50:17.245+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:50:17.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:50:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:50:17.245+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:50:17.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:50:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:50:17.245+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:50:18.201 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:50:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:50:17.911+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:50:18.202 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:50:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:50:17.911+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:50:18.202 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:50:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:50:17.911+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:50:18.202 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:50:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:50:17.999+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:50:18.202 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:50:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:50:17.999+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:50:18.202 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:50:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:50:17.999+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:50:18.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:50:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:50:18.199+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:50:18.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:50:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:50:18.199+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:50:18.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:50:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:50:18.199+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:50:18.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:50:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:50:18.199+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:50:19.243 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:50:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:50:18.742+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 43484 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:50:19.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:50:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:50:18.977+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:50:19.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:50:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:50:18.977+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:50:19.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:50:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:50:18.977+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:50:19.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:50:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:50:18.932+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:50:19.245 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:50:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:50:18.932+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:50:19.245 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:50:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:50:18.932+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:50:19.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:50:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:50:19.246+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:50:19.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:50:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:50:19.246+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:50:19.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:50:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:50:19.246+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:50:19.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:50:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:50:19.246+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:50:20.233 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:50:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:50:19.979+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:50:20.233 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:50:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:50:19.979+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:50:20.234 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:50:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:50:19.979+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:50:20.234 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:50:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:50:19.887+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:50:20.234 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:50:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:50:19.887+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:50:20.234 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:50:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:50:19.887+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:50:20.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:50:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:50:20.230+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:50:20.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:50:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:50:20.230+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:50:20.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:50:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:50:20.230+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:50:20.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:50:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:50:20.230+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:50:21.215 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:50:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:50:20.887+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:50:21.216 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:50:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:50:20.887+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:50:21.216 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:50:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:50:20.887+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:50:21.216 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:50:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:50:20.940+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:50:21.216 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:50:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:50:20.940+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:50:21.216 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:50:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:50:20.940+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:50:21.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:50:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:50:21.213+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:50:21.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:50:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:50:21.213+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:50:21.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:50:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:50:21.213+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:50:21.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:50:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:50:21.213+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:50:22.213 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:50:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:50:21.838+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:50:22.214 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:50:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:50:21.838+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:50:22.214 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:50:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:50:21.838+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:50:22.214 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:50:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:50:21.910+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:50:22.214 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:50:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:50:21.910+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:50:22.214 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:50:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:50:21.910+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:50:22.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:50:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:50:22.211+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:50:22.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:50:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:50:22.211+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:50:22.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:50:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:50:22.211+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:50:22.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:50:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:50:22.211+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:50:23.175 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:50:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:50:22.810+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:50:23.175 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:50:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:50:22.810+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:50:23.175 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:50:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:50:22.810+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:50:23.176 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:50:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:50:22.860+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:50:23.176 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:50:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:50:22.860+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:50:23.176 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:50:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:50:22.860+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:50:23.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:50:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:50:23.171+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:50:23.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:50:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:50:23.171+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:50:23.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:50:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:50:23.171+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:50:23.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:50:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:50:23.171+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:50:24.122 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:50:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:50:23.747+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 43555 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:50:24.122 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:50:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:50:23.858+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:50:24.122 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:50:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:50:23.858+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:50:24.123 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:50:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:50:23.858+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:50:24.123 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:50:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:50:23.807+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:50:24.123 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:50:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:50:23.807+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:50:24.123 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:50:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:50:23.807+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:50:24.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:50:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:50:24.141+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:50:24.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:50:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:50:24.141+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:50:24.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:50:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:50:24.141+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:50:24.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:50:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:50:24.141+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:50:25.144 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:50:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:50:24.887+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:50:25.145 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:50:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:50:24.887+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:50:25.145 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:50:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:50:24.887+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:50:25.145 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:50:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:50:24.778+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:50:25.145 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:50:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:50:24.778+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:50:25.146 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:50:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:50:24.778+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:50:25.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:50:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:50:25.141+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:50:25.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:50:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:50:25.141+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:50:25.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:50:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:50:25.141+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:50:25.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:50:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:50:25.141+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:50:26.101 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:50:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:50:25.860+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:50:26.101 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:50:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:50:25.860+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:50:26.101 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:50:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:50:25.860+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:50:26.101 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:50:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:50:25.822+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:50:26.102 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:50:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:50:25.822+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:50:26.102 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:50:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:50:25.822+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:50:26.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:50:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:50:26.098+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:50:26.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:50:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:50:26.098+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:50:26.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:50:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:50:26.098+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:50:26.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:50:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:50:26.098+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:50:27.114 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:50:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:50:26.869+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:50:27.115 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:50:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:50:26.869+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:50:27.115 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:50:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:50:26.869+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:50:27.115 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:50:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:50:26.824+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:50:27.115 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:50:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:50:26.824+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:50:27.115 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:50:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:50:26.824+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:50:27.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:50:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:50:27.111+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:50:27.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:50:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:50:27.111+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:50:27.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:50:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:50:27.111+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:50:27.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:50:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:50:27.111+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:50:28.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:50:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:50:27.902+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:50:28.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:50:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:50:27.902+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:50:28.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:50:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:50:27.902+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:50:28.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:50:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:50:28.066+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:50:28.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:50:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:50:28.066+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:50:28.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:50:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:50:28.066+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:50:28.245 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:50:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:50:28.067+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:50:28.245 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:50:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:50:27.858+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:50:28.245 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:50:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:50:27.858+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:50:28.245 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:50:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:50:27.858+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:50:29.090 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:50:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:50:28.754+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 43628 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:50:29.090 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:50:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:50:28.877+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:50:29.090 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:50:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:50:28.877+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:50:29.091 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:50:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:50:28.877+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:50:29.091 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:50:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:50:28.821+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:50:29.091 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:50:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:50:28.821+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:50:29.091 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:50:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:50:28.821+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:50:29.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:50:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:50:29.087+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:50:29.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:50:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:50:29.087+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:50:29.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:50:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:50:29.087+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:50:29.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:50:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:50:29.087+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:50:30.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:50:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:50:29.899+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:50:30.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:50:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:50:29.899+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:50:30.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:50:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:50:29.899+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:50:30.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:50:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:50:30.069+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:50:30.245 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:50:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:50:30.069+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:50:30.245 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:50:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:50:30.069+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:50:30.245 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:50:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:50:30.069+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:50:30.245 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:50:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:50:29.849+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:50:30.245 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:50:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:50:29.849+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:50:30.245 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:50:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:50:29.849+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:50:31.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:50:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:50:30.878+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:50:31.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:50:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:50:30.878+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:50:31.245 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:50:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:50:30.878+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:50:31.245 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:50:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:50:30.864+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:50:31.245 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:50:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:50:30.864+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:50:31.245 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:50:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:50:30.864+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:50:31.245 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:50:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:50:31.024+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:50:31.246 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:50:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:50:31.024+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:50:31.246 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:50:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:50:31.024+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:50:31.246 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:50:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:50:31.024+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:50:32.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:50:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:50:31.832+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:50:32.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:50:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:50:31.832+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:50:32.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:50:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:50:31.832+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:50:32.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:50:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:50:31.913+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:50:32.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:50:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:50:31.913+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:50:32.245 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:50:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:50:31.913+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:50:32.245 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:50:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:50:32.062+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:50:32.245 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:50:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:50:32.062+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:50:32.245 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:50:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:50:32.062+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:50:32.245 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:50:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:50:32.062+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:50:33.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:50:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:50:32.857+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:50:33.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:50:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:50:32.857+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:50:33.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:50:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:50:32.857+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:50:33.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:50:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:50:32.945+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:50:33.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:50:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:50:32.945+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:50:33.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:50:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:50:32.945+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:50:33.245 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:50:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:50:33.019+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:50:33.245 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:50:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:50:33.019+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:50:33.245 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:50:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:50:33.019+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:50:33.245 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:50:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:50:33.019+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:50:34.022 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:50:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:50:33.760+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 43699 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:50:34.022 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:50:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:50:33.816+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:50:34.023 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:50:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:50:33.816+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:50:34.023 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:50:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:50:33.816+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:50:34.023 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:50:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:50:33.951+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:50:34.023 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:50:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:50:33.951+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:50:34.023 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:50:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:50:33.951+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:50:34.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:50:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:50:34.018+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:50:34.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:50:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:50:34.018+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:50:34.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:50:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:50:34.018+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:50:34.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:50:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:50:34.018+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:50:35.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:50:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:50:34.901+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:50:35.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:50:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:50:34.902+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:50:35.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:50:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:50:34.902+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:50:35.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:50:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:50:35.014+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:50:35.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:50:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:50:35.014+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:50:35.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:50:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:50:35.014+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:50:35.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:50:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:50:35.014+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:50:35.245 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:50:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:50:34.824+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:50:35.245 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:50:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:50:34.824+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:50:35.245 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:50:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:50:34.824+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:50:36.145 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:50:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:50:35.812+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:50:36.146 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:50:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:50:35.812+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:50:36.146 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:50:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:50:35.812+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:50:36.146 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:50:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:50:36.001+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:50:36.146 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:50:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:50:36.001+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:50:36.146 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:50:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:50:36.001+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:50:36.147 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:50:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:50:36.001+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:50:36.147 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:50:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:50:35.859+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:50:36.147 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:50:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:50:35.859+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:50:36.147 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:50:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:50:35.859+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:50:37.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:50:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:50:37.052+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:50:37.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:50:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:50:37.052+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:50:37.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:50:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:50:37.052+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:50:37.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:50:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:50:37.052+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:50:37.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:50:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:50:36.890+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:50:37.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:50:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:50:36.890+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:50:37.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:50:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:50:36.890+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:50:37.245 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:50:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:50:36.841+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:50:37.245 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:50:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:50:36.841+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:50:37.245 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:50:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:50:36.841+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:50:38.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:50:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:50:37.906+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:50:38.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:50:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:50:37.906+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:50:38.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:50:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:50:37.907+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:50:38.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:50:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:50:37.844+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:50:38.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:50:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:50:37.844+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:50:38.245 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:50:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:50:37.844+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:50:38.245 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:50:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:50:38.038+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:50:38.245 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:50:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:50:38.038+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:50:38.245 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:50:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:50:38.038+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:50:38.245 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:50:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:50:38.038+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:50:39.037 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:50:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:50:38.766+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 43772 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:50:39.037 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:50:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:50:38.885+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:50:39.037 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:50:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:50:38.885+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:50:39.037 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:50:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:50:38.885+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:50:39.038 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:50:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:50:38.905+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:50:39.038 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:50:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:50:38.905+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:50:39.038 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:50:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:50:38.905+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:50:39.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:50:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:50:39.034+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:50:39.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:50:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:50:39.034+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:50:39.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:50:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:50:39.034+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:50:39.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:50:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:50:39.034+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:50:40.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:50:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:50:39.838+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:50:40.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:50:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:50:39.838+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:50:40.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:50:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:50:39.838+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:50:40.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:50:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:50:39.890+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:50:40.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:50:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:50:39.890+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:50:40.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:50:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:50:39.890+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:50:40.245 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:50:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:50:40.075+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:50:40.245 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:50:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:50:40.075+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:50:40.245 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:50:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:50:40.075+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:50:40.245 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:50:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:50:40.075+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:50:41.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:50:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:50:40.854+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:50:41.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:50:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:50:40.854+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:50:41.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:50:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:50:40.854+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:50:41.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:50:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:50:40.858+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:50:41.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:50:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:50:40.858+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:50:41.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:50:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:50:40.858+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:50:41.245 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:50:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:50:41.043+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:50:41.245 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:50:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:50:41.043+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:50:41.245 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:50:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:50:41.043+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:50:41.245 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:50:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:50:41.043+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:50:42.095 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:50:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:50:41.864+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:50:42.096 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:50:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:50:41.864+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:50:42.096 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:50:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:50:41.864+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:50:42.096 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:50:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:50:41.817+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:50:42.096 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:50:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:50:41.817+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:50:42.097 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:50:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:50:41.817+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:50:42.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:50:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:50:42.092+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:50:42.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:50:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:50:42.092+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:50:42.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:50:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:50:42.092+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:50:42.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:50:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:50:42.092+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:50:43.122 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:50:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:50:42.829+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:50:43.122 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:50:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:50:42.829+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:50:43.122 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:50:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:50:42.829+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:50:43.123 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:50:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:50:42.817+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:50:43.123 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:50:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:50:42.817+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:50:43.123 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:50:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:50:42.817+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:50:43.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:50:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:50:43.118+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:50:43.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:50:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:50:43.118+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:50:43.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:50:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:50:43.118+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:50:43.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:50:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:50:43.118+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:50:44.113 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:50:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:50:43.773+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 43843 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:50:44.114 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:50:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:50:43.842+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:50:44.114 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:50:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:50:43.842+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:50:44.114 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:50:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:50:43.842+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:50:44.114 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:50:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:50:43.871+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:50:44.114 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:50:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:50:43.871+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:50:44.114 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:50:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:50:43.871+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:50:44.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:50:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:50:44.109+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:50:44.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:50:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:50:44.109+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:50:44.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:50:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:50:44.109+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:50:44.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:50:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:50:44.110+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:50:45.071 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:50:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:50:44.840+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:50:45.072 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:50:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:50:44.840+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:50:45.072 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:50:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:50:44.840+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:50:45.072 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:50:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:50:44.803+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:50:45.072 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:50:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:50:44.803+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:50:45.072 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:50:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:50:44.803+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:50:45.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:50:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:50:45.068+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:50:45.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:50:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:50:45.068+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:50:45.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:50:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:50:45.068+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:50:45.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:50:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:50:45.068+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:50:46.121 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:50:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:50:45.829+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:50:46.121 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:50:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:50:45.829+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:50:46.121 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:50:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:50:45.829+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:50:46.121 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:50:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:50:45.774+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:50:46.122 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:50:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:50:45.774+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:50:46.122 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:50:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:50:45.774+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:50:46.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:50:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:50:46.118+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:50:46.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:50:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:50:46.118+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:50:46.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:50:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:50:46.118+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:50:46.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:50:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:50:46.118+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:50:47.084 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:50:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:50:46.864+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:50:47.084 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:50:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:50:46.864+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:50:47.085 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:50:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:50:46.864+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:50:47.085 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:50:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:50:46.812+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:50:47.085 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:50:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:50:46.812+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:50:47.085 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:50:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:50:46.812+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:50:47.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:50:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:50:47.081+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:50:47.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:50:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:50:47.081+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:50:47.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:50:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:50:47.081+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:50:47.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:50:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:50:47.081+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:50:48.096 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:50:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:50:47.824+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:50:48.097 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:50:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:50:47.824+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:50:48.097 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:50:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:50:47.824+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:50:48.097 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:50:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:50:47.855+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:50:48.097 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:50:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:50:47.855+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:50:48.097 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:50:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:50:47.855+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:50:48.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:50:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:50:48.093+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:50:48.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:50:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:50:48.093+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:50:48.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:50:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:50:48.093+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:50:48.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:50:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:50:48.093+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:50:49.056 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:50:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:50:48.780+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 43916 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:50:49.057 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:50:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:50:48.902+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:50:49.057 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:50:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:50:48.902+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:50:49.057 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:50:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:50:48.902+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:50:49.058 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:50:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:50:48.810+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:50:49.058 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:50:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:50:48.810+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:50:49.058 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:50:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:50:48.810+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:50:49.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:50:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:50:49.054+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:50:49.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:50:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:50:49.054+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:50:49.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:50:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:50:49.054+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:50:49.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:50:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:50:49.054+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:50:50.047 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:50:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:50:49.776+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:50:50.047 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:50:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:50:49.776+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:50:50.047 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:50:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:50:49.776+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:50:50.048 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:50:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:50:49.945+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:50:50.048 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:50:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:50:49.945+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:50:50.048 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:50:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:50:49.945+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:50:50.321 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:50:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:50:50.044+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:50:50.321 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:50:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:50:50.044+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:50:50.322 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:50:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:50:50.044+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:50:50.322 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:50:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:50:50.044+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:50:51.057 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:50:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:50:50.909+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:50:51.057 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:50:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:50:50.909+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:50:51.058 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:50:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:50:50.909+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:50:51.058 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:50:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:50:50.782+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:50:51.058 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:50:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:50:50.782+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:50:51.058 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:50:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:50:50.782+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:50:51.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:50:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:50:51.054+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:50:51.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:50:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:50:51.054+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:50:51.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:50:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:50:51.054+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:50:51.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:50:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:50:51.054+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:50:52.104 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:50:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:50:51.918+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:50:52.105 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:50:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:50:51.918+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:50:52.105 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:50:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:50:51.918+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:50:52.105 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:50:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:50:51.811+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:50:52.105 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:50:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:50:51.811+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:50:52.105 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:50:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:50:51.811+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:50:52.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:50:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:50:52.102+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:50:52.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:50:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:50:52.102+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:50:52.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:50:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:50:52.102+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:50:52.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:50:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:50:52.102+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:50:53.067 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:50:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:50:52.777+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:50:53.067 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:50:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:50:52.777+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:50:53.068 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:50:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:50:52.777+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:50:53.068 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:50:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:50:52.935+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:50:53.068 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:50:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:50:52.935+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:50:53.068 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:50:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:50:52.935+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:50:53.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:50:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:50:53.063+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:50:53.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:50:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:50:53.063+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:50:53.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:50:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:50:53.063+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:50:53.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:50:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:50:53.064+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:50:54.111 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:50:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:50:53.786+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 43987 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:50:54.112 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:50:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:50:53.977+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:50:54.112 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:50:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:50:53.977+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:50:54.112 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:50:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:50:53.977+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:50:54.112 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:50:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:50:53.777+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:50:54.112 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:50:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:50:53.777+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:50:54.113 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:50:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:50:53.777+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:50:54.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:50:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:50:54.108+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:50:54.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:50:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:50:54.108+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:50:54.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:50:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:50:54.108+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:50:54.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:50:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:50:54.108+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:50:55.068 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:50:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:50:54.982+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:50:55.068 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:50:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:50:54.982+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:50:55.068 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:50:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:50:54.982+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:50:55.068 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:50:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:50:54.801+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:50:55.068 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:50:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:50:54.801+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:50:55.069 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:50:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:50:54.801+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:50:55.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:50:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:50:55.065+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:50:55.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:50:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:50:55.065+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:50:55.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:50:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:50:55.065+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:50:55.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:50:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:50:55.065+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:50:56.039 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:50:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:50:55.969+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:50:56.039 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:50:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:50:55.969+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:50:56.039 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:50:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:50:55.969+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:50:56.040 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:50:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:50:55.777+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:50:56.040 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:50:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:50:55.777+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:50:56.040 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:50:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:50:55.777+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:50:56.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:50:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:50:56.036+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:50:56.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:50:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:50:56.036+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:50:56.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:50:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:50:56.036+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:50:56.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:50:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:50:56.036+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:50:57.079 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:50:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:50:56.921+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:50:57.079 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:50:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:50:56.921+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:50:57.079 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:50:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:50:56.921+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:50:57.079 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:50:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:50:56.799+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:50:57.080 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:50:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:50:56.799+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:50:57.080 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:50:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:50:56.799+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:50:57.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:50:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:50:57.076+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:50:57.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:50:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:50:57.076+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:50:57.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:50:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:50:57.076+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:50:57.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:50:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:50:57.076+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:50:58.050 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:50:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:50:57.955+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:50:58.050 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:50:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:50:57.955+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:50:58.050 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:50:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:50:57.955+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:50:58.050 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:50:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:50:57.781+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:50:58.050 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:50:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:50:57.781+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:50:58.051 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:50:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:50:57.781+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:50:58.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:50:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:50:58.047+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:50:58.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:50:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:50:58.047+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:50:58.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:50:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:50:58.047+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:50:58.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:50:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:50:58.047+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:50:58.993 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:50:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:50:58.793+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 44060 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:50:58.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:50:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:50:58.977+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:50:58.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:50:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:50:58.977+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:50:58.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:50:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:50:58.977+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:50:58.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:50:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:50:58.735+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:50:58.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:50:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:50:58.735+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:50:58.995 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:50:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:50:58.735+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:50:59.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:50:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:50:59.041+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:50:59.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:50:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:50:59.041+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:50:59.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:50:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:50:59.041+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:50:59.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:50:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:50:59.041+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:50:59.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:50:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:50:59.701+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:50:59.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:50:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:50:59.701+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:50:59.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:50:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:50:59.701+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:51:00.322 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:51:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:51:00.012+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:51:00.323 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:51:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:51:00.012+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:51:00.323 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:51:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:51:00.012+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:51:00.323 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:51:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:51:00.012+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:51:00.323 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:50:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:50:59.997+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:51:00.323 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:50:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:50:59.997+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:51:00.324 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:50:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:50:59.997+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:51:00.986 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:51:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:51:00.678+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:51:00.986 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:51:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:51:00.678+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:51:00.986 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:51:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:51:00.678+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:51:01.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:51:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:51:01.057+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:51:01.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:51:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:51:01.057+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:51:01.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:51:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:51:01.057+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:51:01.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:51:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:51:01.057+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:51:01.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:51:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:51:01.002+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:51:01.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:51:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:51:01.002+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:51:01.245 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:51:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:51:01.002+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:51:01.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:51:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:51:01.685+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:51:01.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:51:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:51:01.685+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:51:01.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:51:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:51:01.685+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:51:02.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:51:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:51:02.018+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:51:02.494 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:51:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:51:02.018+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:51:02.494 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:51:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:51:02.018+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:51:02.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:51:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:51:02.044+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:51:02.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:51:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:51:02.044+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:51:02.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:51:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:51:02.044+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:51:02.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:51:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:51:02.044+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:51:02.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:51:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:51:02.731+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:51:02.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:51:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:51:02.731+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:51:02.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:51:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:51:02.731+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:51:03.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:51:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:51:03.015+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:51:03.494 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:51:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:51:03.015+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:51:03.494 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:51:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:51:03.015+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:51:03.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:51:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:51:03.038+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:51:03.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:51:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:51:03.038+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:51:03.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:51:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:51:03.038+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:51:03.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:51:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:51:03.038+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:51:04.044 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:51:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:51:03.800+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 44132 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:51:04.044 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:51:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:51:03.761+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:51:04.045 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:51:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:51:03.761+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:51:04.045 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:51:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:51:03.761+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:51:04.045 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:51:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:51:04.007+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:51:04.045 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:51:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:51:04.007+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:51:04.045 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:51:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:51:04.007+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:51:04.046 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:51:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:51:04.008+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:51:04.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:51:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:51:04.041+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:51:04.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:51:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:51:04.041+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:51:04.494 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:51:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:51:04.041+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:51:05.019 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:51:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:51:04.745+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:51:05.020 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:51:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:51:04.745+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:51:05.020 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:51:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:51:04.745+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:51:05.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:51:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:51:05.018+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:51:05.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:51:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:51:05.018+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:51:05.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:51:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:51:05.018+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:51:05.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:51:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:51:05.018+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:51:05.494 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:51:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:51:05.017+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:51:05.494 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:51:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:51:05.017+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:51:05.494 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:51:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:51:05.017+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:51:05.976 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:51:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:51:05.699+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:51:05.976 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:51:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:51:05.699+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:51:05.976 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:51:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:51:05.699+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:51:06.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:51:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:51:05.973+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:51:06.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:51:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:51:05.973+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:51:06.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:51:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:51:05.973+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:51:06.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:51:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:51:06.000+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:51:06.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:51:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:51:06.000+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:51:06.245 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:51:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:51:06.000+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:51:06.245 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:51:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:51:06.000+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:51:06.981 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:51:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:51:06.686+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:51:06.982 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:51:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:51:06.686+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:51:06.982 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:51:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:51:06.686+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:51:07.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:51:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:51:06.996+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:51:07.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:51:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:51:06.996+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:51:07.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:51:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:51:06.996+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:51:07.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:51:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:51:06.979+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:51:07.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:51:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:51:06.979+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:51:07.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:51:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:51:06.979+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:51:07.245 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:51:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:51:06.979+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:51:07.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:51:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:51:07.678+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:51:07.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:51:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:51:07.678+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:51:07.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:51:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:51:07.678+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:51:08.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:51:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:51:08.007+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:51:08.494 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:51:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:51:08.007+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:51:08.494 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:51:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:51:08.007+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:51:08.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:51:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:51:08.022+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:51:08.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:51:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:51:08.022+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:51:08.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:51:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:51:08.022+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:51:08.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:51:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:51:08.022+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:51:08.931 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:51:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:51:08.806+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 44204 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:51:08.931 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:51:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:51:08.633+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:51:08.931 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:51:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:51:08.633+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:51:08.931 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:51:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:51:08.633+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:51:09.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:51:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:51:09.032+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:51:09.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:51:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:51:09.032+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:51:09.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:51:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:51:09.032+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:51:09.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:51:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:51:08.983+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:51:09.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:51:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:51:08.983+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:51:09.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:51:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:51:08.983+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:51:09.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:51:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:51:08.983+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:51:09.989 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:51:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:51:09.636+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:51:09.989 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:51:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:51:09.636+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:51:09.989 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:51:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:51:09.636+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:51:10.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:51:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:51:09.986+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:51:10.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:51:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:51:09.986+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:51:10.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:51:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:51:09.986+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:51:10.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:51:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:51:10.007+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:51:10.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:51:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:51:10.007+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:51:10.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:51:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:51:10.007+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:51:10.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:51:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:51:10.007+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:51:10.942 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:51:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:51:10.632+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:51:10.942 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:51:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:51:10.632+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:51:10.942 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:51:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:51:10.632+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:51:11.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:51:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:51:10.939+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:51:11.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:51:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:51:10.939+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:51:11.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:51:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:51:10.939+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:51:11.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:51:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:51:11.012+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:51:11.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:51:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:51:11.012+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:51:11.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:51:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:51:11.012+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:51:11.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:51:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:51:11.012+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:51:11.975 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:51:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:51:11.904+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:51:11.975 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:51:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:51:11.904+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:51:11.975 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:51:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:51:11.904+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:51:11.975 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:51:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:51:11.677+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:51:11.976 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:51:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:51:11.677+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:51:11.976 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:51:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:51:11.677+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:51:12.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:51:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:51:11.972+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:51:12.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:51:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:51:11.972+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:51:12.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:51:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:51:11.972+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:51:12.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:51:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:51:11.972+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:51:12.931 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:51:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:51:12.677+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:51:12.932 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:51:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:51:12.677+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:51:12.932 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:51:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:51:12.677+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:51:12.932 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:51:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:51:12.920+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:51:12.932 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:51:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:51:12.920+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:51:12.932 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:51:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:51:12.920+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:51:13.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:51:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:51:12.928+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:51:13.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:51:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:51:12.928+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:51:13.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:51:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:51:12.928+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:51:13.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:51:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:51:12.929+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:51:13.889 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:51:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:51:13.812+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 44275 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:51:13.889 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:51:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:51:13.881+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:51:13.890 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:51:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:51:13.881+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:51:13.890 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:51:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:51:13.881+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:51:13.890 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:51:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:51:13.633+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:51:13.890 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:51:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:51:13.633+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:51:13.890 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:51:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:51:13.633+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:51:14.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:51:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:51:13.886+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:51:14.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:51:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:51:13.886+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:51:14.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:51:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:51:13.886+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:51:14.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:51:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:51:13.886+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:51:14.922 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:51:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:51:14.652+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:51:14.922 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:51:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:51:14.652+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:51:14.922 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:51:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:51:14.652+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:51:15.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:51:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:51:14.919+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:51:15.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:51:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:51:14.919+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:51:15.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:51:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:51:14.919+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:51:15.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:51:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:51:14.931+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:51:15.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:51:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:51:14.931+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:51:15.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:51:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:51:14.931+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:51:15.245 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:51:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:51:14.931+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:51:15.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:51:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:51:15.893+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:51:15.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:51:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:51:15.893+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:51:15.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:51:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:51:15.893+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:51:15.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:51:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:51:15.893+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:51:15.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:51:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:51:15.911+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:51:15.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:51:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:51:15.911+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:51:15.995 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:51:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:51:15.911+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:51:15.995 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:51:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:51:15.669+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:51:15.995 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:51:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:51:15.669+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:51:15.995 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:51:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:51:15.669+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:51:16.963 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:51:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:51:16.905+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:51:16.964 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:51:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:51:16.905+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:51:16.964 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:51:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:51:16.905+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:51:16.964 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:51:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:51:16.905+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:51:16.964 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:51:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:51:16.669+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:51:16.964 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:51:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:51:16.669+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:51:16.964 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:51:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:51:16.669+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:51:17.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:51:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:51:16.959+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:51:17.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:51:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:51:16.959+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:51:17.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:51:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:51:16.959+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:51:17.983 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:51:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:51:17.632+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:51:17.984 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:51:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:51:17.632+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:51:17.984 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:51:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:51:17.632+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:51:17.984 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:51:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:51:17.873+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:51:17.984 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:51:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:51:17.873+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:51:17.984 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:51:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:51:17.873+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:51:17.985 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:51:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:51:17.873+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:51:18.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:51:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:51:17.980+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:51:18.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:51:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:51:17.980+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:51:18.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:51:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:51:17.980+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:51:18.917 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:51:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:51:18.818+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 44350 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:51:18.918 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:51:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:51:18.592+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:51:18.918 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:51:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:51:18.592+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:51:18.918 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:51:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:51:18.592+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:51:19.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:51:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:51:18.913+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:51:19.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:51:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:51:18.913+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:51:19.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:51:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:51:18.914+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:51:19.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:51:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:51:18.914+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:51:19.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:51:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:51:18.983+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:51:19.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:51:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:51:18.983+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:51:19.245 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:51:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:51:18.983+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:51:19.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:51:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:51:19.635+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:51:19.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:51:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:51:19.635+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:51:19.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:51:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:51:19.635+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:51:19.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:51:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:51:19.879+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:51:19.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:51:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:51:19.879+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:51:19.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:51:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:51:19.879+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:51:19.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:51:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:51:19.880+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:51:20.324 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:51:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:51:20.007+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:51:20.324 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:51:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:51:20.007+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:51:20.324 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:51:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:51:20.007+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:51:20.893 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:51:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:51:20.592+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:51:20.893 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:51:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:51:20.592+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:51:20.893 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:51:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:51:20.592+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:51:21.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:51:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:51:21.026+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:51:21.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:51:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:51:21.026+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:51:21.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:51:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:51:21.026+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:51:21.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:51:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:51:20.890+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:51:21.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:51:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:51:20.890+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:51:21.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:51:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:51:20.890+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:51:21.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:51:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:51:20.890+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:51:21.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:51:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:51:21.632+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:51:21.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:51:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:51:21.632+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:51:21.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:51:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:51:21.632+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:51:21.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:51:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:51:21.850+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:51:21.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:51:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:51:21.850+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:51:21.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:51:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:51:21.850+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:51:21.995 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:51:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:51:21.850+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:51:22.490 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:51:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:51:22.008+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:51:22.490 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:51:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:51:22.008+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:51:22.490 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:51:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:51:22.008+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:51:22.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:51:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:51:22.621+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:51:22.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:51:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:51:22.621+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:51:22.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:51:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:51:22.621+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:51:23.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:51:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:51:22.965+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:51:23.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:51:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:51:22.965+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:51:23.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:51:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:51:22.965+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:51:23.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:51:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:51:22.861+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:51:23.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:51:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:51:22.861+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:51:23.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:51:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:51:22.861+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:51:23.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:51:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:51:22.861+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:51:23.953 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:51:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:51:23.825+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 44421 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:51:23.953 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:51:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:51:23.661+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:51:23.953 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:51:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:51:23.661+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:51:23.954 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:51:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:51:23.661+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:51:23.954 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:51:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:51:23.884+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:51:23.954 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:51:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:51:23.884+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:51:23.954 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:51:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:51:23.884+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:51:23.954 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:51:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:51:23.884+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:51:24.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:51:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:51:23.957+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:51:24.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:51:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:51:23.957+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:51:24.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:51:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:51:23.957+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:51:24.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:51:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:51:24.856+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:51:24.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:51:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:51:24.856+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:51:24.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:51:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:51:24.856+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:51:24.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:51:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:51:24.857+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:51:24.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:51:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:51:24.647+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:51:24.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:51:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:51:24.647+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:51:24.995 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:51:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:51:24.647+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:51:25.494 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:51:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:51:24.995+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:51:25.494 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:51:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:51:24.995+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:51:25.494 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:51:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:51:24.995+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:51:25.961 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:51:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:51:25.621+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:51:25.961 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:51:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:51:25.621+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:51:25.961 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:51:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:51:25.621+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:51:25.962 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:51:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:51:25.840+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:51:25.962 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:51:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:51:25.840+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:51:25.962 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:51:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:51:25.840+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:51:25.962 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:51:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:51:25.840+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:51:26.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:51:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:51:25.957+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:51:26.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:51:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:51:25.957+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:51:26.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:51:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:51:25.957+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:51:26.884 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:51:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:51:26.598+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:51:26.885 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:51:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:51:26.598+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:51:26.885 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:51:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:51:26.598+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:51:27.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:51:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:51:26.953+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:51:27.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:51:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:51:26.953+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:51:27.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:51:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:51:26.953+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:51:27.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:51:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:51:26.881+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:51:27.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:51:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:51:26.881+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:51:27.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:51:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:51:26.881+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:51:27.245 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:51:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:51:26.881+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:51:27.945 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:51:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:51:27.604+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:51:27.945 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:51:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:51:27.604+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:51:27.945 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:51:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:51:27.604+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:51:27.946 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:51:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:51:27.846+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:51:27.946 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:51:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:51:27.846+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:51:27.946 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:51:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:51:27.846+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:51:27.946 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:51:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:51:27.846+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:51:28.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:51:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:51:27.942+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:51:28.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:51:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:51:27.942+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:51:28.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:51:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:51:27.942+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:51:28.843 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:51:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:51:28.832+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 44491 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:51:28.843 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:51:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:51:28.587+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:51:28.843 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:51:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:51:28.587+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:51:28.844 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:51:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:51:28.587+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:51:29.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:51:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:51:28.956+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:51:29.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:51:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:51:28.956+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:51:29.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:51:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:51:28.956+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:51:29.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:51:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:51:28.859+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:51:29.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:51:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:51:28.859+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:51:29.245 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:51:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:51:28.859+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:51:29.245 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:51:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:51:28.859+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:51:29.880 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:51:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:51:29.614+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:51:29.880 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:51:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:51:29.614+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:51:29.880 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:51:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:51:29.614+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:51:30.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:51:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:51:29.977+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:51:30.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:51:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:51:29.977+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:51:30.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:51:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:51:29.977+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:51:30.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:51:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:51:29.876+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:51:30.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:51:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:51:29.876+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:51:30.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:51:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:51:29.876+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:51:30.245 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:51:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:51:29.876+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:51:30.901 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:51:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:51:30.573+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:51:30.901 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:51:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:51:30.573+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:51:30.901 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:51:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:51:30.573+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:51:31.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:51:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:51:30.959+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:51:31.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:51:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:51:30.959+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:51:31.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:51:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:51:30.959+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:51:31.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:51:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:51:30.897+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:51:31.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:51:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:51:30.897+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:51:31.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:51:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:51:30.897+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:51:31.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:51:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:51:30.897+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:51:31.853 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:51:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:51:31.538+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:51:31.853 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:51:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:51:31.538+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:51:31.853 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:51:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:51:31.538+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:51:32.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:51:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:51:31.973+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:51:32.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:51:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:51:31.973+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:51:32.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:51:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:51:31.973+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:51:32.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:51:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:51:31.860+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:51:32.245 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:51:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:51:31.860+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:51:32.245 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:51:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:51:31.860+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:51:32.245 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:51:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:51:31.860+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:51:32.855 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:51:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:51:32.537+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:51:32.855 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:51:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:51:32.537+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:51:32.855 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:51:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:51:32.537+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:51:33.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:51:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:51:32.924+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:51:33.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:51:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:51:32.924+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:51:33.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:51:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:51:32.924+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:51:33.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:51:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:51:32.851+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:51:33.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:51:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:51:32.851+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:51:33.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:51:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:51:32.851+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:51:33.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:51:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:51:32.851+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:51:33.819 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:51:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:51:33.491+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:51:33.819 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:51:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:51:33.491+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:51:33.819 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:51:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:51:33.491+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:51:34.243 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:51:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:51:33.839+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 44559 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:51:34.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:51:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:51:33.955+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:51:34.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:51:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:51:33.955+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:51:34.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:51:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:51:33.955+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:51:34.245 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:51:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:51:33.816+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:51:34.245 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:51:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:51:33.816+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:51:34.245 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:51:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:51:33.816+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:51:34.245 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:51:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:51:33.816+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:51:34.778 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:51:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:51:34.515+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:51:34.778 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:51:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:51:34.515+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:51:34.778 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:51:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:51:34.515+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:51:35.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:51:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:51:34.960+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:51:35.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:51:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:51:34.960+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:51:35.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:51:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:51:34.960+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:51:35.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:51:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:51:34.809+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:51:35.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:51:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:51:34.809+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:51:35.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:51:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:51:34.809+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:51:35.245 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:51:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:51:34.809+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:51:35.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:51:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:51:35.476+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:51:35.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:51:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:51:35.476+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:51:35.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:51:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:51:35.476+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:51:36.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:51:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:51:35.920+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:51:36.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:51:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:51:35.920+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:51:36.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:51:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:51:35.920+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:51:36.245 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:51:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:51:35.841+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:51:36.245 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:51:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:51:35.841+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:51:36.245 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:51:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:51:35.841+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:51:36.245 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:51:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:51:35.842+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:51:36.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:51:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:51:36.449+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:51:36.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:51:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:51:36.449+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:51:36.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:51:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:51:36.449+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:51:37.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:51:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:51:36.938+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:51:37.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:51:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:51:36.938+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:51:37.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:51:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:51:36.938+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:51:37.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:51:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:51:36.866+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:51:37.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:51:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:51:36.866+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:51:37.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:51:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:51:36.866+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:51:37.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:51:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:51:36.866+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:51:37.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:51:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:51:37.481+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:51:37.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:51:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:51:37.481+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:51:37.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:51:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:51:37.481+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:51:38.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:51:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:51:37.967+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:51:38.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:51:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:51:37.967+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:51:38.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:51:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:51:37.967+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:51:38.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:51:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:51:37.906+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:51:38.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:51:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:51:37.906+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:51:38.245 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:51:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:51:37.906+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:51:38.245 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:51:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:51:37.906+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:51:38.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:51:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:51:38.488+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:51:38.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:51:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:51:38.488+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:51:38.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:51:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:51:38.488+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:51:39.243 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:51:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:51:38.846+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 44629 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:51:39.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:51:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:51:38.967+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:51:39.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:51:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:51:38.967+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:51:39.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:51:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:51:38.967+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:51:39.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:51:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:51:38.881+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:51:39.245 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:51:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:51:38.881+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:51:39.245 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:51:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:51:38.881+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:51:39.245 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:51:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:51:38.882+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:51:39.841 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:51:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:51:39.491+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:51:39.842 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:51:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:51:39.491+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:51:39.842 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:51:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:51:39.491+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:51:40.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:51:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:51:39.921+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:51:40.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:51:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:51:39.921+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:51:40.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:51:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:51:39.921+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:51:40.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:51:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:51:39.838+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:51:40.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:51:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:51:39.838+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:51:40.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:51:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:51:39.838+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:51:40.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:51:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:51:39.838+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:51:40.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:51:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:51:40.461+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:51:40.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:51:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:51:40.461+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:51:40.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:51:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:51:40.461+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:51:41.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:51:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:51:40.910+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:51:41.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:51:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:51:40.910+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:51:41.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:51:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:51:40.910+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:51:41.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:51:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:51:40.879+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:51:41.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:51:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:51:40.879+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:51:41.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:51:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:51:40.879+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:51:41.245 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:51:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:51:40.879+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:51:41.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:51:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:51:41.415+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:51:41.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:51:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:51:41.415+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:51:41.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:51:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:51:41.415+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:51:42.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:51:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:51:41.949+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:51:42.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:51:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:51:41.949+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:51:42.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:51:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:51:41.949+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:51:42.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:51:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:51:41.857+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:51:42.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:51:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:51:41.857+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:51:42.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:51:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:51:41.857+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:51:42.245 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:51:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:51:41.857+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:51:42.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:51:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:51:42.392+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:51:42.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:51:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:51:42.392+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:51:42.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:51:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:51:42.392+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:51:43.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:51:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:51:42.916+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:51:43.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:51:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:51:42.916+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:51:43.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:51:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:51:42.916+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:51:43.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:51:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:51:42.878+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:51:43.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:51:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:51:42.878+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:51:43.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:51:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:51:42.878+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:51:43.245 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:51:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:51:42.878+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:51:43.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:51:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:51:43.356+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:51:43.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:51:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:51:43.356+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:51:43.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:51:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:51:43.356+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:51:44.243 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:51:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:51:43.852+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 44694 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:51:44.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:51:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:51:43.921+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:51:44.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:51:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:51:43.921+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:51:44.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:51:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:51:43.921+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:51:44.245 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:51:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:51:43.838+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:51:44.245 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:51:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:51:43.838+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:51:44.245 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:51:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:51:43.838+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:51:44.245 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:51:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:51:43.838+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:51:44.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:51:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:51:44.311+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:51:44.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:51:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:51:44.311+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:51:44.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:51:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:51:44.311+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:51:45.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:51:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:51:44.928+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:51:45.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:51:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:51:44.928+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:51:45.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:51:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:51:44.928+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:51:45.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:51:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:51:44.833+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:51:45.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:51:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:51:44.833+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:51:45.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:51:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:51:44.833+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:51:45.245 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:51:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:51:44.833+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:51:45.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:51:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:51:45.297+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:51:45.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:51:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:51:45.297+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:51:45.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:51:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:51:45.297+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:51:46.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:51:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:51:45.968+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:51:46.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:51:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:51:45.968+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:51:46.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:51:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:51:45.968+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:51:46.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:51:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:51:45.834+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:51:46.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:51:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:51:45.834+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:51:46.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:51:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:51:45.834+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:51:46.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:51:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:51:45.834+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:51:46.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:51:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:51:46.292+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:51:46.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:51:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:51:46.292+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:51:46.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:51:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:51:46.292+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:51:47.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:51:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:51:46.965+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:51:47.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:51:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:51:46.965+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:51:47.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:51:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:51:46.965+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:51:47.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:51:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:51:46.826+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:51:47.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:51:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:51:46.826+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:51:47.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:51:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:51:46.826+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:51:47.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:51:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:51:46.826+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:51:47.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:51:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:51:47.261+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:51:47.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:51:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:51:47.261+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:51:47.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:51:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:51:47.261+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:51:48.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:51:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:51:47.958+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:51:48.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:51:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:51:47.958+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:51:48.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:51:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:51:47.958+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:51:48.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:51:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:51:47.864+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:51:48.245 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:51:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:51:47.864+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:51:48.245 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:51:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:51:47.864+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:51:48.245 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:51:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:51:47.864+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:51:48.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:51:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:51:48.290+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:51:48.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:51:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:51:48.290+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:51:48.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:51:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:51:48.290+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:51:49.243 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:51:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:51:48.860+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 44764 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:51:49.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:51:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:51:48.995+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:51:49.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:51:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:51:48.995+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:51:49.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:51:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:51:48.995+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:51:49.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:51:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:51:48.897+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:51:49.245 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:51:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:51:48.897+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:51:49.245 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:51:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:51:48.897+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:51:49.245 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:51:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:51:48.897+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:51:49.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:51:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:51:49.255+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:51:49.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:51:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:51:49.255+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:51:49.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:51:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:51:49.255+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:51:50.222 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:51:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:51:49.962+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:51:50.222 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:51:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:51:49.962+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:51:50.222 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:51:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:51:49.962+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:51:50.222 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:51:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:51:49.859+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:51:50.223 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:51:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:51:49.859+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:51:50.223 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:51:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:51:49.859+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:51:50.223 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:51:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:51:49.860+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:51:50.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:51:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:51:50.219+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:51:50.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:51:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:51:50.219+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:51:50.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:51:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:51:50.219+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:51:51.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:51:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:51:50.999+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:51:51.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:51:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:51:50.999+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:51:51.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:51:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:51:50.999+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:51:51.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:51:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:51:50.861+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:51:51.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:51:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:51:50.861+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:51:51.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:51:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:51:50.861+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:51:51.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:51:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:51:50.861+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:51:51.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:51:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:51:51.262+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:51:51.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:51:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:51:51.262+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:51:51.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:51:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:51:51.263+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:51:52.222 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:51:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:51:51.965+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:51:52.222 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:51:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:51:51.965+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:51:52.222 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:51:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:51:51.965+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:51:52.223 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:51:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:51:51.875+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:51:52.223 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:51:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:51:51.875+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:51:52.223 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:51:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:51:51.875+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:51:52.223 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:51:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:51:51.875+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:51:52.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:51:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:51:52.219+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:51:52.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:51:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:51:52.219+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:51:52.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:51:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:51:52.219+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:51:53.210 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:51:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:51:52.942+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:51:53.210 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:51:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:51:52.942+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:51:53.210 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:51:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:51:52.942+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:51:53.211 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:51:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:51:52.852+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:51:53.211 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:51:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:51:52.852+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:51:53.211 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:51:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:51:52.852+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:51:53.211 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:51:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:51:52.852+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:51:53.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:51:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:51:53.248+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:51:53.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:51:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:51:53.248+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:51:53.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:51:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:51:53.248+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:51:54.218 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:51:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:51:53.866+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 44829 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:51:54.219 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:51:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:51:53.910+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:51:54.219 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:51:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:51:53.910+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:51:54.219 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:51:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:51:53.910+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:51:54.219 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:51:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:51:53.815+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:51:54.220 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:51:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:51:53.815+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:51:54.220 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:51:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:51:53.815+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:51:54.220 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:51:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:51:53.815+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:51:54.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:51:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:51:54.216+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:51:54.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:51:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:51:54.216+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:51:54.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:51:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:51:54.216+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:51:55.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:51:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:51:54.921+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:51:55.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:51:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:51:54.921+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:51:55.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:51:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:51:54.921+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:51:55.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:51:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:51:54.839+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:51:55.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:51:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:51:54.839+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:51:55.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:51:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:51:54.839+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:51:55.245 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:51:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:51:54.839+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:51:55.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:51:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:51:55.252+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:51:55.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:51:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:51:55.252+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:51:55.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:51:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:51:55.252+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:51:56.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:51:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:51:55.888+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:51:56.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:51:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:51:55.888+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:51:56.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:51:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:51:55.888+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:51:56.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:51:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:51:55.804+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:51:56.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:51:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:51:55.804+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:51:56.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:51:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:51:55.804+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:51:56.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:51:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:51:55.804+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:51:56.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:51:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:51:56.300+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:51:56.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:51:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:51:56.300+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:51:56.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:51:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:51:56.300+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:51:57.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:51:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:51:56.883+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:51:57.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:51:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:51:56.883+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:51:57.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:51:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:51:56.883+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:51:57.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:51:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:51:56.786+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:51:57.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:51:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:51:56.786+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:51:57.245 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:51:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:51:56.786+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:51:57.245 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:51:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:51:56.786+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:51:57.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:51:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:51:57.317+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:51:57.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:51:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:51:57.317+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:51:57.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:51:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:51:57.317+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:51:58.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:51:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:51:57.931+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:51:58.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:51:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:51:57.931+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:51:58.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:51:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:51:57.931+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:51:58.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:51:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:51:57.758+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:51:58.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:51:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:51:57.758+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:51:58.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:51:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:51:57.758+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:51:58.245 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:51:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:51:57.758+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:51:58.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:51:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:51:58.271+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:51:58.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:51:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:51:58.272+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:51:58.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:51:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:51:58.272+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:51:59.243 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:51:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:51:58.873+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 44896 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:51:59.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:51:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:51:58.944+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:51:59.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:51:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:51:58.944+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:51:59.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:51:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:51:58.944+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:51:59.245 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:51:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:51:58.803+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:51:59.245 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:51:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:51:58.803+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:51:59.245 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:51:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:51:58.803+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:51:59.245 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:51:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:51:58.803+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:51:59.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:51:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:51:59.281+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:51:59.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:51:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:51:59.281+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:51:59.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:51:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:51:59.281+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:52:00.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:51:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:51:59.960+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:52:00.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:51:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:51:59.960+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:52:00.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:51:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:51:59.960+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:52:00.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:51:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:51:59.786+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:52:00.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:51:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:51:59.786+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:52:00.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:51:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:51:59.786+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:52:00.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:51:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:51:59.786+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:52:00.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:52:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:52:00.323+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:52:00.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:52:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:52:00.323+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:52:00.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:52:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:52:00.323+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:52:01.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:52:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:52:00.980+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:52:01.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:52:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:52:00.980+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:52:01.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:52:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:52:00.980+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:52:01.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:52:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:52:00.777+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:52:01.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:52:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:52:00.777+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:52:01.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:52:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:52:00.777+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:52:01.245 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:52:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:52:00.777+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:52:01.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:52:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:52:01.302+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:52:01.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:52:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:52:01.302+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:52:01.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:52:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:52:01.302+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:52:01.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:52:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:52:01.941+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:52:01.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:52:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:52:01.941+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:52:01.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:52:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:52:01.941+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:52:01.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:52:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:52:01.738+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:52:01.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:52:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:52:01.738+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:52:01.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:52:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:52:01.738+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:52:01.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:52:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:52:01.738+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:52:02.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:52:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:52:02.265+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:52:02.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:52:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:52:02.265+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:52:02.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:52:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:52:02.265+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:52:03.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:52:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:52:02.948+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:52:03.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:52:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:52:02.948+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:52:03.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:52:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:52:02.948+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:52:03.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:52:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:52:02.751+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:52:03.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:52:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:52:02.751+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:52:03.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:52:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:52:02.751+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:52:03.245 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:52:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:52:02.751+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:52:03.717 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:52:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:52:03.313+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:52:03.718 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:52:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:52:03.313+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:52:03.718 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:52:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:52:03.313+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:52:03.991 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:52:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:52:03.880+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 44964 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:52:03.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:52:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:52:03.714+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:52:03.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:52:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:52:03.714+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:52:03.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:52:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:52:03.714+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:52:03.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:52:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:52:03.715+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:52:04.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:52:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:52:03.988+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:52:04.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:52:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:52:03.988+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:52:04.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:52:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:52:03.988+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:52:04.732 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:52:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:52:04.264+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:52:04.733 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:52:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:52:04.264+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:52:04.733 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:52:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:52:04.264+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:52:04.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:52:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:52:04.729+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:52:04.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:52:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:52:04.729+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:52:04.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:52:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:52:04.729+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:52:04.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:52:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:52:04.729+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:52:05.258 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:52:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:52:05.005+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:52:05.259 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:52:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:52:05.005+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:52:05.259 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:52:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:52:05.005+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:52:05.711 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:52:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:52:05.256+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:52:05.711 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:52:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:52:05.256+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:52:05.712 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:52:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:52:05.256+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:52:05.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:52:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:52:05.709+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:52:05.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:52:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:52:05.709+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:52:05.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:52:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:52:05.709+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:52:05.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:52:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:52:05.709+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:52:06.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:52:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:52:06.045+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:52:06.494 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:52:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:52:06.045+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:52:06.494 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:52:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:52:06.045+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:52:06.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:52:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:52:06.235+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:52:06.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:52:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:52:06.235+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:52:06.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:52:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:52:06.235+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:52:07.082 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:52:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:52:06.750+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:52:07.082 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:52:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:52:06.750+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:52:07.082 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:52:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:52:06.750+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:52:07.082 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:52:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:52:06.750+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:52:07.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:52:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:52:07.079+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:52:07.494 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:52:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:52:07.079+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:52:07.494 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:52:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:52:07.079+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:52:07.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:52:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:52:07.207+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:52:07.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:52:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:52:07.207+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:52:07.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:52:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:52:07.207+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:52:07.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:52:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:52:07.718+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:52:07.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:52:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:52:07.718+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:52:07.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:52:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:52:07.718+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:52:07.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:52:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:52:07.718+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:52:08.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:52:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:52:08.057+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:52:08.494 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:52:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:52:08.057+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:52:08.494 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:52:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:52:08.057+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:52:08.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:52:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:52:08.240+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:52:08.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:52:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:52:08.240+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:52:08.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:52:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:52:08.240+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:52:09.023 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:52:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:52:08.887+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 45034 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:52:09.024 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:52:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:52:08.766+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:52:09.024 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:52:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:52:08.766+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:52:09.024 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:52:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:52:08.766+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:52:09.024 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:52:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:52:08.766+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:52:09.494 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:52:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:52:09.021+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:52:09.494 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:52:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:52:09.021+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:52:09.494 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:52:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:52:09.021+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:52:09.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:52:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:52:09.241+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:52:09.495 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:52:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:52:09.241+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:52:09.495 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:52:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:52:09.241+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:52:09.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:52:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:52:09.724+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:52:09.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:52:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:52:09.724+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:52:09.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:52:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:52:09.724+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:52:09.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:52:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:52:09.724+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:52:10.327 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:52:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:52:10.046+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:52:10.327 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:52:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:52:10.046+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:52:10.328 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:52:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:52:10.046+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:52:10.328 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:52:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:52:10.242+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:52:10.328 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:52:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:52:10.242+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:52:10.328 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:52:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:52:10.242+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:52:11.006 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:52:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:52:10.751+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:52:11.007 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:52:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:52:10.751+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:52:11.007 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:52:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:52:10.751+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:52:11.007 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:52:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:52:10.752+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:52:11.263 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:52:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:52:11.002+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:52:11.263 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:52:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:52:11.002+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:52:11.263 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:52:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:52:11.002+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:52:11.642 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:52:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:52:11.259+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:52:11.642 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:52:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:52:11.259+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:52:11.642 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:52:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:52:11.259+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:52:11.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:52:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:52:11.715+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:52:11.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:52:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:52:11.715+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:52:11.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:52:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:52:11.715+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:52:11.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:52:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:52:11.715+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:52:12.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:52:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:52:12.017+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:52:12.494 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:52:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:52:12.017+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:52:12.494 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:52:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:52:12.017+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:52:12.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:52:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:52:12.236+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:52:12.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:52:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:52:12.236+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:52:12.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:52:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:52:12.236+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:52:12.995 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:52:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:52:12.737+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:52:12.996 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:52:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:52:12.737+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:52:12.996 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:52:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:52:12.737+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:52:12.996 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:52:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:52:12.737+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:52:13.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:52:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:52:13.016+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:52:13.494 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:52:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:52:13.016+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:52:13.494 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:52:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:52:13.016+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:52:13.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:52:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:52:13.254+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:52:13.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:52:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:52:13.254+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:52:13.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:52:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:52:13.254+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:52:13.979 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:52:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:52:13.893+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 45102 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:52:13.980 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:52:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:52:13.703+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:52:13.980 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:52:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:52:13.703+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:52:13.980 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:52:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:52:13.703+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:52:13.980 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:52:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:52:13.703+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:52:14.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:52:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:52:13.975+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:52:14.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:52:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:52:13.975+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:52:14.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:52:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:52:13.975+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:52:14.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:52:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:52:14.220+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:52:14.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:52:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:52:14.220+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:52:14.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:52:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:52:14.220+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:52:14.988 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:52:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:52:14.667+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:52:14.988 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:52:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:52:14.667+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:52:14.988 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:52:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:52:14.667+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:52:14.989 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:52:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:52:14.667+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:52:15.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:52:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:52:14.984+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:52:15.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:52:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:52:14.984+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:52:15.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:52:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:52:14.984+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:52:15.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:52:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:52:15.187+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:52:15.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:52:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:52:15.187+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:52:15.245 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:52:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:52:15.187+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:52:15.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:52:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:52:15.709+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:52:15.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:52:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:52:15.709+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:52:15.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:52:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:52:15.709+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:52:15.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:52:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:52:15.709+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:52:16.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:52:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:52:15.998+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:52:16.494 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:52:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:52:15.998+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:52:16.494 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:52:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:52:15.998+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:52:16.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:52:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:52:16.215+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:52:16.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:52:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:52:16.215+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:52:16.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:52:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:52:16.215+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:52:16.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:52:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:52:16.693+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:52:16.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:52:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:52:16.693+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:52:16.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:52:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:52:16.693+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:52:16.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:52:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:52:16.693+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:52:17.266 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:52:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:52:16.997+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:52:17.266 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:52:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:52:16.997+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:52:17.266 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:52:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:52:16.997+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:52:17.692 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:52:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:52:17.262+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:52:17.692 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:52:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:52:17.262+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:52:17.692 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:52:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:52:17.262+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:52:17.961 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:52:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:52:17.688+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:52:17.961 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:52:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:52:17.688+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:52:17.961 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:52:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:52:17.688+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:52:17.961 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:52:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:52:17.688+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:52:18.229 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:52:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:52:17.957+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:52:18.229 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:52:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:52:17.957+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:52:18.229 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:52:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:52:17.957+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:52:18.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:52:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:52:18.226+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:52:18.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:52:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:52:18.226+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:52:18.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:52:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:52:18.226+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:52:18.913 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:52:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:52:18.899+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 45175 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:52:18.913 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:52:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:52:18.652+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:52:18.913 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:52:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:52:18.652+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:52:18.914 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:52:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:52:18.652+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:52:18.914 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:52:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:52:18.652+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:52:19.181 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:52:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:52:18.915+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:52:19.182 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:52:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:52:18.915+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:52:19.182 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:52:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:52:18.915+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:52:19.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:52:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:52:19.177+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:52:19.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:52:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:52:19.177+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:52:19.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:52:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:52:19.177+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:52:19.887 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:52:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:52:19.631+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:52:19.887 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:52:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:52:19.631+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:52:19.887 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:52:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:52:19.631+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:52:19.887 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:52:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:52:19.631+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:52:20.177 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:52:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:52:19.887+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:52:20.178 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:52:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:52:19.887+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:52:20.178 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:52:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:52:19.887+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:52:20.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:52:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:52:20.174+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:52:20.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:52:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:52:20.174+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:52:20.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:52:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:52:20.174+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:52:20.900 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:52:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:52:20.606+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:52:20.900 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:52:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:52:20.606+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:52:20.901 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:52:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:52:20.606+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:52:20.901 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:52:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:52:20.606+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:52:21.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:52:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:52:20.896+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:52:21.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:52:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:52:20.896+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:52:21.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:52:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:52:20.896+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:52:21.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:52:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:52:21.131+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:52:21.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:52:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:52:21.131+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:52:21.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:52:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:52:21.131+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:52:21.884 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:52:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:52:21.629+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:52:21.884 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:52:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:52:21.629+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:52:21.885 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:52:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:52:21.629+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:52:21.885 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:52:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:52:21.630+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:52:22.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:52:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:52:21.880+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:52:22.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:52:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:52:21.880+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:52:22.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:52:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:52:21.880+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:52:22.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:52:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:52:22.084+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:52:22.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:52:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:52:22.084+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:52:22.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:52:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:52:22.084+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:52:22.913 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:52:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:52:22.626+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:52:22.913 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:52:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:52:22.626+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:52:22.914 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:52:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:52:22.626+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:52:22.914 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:52:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:52:22.626+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:52:23.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:52:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:52:22.910+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:52:23.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:52:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:52:22.910+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:52:23.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:52:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:52:22.910+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:52:23.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:52:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:52:23.118+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:52:23.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:52:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:52:23.118+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:52:23.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:52:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:52:23.118+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:52:23.911 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:52:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:52:23.886+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:52:23.911 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:52:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:52:23.886+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:52:23.911 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:52:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:52:23.886+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:52:23.912 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:52:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:52:23.645+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:52:23.912 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:52:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:52:23.645+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:52:23.912 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:52:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:52:23.645+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:52:23.912 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:52:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:52:23.645+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:52:24.243 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:52:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:52:23.907+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 45246 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:52:24.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:52:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:52:24.141+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:52:24.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:52:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:52:24.141+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:52:24.247 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:52:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:52:24.141+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:52:24.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:52:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:52:24.849+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:52:24.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:52:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:52:24.849+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:52:24.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:52:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:52:24.849+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:52:24.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:52:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:52:24.606+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:52:24.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:52:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:52:24.606+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:52:24.995 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:52:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:52:24.606+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:52:24.995 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:52:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:52:24.606+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:52:25.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:52:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:52:25.104+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:52:25.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:52:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:52:25.104+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:52:25.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:52:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:52:25.104+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:52:25.864 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:52:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:52:25.603+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:52:25.864 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:52:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:52:25.603+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:52:25.864 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:52:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:52:25.603+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:52:25.865 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:52:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:52:25.603+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:52:26.127 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:52:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:52:25.861+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:52:26.127 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:52:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:52:25.861+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:52:26.128 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:52:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:52:25.861+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:52:26.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:52:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:52:26.124+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:52:26.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:52:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:52:26.124+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:52:26.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:52:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:52:26.124+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:52:26.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:52:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:52:26.824+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:52:26.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:52:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:52:26.824+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:52:26.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:52:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:52:26.824+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:52:26.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:52:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:52:26.592+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:52:26.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:52:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:52:26.592+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:52:26.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:52:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:52:26.592+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:52:26.995 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:52:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:52:26.592+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:52:27.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:52:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:52:27.138+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:52:27.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:52:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:52:27.138+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:52:27.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:52:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:52:27.138+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:52:27.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:52:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:52:27.787+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:52:27.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:52:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:52:27.787+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:52:27.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:52:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:52:27.787+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:52:27.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:52:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:52:27.608+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:52:27.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:52:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:52:27.608+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:52:27.995 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:52:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:52:27.608+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:52:27.995 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:52:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:52:27.608+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:52:28.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:52:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:52:28.100+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:52:28.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:52:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:52:28.100+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:52:28.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:52:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:52:28.100+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:52:28.918 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:52:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:52:28.752+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:52:28.919 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:52:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:52:28.752+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:52:28.919 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:52:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:52:28.752+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:52:28.919 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:52:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:52:28.593+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:52:28.919 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:52:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:52:28.593+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:52:28.919 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:52:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:52:28.593+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:52:28.919 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:52:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:52:28.593+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:52:29.243 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:52:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:52:28.914+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 45319 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:52:29.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:52:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:52:29.052+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:52:29.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:52:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:52:29.052+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:52:29.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:52:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:52:29.052+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:52:29.887 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:52:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:52:29.749+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:52:29.887 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:52:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:52:29.749+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:52:29.887 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:52:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:52:29.749+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:52:29.888 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:52:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:52:29.559+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:52:29.888 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:52:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:52:29.559+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:52:29.888 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:52:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:52:29.559+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:52:29.888 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:52:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:52:29.559+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:52:30.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:52:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:52:30.060+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:52:30.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:52:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:52:30.060+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:52:30.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:52:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:52:30.060+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:52:30.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:52:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:52:30.601+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:52:30.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:52:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:52:30.601+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:52:30.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:52:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:52:30.601+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:52:30.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:52:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:52:30.601+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:52:30.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:52:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:52:30.706+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:52:30.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:52:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:52:30.706+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:52:30.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:52:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:52:30.706+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:52:31.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:52:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:52:31.075+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:52:31.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:52:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:52:31.075+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:52:31.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:52:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:52:31.075+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:52:31.856 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:52:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:52:31.753+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:52:31.857 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:52:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:52:31.753+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:52:31.857 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:52:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:52:31.753+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:52:31.857 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:52:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:52:31.601+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:52:31.857 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:52:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:52:31.601+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:52:31.857 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:52:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:52:31.602+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:52:31.857 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:52:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:52:31.602+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:52:32.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:52:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:52:32.124+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:52:32.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:52:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:52:32.124+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:52:32.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:52:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:52:32.124+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:52:32.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:52:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:52:32.726+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:52:32.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:52:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:52:32.726+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:52:32.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:52:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:52:32.726+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:52:32.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:52:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:52:32.634+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:52:32.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:52:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:52:32.634+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:52:32.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:52:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:52:32.634+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:52:32.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:52:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:52:32.634+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:52:33.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:52:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:52:33.090+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:52:33.523 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:52:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:52:33.090+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:52:33.523 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:52:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:52:33.090+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:52:33.925 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:52:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:52:33.698+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:52:33.925 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:52:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:52:33.698+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:52:33.926 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:52:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:52:33.698+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:52:33.926 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:52:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:52:33.626+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:52:33.926 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:52:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:52:33.626+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:52:33.926 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:52:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:52:33.626+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:52:33.926 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:52:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:52:33.626+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:52:34.243 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:52:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:52:33.921+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 45390 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:52:34.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:52:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:52:34.127+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:52:34.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:52:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:52:34.127+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:52:34.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:52:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:52:34.127+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:52:34.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:52:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:52:34.656+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:52:34.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:52:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:52:34.656+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:52:34.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:52:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:52:34.656+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:52:34.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:52:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:52:34.630+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:52:34.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:52:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:52:34.630+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:52:34.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:52:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:52:34.630+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:52:34.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:52:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:52:34.630+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:52:35.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:52:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:52:35.147+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:52:35.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:52:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:52:35.147+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:52:35.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:52:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:52:35.147+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:52:35.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:52:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:52:35.614+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:52:35.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:52:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:52:35.614+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:52:35.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:52:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:52:35.614+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:52:35.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:52:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:52:35.606+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:52:35.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:52:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:52:35.606+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:52:35.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:52:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:52:35.606+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:52:35.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:52:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:52:35.606+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:52:36.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:52:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:52:36.151+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:52:36.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:52:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:52:36.151+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:52:36.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:52:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:52:36.151+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:52:36.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:52:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:52:36.624+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:52:36.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:52:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:52:36.624+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:52:36.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:52:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:52:36.624+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:52:36.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:52:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:52:36.596+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:52:36.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:52:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:52:36.596+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:52:36.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:52:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:52:36.596+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:52:36.995 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:52:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:52:36.596+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:52:37.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:52:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:52:37.118+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:52:37.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:52:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:52:37.118+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:52:37.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:52:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:52:37.118+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:52:37.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:52:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:52:37.651+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:52:37.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:52:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:52:37.651+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:52:37.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:52:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:52:37.651+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:52:37.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:52:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:52:37.553+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:52:37.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:52:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:52:37.553+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:52:37.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:52:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:52:37.553+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:52:37.995 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:52:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:52:37.553+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:52:38.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:52:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:52:38.152+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:52:38.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:52:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:52:38.152+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:52:38.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:52:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:52:38.152+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:52:38.933 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:52:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:52:38.654+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:52:38.933 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:52:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:52:38.654+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:52:38.933 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:52:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:52:38.654+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:52:38.933 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:52:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:52:38.563+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:52:38.933 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:52:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:52:38.563+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:52:38.933 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:52:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:52:38.563+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:52:38.934 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:52:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:52:38.563+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:52:39.186 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:52:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:52:38.929+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 45463 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:52:39.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:52:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:52:39.183+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:52:39.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:52:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:52:39.183+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:52:39.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:52:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:52:39.183+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:52:39.887 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:52:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:52:39.541+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:52:39.888 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:52:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:52:39.541+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:52:39.888 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:52:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:52:39.541+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:52:39.888 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:52:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:52:39.541+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:52:39.888 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:52:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:52:39.648+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:52:39.888 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:52:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:52:39.648+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:52:39.889 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:52:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:52:39.648+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:52:40.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:52:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:52:40.147+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:52:40.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:52:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:52:40.147+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:52:40.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:52:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:52:40.147+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:52:40.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:52:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:52:40.502+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:52:40.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:52:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:52:40.502+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:52:40.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:52:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:52:40.502+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:52:40.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:52:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:52:40.502+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:52:40.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:52:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:52:40.626+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:52:40.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:52:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:52:40.626+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:52:40.995 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:52:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:52:40.626+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:52:41.489 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:52:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:52:41.113+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:52:41.489 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:52:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:52:41.113+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:52:41.489 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:52:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:52:41.113+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:52:41.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:52:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:52:41.486+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:52:41.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:52:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:52:41.486+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:52:41.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:52:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:52:41.486+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:52:41.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:52:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:52:41.486+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:52:41.744 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:52:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:52:41.677+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:52:41.744 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:52:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:52:41.677+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:52:41.744 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:52:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:52:41.677+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:52:42.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:52:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:52:42.152+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:52:42.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:52:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:52:42.152+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:52:42.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:52:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:52:42.152+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:52:42.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:52:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:52:42.536+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:52:42.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:52:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:52:42.536+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:52:42.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:52:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:52:42.536+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:52:42.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:52:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:52:42.536+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:52:42.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:52:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:52:42.696+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:52:42.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:52:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:52:42.696+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:52:42.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:52:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:52:42.696+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:52:43.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:52:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:52:43.175+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:52:43.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:52:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:52:43.175+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:52:43.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:52:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:52:43.175+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:52:43.939 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:52:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:52:43.721+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:52:43.939 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:52:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:52:43.721+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:52:43.940 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:52:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:52:43.721+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:52:43.940 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:52:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:52:43.538+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:52:43.940 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:52:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:52:43.538+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:52:43.940 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:52:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:52:43.538+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:52:43.940 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:52:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:52:43.538+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:52:44.211 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:52:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:52:43.936+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 45534 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:52:44.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:52:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:52:44.208+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:52:44.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:52:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:52:44.208+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:52:44.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:52:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:52:44.208+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:52:44.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:52:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:52:44.751+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:52:44.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:52:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:52:44.751+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:52:44.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:52:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:52:44.751+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:52:44.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:52:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:52:44.581+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:52:44.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:52:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:52:44.581+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:52:44.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:52:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:52:44.581+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:52:44.995 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:52:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:52:44.581+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:52:45.555 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:52:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:52:45.246+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:52:45.556 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:52:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:52:45.246+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:52:45.556 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:52:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:52:45.246+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:52:45.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:52:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:52:45.762+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:52:45.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:52:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:52:45.762+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:52:45.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:52:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:52:45.762+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:52:45.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:52:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:52:45.552+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:52:45.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:52:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:52:45.552+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:52:45.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:52:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:52:45.552+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:52:45.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:52:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:52:45.553+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:52:46.563 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:52:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:52:46.282+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:52:46.563 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:52:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:52:46.282+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:52:46.563 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:52:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:52:46.282+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:52:46.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:52:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:52:46.560+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:52:46.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:52:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:52:46.560+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:52:46.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:52:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:52:46.560+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:52:46.995 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:52:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:52:46.560+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:52:46.995 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:52:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:52:46.759+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:52:46.995 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:52:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:52:46.759+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:52:46.995 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:52:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:52:46.759+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:52:47.569 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:52:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:52:47.261+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:52:47.569 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:52:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:52:47.261+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:52:47.570 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:52:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:52:47.261+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:52:47.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:52:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:52:47.566+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:52:47.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:52:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:52:47.566+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:52:47.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:52:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:52:47.566+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:52:47.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:52:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:52:47.566+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:52:47.995 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:52:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:52:47.761+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:52:47.995 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:52:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:52:47.761+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:52:47.995 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:52:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:52:47.761+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:52:48.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:52:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:52:48.302+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:52:48.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:52:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:52:48.302+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:52:48.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:52:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:52:48.302+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:52:48.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:52:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:52:48.551+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:52:48.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:52:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:52:48.551+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:52:48.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:52:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:52:48.551+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:52:48.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:52:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:52:48.551+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:52:49.065 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:52:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:52:48.943+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 45607 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:52:49.066 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:52:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:52:48.750+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:52:49.066 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:52:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:52:48.750+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:52:49.066 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:52:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:52:48.750+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:52:49.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:52:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:52:49.292+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:52:49.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:52:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:52:49.292+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:52:49.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:52:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:52:49.292+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:52:49.887 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:52:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:52:49.748+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:52:49.887 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:52:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:52:49.748+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:52:49.888 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:52:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:52:49.748+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:52:49.888 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:52:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:52:49.594+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:52:49.888 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:52:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:52:49.594+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:52:49.888 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:52:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:52:49.594+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:52:49.888 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:52:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:52:49.594+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:52:50.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:52:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:52:50.556+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:52:50.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:52:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:52:50.556+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:52:50.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:52:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:52:50.556+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:52:50.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:52:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:52:50.556+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:52:50.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:52:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:52:50.324+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:52:50.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:52:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:52:50.324+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:52:50.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:52:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:52:50.324+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:52:51.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:52:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:52:50.746+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:52:51.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:52:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:52:50.746+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:52:51.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:52:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:52:50.746+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:52:51.726 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:52:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:52:51.339+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:52:51.726 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:52:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:52:51.339+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:52:51.726 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:52:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:52:51.339+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:52:51.726 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:52:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:52:51.523+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:52:51.727 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:52:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:52:51.523+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:52:51.727 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:52:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:52:51.523+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:52:51.727 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:52:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:52:51.523+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:52:51.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:52:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:52:51.723+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:52:51.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:52:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:52:51.723+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:52:51.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:52:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:52:51.723+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:52:52.721 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:52:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:52:52.310+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:52:52.722 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:52:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:52:52.310+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:52:52.722 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:52:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:52:52.310+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:52:52.722 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:52:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:52:52.473+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:52:52.722 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:52:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:52:52.473+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:52:52.722 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:52:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:52:52.473+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:52:52.722 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:52:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:52:52.473+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:52:52.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:52:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:52:52.718+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:52:52.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:52:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:52:52.718+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:52:52.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:52:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:52:52.718+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:52:53.740 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:52:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:52:53.357+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:52:53.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:52:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:52:53.357+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:52:53.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:52:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:52:53.357+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:52:53.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:52:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:52:53.523+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:52:53.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:52:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:52:53.523+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:52:53.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:52:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:52:53.523+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:52:53.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:52:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:52:53.523+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:52:53.993 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:52:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:52:53.950+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 45678 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:52:54.014 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:52:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:52:53.737+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:52:54.014 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:52:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:52:53.737+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:52:54.015 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:52:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:52:53.737+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:52:54.650 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:52:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:52:54.507+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:52:54.650 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:52:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:52:54.507+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:52:54.650 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:52:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:52:54.507+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:52:54.650 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:52:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:52:54.507+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:52:54.650 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:52:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:52:54.374+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:52:54.651 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:52:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:52:54.374+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:52:54.651 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:52:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:52:54.374+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:52:54.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:52:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:52:54.713+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:52:54.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:52:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:52:54.713+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:52:54.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:52:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:52:54.713+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:52:55.734 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:52:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:52:55.404+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:52:55.734 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:52:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:52:55.404+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:52:55.734 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:52:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:52:55.404+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:52:55.734 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:52:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:52:55.539+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:52:55.735 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:52:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:52:55.539+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:52:55.735 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:52:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:52:55.539+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:52:55.735 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:52:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:52:55.539+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:52:55.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:52:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:52:55.731+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:52:55.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:52:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:52:55.731+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:52:55.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:52:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:52:55.731+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:52:56.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:52:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:52:56.381+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:52:56.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:52:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:52:56.381+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:52:56.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:52:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:52:56.381+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:52:56.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:52:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:52:56.523+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:52:56.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:52:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:52:56.523+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:52:56.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:52:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:52:56.523+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:52:56.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:52:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:52:56.523+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:52:57.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:52:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:52:56.772+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:52:57.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:52:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:52:56.772+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:52:57.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:52:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:52:56.772+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:52:57.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:52:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:52:57.430+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:52:57.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:52:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:52:57.430+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:52:57.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:52:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:52:57.430+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:52:57.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:52:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:52:57.544+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:52:57.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:52:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:52:57.544+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:52:57.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:52:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:52:57.544+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:52:57.745 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:52:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:52:57.544+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:52:58.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:52:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:52:57.759+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:52:58.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:52:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:52:57.759+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:52:58.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:52:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:52:57.759+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:52:58.733 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:52:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:52:58.570+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:52:58.734 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:52:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:52:58.570+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:52:58.734 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:52:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:52:58.570+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:52:58.734 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:52:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:52:58.570+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:52:58.734 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:52:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:52:58.440+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:52:58.734 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:52:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:52:58.440+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:52:58.734 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:52:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:52:58.440+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:52:58.993 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:52:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:52:58.956+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 45751 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:52:58.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:52:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:52:58.731+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:52:58.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:52:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:52:58.731+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:52:58.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:52:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:52:58.731+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:52:59.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:52:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:52:59.567+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:52:59.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:52:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:52:59.567+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:52:59.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:52:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:52:59.567+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:52:59.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:52:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:52:59.568+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:52:59.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:52:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:52:59.441+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:52:59.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:52:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:52:59.441+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:52:59.745 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:52:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:52:59.441+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:53:00.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:52:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:52:59.779+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:53:00.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:52:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:52:59.779+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:53:00.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:52:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:52:59.779+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:53:00.734 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:53:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:53:00.417+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:53:00.734 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:53:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:53:00.417+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:53:00.734 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:53:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:53:00.417+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:53:00.735 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:53:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:53:00.523+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:53:00.735 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:53:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:53:00.523+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:53:00.735 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:53:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:53:00.523+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:53:00.735 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:53:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:53:00.523+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:53:00.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:53:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:53:00.731+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:53:00.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:53:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:53:00.731+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:53:00.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:53:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:53:00.731+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:53:01.703 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:53:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:53:01.435+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:53:01.703 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:53:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:53:01.435+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:53:01.703 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:53:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:53:01.435+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:53:01.703 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:53:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:53:01.570+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:53:01.704 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:53:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:53:01.570+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:53:01.704 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:53:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:53:01.570+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:53:01.704 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:53:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:53:01.571+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:53:01.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:53:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:53:01.700+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:53:01.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:53:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:53:01.700+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:53:01.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:53:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:53:01.700+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:53:02.722 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:53:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:53:02.431+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:53:02.723 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:53:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:53:02.431+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:53:02.723 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:53:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:53:02.431+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:53:02.723 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:53:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:53:02.606+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:53:02.723 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:53:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:53:02.606+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:53:02.723 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:53:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:53:02.606+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:53:02.723 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:53:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:53:02.606+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:53:02.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:53:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:53:02.719+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:53:02.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:53:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:53:02.719+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:53:02.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:53:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:53:02.719+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:53:03.733 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:53:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:53:03.444+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:53:03.734 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:53:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:53:03.444+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:53:03.734 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:53:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:53:03.444+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:53:03.734 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:53:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:53:03.613+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:53:03.734 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:53:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:53:03.613+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:53:03.734 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:53:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:53:03.613+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:53:03.735 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:53:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:53:03.613+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:53:03.987 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:53:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:53:03.962+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 45822 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:53:03.988 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:53:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:53:03.730+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:53:03.988 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:53:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:53:03.730+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:53:03.989 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:53:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:53:03.730+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:53:04.689 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:53:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:53:04.423+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:53:04.689 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:53:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:53:04.423+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:53:04.689 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:53:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:53:04.423+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:53:04.690 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:53:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:53:04.647+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:53:04.690 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:53:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:53:04.647+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:53:04.690 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:53:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:53:04.647+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:53:04.690 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:53:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:53:04.647+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:53:04.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:53:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:53:04.685+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:53:04.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:53:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:53:04.685+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:53:04.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:53:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:53:04.685+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:53:05.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:53:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:53:05.465+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:53:05.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:53:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:53:05.465+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:53:05.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:53:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:53:05.465+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:53:05.866 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:53:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:53:05.726+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:53:05.866 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:53:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:53:05.726+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:53:05.866 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:53:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:53:05.726+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:53:05.867 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:53:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:53:05.617+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:53:05.867 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:53:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:53:05.617+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:53:05.867 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:53:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:53:05.617+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:53:05.867 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:53:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:53:05.617+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:53:06.637 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:53:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:53:06.464+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:53:06.637 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:53:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:53:06.464+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:53:06.637 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:53:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:53:06.464+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:53:06.957 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:53:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:53:06.677+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:53:06.957 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:53:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:53:06.677+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:53:06.957 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:53:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:53:06.677+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:53:06.957 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:53:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:53:06.633+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:53:06.957 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:53:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:53:06.633+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:53:06.957 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:53:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:53:06.633+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:53:06.958 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:53:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:53:06.633+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:53:07.675 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:53:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:53:07.474+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:53:07.675 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:53:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:53:07.474+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:53:07.675 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:53:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:53:07.474+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:53:07.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:53:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:53:07.709+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:53:07.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:53:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:53:07.709+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:53:07.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:53:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:53:07.709+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:53:07.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:53:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:53:07.671+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:53:07.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:53:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:53:07.671+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:53:07.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:53:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:53:07.671+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:53:07.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:53:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:53:07.671+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:53:08.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:53:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:53:08.691+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:53:08.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:53:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:53:08.691+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:53:08.744 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:53:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:53:08.691+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:53:08.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:53:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:53:08.483+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:53:08.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:53:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:53:08.483+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:53:08.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:53:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:53:08.483+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:53:08.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:53:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:53:08.680+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:53:08.745 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:53:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:53:08.680+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:53:08.745 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:53:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:53:08.680+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:53:08.745 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:53:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:53:08.680+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:53:09.243 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:53:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:53:08.969+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 45895 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:53:09.887 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:53:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:53:09.701+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:53:09.887 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:53:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:53:09.701+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:53:09.888 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:53:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:53:09.701+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:53:09.888 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:53:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:53:09.505+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:53:09.888 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:53:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:53:09.505+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:53:09.888 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:53:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:53:09.505+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:53:09.889 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:53:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:53:09.706+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:53:09.889 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:53:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:53:09.706+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:53:09.889 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:53:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:53:09.706+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:53:09.889 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:53:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:53:09.706+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:53:10.724 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:53:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:53:10.487+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:53:10.725 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:53:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:53:10.487+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:53:10.725 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:53:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:53:10.487+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:53:10.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:53:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:53:10.722+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:53:10.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:53:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:53:10.722+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:53:10.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:53:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:53:10.722+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:53:10.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:53:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:53:10.721+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:53:10.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:53:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:53:10.721+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:53:10.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:53:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:53:10.721+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:53:10.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:53:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:53:10.721+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:53:11.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:53:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:53:11.472+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:53:11.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:53:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:53:11.472+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:53:11.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:53:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:53:11.472+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:53:11.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:53:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:53:11.682+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:53:11.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:53:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:53:11.682+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:53:11.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:53:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:53:11.682+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:53:11.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:53:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:53:11.682+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:53:12.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:53:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:53:11.760+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:53:12.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:53:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:53:11.760+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:53:12.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:53:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:53:11.760+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:53:12.714 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:53:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:53:12.461+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:53:12.714 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:53:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:53:12.461+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:53:12.715 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:53:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:53:12.461+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:53:12.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:53:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:53:12.798+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:53:12.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:53:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:53:12.798+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:53:12.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:53:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:53:12.798+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:53:12.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:53:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:53:12.711+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:53:12.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:53:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:53:12.711+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:53:12.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:53:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:53:12.711+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:53:12.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:53:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:53:12.711+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:53:13.755 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:53:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:53:13.496+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:53:13.755 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:53:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:53:13.496+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:53:13.755 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:53:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:53:13.496+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:53:13.755 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:53:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:53:13.720+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:53:13.756 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:53:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:53:13.720+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:53:13.756 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:53:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:53:13.720+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:53:13.756 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:53:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:53:13.720+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:53:14.103 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:53:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:53:13.977+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 45962 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:53:14.103 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:53:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:53:13.751+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:53:14.103 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:53:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:53:13.751+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:53:14.103 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:53:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:53:13.751+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:53:14.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:53:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:53:14.738+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:53:14.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:53:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:53:14.738+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:53:14.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:53:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:53:14.738+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:53:14.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:53:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:53:14.492+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:53:14.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:53:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:53:14.492+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:53:14.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:53:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:53:14.492+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:53:14.995 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:53:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:53:14.715+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:53:14.995 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:53:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:53:14.715+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:53:14.995 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:53:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:53:14.715+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:53:14.995 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:53:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:53:14.715+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:53:15.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:53:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:53:15.449+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:53:15.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:53:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:53:15.449+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:53:15.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:53:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:53:15.449+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:53:15.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:53:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:53:15.739+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:53:15.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:53:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:53:15.739+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:53:15.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:53:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:53:15.739+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:53:15.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:53:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:53:15.739+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:53:15.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:53:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:53:15.775+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:53:15.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:53:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:53:15.775+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:53:15.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:53:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:53:15.775+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:53:16.713 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:53:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:53:16.422+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:53:16.713 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:53:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:53:16.422+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:53:16.714 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:53:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:53:16.422+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:53:16.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:53:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:53:16.771+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:53:16.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:53:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:53:16.771+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:53:16.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:53:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:53:16.771+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:53:16.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:53:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:53:16.710+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:53:16.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:53:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:53:16.710+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:53:16.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:53:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:53:16.710+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:53:16.995 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:53:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:53:16.710+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:53:17.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:53:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:53:17.428+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:53:17.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:53:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:53:17.428+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:53:17.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:53:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:53:17.428+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:53:18.197 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:53:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:53:17.748+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:53:18.198 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:53:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:53:17.748+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:53:18.198 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:53:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:53:17.748+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:53:18.198 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:53:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:53:17.750+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:53:18.198 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:53:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:53:17.750+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:53:18.198 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:53:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:53:17.750+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:53:18.199 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:53:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:53:17.750+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:53:18.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:53:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:53:18.387+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:53:18.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:53:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:53:18.387+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:53:18.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:53:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:53:18.387+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:53:18.987 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:53:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:53:18.713+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:53:18.987 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:53:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:53:18.713+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:53:18.988 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:53:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:53:18.713+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:53:18.988 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:53:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:53:18.792+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:53:18.988 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:53:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:53:18.792+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:53:18.988 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:53:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:53:18.792+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:53:18.988 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:53:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:53:18.792+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:53:19.243 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:53:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:53:18.983+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 46034 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:53:19.708 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:53:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:53:19.337+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:53:19.708 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:53:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:53:19.337+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:53:19.708 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:53:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:53:19.337+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:53:19.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:53:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:53:19.703+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:53:19.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:53:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:53:19.703+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:53:19.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:53:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:53:19.703+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:53:19.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:53:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:53:19.841+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:53:19.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:53:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:53:19.841+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:53:19.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:53:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:53:19.841+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:53:19.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:53:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:53:19.841+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:53:20.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:53:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:53:20.293+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:53:20.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:53:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:53:20.293+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:53:20.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:53:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:53:20.293+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:53:21.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:53:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:53:20.887+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:53:21.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:53:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:53:20.887+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:53:21.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:53:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:53:20.887+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:53:21.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:53:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:53:20.888+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:53:21.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:53:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:53:20.740+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:53:21.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:53:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:53:20.740+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:53:21.245 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:53:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:53:20.740+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:53:21.735 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:53:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:53:21.305+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:53:21.736 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:53:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:53:21.305+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:53:21.736 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:53:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:53:21.305+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:53:21.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:53:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:53:21.894+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:53:21.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:53:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:53:21.894+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:53:21.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:53:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:53:21.894+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:53:21.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:53:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:53:21.894+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:53:21.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:53:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:53:21.732+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:53:21.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:53:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:53:21.732+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:53:21.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:53:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:53:21.732+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:53:22.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:53:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:53:22.305+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:53:22.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:53:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:53:22.305+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:53:22.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:53:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:53:22.305+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:53:23.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:53:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:53:22.757+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:53:23.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:53:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:53:22.757+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:53:23.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:53:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:53:22.757+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:53:23.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:53:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:53:22.941+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:53:23.245 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:53:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:53:22.941+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:53:23.245 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:53:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:53:22.941+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:53:23.245 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:53:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:53:22.941+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:53:23.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:53:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:53:23.262+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:53:23.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:53:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:53:23.262+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:53:23.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:53:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:53:23.262+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:53:24.003 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:53:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:53:23.991+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 46106 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:53:24.004 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:53:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:53:23.745+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:53:24.004 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:53:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:53:23.745+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:53:24.004 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:53:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:53:23.745+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:53:24.004 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:53:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:53:23.953+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:53:24.004 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:53:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:53:23.953+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:53:24.004 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:53:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:53:23.953+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:53:24.005 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:53:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:53:23.953+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:53:24.253 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:53:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:53:24.250+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:53:24.254 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:53:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:53:24.250+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:53:24.254 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:53:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:53:24.250+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:53:25.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:53:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:53:24.745+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:53:25.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:53:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:53:24.745+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:53:25.245 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:53:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:53:24.745+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:53:25.245 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:53:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:53:24.981+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:53:25.245 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:53:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:53:24.981+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:53:25.245 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:53:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:53:24.981+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:53:25.245 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:53:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:53:24.981+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:53:25.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:53:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:53:25.275+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:53:25.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:53:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:53:25.275+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:53:25.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:53:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:53:25.275+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:53:26.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:53:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:53:25.741+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:53:26.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:53:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:53:25.741+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:53:26.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:53:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:53:25.741+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:53:26.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:53:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:53:25.943+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:53:26.245 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:53:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:53:25.943+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:53:26.245 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:53:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:53:25.943+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:53:26.245 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:53:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:53:25.943+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:53:26.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:53:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:53:26.322+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:53:26.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:53:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:53:26.322+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:53:26.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:53:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:53:26.322+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:53:27.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:53:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:53:26.763+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:53:27.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:53:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:53:26.763+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:53:27.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:53:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:53:26.763+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:53:27.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:53:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:53:26.915+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:53:27.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:53:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:53:26.915+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:53:27.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:53:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:53:26.915+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:53:27.245 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:53:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:53:26.915+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:53:27.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:53:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:53:27.330+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:53:27.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:53:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:53:27.330+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:53:27.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:53:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:53:27.330+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:53:28.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:53:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:53:27.763+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:53:28.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:53:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:53:27.763+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:53:28.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:53:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:53:27.763+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:53:28.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:53:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:53:27.870+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:53:28.245 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:53:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:53:27.870+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:53:28.245 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:53:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:53:27.870+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:53:28.245 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:53:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:53:27.870+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:53:28.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:53:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:53:28.329+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:53:28.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:53:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:53:28.329+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:53:28.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:53:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:53:28.329+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:53:29.020 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:53:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:53:28.997+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 46177 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:53:29.020 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:53:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:53:28.759+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:53:29.021 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:53:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:53:28.759+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:53:29.021 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:53:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:53:28.759+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:53:29.021 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:53:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:53:28.877+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:53:29.021 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:53:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:53:28.877+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:53:29.021 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:53:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:53:28.877+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:53:29.021 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:53:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:53:28.877+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:53:29.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:53:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:53:29.320+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:53:29.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:53:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:53:29.320+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:53:29.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:53:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:53:29.320+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:53:30.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:53:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:53:29.741+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:53:30.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:53:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:53:29.741+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:53:30.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:53:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:53:29.741+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:53:30.245 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:53:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:53:29.830+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:53:30.245 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:53:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:53:29.830+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:53:30.245 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:53:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:53:29.830+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:53:30.245 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:53:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:53:29.830+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:53:30.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:53:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:53:30.290+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:53:30.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:53:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:53:30.290+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:53:30.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:53:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:53:30.290+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:53:31.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:53:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:53:30.750+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:53:31.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:53:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:53:30.750+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:53:31.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:53:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:53:30.750+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:53:31.245 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:53:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:53:30.866+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:53:31.245 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:53:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:53:30.866+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:53:31.245 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:53:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:53:30.866+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:53:31.245 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:53:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:53:30.866+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:53:31.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:53:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:53:31.295+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:53:31.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:53:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:53:31.295+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:53:31.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:53:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:53:31.295+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:53:32.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:53:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:53:31.770+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:53:32.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:53:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:53:31.770+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:53:32.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:53:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:53:31.770+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:53:32.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:53:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:53:31.867+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:53:32.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:53:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:53:31.867+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:53:32.245 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:53:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:53:31.867+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:53:32.245 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:53:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:53:31.867+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:53:32.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:53:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:53:32.309+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:53:32.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:53:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:53:32.309+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:53:32.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:53:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:53:32.309+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:53:33.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:53:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:53:32.791+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:53:33.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:53:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:53:32.791+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:53:33.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:53:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:53:32.791+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:53:33.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:53:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:53:32.831+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:53:33.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:53:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:53:32.831+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:53:33.245 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:53:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:53:32.831+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:53:33.245 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:53:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:53:32.832+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:53:33.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:53:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:53:33.335+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:53:33.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:53:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:53:33.335+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:53:33.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:53:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:53:33.335+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:53:34.131 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:53:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:53:34.004+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 46249 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:53:34.131 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:53:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:53:33.795+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:53:34.131 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:53:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:53:33.795+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:53:34.132 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:53:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:53:33.795+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:53:34.132 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:53:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:53:33.815+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:53:34.132 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:53:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:53:33.815+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:53:34.132 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:53:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:53:33.815+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:53:34.133 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:53:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:53:33.815+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:53:34.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:53:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:53:34.344+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:53:34.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:53:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:53:34.344+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:53:34.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:53:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:53:34.344+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:53:35.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:53:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:53:34.841+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:53:35.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:53:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:53:34.841+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:53:35.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:53:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:53:34.841+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:53:35.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:53:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:53:34.845+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:53:35.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:53:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:53:34.845+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:53:35.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:53:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:53:34.845+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:53:35.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:53:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:53:34.846+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:53:35.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:53:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:53:35.323+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:53:35.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:53:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:53:35.323+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:53:35.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:53:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:53:35.323+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:53:36.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:53:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:53:35.855+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:53:36.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:53:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:53:35.855+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:53:36.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:53:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:53:35.855+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:53:36.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:53:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:53:35.823+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:53:36.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:53:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:53:35.823+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:53:36.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:53:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:53:35.823+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:53:36.245 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:53:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:53:35.824+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:53:36.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:53:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:53:36.343+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:53:36.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:53:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:53:36.343+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:53:36.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:53:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:53:36.343+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:53:37.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:53:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:53:36.817+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:53:37.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:53:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:53:36.817+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:53:37.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:53:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:53:36.817+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:53:37.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:53:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:53:36.872+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:53:37.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:53:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:53:36.872+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:53:37.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:53:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:53:36.872+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:53:37.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:53:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:53:36.872+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:53:37.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:53:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:53:37.358+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:53:37.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:53:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:53:37.358+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:53:37.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:53:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:53:37.358+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:53:38.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:53:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:53:37.798+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:53:38.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:53:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:53:37.798+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:53:38.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:53:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:53:37.798+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:53:38.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:53:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:53:37.872+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:53:38.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:53:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:53:37.872+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:53:38.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:53:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:53:37.872+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:53:38.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:53:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:53:37.872+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:53:38.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:53:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:53:38.388+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:53:38.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:53:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:53:38.388+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:53:38.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:53:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:53:38.388+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:53:39.134 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:53:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:53:39.011+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 46320 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:53:39.135 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:53:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:53:38.788+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:53:39.135 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:53:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:53:38.788+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:53:39.135 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:53:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:53:38.788+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:53:39.136 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:53:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:53:38.880+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:53:39.136 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:53:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:53:38.880+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:53:39.136 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:53:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:53:38.880+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:53:39.136 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:53:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:53:38.880+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:53:39.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:53:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:53:39.423+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:53:39.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:53:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:53:39.423+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:53:39.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:53:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:53:39.423+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:53:40.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:53:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:53:39.748+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:53:40.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:53:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:53:39.749+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:53:40.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:53:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:53:39.749+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:53:40.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:53:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:53:39.877+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:53:40.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:53:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:53:39.877+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:53:40.245 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:53:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:53:39.877+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:53:40.245 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:53:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:53:39.878+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:53:40.730 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:53:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:53:40.383+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:53:40.730 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:53:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:53:40.383+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:53:40.730 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:53:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:53:40.383+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:53:40.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:53:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:53:40.731+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:53:40.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:53:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:53:40.731+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:53:40.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:53:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:53:40.731+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:53:40.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:53:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:53:40.885+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:53:40.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:53:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:53:40.885+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:53:40.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:53:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:53:40.885+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:53:40.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:53:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:53:40.885+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:53:41.732 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:53:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:53:41.334+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:53:41.732 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:53:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:53:41.334+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:53:41.733 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:53:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:53:41.334+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:53:41.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:53:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:53:41.729+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:53:41.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:53:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:53:41.729+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:53:41.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:53:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:53:41.729+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:53:41.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:53:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:53:41.914+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:53:41.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:53:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:53:41.914+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:53:41.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:53:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:53:41.914+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:53:41.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:53:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:53:41.914+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:53:42.684 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:53:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:53:42.324+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:53:42.685 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:53:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:53:42.324+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:53:42.685 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:53:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:53:42.324+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:53:42.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:53:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:53:42.681+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:53:42.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:53:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:53:42.681+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:53:42.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:53:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:53:42.681+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:53:42.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:53:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:53:42.924+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:53:42.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:53:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:53:42.924+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:53:42.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:53:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:53:42.924+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:53:42.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:53:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:53:42.924+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:53:43.727 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:53:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:53:43.357+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:53:43.728 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:53:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:53:43.357+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:53:43.728 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:53:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:53:43.357+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:53:43.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:53:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:53:43.724+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:53:43.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:53:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:53:43.724+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:53:43.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:53:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:53:43.724+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:53:43.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:53:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:53:43.910+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:53:43.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:53:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:53:43.910+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:53:43.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:53:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:53:43.910+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:53:43.995 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:53:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:53:43.910+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:53:44.384 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:53:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:53:44.019+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 46391 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:53:44.678 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:53:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:53:44.381+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:53:44.678 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:53:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:53:44.381+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:53:44.678 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:53:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:53:44.381+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:53:44.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:53:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:53:44.675+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:53:44.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:53:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:53:44.675+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:53:44.995 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:53:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:53:44.675+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:53:44.995 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:53:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:53:44.894+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:53:44.995 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:53:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:53:44.894+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:53:44.995 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:53:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:53:44.894+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:53:44.995 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:53:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:53:44.894+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:53:45.689 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:53:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:53:45.334+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:53:45.689 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:53:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:53:45.334+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:53:45.689 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:53:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:53:45.334+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:53:45.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:53:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:53:45.686+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:53:45.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:53:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:53:45.686+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:53:45.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:53:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:53:45.686+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:53:45.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:53:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:53:45.909+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:53:45.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:53:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:53:45.909+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:53:45.995 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:53:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:53:45.909+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:53:45.995 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:53:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:53:45.909+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:53:46.738 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:53:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:53:46.375+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:53:46.738 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:53:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:53:46.375+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:53:46.738 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:53:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:53:46.375+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:53:46.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:53:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:53:46.735+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:53:46.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:53:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:53:46.735+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:53:46.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:53:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:53:46.735+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:53:46.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:53:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:53:46.910+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:53:46.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:53:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:53:46.910+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:53:46.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:53:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:53:46.910+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:53:46.995 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:53:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:53:46.910+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:53:47.701 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:53:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:53:47.344+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:53:47.701 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:53:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:53:47.344+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:53:47.701 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:53:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:53:47.344+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:53:47.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:53:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:53:47.698+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:53:47.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:53:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:53:47.698+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:53:47.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:53:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:53:47.698+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:53:47.995 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:53:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:53:47.862+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:53:47.995 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:53:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:53:47.862+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:53:47.995 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:53:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:53:47.862+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:53:47.995 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:53:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:53:47.862+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:53:48.667 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:53:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:53:48.352+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:53:48.667 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:53:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:53:48.352+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:53:48.668 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:53:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:53:48.352+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:53:48.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:53:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:53:48.664+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:53:48.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:53:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:53:48.664+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:53:48.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:53:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:53:48.664+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:53:48.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:53:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:53:48.870+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:53:48.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:53:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:53:48.870+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:53:48.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:53:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:53:48.870+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:53:48.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:53:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:53:48.870+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:53:49.362 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:53:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:53:49.026+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 46463 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:53:49.692 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:53:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:53:49.360+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:53:49.692 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:53:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:53:49.360+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:53:49.692 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:53:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:53:49.360+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:53:49.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:53:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:53:49.689+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:53:49.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:53:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:53:49.689+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:53:49.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:53:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:53:49.689+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:53:49.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:53:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:53:49.875+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:53:49.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:53:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:53:49.875+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:53:49.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:53:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:53:49.875+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:53:49.995 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:53:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:53:49.875+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:53:50.687 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:53:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:53:50.393+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:53:50.688 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:53:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:53:50.393+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:53:50.688 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:53:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:53:50.393+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:53:50.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:53:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:53:50.685+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:53:50.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:53:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:53:50.685+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:53:50.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:53:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:53:50.685+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:53:50.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:53:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:53:50.860+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:53:50.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:53:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:53:50.860+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:53:50.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:53:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:53:50.860+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:53:50.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:53:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:53:50.860+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:53:51.682 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:53:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:53:51.420+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:53:51.682 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:53:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:53:51.420+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:53:51.683 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:53:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:53:51.420+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:53:51.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:53:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:53:51.679+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:53:51.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:53:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:53:51.679+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:53:51.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:53:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:53:51.679+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:53:51.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:53:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:53:51.841+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:53:51.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:53:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:53:51.841+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:53:51.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:53:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:53:51.841+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:53:51.995 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:53:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:53:51.841+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:53:52.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:53:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:53:52.651+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:53:52.744 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:53:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:53:52.651+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:53:52.744 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:53:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:53:52.651+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:53:52.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:53:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:53:52.461+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:53:52.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:53:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:53:52.461+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:53:52.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:53:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:53:52.461+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:53:53.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:53:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:53:52.883+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:53:53.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:53:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:53:52.883+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:53:53.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:53:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:53:52.883+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:53:53.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:53:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:53:52.883+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:53:53.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:53:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:53:53.648+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:53:53.744 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:53:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:53:53.648+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:53:53.744 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:53:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:53:53.648+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:53:53.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:53:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:53:53.474+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:53:53.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:53:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:53:53.474+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:53:53.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:53:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:53:53.474+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:53:54.158 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:53:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:53:54.033+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 46535 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:53:54.159 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:53:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:53:53.850+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:53:54.159 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:53:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:53:53.850+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:53:54.159 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:53:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:53:53.850+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:53:54.159 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:53:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:53:53.850+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:53:54.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:53:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:53:54.427+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:53:54.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:53:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:53:54.427+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:53:54.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:53:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:53:54.427+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:53:54.744 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:53:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:53:54.638+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:53:54.744 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:53:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:53:54.638+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:53:54.745 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:53:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:53:54.638+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:53:55.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:53:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:53:54.858+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:53:55.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:53:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:53:54.858+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:53:55.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:53:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:53:54.858+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:53:55.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:53:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:53:54.858+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:53:55.666 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:53:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:53:55.381+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:53:55.666 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:53:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:53:55.381+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:53:55.666 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:53:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:53:55.381+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:53:55.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:53:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:53:55.663+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:53:55.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:53:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:53:55.663+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:53:55.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:53:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:53:55.663+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:53:55.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:53:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:53:55.847+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:53:55.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:53:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:53:55.847+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:53:55.995 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:53:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:53:55.847+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:53:55.995 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:53:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:53:55.847+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:53:56.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:53:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:53:56.619+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:53:56.744 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:53:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:53:56.619+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:53:56.744 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:53:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:53:56.619+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:53:56.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:53:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:53:56.387+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:53:56.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:53:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:53:56.387+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:53:56.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:53:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:53:56.387+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:53:57.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:53:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:53:56.869+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:53:57.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:53:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:53:56.869+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:53:57.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:53:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:53:56.869+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:53:57.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:53:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:53:56.869+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:53:57.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:53:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:53:57.621+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:53:57.744 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:53:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:53:57.621+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:53:57.744 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:53:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:53:57.621+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:53:57.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:53:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:53:57.413+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:53:57.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:53:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:53:57.413+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:53:57.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:53:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:53:57.413+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:53:58.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:53:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:53:57.864+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:53:58.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:53:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:53:57.864+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:53:58.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:53:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:53:57.864+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:53:58.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:53:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:53:57.865+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:53:58.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:53:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:53:58.579+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:53:58.744 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:53:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:53:58.579+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:53:58.744 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:53:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:53:58.579+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:53:58.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:53:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:53:58.418+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:53:58.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:53:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:53:58.418+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:53:58.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:53:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:53:58.418+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:53:59.163 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:53:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:53:59.039+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 46607 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:53:59.163 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:53:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:53:58.905+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:53:59.164 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:53:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:53:58.905+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:53:59.164 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:53:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:53:58.905+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:53:59.164 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:53:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:53:58.906+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:53:59.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:53:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:53:59.378+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:53:59.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:53:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:53:59.378+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:53:59.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:53:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:53:59.378+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:53:59.887 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:53:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:53:59.548+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:53:59.888 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:53:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:53:59.548+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:53:59.888 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:53:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:53:59.548+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:54:00.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:53:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:53:59.935+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:54:00.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:53:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:53:59.935+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:54:00.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:53:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:53:59.935+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:54:00.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:53:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:53:59.936+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:54:00.586 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:54:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:54:00.419+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:54:00.586 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:54:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:54:00.419+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:54:00.587 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:54:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:54:00.419+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:54:00.927 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:54:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:54:00.582+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:54:00.927 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:54:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:54:00.582+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:54:00.927 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:54:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:54:00.582+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:54:01.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:54:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:54:00.923+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:54:01.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:54:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:54:00.923+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:54:01.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:54:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:54:00.923+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:54:01.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:54:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:54:00.923+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:54:01.744 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:54:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:54:01.544+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:54:01.744 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:54:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:54:01.544+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:54:01.744 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:54:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:54:01.544+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:54:01.745 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:54:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:54:01.446+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:54:01.745 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:54:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:54:01.446+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:54:01.745 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:54:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:54:01.446+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:54:02.234 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:54:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:54:01.926+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:54:02.234 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:54:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:54:01.926+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:54:02.234 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:54:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:54:01.926+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:54:02.235 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:54:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:54:01.926+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:54:02.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:54:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:54:02.469+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:54:02.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:54:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:54:02.469+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:54:02.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:54:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:54:02.469+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:54:02.938 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:54:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:54:02.501+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:54:02.939 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:54:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:54:02.501+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:54:02.939 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:54:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:54:02.501+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:54:03.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:54:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:54:02.935+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:54:03.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:54:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:54:02.935+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:54:03.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:54:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:54:02.935+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:54:03.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:54:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:54:02.935+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:54:03.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:54:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:54:03.468+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:54:03.744 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:54:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:54:03.468+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:54:03.744 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:54:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:54:03.468+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:54:03.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:54:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:54:03.470+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:54:03.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:54:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:54:03.470+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:54:03.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:54:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:54:03.470+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:54:04.244 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:54:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:54:04.046+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 46679 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:54:04.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:54:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:54:03.963+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:54:04.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:54:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:54:03.963+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:54:04.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:54:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:54:03.963+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:54:04.245 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:54:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:54:03.963+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:54:04.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:54:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:54:04.441+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:54:04.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:54:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:54:04.441+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:54:04.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:54:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:54:04.441+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:54:04.744 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:54:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:54:04.514+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:54:04.744 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:54:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:54:04.514+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:54:04.744 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:54:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:54:04.514+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:54:05.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:54:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:54:04.973+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:54:05.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:54:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:54:04.973+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:54:05.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:54:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:54:04.973+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:54:05.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:54:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:54:04.973+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:54:05.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:54:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:54:05.426+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:54:05.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:54:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:54:05.426+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:54:05.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:54:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:54:05.426+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:54:05.744 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:54:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:54:05.521+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:54:05.744 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:54:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:54:05.521+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:54:05.744 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:54:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:54:05.521+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:54:06.453 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:54:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:54:06.017+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:54:06.453 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:54:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:54:06.017+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:54:06.454 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:54:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:54:06.017+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:54:06.454 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:54:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:54:06.017+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:54:06.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:54:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:54:06.511+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:54:06.744 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:54:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:54:06.511+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:54:06.744 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:54:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:54:06.511+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:54:06.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:54:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:54:06.450+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:54:06.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:54:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:54:06.450+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:54:06.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:54:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:54:06.450+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:54:07.489 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:54:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:54:07.000+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:54:07.489 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:54:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:54:07.000+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:54:07.489 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:54:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:54:07.000+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:54:07.489 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:54:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:54:07.000+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:54:07.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:54:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:54:07.485+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:54:07.744 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:54:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:54:07.485+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:54:07.744 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:54:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:54:07.485+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:54:07.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:54:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:54:07.485+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:54:07.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:54:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:54:07.485+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:54:07.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:54:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:54:07.485+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:54:08.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:54:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:54:07.963+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:54:08.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:54:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:54:07.963+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:54:08.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:54:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:54:07.963+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:54:08.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:54:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:54:07.963+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:54:08.941 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:54:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:54:08.526+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:54:08.941 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:54:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:54:08.526+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:54:08.941 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:54:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:54:08.526+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:54:08.942 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:54:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:54:08.495+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:54:08.942 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:54:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:54:08.495+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:54:08.942 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:54:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:54:08.495+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:54:09.243 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:54:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:54:09.054+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 46751 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:54:09.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:54:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:54:08.937+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:54:09.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:54:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:54:08.937+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:54:09.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:54:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:54:08.937+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:54:09.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:54:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:54:08.937+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:54:09.744 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:54:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:54:09.576+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:54:09.744 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:54:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:54:09.576+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:54:09.744 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:54:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:54:09.576+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:54:09.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:54:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:54:09.460+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:54:09.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:54:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:54:09.460+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:54:09.745 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:54:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:54:09.460+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:54:10.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:54:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:54:09.922+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:54:10.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:54:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:54:09.922+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:54:10.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:54:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:54:09.922+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:54:10.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:54:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:54:09.922+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:54:10.612 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:54:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:54:10.469+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:54:10.612 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:54:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:54:10.469+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:54:10.612 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:54:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:54:10.469+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:54:10.910 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:54:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:54:10.609+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:54:10.910 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:54:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:54:10.609+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:54:10.910 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:54:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:54:10.609+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:54:11.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:54:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:54:10.906+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:54:11.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:54:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:54:10.906+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:54:11.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:54:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:54:10.906+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:54:11.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:54:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:54:10.906+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:54:11.744 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:54:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:54:11.650+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:54:11.744 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:54:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:54:11.650+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:54:11.744 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:54:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:54:11.650+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:54:11.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:54:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:54:11.453+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:54:11.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:54:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:54:11.453+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:54:11.745 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:54:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:54:11.453+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:54:12.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:54:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:54:11.928+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:54:12.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:54:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:54:11.928+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:54:12.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:54:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:54:11.928+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:54:12.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:54:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:54:11.928+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:54:12.744 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:54:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:54:12.681+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:54:12.744 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:54:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:54:12.681+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:54:12.744 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:54:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:54:12.681+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:54:12.745 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:54:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:54:12.489+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:54:12.745 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:54:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:54:12.489+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:54:12.745 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:54:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:54:12.489+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:54:13.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:54:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:54:12.937+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:54:13.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:54:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:54:12.937+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:54:13.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:54:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:54:12.937+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:54:13.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:54:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:54:12.937+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:54:13.960 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:54:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:54:13.642+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:54:13.960 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:54:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:54:13.642+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:54:13.960 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:54:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:54:13.642+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:54:13.961 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:54:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:54:13.501+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:54:13.961 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:54:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:54:13.501+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:54:13.961 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:54:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:54:13.501+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:54:14.243 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:54:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:54:14.060+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 46823 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:54:14.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:54:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:54:13.955+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:54:14.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:54:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:54:13.955+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:54:14.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:54:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:54:13.955+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:54:14.245 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:54:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:54:13.955+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:54:14.927 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:54:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:54:14.609+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:54:14.927 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:54:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:54:14.609+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:54:14.928 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:54:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:54:14.609+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:54:14.928 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:54:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:54:14.518+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:54:14.928 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:54:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:54:14.518+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:54:14.928 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:54:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:54:14.518+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:54:15.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:54:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:54:14.924+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:54:15.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:54:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:54:14.924+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:54:15.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:54:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:54:14.924+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:54:15.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:54:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:54:14.924+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:54:15.891 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:54:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:54:15.595+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:54:15.891 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:54:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:54:15.595+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:54:15.891 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:54:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:54:15.595+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:54:15.891 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:54:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:54:15.563+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:54:15.892 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:54:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:54:15.563+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:54:15.892 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:54:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:54:15.563+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:54:16.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:54:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:54:15.888+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:54:16.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:54:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:54:15.888+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:54:16.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:54:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:54:15.888+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:54:16.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:54:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:54:15.888+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:54:16.871 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:54:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:54:16.618+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:54:16.872 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:54:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:54:16.618+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:54:16.872 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:54:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:54:16.618+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:54:16.872 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:54:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:54:16.567+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:54:16.872 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:54:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:54:16.567+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:54:16.872 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:54:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:54:16.567+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:54:17.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:54:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:54:16.868+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:54:17.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:54:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:54:16.868+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:54:17.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:54:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:54:16.868+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:54:17.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:54:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:54:16.868+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:54:17.877 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:54:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:54:17.604+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:54:17.877 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:54:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:54:17.604+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:54:17.877 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:54:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:54:17.604+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:54:17.878 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:54:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:54:17.557+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:54:17.878 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:54:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:54:17.557+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:54:17.878 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:54:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:54:17.557+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:54:18.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:54:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:54:17.873+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:54:18.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:54:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:54:17.873+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:54:18.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:54:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:54:17.873+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:54:18.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:54:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:54:17.873+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:54:18.833 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:54:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:54:18.568+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:54:18.833 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:54:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:54:18.568+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:54:18.833 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:54:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:54:18.568+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:54:18.833 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:54:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:54:18.564+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:54:18.834 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:54:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:54:18.564+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:54:18.834 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:54:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:54:18.564+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:54:19.091 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:54:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:54:19.068+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 46897 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:54:19.091 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:54:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:54:18.829+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:54:19.091 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:54:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:54:18.829+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:54:19.091 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:54:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:54:18.829+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:54:19.091 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:54:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:54:18.829+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:54:19.887 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:54:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:54:19.614+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:54:19.887 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:54:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:54:19.614+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:54:19.887 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:54:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:54:19.614+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:54:19.888 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:54:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:54:19.589+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:54:19.888 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:54:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:54:19.589+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:54:19.888 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:54:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:54:19.589+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:54:19.888 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:54:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:54:19.829+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:54:19.888 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:54:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:54:19.829+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:54:19.888 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:54:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:54:19.829+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:54:19.889 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:54:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:54:19.829+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:54:20.650 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:54:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:54:20.544+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:54:20.650 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:54:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:54:20.544+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:54:20.650 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:54:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:54:20.544+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:54:20.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:54:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:54:20.646+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:54:21.039 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:54:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:54:20.646+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:54:21.039 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:54:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:54:20.646+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:54:21.039 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:54:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:54:20.793+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:54:21.040 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:54:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:54:20.793+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:54:21.040 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:54:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:54:20.793+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:54:21.040 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:54:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:54:20.793+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:54:21.865 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:54:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:54:21.579+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:54:21.865 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:54:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:54:21.579+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:54:21.865 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:54:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:54:21.579+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:54:21.865 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:54:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:54:21.608+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:54:21.866 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:54:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:54:21.608+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:54:21.866 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:54:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:54:21.608+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:54:21.866 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:54:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:54:21.776+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:54:21.866 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:54:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:54:21.776+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:54:21.866 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:54:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:54:21.776+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:54:21.866 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:54:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:54:21.776+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:54:22.826 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:54:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:54:22.602+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:54:22.827 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:54:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:54:22.602+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:54:22.827 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:54:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:54:22.602+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:54:22.827 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:54:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:54:22.539+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:54:22.827 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:54:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:54:22.539+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:54:22.828 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:54:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:54:22.539+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:54:23.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:54:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:54:22.824+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:54:23.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:54:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:54:22.824+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:54:23.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:54:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:54:22.824+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:54:23.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:54:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:54:22.824+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:54:23.804 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:54:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:54:23.630+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:54:23.804 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:54:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:54:23.630+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:54:23.804 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:54:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:54:23.630+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:54:23.804 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:54:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:54:23.538+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:54:23.804 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:54:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:54:23.538+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:54:23.805 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:54:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:54:23.538+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:54:24.080 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:54:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:54:23.800+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:54:24.080 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:54:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:54:23.800+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:54:24.080 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:54:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:54:23.800+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:54:24.081 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:54:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:54:23.800+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:54:24.493 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:54:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:54:24.076+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 46969 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:54:24.829 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:54:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:54:24.638+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:54:24.830 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:54:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:54:24.638+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:54:24.830 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:54:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:54:24.638+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:54:24.830 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:54:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:54:24.491+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:54:24.830 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:54:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:54:24.491+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:54:24.830 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:54:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:54:24.491+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:54:25.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:54:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:54:24.826+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:54:25.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:54:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:54:24.826+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:54:25.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:54:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:54:24.826+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:54:25.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:54:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:54:24.826+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:54:25.874 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:54:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:54:25.601+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:54:25.874 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:54:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:54:25.601+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:54:25.874 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:54:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:54:25.601+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:54:25.875 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:54:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:54:25.525+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:54:25.875 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:54:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:54:25.525+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:54:25.875 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:54:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:54:25.525+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:54:26.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:54:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:54:25.871+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:54:26.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:54:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:54:25.871+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:54:26.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:54:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:54:25.871+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:54:26.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:54:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:54:25.871+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:54:26.899 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:54:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:54:26.549+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:54:26.900 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:54:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:54:26.549+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:54:26.900 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:54:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:54:26.549+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:54:26.900 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:54:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:54:26.634+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:54:26.900 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:54:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:54:26.634+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:54:26.901 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:54:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:54:26.634+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:54:27.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:54:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:54:26.895+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:54:27.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:54:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:54:26.895+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:54:27.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:54:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:54:26.895+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:54:27.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:54:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:54:26.895+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:54:27.918 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:54:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:54:27.648+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:54:27.918 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:54:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:54:27.648+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:54:27.918 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:54:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:54:27.648+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:54:27.919 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:54:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:54:27.568+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:54:27.919 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:54:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:54:27.568+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:54:27.919 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:54:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:54:27.568+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:54:28.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:54:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:54:27.915+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:54:28.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:54:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:54:27.915+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:54:28.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:54:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:54:27.915+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:54:28.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:54:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:54:27.915+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:54:28.894 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:54:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:54:28.688+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:54:28.894 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:54:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:54:28.688+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:54:28.894 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:54:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:54:28.688+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:54:28.894 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:54:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:54:28.604+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:54:28.894 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:54:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:54:28.604+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:54:28.895 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:54:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:54:28.604+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:54:29.209 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:54:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:54:29.084+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 47041 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:54:29.209 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:54:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:54:28.891+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:54:29.209 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:54:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:54:28.891+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:54:29.209 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:54:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:54:28.891+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:54:29.210 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:54:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:54:28.891+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:54:29.887 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:54:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:54:29.565+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:54:29.888 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:54:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:54:29.565+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:54:29.888 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:54:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:54:29.565+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:54:29.888 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:54:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:54:29.737+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:54:29.888 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:54:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:54:29.737+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:54:29.888 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:54:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:54:29.737+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:54:30.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:54:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:54:29.904+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:54:30.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:54:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:54:29.904+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:54:30.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:54:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:54:29.904+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:54:30.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:54:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:54:29.904+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:54:30.744 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:54:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:54:30.517+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:54:30.744 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:54:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:54:30.517+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:54:30.744 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:54:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:54:30.517+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:54:31.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:54:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:54:30.920+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:54:31.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:54:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:54:30.920+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:54:31.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:54:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:54:30.920+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:54:31.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:54:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:54:30.920+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:54:31.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:54:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:54:30.761+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:54:31.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:54:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:54:30.761+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:54:31.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:54:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:54:30.761+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:54:31.865 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:54:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:54:31.804+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:54:31.866 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:54:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:54:31.804+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:54:31.866 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:54:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:54:31.804+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:54:31.866 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:54:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:54:31.557+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:54:31.866 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:54:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:54:31.557+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:54:31.866 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:54:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:54:31.557+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:54:32.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:54:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:54:31.957+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:54:32.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:54:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:54:31.957+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:54:32.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:54:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:54:31.957+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:54:32.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:54:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:54:31.957+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:54:32.922 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:54:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:54:32.555+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:54:32.923 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:54:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:54:32.555+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:54:32.923 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:54:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:54:32.555+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:54:32.923 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:54:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:54:32.804+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:54:32.923 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:54:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:54:32.804+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:54:32.923 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:54:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:54:32.804+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:54:33.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:54:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:54:32.920+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:54:33.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:54:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:54:32.920+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:54:33.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:54:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:54:32.920+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:54:33.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:54:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:54:32.920+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:54:33.936 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:54:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:54:33.581+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:54:33.936 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:54:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:54:33.581+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:54:33.936 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:54:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:54:33.581+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:54:33.937 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:54:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:54:33.802+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:54:33.937 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:54:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:54:33.802+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:54:33.937 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:54:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:54:33.802+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:54:34.215 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:54:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:54:34.091+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 47114 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:54:34.216 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:54:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:54:33.933+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:54:34.216 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:54:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:54:33.933+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:54:34.216 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:54:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:54:33.933+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:54:34.216 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:54:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:54:33.933+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:54:34.846 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:54:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:54:34.572+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:54:34.846 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:54:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:54:34.572+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:54:34.846 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:54:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:54:34.572+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:54:35.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:54:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:54:34.843+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:54:35.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:54:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:54:34.843+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:54:35.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:54:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:54:34.843+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:54:35.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:54:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:54:34.972+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:54:35.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:54:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:54:34.972+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:54:35.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:54:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:54:34.972+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:54:35.245 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:54:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:54:34.973+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:54:35.826 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:54:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:54:35.544+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:54:35.827 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:54:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:54:35.544+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:54:35.827 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:54:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:54:35.544+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:54:36.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:54:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:54:35.823+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:54:36.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:54:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:54:35.823+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:54:36.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:54:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:54:35.823+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:54:36.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:54:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:54:36.010+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:54:36.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:54:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:54:36.010+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:54:36.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:54:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:54:36.010+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:54:36.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:54:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:54:36.010+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:54:36.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:54:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:54:36.582+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:54:36.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:54:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:54:36.582+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:54:36.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:54:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:54:36.582+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:54:36.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:54:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:54:36.812+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:54:36.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:54:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:54:36.812+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:54:36.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:54:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:54:36.812+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:54:37.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:54:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:54:36.998+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:54:37.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:54:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:54:36.998+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:54:37.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:54:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:54:36.998+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:54:37.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:54:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:54:36.998+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:54:37.808 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:54:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:54:37.543+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:54:37.808 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:54:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:54:37.543+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:54:37.808 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:54:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:54:37.543+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:54:38.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:54:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:54:37.805+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:54:38.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:54:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:54:37.805+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:54:38.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:54:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:54:37.805+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:54:38.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:54:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:54:37.950+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:54:38.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:54:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:54:37.950+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:54:38.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:54:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:54:37.950+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:54:38.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:54:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:54:37.950+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:54:38.942 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:54:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:54:38.536+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:54:38.942 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:54:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:54:38.536+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:54:38.942 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:54:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:54:38.536+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:54:38.942 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:54:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:54:38.765+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:54:38.942 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:54:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:54:38.765+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:54:38.943 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:54:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:54:38.765+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:54:39.220 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:54:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:54:39.098+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 47186 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:54:39.221 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:54:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:54:38.938+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:54:39.221 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:54:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:54:38.938+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:54:39.221 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:54:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:54:38.938+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:54:39.221 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:54:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:54:38.939+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:54:39.887 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:54:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:54:39.535+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:54:39.888 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:54:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:54:39.535+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:54:39.888 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:54:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:54:39.535+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:54:39.888 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:54:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:54:39.737+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:54:39.888 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:54:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:54:39.737+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:54:39.888 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:54:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:54:39.737+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:54:40.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:54:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:54:39.906+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:54:40.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:54:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:54:39.906+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:54:40.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:54:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:54:39.906+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:54:40.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:54:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:54:39.906+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:54:40.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:54:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:54:40.510+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:54:40.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:54:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:54:40.510+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:54:40.744 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:54:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:54:40.510+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:54:41.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:54:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:54:40.918+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:54:41.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:54:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:54:40.918+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:54:41.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:54:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:54:40.918+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:54:41.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:54:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:54:40.918+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:54:41.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:54:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:54:40.786+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:54:41.245 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:54:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:54:40.786+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:54:41.245 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:54:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:54:40.786+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:54:41.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:54:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:54:41.485+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:54:41.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:54:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:54:41.485+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:54:41.744 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:54:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:54:41.485+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:54:42.170 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:54:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:54:41.803+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:54:42.170 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:54:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:54:41.803+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:54:42.170 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:54:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:54:41.803+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:54:42.170 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:54:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:54:41.872+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:54:42.170 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:54:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:54:41.872+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:54:42.171 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:54:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:54:41.872+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:54:42.171 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:54:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:54:41.872+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:54:42.802 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:54:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:54:42.513+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:54:42.802 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:54:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:54:42.513+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:54:42.802 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:54:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:54:42.513+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:54:43.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:54:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:54:42.799+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:54:43.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:54:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:54:42.799+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:54:43.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:54:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:54:42.799+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:54:43.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:54:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:54:42.844+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:54:43.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:54:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:54:42.844+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:54:43.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:54:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:54:42.844+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:54:43.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:54:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:54:42.844+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:54:43.825 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:54:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:54:43.556+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:54:43.825 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:54:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:54:43.556+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:54:43.825 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:54:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:54:43.556+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:54:44.108 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:54:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:54:43.822+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:54:44.108 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:54:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:54:43.822+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:54:44.108 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:54:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:54:43.822+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:54:44.109 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:54:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:54:43.827+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:54:44.109 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:54:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:54:43.827+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:54:44.109 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:54:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:54:43.827+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:54:44.109 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:54:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:54:43.827+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:54:44.493 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:54:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:54:44.105+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 47258 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:54:44.824 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:54:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:54:44.552+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:54:44.825 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:54:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:54:44.552+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:54:44.825 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:54:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:54:44.552+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:54:45.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:54:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:54:44.867+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:54:45.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:54:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:54:44.867+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:54:45.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:54:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:54:44.867+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:54:45.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:54:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:54:44.822+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:54:45.245 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:54:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:54:44.822+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:54:45.245 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:54:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:54:44.822+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:54:45.245 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:54:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:54:44.822+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:54:45.829 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:54:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:54:45.513+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:54:45.829 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:54:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:54:45.513+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:54:45.829 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:54:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:54:45.513+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:54:46.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:54:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:54:45.826+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:54:46.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:54:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:54:45.826+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:54:46.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:54:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:54:45.826+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:54:46.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:54:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:54:45.826+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:54:46.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:54:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:54:45.826+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:54:46.245 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:54:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:54:45.826+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:54:46.245 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:54:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:54:45.827+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:54:46.798 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:54:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:54:46.512+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:54:46.799 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:54:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:54:46.512+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:54:46.799 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:54:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:54:46.512+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:54:47.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:54:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:54:46.843+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:54:47.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:54:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:54:46.843+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:54:47.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:54:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:54:46.843+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:54:47.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:54:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:54:46.795+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:54:47.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:54:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:54:46.795+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:54:47.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:54:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:54:46.795+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:54:47.245 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:54:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:54:46.796+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:54:47.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:54:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:54:47.483+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:54:47.744 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:54:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:54:47.483+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:54:47.744 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:54:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:54:47.483+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:54:48.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:54:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:54:47.859+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:54:48.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:54:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:54:47.859+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:54:48.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:54:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:54:47.859+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:54:48.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:54:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:54:47.784+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:54:48.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:54:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:54:47.784+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:54:48.245 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:54:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:54:47.784+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:54:48.245 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:54:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:54:47.784+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:54:48.781 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:54:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:54:48.528+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:54:48.782 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:54:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:54:48.528+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:54:48.782 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:54:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:54:48.528+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:54:48.782 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:54:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:54:48.778+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:54:48.782 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:54:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:54:48.778+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:54:48.782 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:54:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:54:48.778+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:54:48.782 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:54:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:54:48.778+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:54:49.115 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:54:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:54:48.815+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:54:49.115 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:54:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:54:48.815+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:54:49.115 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:54:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:54:48.815+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:54:49.493 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:54:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:54:49.111+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 47326 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:54:49.829 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:54:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:54:49.776+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:54:49.829 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:54:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:54:49.776+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:54:49.829 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:54:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:54:49.776+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:54:49.829 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:54:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:54:49.776+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:54:49.830 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:54:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:54:49.527+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:54:49.830 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:54:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:54:49.527+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:54:49.830 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:54:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:54:49.527+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:54:50.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:54:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:54:49.825+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:54:50.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:54:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:54:49.825+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:54:50.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:54:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:54:49.825+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:54:50.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:54:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:54:50.559+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:54:50.744 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:54:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:54:50.559+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:54:50.744 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:54:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:54:50.559+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:54:51.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:54:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:54:50.758+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:54:51.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:54:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:54:50.758+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:54:51.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:54:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:54:50.758+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:54:51.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:54:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:54:50.758+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:54:51.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:54:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:54:50.848+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:54:51.245 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:54:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:54:50.848+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:54:51.245 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:54:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:54:50.848+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:54:51.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:54:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:54:51.601+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:54:51.744 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:54:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:54:51.601+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:54:51.744 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:54:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:54:51.601+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:54:52.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:54:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:54:51.761+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:54:52.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:54:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:54:51.761+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:54:52.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:54:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:54:51.761+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:54:52.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:54:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:54:51.761+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:54:52.245 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:54:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:54:51.839+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:54:52.245 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:54:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:54:51.839+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:54:52.245 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:54:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:54:51.839+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:54:52.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:54:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:54:52.649+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:54:52.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:54:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:54:52.649+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:54:52.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:54:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:54:52.649+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:54:52.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:54:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:54:52.845+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:54:52.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:54:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:54:52.845+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:54:52.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:54:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:54:52.845+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:54:52.995 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:54:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:54:52.756+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:54:52.995 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:54:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:54:52.756+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:54:52.995 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:54:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:54:52.756+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:54:52.995 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:54:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:54:52.756+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:54:53.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:54:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:54:53.847+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:54:53.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:54:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:54:53.847+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:54:53.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:54:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:54:53.847+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:54:53.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:54:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:54:53.616+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:54:53.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:54:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:54:53.616+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:54:53.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:54:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:54:53.616+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:54:53.995 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:54:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:54:53.799+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:54:53.995 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:54:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:54:53.799+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:54:53.995 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:54:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:54:53.799+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:54:53.995 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:54:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:54:53.799+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:54:54.458 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:54:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:54:54.118+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 47398 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:54:54.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:54:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:54:54.594+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:54:54.744 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:54:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:54:54.594+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:54:54.744 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:54:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:54:54.594+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:54:55.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:54:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:54:54.892+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:54:55.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:54:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:54:54.892+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:54:55.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:54:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:54:54.892+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:54:55.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:54:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:54:54.769+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:54:55.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:54:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:54:54.769+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:54:55.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:54:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:54:54.769+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:54:55.245 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:54:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:54:54.769+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:54:55.905 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:54:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:54:55.602+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:54:55.906 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:54:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:54:55.602+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:54:55.906 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:54:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:54:55.602+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:54:55.906 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:54:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:54:55.786+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:54:55.906 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:54:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:54:55.786+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:54:55.907 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:54:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:54:55.786+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:54:55.907 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:54:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:54:55.786+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:54:56.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:54:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:54:55.902+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:54:56.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:54:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:54:55.902+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:54:56.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:54:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:54:55.902+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:54:56.891 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:54:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:54:56.571+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:54:56.891 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:54:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:54:56.571+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:54:56.891 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:54:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:54:56.571+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:54:56.891 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:54:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:54:56.785+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:54:56.892 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:54:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:54:56.785+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:54:56.892 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:54:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:54:56.785+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:54:56.892 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:54:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:54:56.785+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:54:57.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:54:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:54:56.887+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:54:57.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:54:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:54:56.887+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:54:57.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:54:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:54:56.887+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:54:57.933 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:54:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:54:57.603+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:54:57.933 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:54:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:54:57.603+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:54:57.933 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:54:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:54:57.603+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:54:57.934 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:54:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:54:57.834+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:54:57.934 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:54:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:54:57.834+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:54:57.934 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:54:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:54:57.834+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:54:57.934 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:54:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:54:57.834+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:54:58.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:54:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:54:57.929+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:54:58.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:54:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:54:57.929+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:54:58.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:54:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:54:57.929+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:54:58.951 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:54:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:54:58.653+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:54:58.951 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:54:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:54:58.653+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:54:58.951 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:54:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:54:58.653+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:54:58.951 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:54:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:54:58.819+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:54:58.951 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:54:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:54:58.819+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:54:58.952 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:54:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:54:58.819+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:54:58.952 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:54:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:54:58.819+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:54:59.243 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:54:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:54:59.125+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 47470 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:54:59.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:54:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:54:58.947+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:54:59.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:54:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:54:58.947+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:54:59.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:54:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:54:58.947+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:54:59.974 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:54:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:54:59.665+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:54:59.974 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:54:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:54:59.665+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:54:59.974 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:54:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:54:59.665+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:54:59.975 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:54:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:54:59.838+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:54:59.975 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:54:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:54:59.838+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:54:59.975 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:54:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:54:59.838+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:54:59.975 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:54:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:54:59.838+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:55:00.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:54:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:54:59.971+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:55:00.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:54:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:54:59.971+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:55:00.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:54:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:54:59.971+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:55:00.947 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:55:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:55:00.691+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:55:00.948 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:55:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:55:00.691+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:55:00.948 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:55:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:55:00.691+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:55:00.948 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:55:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:55:00.838+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:55:00.948 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:55:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:55:00.838+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:55:00.948 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:55:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:55:00.838+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:55:00.948 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:55:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:55:00.839+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:55:01.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:55:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:55:00.944+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:55:01.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:55:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:55:00.944+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:55:01.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:55:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:55:00.944+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:55:01.976 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:55:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:55:01.675+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:55:01.976 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:55:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:55:01.675+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:55:01.977 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:55:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:55:01.675+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:55:01.977 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:55:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:55:01.795+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:55:01.977 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:55:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:55:01.795+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:55:01.977 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:55:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:55:01.795+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:55:01.977 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:55:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:55:01.795+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:55:02.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:55:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:55:01.973+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:55:02.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:55:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:55:01.973+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:55:02.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:55:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:55:01.973+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:55:02.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:55:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:55:02.678+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:55:02.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:55:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:55:02.678+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:55:02.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:55:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:55:02.678+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:55:02.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:55:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:55:02.813+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:55:02.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:55:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:55:02.813+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:55:02.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:55:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:55:02.813+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:55:02.995 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:55:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:55:02.813+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:55:03.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:55:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:55:03.001+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:55:03.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:55:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:55:03.001+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:55:03.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:55:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:55:03.001+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:55:03.989 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:55:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:55:03.669+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:55:03.990 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:55:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:55:03.669+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:55:03.990 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:55:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:55:03.669+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:55:03.990 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:55:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:55:03.775+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:55:03.990 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:55:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:55:03.775+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:55:03.990 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:55:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:55:03.775+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:55:03.990 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:55:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:55:03.775+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:55:04.243 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:55:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:55:04.132+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 47542 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:55:04.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:55:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:55:03.986+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:55:04.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:55:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:55:03.986+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:55:04.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:55:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:55:03.986+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:55:04.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:55:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:55:04.637+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:55:04.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:55:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:55:04.637+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:55:04.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:55:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:55:04.637+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:55:04.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:55:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:55:04.811+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:55:04.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:55:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:55:04.811+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:55:04.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:55:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:55:04.811+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:55:04.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:55:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:55:04.811+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:55:05.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:55:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:55:04.990+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:55:05.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:55:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:55:04.990+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:55:05.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:55:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:55:04.990+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:55:05.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:55:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:55:05.615+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:55:05.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:55:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:55:05.615+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:55:05.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:55:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:55:05.615+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:55:05.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:55:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:55:05.795+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:55:05.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:55:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:55:05.795+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:55:05.995 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:55:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:55:05.795+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:55:05.995 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:55:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:55:05.796+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:55:06.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:55:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:55:06.037+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:55:06.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:55:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:55:06.037+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:55:06.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:55:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:55:06.037+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:55:06.845 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:55:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:55:06.583+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:55:06.846 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:55:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:55:06.583+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:55:06.846 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:55:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:55:06.583+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:55:07.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:55:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:55:07.084+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:55:07.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:55:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:55:07.084+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:55:07.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:55:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:55:07.084+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:55:07.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:55:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:55:06.843+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:55:07.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:55:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:55:06.843+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:55:07.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:55:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:55:06.843+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:55:07.245 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:55:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:55:06.843+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:55:07.894 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:55:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:55:07.615+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:55:07.894 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:55:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:55:07.615+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:55:07.894 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:55:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:55:07.615+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:55:08.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:55:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:55:08.089+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:55:08.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:55:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:55:08.089+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:55:08.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:55:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:55:08.089+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:55:08.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:55:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:55:07.891+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:55:08.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:55:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:55:07.891+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:55:08.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:55:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:55:07.891+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:55:08.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:55:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:55:07.891+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:55:08.868 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:55:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:55:08.597+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:55:08.868 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:55:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:55:08.597+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:55:08.868 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:55:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:55:08.597+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:55:09.140 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:55:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:55:08.865+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:55:09.140 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:55:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:55:08.865+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:55:09.140 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:55:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:55:08.865+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:55:09.140 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:55:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:55:08.865+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:55:09.493 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:55:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:55:09.140+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 47614 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:55:09.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:55:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:55:09.136+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:55:09.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:55:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:55:09.136+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:55:09.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:55:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:55:09.136+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:55:09.887 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:55:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:55:09.600+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:55:09.888 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:55:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:55:09.600+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:55:09.888 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:55:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:55:09.600+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:55:10.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:55:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:55:10.094+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:55:10.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:55:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:55:10.094+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:55:10.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:55:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:55:10.094+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:55:10.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:55:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:55:09.897+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:55:10.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:55:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:55:09.897+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:55:10.245 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:55:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:55:09.897+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:55:10.245 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:55:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:55:09.897+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:55:10.916 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:55:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:55:10.643+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:55:10.917 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:55:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:55:10.643+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:55:10.917 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:55:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:55:10.643+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:55:11.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:55:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:55:11.068+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:55:11.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:55:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:55:11.068+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:55:11.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:55:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:55:11.068+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:55:11.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:55:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:55:10.912+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:55:11.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:55:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:55:10.912+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:55:11.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:55:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:55:10.912+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:55:11.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:55:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:55:10.913+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:55:11.868 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:55:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:55:11.613+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:55:11.868 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:55:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:55:11.613+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:55:11.868 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:55:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:55:11.613+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:55:12.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:55:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:55:12.040+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:55:12.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:55:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:55:12.040+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:55:12.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:55:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:55:12.040+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:55:12.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:55:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:55:11.901+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:55:12.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:55:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:55:11.901+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:55:12.245 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:55:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:55:11.901+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:55:12.245 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:55:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:55:11.901+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:55:12.881 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:55:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:55:12.594+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:55:12.881 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:55:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:55:12.594+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:55:12.881 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:55:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:55:12.594+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:55:13.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:55:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:55:13.071+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:55:13.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:55:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:55:13.071+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:55:13.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:55:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:55:13.071+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:55:13.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:55:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:55:12.878+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:55:13.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:55:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:55:12.878+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:55:13.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:55:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:55:12.878+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:55:13.245 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:55:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:55:12.878+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:55:13.851 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:55:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:55:13.577+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:55:13.851 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:55:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:55:13.577+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:55:13.851 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:55:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:55:13.577+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:55:14.151 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:55:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:55:14.030+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:55:14.152 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:55:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:55:14.030+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:55:14.152 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:55:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:55:14.030+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:55:14.152 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:55:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:55:13.848+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:55:14.152 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:55:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:55:13.848+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:55:14.152 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:55:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:55:13.848+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:55:14.152 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:55:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:55:13.848+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:55:14.493 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:55:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:55:14.147+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 47686 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:55:14.881 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:55:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:55:14.587+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:55:14.882 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:55:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:55:14.587+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:55:14.882 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:55:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:55:14.587+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:55:15.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:55:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:55:15.048+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:55:15.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:55:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:55:15.048+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:55:15.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:55:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:55:15.048+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:55:15.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:55:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:55:14.878+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:55:15.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:55:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:55:14.878+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:55:15.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:55:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:55:14.878+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:55:15.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:55:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:55:14.878+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:55:15.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:55:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:55:15.832+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:55:15.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:55:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:55:15.832+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:55:15.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:55:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:55:15.832+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:55:15.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:55:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:55:15.832+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:55:15.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:55:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:55:15.621+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:55:15.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:55:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:55:15.621+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:55:15.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:55:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:55:15.621+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:55:16.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:55:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:55:16.084+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:55:16.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:55:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:55:16.084+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:55:16.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:55:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:55:16.084+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:55:16.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:55:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:55:16.839+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:55:16.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:55:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:55:16.839+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:55:16.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:55:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:55:16.839+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:55:16.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:55:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:55:16.839+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:55:16.995 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:55:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:55:16.600+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:55:16.995 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:55:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:55:16.600+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:55:16.995 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:55:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:55:16.600+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:55:17.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:55:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:55:17.101+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:55:17.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:55:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:55:17.101+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:55:17.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:55:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:55:17.101+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:55:17.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:55:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:55:17.610+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:55:17.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:55:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:55:17.610+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:55:17.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:55:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:55:17.610+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:55:17.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:55:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:55:17.842+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:55:17.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:55:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:55:17.842+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:55:17.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:55:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:55:17.842+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:55:17.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:55:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:55:17.842+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:55:18.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:55:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:55:18.146+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:55:18.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:55:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:55:18.146+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:55:18.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:55:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:55:18.146+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:55:18.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:55:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:55:18.655+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:55:18.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:55:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:55:18.655+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:55:18.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:55:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:55:18.655+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:55:18.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:55:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:55:18.813+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:55:18.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:55:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:55:18.813+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:55:18.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:55:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:55:18.813+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:55:18.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:55:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:55:18.813+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:55:19.493 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:55:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:55:19.154+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 47758 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:55:19.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:55:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:55:19.152+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:55:19.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:55:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:55:19.152+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:55:19.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:55:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:55:19.152+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:55:19.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:55:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:55:19.824+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:55:19.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:55:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:55:19.824+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:55:19.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:55:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:55:19.824+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:55:19.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:55:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:55:19.824+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:55:19.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:55:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:55:19.684+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:55:19.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:55:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:55:19.684+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:55:19.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:55:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:55:19.684+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:55:20.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:55:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:55:20.135+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:55:20.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:55:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:55:20.135+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:55:20.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:55:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:55:20.135+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:55:20.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:55:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:55:20.839+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:55:20.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:55:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:55:20.839+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:55:20.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:55:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:55:20.839+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:55:20.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:55:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:55:20.839+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:55:20.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:55:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:55:20.733+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:55:20.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:55:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:55:20.733+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:55:20.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:55:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:55:20.733+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:55:21.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:55:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:55:21.087+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:55:21.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:55:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:55:21.087+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:55:21.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:55:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:55:21.087+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:55:21.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:55:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:55:21.688+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:55:21.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:55:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:55:21.688+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:55:21.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:55:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:55:21.688+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:55:21.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:55:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:55:21.854+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:55:21.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:55:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:55:21.854+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:55:21.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:55:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:55:21.854+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:55:21.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:55:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:55:21.855+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:55:22.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:55:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:55:22.083+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:55:22.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:55:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:55:22.083+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:55:22.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:55:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:55:22.083+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:55:22.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:55:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:55:22.682+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:55:22.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:55:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:55:22.682+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:55:22.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:55:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:55:22.682+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:55:22.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:55:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:55:22.899+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:55:22.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:55:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:55:22.899+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:55:22.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:55:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:55:22.899+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:55:22.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:55:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:55:22.899+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:55:23.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:55:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:55:23.128+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:55:23.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:55:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:55:23.128+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:55:23.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:55:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:55:23.128+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:55:23.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:55:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:55:23.684+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:55:23.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:55:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:55:23.684+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:55:23.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:55:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:55:23.684+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:55:23.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:55:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:55:23.855+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:55:23.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:55:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:55:23.855+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:55:23.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:55:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:55:23.855+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:55:23.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:55:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:55:23.855+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:55:24.493 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:55:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:55:24.163+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 47830 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:55:24.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:55:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:55:24.161+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:55:24.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:55:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:55:24.161+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:55:24.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:55:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:55:24.161+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:55:24.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:55:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:55:24.876+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:55:24.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:55:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:55:24.876+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:55:24.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:55:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:55:24.876+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:55:24.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:55:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:55:24.876+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:55:24.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:55:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:55:24.721+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:55:24.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:55:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:55:24.721+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:55:24.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:55:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:55:24.721+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:55:25.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:55:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:55:25.136+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:55:25.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:55:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:55:25.136+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:55:25.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:55:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:55:25.136+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:55:25.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:55:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:55:25.910+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:55:25.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:55:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:55:25.910+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:55:25.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:55:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:55:25.910+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:55:25.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:55:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:55:25.910+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:55:25.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:55:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:55:25.696+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:55:25.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:55:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:55:25.696+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:55:25.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:55:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:55:25.696+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:55:26.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:55:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:55:26.153+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:55:26.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:55:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:55:26.153+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:55:26.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:55:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:55:26.153+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:55:26.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:55:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:55:26.724+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:55:26.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:55:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:55:26.724+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:55:26.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:55:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:55:26.724+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:55:26.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:55:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:55:26.940+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:55:26.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:55:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:55:26.940+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:55:26.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:55:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:55:26.940+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:55:26.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:55:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:55:26.940+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:55:27.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:55:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:55:27.185+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:55:27.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:55:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:55:27.185+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:55:27.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:55:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:55:27.185+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:55:27.982 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:55:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:55:27.717+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:55:27.982 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:55:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:55:27.717+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:55:27.982 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:55:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:55:27.717+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:55:28.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:55:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:55:27.979+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:55:28.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:55:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:55:27.979+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:55:28.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:55:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:55:27.979+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:55:28.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:55:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:55:27.979+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:55:28.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:55:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:55:28.216+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:55:28.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:55:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:55:28.216+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:55:28.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:55:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:55:28.216+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:55:28.977 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:55:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:55:28.677+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:55:28.978 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:55:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:55:28.677+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:55:28.978 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:55:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:55:28.677+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:55:29.243 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:55:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:55:29.170+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 47902 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:55:29.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:55:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:55:29.193+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:55:29.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:55:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:55:29.193+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:55:29.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:55:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:55:29.193+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:55:29.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:55:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:55:28.975+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:55:29.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:55:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:55:28.975+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:55:29.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:55:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:55:28.975+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:55:29.245 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:55:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:55:28.975+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:55:29.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:55:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:55:29.682+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:55:29.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:55:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:55:29.682+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:55:29.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:55:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:55:29.682+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:55:30.342 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:55:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:55:30.179+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:55:30.342 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:55:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:55:30.179+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:55:30.343 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:55:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:55:30.179+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:55:30.343 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:55:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:55:30.020+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:55:30.343 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:55:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:55:30.020+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:55:30.343 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:55:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:55:30.020+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:55:30.343 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:55:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:55:30.021+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:55:30.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:55:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:55:30.658+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:55:30.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:55:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:55:30.658+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:55:30.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:55:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:55:30.658+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:55:31.322 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:55:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:55:31.062+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:55:31.322 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:55:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:55:31.062+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:55:31.322 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:55:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:55:31.062+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:55:31.322 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:55:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:55:31.062+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:55:31.323 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:55:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:55:31.167+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:55:31.323 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:55:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:55:31.167+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:55:31.323 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:55:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:55:31.167+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:55:31.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:55:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:55:31.699+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:55:31.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:55:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:55:31.699+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:55:31.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:55:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:55:31.699+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:55:32.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:55:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:55:32.184+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:55:32.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:55:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:55:32.184+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:55:32.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:55:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:55:32.184+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:55:32.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:55:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:55:32.096+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:55:32.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:55:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:55:32.096+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:55:32.495 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:55:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:55:32.096+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:55:32.495 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:55:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:55:32.096+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:55:32.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:55:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:55:32.735+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:55:32.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:55:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:55:32.735+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:55:32.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:55:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:55:32.735+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:55:33.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:55:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:55:33.217+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:55:33.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:55:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:55:33.217+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:55:33.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:55:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:55:33.217+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:55:33.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:55:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:55:33.069+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:55:33.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:55:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:55:33.069+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:55:33.495 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:55:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:55:33.069+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:55:33.495 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:55:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:55:33.069+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:55:33.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:55:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:55:33.715+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:55:33.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:55:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:55:33.715+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:55:33.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:55:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:55:33.715+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:55:34.300 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:55:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:55:34.214+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:55:34.301 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:55:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:55:34.214+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:55:34.301 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:55:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:55:34.214+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:55:34.301 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:55:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:55:34.177+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 47974 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:55:34.301 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:55:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:55:34.025+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:55:34.302 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:55:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:55:34.025+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:55:34.302 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:55:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:55:34.025+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:55:34.302 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:55:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:55:34.025+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:55:34.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:55:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:55:34.717+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:55:34.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:55:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:55:34.717+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:55:34.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:55:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:55:34.717+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:55:35.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:55:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:55:35.241+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:55:35.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:55:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:55:35.241+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:55:35.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:55:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:55:35.241+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:55:35.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:55:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:55:35.056+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:55:35.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:55:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:55:35.056+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:55:35.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:55:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:55:35.056+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:55:35.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:55:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:55:35.056+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:55:36.016 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:55:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:55:35.764+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:55:36.019 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:55:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:55:35.764+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:55:36.019 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:55:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:55:35.764+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:55:36.286 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:55:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:55:36.014+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:55:36.287 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:55:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:55:36.014+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:55:36.287 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:55:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:55:36.014+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:55:36.287 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:55:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:55:36.014+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:55:36.721 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:55:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:55:36.284+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:55:36.721 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:55:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:55:36.284+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:55:36.722 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:55:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:55:36.284+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:55:36.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:55:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:55:36.719+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:55:36.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:55:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:55:36.719+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:55:36.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:55:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:55:36.719+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:55:37.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:55:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:55:37.243+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:55:37.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:55:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:55:37.243+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:55:37.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:55:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:55:37.243+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:55:37.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:55:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:55:37.023+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:55:37.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:55:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:55:37.023+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:55:37.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:55:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:55:37.023+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:55:37.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:55:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:55:37.023+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:55:38.041 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:55:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:55:37.757+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:55:38.041 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:55:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:55:37.757+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:55:38.041 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:55:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:55:37.757+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:55:38.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:55:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:55:38.229+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:55:38.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:55:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:55:38.229+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:55:38.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:55:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:55:38.229+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:55:38.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:55:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:55:38.038+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:55:38.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:55:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:55:38.038+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:55:38.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:55:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:55:38.038+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:55:38.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:55:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:55:38.038+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:55:39.187 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:55:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:55:38.765+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:55:39.187 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:55:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:55:38.765+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:55:39.187 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:55:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:55:38.765+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:55:39.187 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:55:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:55:39.009+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:55:39.188 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:55:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:55:39.009+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:55:39.188 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:55:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:55:39.009+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:55:39.188 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:55:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:55:39.010+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:55:39.493 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:55:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:55:39.184+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 48046 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:55:39.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:55:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:55:39.258+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:55:39.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:55:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:55:39.258+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:55:39.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:55:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:55:39.258+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:55:39.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:55:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:55:39.730+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:55:39.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:55:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:55:39.730+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:55:39.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:55:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:55:39.730+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:55:40.343 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:55:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:55:40.256+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:55:40.343 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:55:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:55:40.256+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:55:40.343 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:55:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:55:40.256+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:55:40.344 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:55:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:55:40.044+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:55:40.344 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:55:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:55:40.044+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:55:40.344 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:55:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:55:40.044+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:55:40.344 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:55:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:55:40.044+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:55:40.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:55:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:55:40.706+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:55:40.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:55:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:55:40.706+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:55:40.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:55:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:55:40.706+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:55:41.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:55:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:55:41.299+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:55:41.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:55:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:55:41.299+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:55:41.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:55:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:55:41.299+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:55:41.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:55:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:55:41.092+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:55:41.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:55:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:55:41.092+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:55:41.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:55:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:55:41.092+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:55:41.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:55:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:55:41.092+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:55:41.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:55:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:55:41.669+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:55:41.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:55:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:55:41.669+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:55:41.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:55:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:55:41.669+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:55:42.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:55:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:55:42.300+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:55:42.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:55:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:55:42.300+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:55:42.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:55:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:55:42.300+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:55:42.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:55:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:55:42.094+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:55:42.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:55:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:55:42.094+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:55:42.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:55:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:55:42.094+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:55:42.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:55:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:55:42.094+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:55:42.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:55:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:55:42.698+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:55:42.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:55:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:55:42.698+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:55:42.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:55:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:55:42.698+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:55:43.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:55:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:55:43.267+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:55:43.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:55:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:55:43.267+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:55:43.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:55:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:55:43.267+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:55:43.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:55:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:55:43.098+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:55:43.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:55:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:55:43.098+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:55:43.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:55:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:55:43.098+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:55:43.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:55:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:55:43.098+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:55:43.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:55:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:55:43.676+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:55:43.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:55:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:55:43.676+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:55:43.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:55:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:55:43.676+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:55:44.493 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:55:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:55:44.190+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 48118 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:55:44.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:55:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:55:44.283+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:55:44.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:55:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:55:44.283+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:55:44.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:55:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:55:44.283+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:55:44.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:55:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:55:44.085+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:55:44.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:55:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:55:44.085+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:55:44.495 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:55:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:55:44.085+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:55:44.495 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:55:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:55:44.085+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:55:44.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:55:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:55:44.683+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:55:44.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:55:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:55:44.683+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:55:44.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:55:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:55:44.683+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:55:45.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:55:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:55:45.302+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:55:45.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:55:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:55:45.302+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:55:45.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:55:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:55:45.302+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:55:45.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:55:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:55:45.099+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:55:45.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:55:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:55:45.099+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:55:45.495 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:55:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:55:45.099+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:55:45.495 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:55:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:55:45.100+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:55:45.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:55:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:55:45.641+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:55:45.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:55:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:55:45.641+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:55:45.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:55:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:55:45.641+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:55:46.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:55:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:55:46.305+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:55:46.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:55:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:55:46.305+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:55:46.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:55:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:55:46.305+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:55:46.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:55:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:55:46.131+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:55:46.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:55:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:55:46.131+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:55:46.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:55:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:55:46.131+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:55:46.495 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:55:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:55:46.131+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:55:46.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:55:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:55:46.687+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:55:46.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:55:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:55:46.687+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:55:46.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:55:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:55:46.687+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:55:47.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:55:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:55:47.271+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:55:47.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:55:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:55:47.271+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:55:47.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:55:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:55:47.271+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:55:47.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:55:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:55:47.150+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:55:47.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:55:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:55:47.150+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:55:47.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:55:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:55:47.150+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:55:47.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:55:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:55:47.150+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:55:47.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:55:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:55:47.719+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:55:47.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:55:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:55:47.719+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:55:47.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:55:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:55:47.719+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:55:48.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:55:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:55:48.252+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:55:48.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:55:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:55:48.252+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:55:48.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:55:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:55:48.252+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:55:48.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:55:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:55:48.178+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:55:48.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:55:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:55:48.178+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:55:48.495 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:55:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:55:48.178+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:55:48.495 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:55:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:55:48.178+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:55:48.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:55:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:55:48.710+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:55:48.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:55:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:55:48.710+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:55:48.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:55:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:55:48.710+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:55:49.494 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:55:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:55:49.197+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 48190 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:55:49.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:55:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:55:49.143+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:55:49.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:55:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:55:49.143+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:55:49.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:55:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:55:49.143+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:55:49.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:55:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:55:49.143+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:55:49.495 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:55:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:55:49.272+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:55:49.495 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:55:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:55:49.272+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:55:49.495 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:55:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:55:49.272+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:55:49.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:55:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:55:49.726+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:55:49.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:55:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:55:49.726+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:55:49.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:55:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:55:49.726+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:55:50.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:55:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:55:50.298+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:55:50.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:55:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:55:50.298+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:55:50.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:55:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:55:50.298+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:55:50.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:55:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:55:50.094+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:55:50.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:55:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:55:50.094+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:55:50.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:55:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:55:50.094+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:55:50.495 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:55:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:55:50.094+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:55:50.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:55:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:55:50.724+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:55:50.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:55:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:55:50.724+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:55:50.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:55:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:55:50.724+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:55:51.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:55:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:55:51.281+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:55:51.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:55:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:55:51.281+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:55:51.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:55:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:55:51.281+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:55:51.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:55:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:55:51.122+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:55:51.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:55:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:55:51.122+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:55:51.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:55:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:55:51.122+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:55:51.495 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:55:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:55:51.122+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:55:52.103 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:55:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:55:51.751+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:55:52.103 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:55:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:55:51.751+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:55:52.103 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:55:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:55:51.751+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:55:52.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:55:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:55:52.246+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:55:52.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:55:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:55:52.246+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:55:52.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:55:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:55:52.246+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:55:52.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:55:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:55:52.099+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:55:52.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:55:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:55:52.099+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:55:52.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:55:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:55:52.099+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:55:52.495 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:55:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:55:52.099+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:55:53.057 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:55:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:55:52.790+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:55:53.057 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:55:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:55:52.790+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:55:53.057 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:55:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:55:52.790+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:55:53.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:55:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:55:53.232+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:55:53.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:55:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:55:53.232+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:55:53.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:55:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:55:53.232+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:55:53.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:55:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:55:53.054+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:55:53.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:55:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:55:53.054+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:55:53.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:55:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:55:53.054+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:55:53.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:55:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:55:53.054+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:55:54.060 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:55:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:55:53.789+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:55:54.061 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:55:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:55:53.789+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:55:54.061 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:55:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:55:53.789+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:55:54.328 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:55:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:55:54.204+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 48262 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:55:54.328 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:55:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:55:54.227+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:55:54.329 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:55:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:55:54.227+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:55:54.329 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:55:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:55:54.227+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:55:54.329 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:55:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:55:54.057+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:55:54.329 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:55:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:55:54.057+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:55:54.329 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:55:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:55:54.057+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:55:54.329 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:55:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:55:54.057+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:55:55.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:55:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:55:54.801+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:55:55.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:55:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:55:54.801+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:55:55.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:55:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:55:54.801+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:55:55.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:55:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:55:55.013+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:55:55.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:55:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:55:55.013+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:55:55.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:55:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:55:55.013+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:55:55.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:55:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:55:55.013+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:55:55.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:55:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:55:55.270+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:55:55.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:55:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:55:55.270+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:55:55.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:55:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:55:55.270+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:55:56.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:55:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:55:55.847+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:55:56.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:55:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:55:55.847+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:55:56.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:55:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:55:55.847+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:55:56.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:55:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:55:56.013+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:55:56.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:55:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:55:56.013+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:55:56.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:55:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:55:56.013+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:55:56.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:55:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:55:56.013+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:55:56.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:55:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:55:56.255+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:55:56.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:55:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:55:56.255+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:55:56.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:55:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:55:56.255+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:55:57.214 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:55:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:55:56.889+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:55:57.214 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:55:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:55:56.889+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:55:57.214 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:55:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:55:56.889+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:55:57.214 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:55:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:55:57.053+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:55:57.214 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:55:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:55:57.053+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:55:57.215 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:55:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:55:57.053+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:55:57.215 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:55:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:55:57.053+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:55:57.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:55:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:55:57.210+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:55:57.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:55:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:55:57.210+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:55:57.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:55:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:55:57.210+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:55:58.207 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:55:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:55:57.861+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:55:58.208 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:55:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:55:57.861+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:55:58.208 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:55:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:55:57.861+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:55:58.208 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:55:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:55:58.070+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:55:58.208 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:55:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:55:58.070+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:55:58.209 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:55:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:55:58.070+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:55:58.209 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:55:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:55:58.070+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:55:58.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:55:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:55:58.203+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:55:58.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:55:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:55:58.204+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:55:58.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:55:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:55:58.204+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:55:59.197 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:55:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:55:58.905+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:55:59.197 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:55:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:55:58.905+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:55:59.197 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:55:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:55:58.905+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:55:59.197 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:55:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:55:59.111+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:55:59.197 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:55:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:55:59.111+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:55:59.198 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:55:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:55:59.111+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:55:59.198 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:55:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:55:59.111+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:55:59.493 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:55:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:55:59.212+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 48334 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:55:59.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:55:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:55:59.193+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:55:59.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:55:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:55:59.193+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:55:59.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:55:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:55:59.193+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:56:00.169 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:55:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:55:59.880+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:56:00.169 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:55:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:55:59.880+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:56:00.169 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:55:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:55:59.880+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:56:00.169 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:56:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:56:00.090+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:56:00.170 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:56:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:56:00.090+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:56:00.170 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:56:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:56:00.090+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:56:00.170 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:56:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:56:00.090+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:56:00.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:56:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:56:00.165+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:56:00.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:56:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:56:00.165+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:56:00.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:56:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:56:00.165+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:56:01.168 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:56:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:56:00.861+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:56:01.169 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:56:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:56:00.861+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:56:01.169 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:56:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:56:00.861+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:56:01.169 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:56:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:56:01.090+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:56:01.169 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:56:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:56:01.090+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:56:01.169 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:56:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:56:01.090+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:56:01.169 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:56:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:56:01.090+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:56:01.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:56:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:56:01.164+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:56:01.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:56:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:56:01.164+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:56:01.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:56:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:56:01.164+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:56:02.170 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:56:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:56:01.889+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:56:02.170 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:56:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:56:01.889+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:56:02.170 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:56:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:56:01.889+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:56:02.171 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:56:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:56:02.059+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:56:02.171 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:56:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:56:02.059+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:56:02.171 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:56:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:56:02.059+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:56:02.171 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:56:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:56:02.059+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:56:02.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:56:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:56:02.167+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:56:02.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:56:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:56:02.167+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:56:02.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:56:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:56:02.167+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:56:03.188 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:56:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:56:02.890+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:56:03.188 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:56:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:56:02.890+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:56:03.188 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:56:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:56:02.890+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:56:03.188 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:56:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:56:03.074+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:56:03.188 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:56:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:56:03.074+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:56:03.189 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:56:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:56:03.074+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:56:03.189 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:56:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:56:03.074+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:56:03.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:56:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:56:03.184+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:56:03.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:56:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:56:03.184+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:56:03.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:56:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:56:03.184+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:56:04.171 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:56:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:56:03.867+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:56:04.172 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:56:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:56:03.867+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:56:04.172 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:56:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:56:03.867+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:56:04.172 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:56:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:56:04.042+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:56:04.172 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:56:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:56:04.042+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:56:04.172 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:56:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:56:04.042+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:56:04.172 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:56:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:56:04.042+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:56:04.493 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:56:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:56:04.220+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 48406 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:56:04.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:56:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:56:04.168+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:56:04.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:56:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:56:04.168+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:56:04.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:56:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:56:04.168+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:56:05.148 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:56:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:56:05.014+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:56:05.149 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:56:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:56:05.014+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:56:05.149 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:56:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:56:05.014+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:56:05.149 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:56:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:56:05.014+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:56:05.149 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:56:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:56:04.852+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:56:05.149 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:56:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:56:04.853+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:56:05.149 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:56:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:56:04.853+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:56:05.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:56:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:56:05.145+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:56:05.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:56:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:56:05.145+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:56:05.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:56:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:56:05.145+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:56:06.166 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:56:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:56:06.006+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:56:06.167 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:56:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:56:06.006+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:56:06.167 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:56:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:56:06.006+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:56:06.167 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:56:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:56:06.006+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:56:06.167 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:56:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:56:05.818+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:56:06.167 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:56:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:56:05.818+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:56:06.167 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:56:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:56:05.818+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:56:06.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:56:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:56:06.162+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:56:06.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:56:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:56:06.162+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:56:06.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:56:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:56:06.162+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:56:07.182 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:56:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:56:06.972+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:56:07.182 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:56:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:56:06.972+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:56:07.182 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:56:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:56:06.972+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:56:07.183 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:56:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:56:06.972+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:56:07.183 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:56:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:56:06.774+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:56:07.183 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:56:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:56:06.774+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:56:07.183 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:56:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:56:06.774+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:56:07.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:56:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:56:07.179+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:56:07.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:56:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:56:07.179+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:56:07.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:56:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:56:07.179+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:56:08.151 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:56:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:56:07.934+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:56:08.151 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:56:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:56:07.934+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:56:08.151 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:56:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:56:07.934+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:56:08.151 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:56:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:56:07.934+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:56:08.152 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:56:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:56:07.811+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:56:08.152 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:56:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:56:07.811+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:56:08.152 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:56:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:56:07.811+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:56:08.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:56:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:56:08.148+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:56:08.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:56:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:56:08.148+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:56:08.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:56:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:56:08.148+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:56:09.130 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:56:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:56:08.945+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:56:09.130 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:56:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:56:08.945+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:56:09.130 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:56:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:56:08.945+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:56:09.130 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:56:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:56:08.945+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:56:09.131 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:56:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:56:08.838+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:56:09.131 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:56:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:56:08.838+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:56:09.131 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:56:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:56:08.838+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:56:09.493 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:56:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:56:09.227+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 48478 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:56:09.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:56:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:56:09.127+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:56:09.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:56:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:56:09.127+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:56:09.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:56:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:56:09.127+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:56:10.096 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:56:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:56:09.986+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:56:10.096 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:56:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:56:09.986+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:56:10.097 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:56:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:56:09.986+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:56:10.097 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:56:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:56:09.986+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:56:10.097 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:56:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:56:09.791+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:56:10.097 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:56:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:56:09.791+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:56:10.097 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:56:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:56:09.791+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:56:10.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:56:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:56:10.092+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:56:10.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:56:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:56:10.092+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:56:10.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:56:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:56:10.092+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:56:11.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:56:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:56:11.058+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:56:11.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:56:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:56:11.058+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:56:11.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:56:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:56:11.058+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:56:11.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:56:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:56:10.819+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:56:11.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:56:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:56:10.819+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:56:11.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:56:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:56:10.819+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:56:11.245 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:56:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:56:10.952+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:56:11.245 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:56:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:56:10.952+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:56:11.245 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:56:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:56:10.952+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:56:11.245 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:56:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:56:10.952+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:56:12.105 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:56:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:56:11.916+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:56:12.105 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:56:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:56:11.916+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:56:12.106 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:56:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:56:11.916+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:56:12.106 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:56:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:56:11.917+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:56:12.106 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:56:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:56:11.774+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:56:12.106 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:56:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:56:11.774+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:56:12.106 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:56:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:56:11.774+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:56:12.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:56:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:56:12.102+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:56:12.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:56:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:56:12.102+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:56:12.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:56:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:56:12.102+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:56:13.084 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:56:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:56:12.913+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:56:13.084 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:56:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:56:12.913+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:56:13.084 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:56:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:56:12.913+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:56:13.085 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:56:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:56:12.913+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:56:13.085 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:56:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:56:12.798+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:56:13.085 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:56:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:56:12.798+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:56:13.085 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:56:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:56:12.798+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:56:13.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:56:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:56:13.081+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:56:13.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:56:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:56:13.081+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:56:13.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:56:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:56:13.081+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:56:14.125 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:56:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:56:13.935+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:56:14.125 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:56:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:56:13.935+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:56:14.125 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:56:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:56:13.935+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:56:14.125 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:56:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:56:13.935+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:56:14.125 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:56:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:56:13.782+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:56:14.126 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:56:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:56:13.782+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:56:14.126 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:56:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:56:13.782+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:56:14.494 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:56:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:56:14.234+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 48550 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:56:14.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:56:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:56:14.122+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:56:14.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:56:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:56:14.122+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:56:14.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:56:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:56:14.122+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:56:15.136 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:56:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:56:14.802+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:56:15.136 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:56:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:56:14.802+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:56:15.137 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:56:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:56:14.802+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:56:15.137 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:56:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:56:14.954+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:56:15.137 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:56:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:56:14.954+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:56:15.137 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:56:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:56:14.954+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:56:15.137 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:56:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:56:14.955+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:56:15.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:56:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:56:15.133+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:56:15.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:56:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:56:15.133+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:56:15.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:56:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:56:15.133+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:56:16.170 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:56:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:56:15.822+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:56:16.170 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:56:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:56:15.822+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:56:16.171 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:56:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:56:15.822+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:56:16.171 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:56:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:56:15.995+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:56:16.171 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:56:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:56:15.995+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:56:16.171 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:56:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:56:15.995+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:56:16.171 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:56:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:56:15.995+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:56:16.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:56:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:56:16.167+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:56:16.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:56:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:56:16.167+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:56:16.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:56:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:56:16.167+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:56:17.179 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:56:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:56:16.810+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:56:17.180 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:56:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:56:16.810+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:56:17.180 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:56:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:56:16.810+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:56:17.180 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:56:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:56:16.969+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:56:17.180 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:56:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:56:16.969+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:56:17.180 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:56:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:56:16.969+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:56:17.180 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:56:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:56:16.969+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:56:17.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:56:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:56:17.176+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:56:17.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:56:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:56:17.176+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:56:17.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:56:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:56:17.176+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:56:18.209 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:56:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:56:17.817+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:56:18.209 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:56:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:56:17.817+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:56:18.209 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:56:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:56:17.817+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:56:18.210 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:56:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:56:17.929+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:56:18.210 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:56:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:56:17.929+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:56:18.210 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:56:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:56:17.929+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:56:18.210 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:56:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:56:17.929+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:56:18.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:56:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:56:18.206+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:56:18.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:56:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:56:18.206+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:56:18.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:56:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:56:18.206+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:56:19.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:56:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:56:18.920+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:56:19.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:56:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:56:18.920+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:56:19.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:56:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:56:18.920+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:56:19.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:56:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:56:18.920+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:56:19.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:56:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:56:18.826+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:56:19.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:56:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:56:18.826+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:56:19.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:56:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:56:18.826+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:56:19.743 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:56:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:56:19.242+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 48618 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:56:19.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:56:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:56:19.249+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:56:19.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:56:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:56:19.249+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:56:19.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:56:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:56:19.249+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:56:20.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:56:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:56:19.781+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:56:20.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:56:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:56:19.781+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:56:20.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:56:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:56:19.781+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:56:20.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:56:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:56:19.889+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:56:20.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:56:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:56:19.889+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:56:20.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:56:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:56:19.889+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:56:20.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:56:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:56:19.890+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:56:20.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:56:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:56:20.284+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:56:20.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:56:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:56:20.284+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:56:20.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:56:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:56:20.284+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:56:21.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:56:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:56:20.777+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:56:21.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:56:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:56:20.777+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:56:21.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:56:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:56:20.777+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:56:21.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:56:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:56:20.882+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:56:21.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:56:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:56:20.882+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:56:21.245 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:56:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:56:20.882+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:56:21.245 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:56:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:56:20.882+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:56:21.731 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:56:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:56:21.304+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:56:21.732 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:56:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:56:21.304+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:56:21.732 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:56:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:56:21.304+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:56:21.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:56:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:56:21.848+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:56:21.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:56:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:56:21.848+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:56:21.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:56:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:56:21.848+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:56:21.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:56:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:56:21.848+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:56:21.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:56:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:56:21.729+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:56:21.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:56:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:56:21.729+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:56:21.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:56:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:56:21.729+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:56:22.688 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:56:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:56:22.309+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:56:22.688 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:56:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:56:22.309+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:56:22.688 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:56:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:56:22.309+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:56:22.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:56:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:56:22.685+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:56:22.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:56:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:56:22.685+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:56:22.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:56:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:56:22.685+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:56:22.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:56:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:56:22.845+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:56:22.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:56:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:56:22.845+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:56:22.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:56:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:56:22.845+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:56:22.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:56:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:56:22.845+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:56:23.657 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:56:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:56:23.297+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:56:23.657 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:56:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:56:23.297+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:56:23.657 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:56:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:56:23.297+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:56:23.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:56:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:56:23.654+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:56:23.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:56:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:56:23.654+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:56:23.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:56:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:56:23.654+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:56:23.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:56:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:56:23.854+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:56:23.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:56:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:56:23.854+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:56:23.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:56:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:56:23.854+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:56:23.995 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:56:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:56:23.854+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:56:24.657 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:56:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:56:24.248+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 48690 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:56:24.657 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:56:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:56:24.314+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:56:24.657 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:56:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:56:24.314+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:56:24.658 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:56:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:56:24.314+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:56:24.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:56:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:56:24.654+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:56:24.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:56:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:56:24.654+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:56:24.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:56:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:56:24.654+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:56:24.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:56:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:56:24.899+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:56:24.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:56:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:56:24.899+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:56:24.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:56:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:56:24.899+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:56:24.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:56:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:56:24.899+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:56:25.674 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:56:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:56:25.359+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:56:25.675 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:56:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:56:25.359+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:56:25.675 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:56:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:56:25.359+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:56:25.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:56:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:56:25.671+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:56:25.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:56:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:56:25.671+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:56:25.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:56:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:56:25.671+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:56:25.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:56:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:56:25.891+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:56:25.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:56:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:56:25.891+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:56:25.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:56:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:56:25.891+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:56:25.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:56:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:56:25.891+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:56:26.687 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:56:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:56:26.356+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:56:26.687 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:56:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:56:26.356+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:56:26.688 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:56:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:56:26.356+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:56:26.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:56:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:56:26.684+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:56:26.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:56:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:56:26.684+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:56:26.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:56:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:56:26.684+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:56:26.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:56:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:56:26.849+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:56:26.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:56:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:56:26.849+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:56:26.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:56:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:56:26.849+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:56:26.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:56:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:56:26.849+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:56:27.671 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:56:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:56:27.380+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:56:27.672 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:56:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:56:27.380+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:56:27.672 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:56:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:56:27.380+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:56:27.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:56:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:56:27.668+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:56:27.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:56:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:56:27.668+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:56:27.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:56:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:56:27.668+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:56:27.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:56:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:56:27.870+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:56:27.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:56:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:56:27.870+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:56:27.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:56:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:56:27.870+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:56:27.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:56:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:56:27.870+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:56:28.688 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:56:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:56:28.379+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:56:28.688 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:56:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:56:28.379+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:56:28.688 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:56:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:56:28.379+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:56:28.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:56:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:56:28.685+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:56:28.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:56:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:56:28.685+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:56:28.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:56:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:56:28.685+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:56:28.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:56:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:56:28.896+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:56:28.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:56:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:56:28.896+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:56:28.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:56:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:56:28.896+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:56:28.995 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:56:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:56:28.896+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:56:29.657 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:56:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:56:29.255+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 48762 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:56:29.657 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:56:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:56:29.416+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:56:29.657 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:56:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:56:29.416+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:56:29.657 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:56:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:56:29.416+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:56:29.931 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:56:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:56:29.654+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:56:29.931 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:56:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:56:29.654+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:56:29.932 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:56:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:56:29.654+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:56:30.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:56:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:56:29.928+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:56:30.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:56:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:56:29.928+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:56:30.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:56:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:56:29.928+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:56:30.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:56:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:56:29.928+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:56:30.672 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:56:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:56:30.450+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:56:30.673 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:56:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:56:30.450+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:56:30.673 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:56:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:56:30.450+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:56:30.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:56:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:56:30.670+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:56:30.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:56:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:56:30.670+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:56:30.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:56:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:56:30.670+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:56:30.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:56:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:56:30.913+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:56:30.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:56:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:56:30.913+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:56:30.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:56:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:56:30.913+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:56:30.995 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:56:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:56:30.913+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:56:31.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:56:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:56:31.649+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:56:31.744 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:56:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:56:31.649+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:56:31.744 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:56:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:56:31.649+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:56:31.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:56:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:56:31.463+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:56:31.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:56:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:56:31.463+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:56:31.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:56:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:56:31.463+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:56:32.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:56:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:56:31.865+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:56:32.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:56:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:56:31.865+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:56:32.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:56:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:56:31.865+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:56:32.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:56:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:56:31.865+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:56:32.697 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:56:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:56:32.441+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:56:32.697 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:56:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:56:32.441+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:56:32.697 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:56:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:56:32.441+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:56:32.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:56:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:56:32.694+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:56:32.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:56:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:56:32.694+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:56:32.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:56:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:56:32.694+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:56:32.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:56:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:56:32.867+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:56:32.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:56:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:56:32.867+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:56:32.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:56:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:56:32.867+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:56:32.995 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:56:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:56:32.867+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:56:33.744 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:56:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:56:33.649+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:56:33.744 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:56:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:56:33.649+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:56:33.744 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:56:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:56:33.649+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:56:33.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:56:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:56:33.407+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:56:33.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:56:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:56:33.407+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:56:33.745 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:56:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:56:33.407+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:56:34.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:56:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:56:33.885+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:56:34.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:56:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:56:33.885+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:56:34.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:56:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:56:33.885+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:56:34.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:56:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:56:33.885+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:56:34.689 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:56:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:56:34.262+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 48834 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:56:34.689 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:56:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:56:34.421+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:56:34.690 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:56:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:56:34.421+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:56:34.690 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:56:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:56:34.421+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:56:34.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:56:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:56:34.846+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:56:34.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:56:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:56:34.846+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:56:34.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:56:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:56:34.846+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:56:34.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:56:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:56:34.846+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:56:34.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:56:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:56:34.686+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:56:34.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:56:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:56:34.686+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:56:34.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:56:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:56:34.686+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:56:35.653 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:56:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:56:35.397+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:56:35.653 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:56:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:56:35.397+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:56:35.653 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:56:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:56:35.397+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:56:35.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:56:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:56:35.841+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:56:35.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:56:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:56:35.841+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:56:35.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:56:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:56:35.841+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:56:35.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:56:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:56:35.841+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:56:35.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:56:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:56:35.650+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:56:35.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:56:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:56:35.650+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:56:35.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:56:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:56:35.650+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:56:36.682 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:56:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:56:36.409+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:56:36.682 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:56:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:56:36.409+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:56:36.682 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:56:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:56:36.409+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:56:36.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:56:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:56:36.888+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:56:36.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:56:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:56:36.888+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:56:36.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:56:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:56:36.888+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:56:36.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:56:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:56:36.888+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:56:36.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:56:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:56:36.678+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:56:36.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:56:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:56:36.678+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:56:36.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:56:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:56:36.678+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:56:37.726 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:56:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:56:37.454+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:56:37.726 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:56:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:56:37.454+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:56:37.726 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:56:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:56:37.454+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:56:37.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:56:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:56:37.846+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:56:37.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:56:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:56:37.846+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:56:37.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:56:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:56:37.846+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:56:37.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:56:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:56:37.846+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:56:37.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:56:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:56:37.722+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:56:37.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:56:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:56:37.722+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:56:37.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:56:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:56:37.722+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:56:38.733 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:56:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:56:38.408+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:56:38.733 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:56:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:56:38.408+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:56:38.733 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:56:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:56:38.408+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:56:38.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:56:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:56:38.886+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:56:38.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:56:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:56:38.886+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:56:38.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:56:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:56:38.886+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:56:38.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:56:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:56:38.886+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:56:38.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:56:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:56:38.729+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:56:38.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:56:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:56:38.729+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:56:38.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:56:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:56:38.729+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:56:39.743 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:56:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:56:39.269+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 48906 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:56:39.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:56:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:56:39.408+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:56:39.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:56:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:56:39.408+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:56:39.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:56:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:56:39.408+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:56:40.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:56:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:56:39.767+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:56:40.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:56:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:56:39.767+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:56:40.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:56:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:56:39.767+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:56:40.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:56:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:56:39.853+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:56:40.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:56:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:56:39.853+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:56:40.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:56:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:56:39.853+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:56:40.245 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:56:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:56:39.853+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:56:40.736 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:56:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:56:40.414+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:56:40.736 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:56:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:56:40.414+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:56:40.736 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:56:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:56:40.414+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:56:40.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:56:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:56:40.732+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:56:40.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:56:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:56:40.732+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:56:40.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:56:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:56:40.732+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:56:40.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:56:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:56:40.809+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:56:40.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:56:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:56:40.809+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:56:40.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:56:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:56:40.809+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:56:40.995 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:56:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:56:40.810+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:56:41.710 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:56:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:56:41.416+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:56:41.711 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:56:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:56:41.416+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:56:41.711 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:56:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:56:41.416+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:56:41.978 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:56:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:56:41.839+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:56:41.978 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:56:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:56:41.839+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:56:41.978 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:56:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:56:41.839+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:56:41.978 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:56:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:56:41.839+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:56:41.978 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:56:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:56:41.706+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:56:41.979 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:56:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:56:41.706+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:56:41.979 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:56:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:56:41.706+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:56:42.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:56:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:56:42.668+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:56:42.744 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:56:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:56:42.668+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:56:42.744 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:56:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:56:42.668+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:56:42.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:56:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:56:42.425+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:56:42.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:56:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:56:42.425+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:56:42.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:56:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:56:42.425+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:56:43.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:56:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:56:42.849+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:56:43.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:56:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:56:42.849+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:56:43.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:56:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:56:42.849+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:56:43.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:56:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:56:42.849+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:56:43.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:56:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:56:43.428+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:56:43.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:56:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:56:43.428+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:56:43.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:56:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:56:43.428+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:56:43.744 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:56:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:56:43.634+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:56:43.744 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:56:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:56:43.634+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:56:43.744 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:56:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:56:43.634+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:56:44.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:56:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:56:43.838+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:56:44.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:56:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:56:43.838+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:56:44.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:56:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:56:43.838+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:56:44.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:56:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:56:43.838+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:56:44.603 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:56:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:56:44.276+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 48978 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:56:44.603 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:56:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:56:44.393+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:56:44.603 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:56:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:56:44.393+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:56:44.603 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:56:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:56:44.393+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:56:44.890 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:56:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:56:44.599+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:56:44.890 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:56:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:56:44.599+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:56:44.890 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:56:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:56:44.599+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:56:45.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:56:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:56:44.886+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:56:45.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:56:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:56:44.886+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:56:45.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:56:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:56:44.886+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:56:45.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:56:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:56:44.886+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:56:45.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:56:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:56:45.631+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:56:45.744 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:56:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:56:45.631+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:56:45.744 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:56:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:56:45.631+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:56:45.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:56:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:56:45.397+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:56:45.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:56:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:56:45.397+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:56:45.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:56:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:56:45.397+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:56:46.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:56:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:56:45.913+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:56:46.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:56:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:56:45.913+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:56:46.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:56:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:56:45.913+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:56:46.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:56:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:56:45.913+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:56:46.672 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:56:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:56:46.383+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:56:46.672 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:56:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:56:46.383+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:56:46.673 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:56:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:56:46.383+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:56:46.954 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:56:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:56:46.669+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:56:46.954 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:56:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:56:46.669+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:56:46.954 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:56:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:56:46.669+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:56:47.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:56:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:56:46.950+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:56:47.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:56:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:56:46.950+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:56:47.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:56:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:56:46.950+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:56:47.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:56:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:56:46.950+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:56:47.653 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:56:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:56:47.387+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:56:47.653 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:56:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:56:47.387+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:56:47.653 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:56:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:56:47.387+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:56:47.976 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:56:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:56:47.649+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:56:47.976 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:56:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:56:47.649+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:56:47.976 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:56:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:56:47.649+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:56:48.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:56:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:56:47.973+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:56:48.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:56:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:56:47.973+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:56:48.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:56:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:56:47.973+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:56:48.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:56:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:56:47.973+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:56:48.639 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:56:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:56:48.351+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:56:48.639 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:56:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:56:48.351+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:56:48.639 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:56:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:56:48.351+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:56:48.978 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:56:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:56:48.635+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:56:48.979 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:56:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:56:48.635+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:56:48.979 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:56:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:56:48.635+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:56:49.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:56:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:56:48.975+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:56:49.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:56:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:56:48.975+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:56:49.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:56:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:56:48.975+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:56:49.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:56:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:56:48.975+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:56:49.636 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:56:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:56:49.283+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 49050 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:56:49.636 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:56:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:56:49.361+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:56:49.636 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:56:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:56:49.361+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:56:49.636 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:56:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:56:49.361+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:56:49.887 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:56:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:56:49.632+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:56:49.887 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:56:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:56:49.632+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:56:49.887 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:56:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:56:49.632+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:56:50.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:56:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:56:49.963+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:56:50.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:56:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:56:49.963+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:56:50.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:56:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:56:49.963+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:56:50.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:56:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:56:49.963+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:56:50.678 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:56:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:56:50.407+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:56:50.679 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:56:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:56:50.407+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:56:50.679 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:56:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:56:50.407+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:56:50.948 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:56:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:56:50.675+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:56:50.948 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:56:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:56:50.675+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:56:50.948 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:56:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:56:50.675+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:56:51.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:56:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:56:50.944+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:56:51.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:56:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:56:50.944+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:56:51.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:56:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:56:50.944+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:56:51.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:56:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:56:50.944+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:56:51.674 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:56:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:56:51.381+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:56:51.674 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:56:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:56:51.381+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:56:51.674 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:56:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:56:51.381+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:56:51.957 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:56:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:56:51.671+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:56:51.957 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:56:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:56:51.671+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:56:51.957 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:56:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:56:51.671+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:56:52.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:56:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:56:51.953+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:56:52.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:56:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:56:51.953+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:56:52.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:56:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:56:51.953+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:56:52.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:56:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:56:51.953+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:56:52.704 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:56:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:56:52.369+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:56:52.704 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:56:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:56:52.369+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:56:52.704 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:56:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:56:52.369+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:56:52.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:56:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:56:52.700+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:56:52.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:56:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:56:52.700+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:56:52.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:56:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:56:52.700+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:56:53.336 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:56:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:56:52.997+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:56:53.336 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:56:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:56:52.997+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:56:53.337 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:56:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:56:52.997+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:56:53.337 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:56:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:56:52.997+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:56:53.678 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:56:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:56:53.333+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:56:53.678 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:56:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:56:53.333+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:56:53.678 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:56:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:56:53.333+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:56:53.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:56:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:56:53.674+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:56:53.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:56:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:56:53.674+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:56:53.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:56:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:56:53.674+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:56:54.302 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:56:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:56:54.290+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 49122 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:56:54.302 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:56:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:56:54.043+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:56:54.302 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:56:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:56:54.043+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:56:54.303 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:56:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:56:54.043+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:56:54.303 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:56:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:56:54.043+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:56:54.654 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:56:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:56:54.338+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:56:54.655 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:56:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:56:54.338+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:56:54.655 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:56:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:56:54.338+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:56:54.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:56:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:56:54.651+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:56:54.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:56:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:56:54.651+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:56:54.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:56:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:56:54.651+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:56:55.357 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:56:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:56:55.002+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:56:55.357 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:56:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:56:55.002+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:56:55.358 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:56:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:56:55.002+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:56:55.358 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:56:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:56:55.002+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:56:55.686 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:56:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:56:55.354+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:56:55.687 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:56:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:56:55.354+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:56:55.687 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:56:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:56:55.354+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:56:55.982 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:56:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:56:55.683+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:56:55.982 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:56:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:56:55.683+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:56:55.982 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:56:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:56:55.683+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:56:56.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:56:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:56:55.978+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:56:56.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:56:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:56:55.978+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:56:56.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:56:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:56:55.978+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:56:56.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:56:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:56:55.978+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:56:56.697 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:56:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:56:56.350+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:56:56.697 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:56:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:56:56.350+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:56:56.697 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:56:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:56:56.350+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:56:56.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:56:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:56:56.942+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:56:56.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:56:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:56:56.942+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:56:56.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:56:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:56:56.942+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:56:56.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:56:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:56:56.943+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:56:56.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:56:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:56:56.694+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:56:56.995 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:56:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:56:56.694+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:56:56.995 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:56:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:56:56.694+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:56:57.689 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:56:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:56:57.373+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:56:57.689 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:56:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:56:57.373+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:56:57.689 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:56:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:56:57.373+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:56:57.982 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:56:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:56:57.686+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:56:57.982 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:56:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:56:57.686+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:56:57.983 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:56:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:56:57.686+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:56:58.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:56:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:56:57.978+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:56:58.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:56:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:56:57.978+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:56:58.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:56:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:56:57.978+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:56:58.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:56:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:56:57.979+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:56:58.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:56:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:56:58.661+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:56:58.744 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:56:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:56:58.661+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:56:58.744 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:56:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:56:58.661+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:56:58.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:56:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:56:58.414+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:56:58.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:56:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:56:58.414+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:56:58.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:56:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:56:58.414+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:56:59.302 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:56:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:56:59.022+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:56:59.302 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:56:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:56:59.022+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:56:59.302 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:56:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:56:59.022+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:56:59.302 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:56:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:56:59.022+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:56:59.658 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:56:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:56:59.298+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 49194 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:56:59.658 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:56:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:56:59.403+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:56:59.658 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:56:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:56:59.403+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:56:59.658 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:56:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:56:59.403+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:56:59.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:56:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:56:59.655+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:56:59.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:56:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:56:59.655+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:56:59.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:56:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:56:59.655+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:57:00.348 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:57:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:57:00.021+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:57:00.349 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:57:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:57:00.021+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:57:00.349 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:57:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:57:00.021+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:57:00.349 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:57:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:57:00.021+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:57:00.680 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:57:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:57:00.449+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:57:00.680 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:57:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:57:00.449+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:57:00.680 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:57:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:57:00.449+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:57:00.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:57:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:57:00.677+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:57:00.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:57:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:57:00.677+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:57:00.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:57:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:57:00.677+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:57:01.424 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:57:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:57:00.995+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:57:01.424 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:57:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:57:00.995+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:57:01.425 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:57:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:57:00.995+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:57:01.425 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:57:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:57:00.996+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:57:01.716 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:57:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:57:01.421+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:57:01.716 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:57:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:57:01.421+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:57:01.716 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:57:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:57:01.421+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:57:01.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:57:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:57:01.713+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:57:01.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:57:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:57:01.713+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:57:01.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:57:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:57:01.713+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:57:01.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:57:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:57:01.961+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:57:01.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:57:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:57:01.961+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:57:01.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:57:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:57:01.961+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:57:01.995 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:57:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:57:01.961+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:57:02.744 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:57:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:57:02.668+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:57:02.744 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:57:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:57:02.668+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:57:02.744 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:57:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:57:02.668+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:57:02.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:57:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:57:02.441+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:57:02.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:57:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:57:02.441+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:57:02.745 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:57:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:57:02.441+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:57:03.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:57:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:57:02.944+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:57:03.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:57:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:57:02.944+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:57:03.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:57:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:57:02.944+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:57:03.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:57:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:57:02.944+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:57:03.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:57:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:57:03.690+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:57:03.744 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:57:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:57:03.690+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:57:03.744 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:57:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:57:03.690+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:57:03.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:57:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:57:03.442+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:57:03.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:57:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:57:03.442+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:57:03.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:57:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:57:03.442+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:57:04.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:57:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:57:03.961+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:57:04.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:57:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:57:03.961+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:57:04.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:57:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:57:03.961+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:57:04.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:57:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:57:03.961+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:57:04.660 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:57:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:57:04.305+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 49265 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:57:04.661 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:57:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:57:04.398+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:57:04.661 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:57:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:57:04.398+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:57:04.661 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:57:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:57:04.398+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:57:04.987 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:57:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:57:04.657+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:57:04.987 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:57:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:57:04.657+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:57:04.987 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:57:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:57:04.657+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:57:05.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:57:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:57:04.984+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:57:05.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:57:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:57:04.984+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:57:05.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:57:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:57:04.984+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:57:05.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:57:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:57:04.984+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:57:05.668 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:57:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:57:05.391+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:57:05.668 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:57:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:57:05.391+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:57:05.668 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:57:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:57:05.391+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:57:05.969 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:57:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:57:05.664+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:57:05.970 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:57:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:57:05.664+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:57:05.970 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:57:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:57:05.664+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:57:06.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:57:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:57:05.966+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:57:06.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:57:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:57:05.966+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:57:06.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:57:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:57:05.966+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:57:06.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:57:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:57:05.966+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:57:06.622 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:57:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:57:06.425+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:57:06.622 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:57:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:57:06.425+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:57:06.622 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:57:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:57:06.425+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:57:06.939 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:57:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:57:06.618+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:57:06.939 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:57:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:57:06.618+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:57:06.939 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:57:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:57:06.618+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:57:07.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:57:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:57:06.936+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:57:07.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:57:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:57:06.936+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:57:07.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:57:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:57:06.936+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:57:07.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:57:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:57:06.936+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:57:07.744 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:57:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:57:07.589+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:57:07.744 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:57:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:57:07.589+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:57:07.744 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:57:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:57:07.589+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:57:07.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:57:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:57:07.434+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:57:07.745 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:57:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:57:07.434+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:57:07.745 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:57:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:57:07.434+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:57:08.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:57:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:57:07.930+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:57:08.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:57:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:57:07.930+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:57:08.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:57:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:57:07.930+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:57:08.245 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:57:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:57:07.930+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:57:08.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:57:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:57:08.543+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:57:08.744 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:57:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:57:08.543+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:57:08.744 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:57:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:57:08.543+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:57:08.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:57:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:57:08.388+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:57:08.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:57:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:57:08.388+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:57:08.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:57:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:57:08.388+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:57:09.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:57:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:57:08.960+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:57:09.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:57:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:57:08.960+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:57:09.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:57:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:57:08.960+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:57:09.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:57:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:57:08.960+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:57:09.743 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:57:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:57:09.314+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 49338 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:57:09.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:57:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:57:09.341+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:57:09.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:57:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:57:09.341+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:57:09.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:57:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:57:09.341+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:57:09.744 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:57:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:57:09.514+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:57:09.744 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:57:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:57:09.514+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:57:09.744 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:57:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:57:09.514+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:57:10.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:57:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:57:09.950+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:57:10.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:57:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:57:09.950+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:57:10.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:57:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:57:09.950+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:57:10.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:57:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:57:09.950+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:57:10.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:57:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:57:10.532+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:57:10.744 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:57:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:57:10.532+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:57:10.744 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:57:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:57:10.532+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:57:10.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:57:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:57:10.327+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:57:10.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:57:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:57:10.327+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:57:10.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:57:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:57:10.327+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:57:11.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:57:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:57:10.924+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:57:11.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:57:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:57:10.924+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:57:11.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:57:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:57:10.924+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:57:11.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:57:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:57:10.924+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:57:11.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:57:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:57:11.490+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:57:11.744 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:57:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:57:11.490+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:57:11.744 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:57:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:57:11.490+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:57:11.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:57:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:57:11.327+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:57:11.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:57:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:57:11.327+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:57:11.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:57:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:57:11.327+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:57:12.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:57:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:57:11.923+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:57:12.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:57:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:57:11.923+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:57:12.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:57:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:57:11.923+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:57:12.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:57:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:57:11.924+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:57:12.698 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:57:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:57:12.497+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:57:12.698 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:57:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:57:12.497+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:57:12.698 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:57:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:57:12.497+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:57:12.698 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:57:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:57:12.279+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:57:12.698 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:57:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:57:12.279+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:57:12.698 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:57:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:57:12.279+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:57:12.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:57:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:57:12.894+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:57:12.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:57:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:57:12.894+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:57:12.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:57:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:57:12.894+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:57:12.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:57:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:57:12.894+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:57:13.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:57:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:57:13.499+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:57:13.744 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:57:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:57:13.499+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:57:13.744 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:57:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:57:13.499+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:57:13.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:57:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:57:13.256+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:57:13.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:57:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:57:13.256+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:57:13.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:57:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:57:13.256+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:57:14.233 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:57:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:57:13.862+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:57:14.234 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:57:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:57:13.862+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:57:14.234 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:57:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:57:13.862+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:57:14.234 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:57:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:57:13.862+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:57:14.493 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:57:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:57:14.321+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 49409 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:57:14.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:57:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:57:14.230+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:57:14.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:57:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:57:14.230+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:57:14.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:57:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:57:14.230+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:57:14.903 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:57:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:57:14.506+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:57:14.903 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:57:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:57:14.506+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:57:14.903 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:57:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:57:14.506+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:57:15.237 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:57:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:57:14.899+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:57:15.237 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:57:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:57:14.899+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:57:15.237 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:57:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:57:14.899+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:57:15.237 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:57:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:57:14.900+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:57:15.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:57:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:57:15.460+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:57:15.494 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:57:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:57:15.460+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:57:15.494 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:57:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:57:15.460+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:57:15.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:57:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:57:15.234+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:57:15.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:57:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:57:15.234+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:57:15.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:57:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:57:15.234+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:57:16.228 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:57:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:57:15.903+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:57:16.229 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:57:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:57:15.903+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:57:16.229 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:57:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:57:15.903+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:57:16.229 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:57:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:57:15.903+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:57:16.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:57:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:57:16.473+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:57:16.494 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:57:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:57:16.473+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:57:16.494 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:57:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:57:16.473+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:57:16.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:57:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:57:16.226+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:57:16.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:57:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:57:16.226+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:57:16.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:57:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:57:16.226+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:57:17.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:57:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:57:16.910+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:57:17.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:57:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:57:16.910+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:57:17.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:57:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:57:16.910+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:57:17.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:57:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:57:16.910+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:57:17.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:57:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:57:17.479+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:57:17.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:57:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:57:17.479+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:57:17.744 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:57:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:57:17.479+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:57:17.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:57:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:57:17.268+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:57:17.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:57:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:57:17.268+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:57:17.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:57:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:57:17.268+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:57:18.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:57:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:57:17.925+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:57:18.245 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:57:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:57:17.925+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:57:18.245 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:57:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:57:17.925+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:57:18.245 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:57:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:57:17.925+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:57:18.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:57:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:57:18.521+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:57:18.744 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:57:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:57:18.521+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:57:18.744 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:57:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:57:18.521+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:57:18.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:57:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:57:18.288+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:57:18.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:57:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:57:18.288+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:57:18.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:57:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:57:18.288+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:57:19.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:57:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:57:18.886+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:57:19.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:57:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:57:18.886+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:57:19.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:57:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:57:18.886+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:57:19.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:57:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:57:18.886+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:57:19.538 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:57:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:57:19.329+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 49483 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:57:19.539 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:57:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:57:19.280+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:57:19.539 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:57:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:57:19.280+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:57:19.539 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:57:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:57:19.280+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:57:19.865 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:57:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:57:19.536+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:57:19.866 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:57:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:57:19.536+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:57:19.866 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:57:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:57:19.536+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:57:20.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:57:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:57:19.863+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:57:20.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:57:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:57:19.863+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:57:20.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:57:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:57:19.863+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:57:20.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:57:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:57:19.863+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:57:20.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:57:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:57:20.495+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:57:20.744 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:57:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:57:20.495+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:57:20.744 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:57:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:57:20.495+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:57:20.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:57:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:57:20.297+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:57:20.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:57:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:57:20.297+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:57:20.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:57:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:57:20.297+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:57:21.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:57:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:57:20.833+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:57:21.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:57:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:57:20.833+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:57:21.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:57:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:57:20.833+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:57:21.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:57:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:57:20.833+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:57:21.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:57:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:57:21.452+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:57:21.744 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:57:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:57:21.452+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:57:21.744 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:57:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:57:21.452+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:57:21.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:57:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:57:21.328+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:57:21.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:57:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:57:21.328+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:57:21.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:57:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:57:21.328+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:57:22.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:57:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:57:21.792+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:57:22.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:57:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:57:21.792+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:57:22.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:57:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:57:21.792+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:57:22.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:57:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:57:21.793+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:57:22.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:57:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:57:22.478+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:57:22.744 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:57:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:57:22.478+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:57:22.744 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:57:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:57:22.478+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:57:22.747 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:57:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:57:22.369+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:57:22.747 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:57:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:57:22.369+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:57:22.747 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:57:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:57:22.369+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:57:23.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:57:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:57:22.814+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:57:23.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:57:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:57:22.814+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:57:23.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:57:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:57:22.814+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:57:23.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:57:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:57:22.814+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:57:23.744 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:57:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:57:23.441+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:57:23.744 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:57:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:57:23.441+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:57:23.744 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:57:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:57:23.441+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:57:23.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:57:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:57:23.381+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:57:23.745 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:57:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:57:23.381+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:57:23.745 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:57:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:57:23.381+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:57:24.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:57:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:57:23.780+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:57:24.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:57:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:57:23.780+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:57:24.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:57:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:57:23.780+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:57:24.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:57:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:57:23.780+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:57:24.744 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:57:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:57:24.336+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 49554 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:57:24.744 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:57:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:57:24.445+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:57:24.744 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:57:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:57:24.445+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:57:24.744 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:57:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:57:24.445+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:57:24.745 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:57:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:57:24.377+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:57:24.745 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:57:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:57:24.377+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:57:24.745 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:57:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:57:24.377+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:57:25.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:57:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:57:24.753+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:57:25.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:57:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:57:24.753+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:57:25.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:57:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:57:24.753+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:57:25.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:57:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:57:24.753+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:57:25.721 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:57:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:57:25.421+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:57:25.722 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:57:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:57:25.421+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:57:25.722 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:57:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:57:25.421+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:57:25.722 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:57:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:57:25.426+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:57:25.722 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:57:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:57:25.426+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:57:25.722 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:57:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:57:25.426+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:57:25.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:57:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:57:25.719+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:57:25.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:57:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:57:25.719+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:57:25.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:57:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:57:25.719+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:57:25.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:57:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:57:25.719+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:57:26.740 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:57:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:57:26.391+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:57:26.740 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:57:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:57:26.391+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:57:26.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:57:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:57:26.391+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:57:26.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:57:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:57:26.452+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:57:26.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:57:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:57:26.452+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:57:26.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:57:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:57:26.452+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:57:26.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:57:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:57:26.737+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:57:26.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:57:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:57:26.737+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:57:26.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:57:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:57:26.737+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:57:26.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:57:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:57:26.737+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:57:27.707 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:57:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:57:27.494+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:57:27.708 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:57:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:57:27.494+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:57:27.708 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:57:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:57:27.494+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:57:27.708 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:57:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:57:27.407+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:57:27.708 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:57:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:57:27.407+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:57:27.708 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:57:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:57:27.407+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:57:27.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:57:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:57:27.705+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:57:27.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:57:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:57:27.705+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:57:27.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:57:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:57:27.705+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:57:27.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:57:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:57:27.705+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:57:28.691 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:57:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:57:28.467+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:57:28.691 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:57:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:57:28.467+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:57:28.691 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:57:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:57:28.467+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:57:28.691 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:57:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:57:28.360+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:57:28.692 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:57:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:57:28.360+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:57:28.692 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:57:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:57:28.360+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:57:28.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:57:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:57:28.688+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:57:28.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:57:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:57:28.688+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:57:28.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:57:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:57:28.688+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:57:28.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:57:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:57:28.688+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:57:29.717 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:57:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:57:29.341+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 49627 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:57:29.718 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:57:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:57:29.462+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:57:29.718 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:57:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:57:29.462+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:57:29.718 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:57:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:57:29.462+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:57:29.718 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:57:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:57:29.355+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:57:29.719 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:57:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:57:29.355+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:57:29.719 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:57:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:57:29.355+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:57:29.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:57:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:57:29.714+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:57:29.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:57:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:57:29.714+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:57:29.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:57:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:57:29.714+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:57:29.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:57:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:57:29.714+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:57:30.683 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:57:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:57:30.453+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:57:30.683 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:57:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:57:30.453+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:57:30.684 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:57:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:57:30.453+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:57:30.684 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:57:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:57:30.341+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:57:30.684 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:57:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:57:30.341+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:57:30.684 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:57:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:57:30.341+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:57:30.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:57:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:57:30.679+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:57:30.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:57:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:57:30.679+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:57:30.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:57:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:57:30.679+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:57:30.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:57:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:57:30.679+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:57:31.420 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:57:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:57:31.343+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:57:31.421 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:57:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:57:31.343+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:57:31.421 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:57:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:57:31.343+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:57:31.725 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:57:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:57:31.417+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:57:31.726 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:57:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:57:31.417+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:57:31.726 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:57:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:57:31.417+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:57:31.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:57:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:57:31.722+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:57:31.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:57:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:57:31.722+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:57:31.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:57:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:57:31.722+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:57:31.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:57:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:57:31.722+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:57:32.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:57:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:57:32.310+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:57:32.744 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:57:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:57:32.310+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:57:32.744 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:57:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:57:32.310+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:57:32.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:57:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:57:32.388+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:57:32.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:57:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:57:32.388+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:57:32.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:57:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:57:32.388+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:57:33.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:57:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:57:32.770+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:57:33.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:57:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:57:32.770+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:57:33.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:57:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:57:32.770+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:57:33.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:57:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:57:32.770+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:57:33.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:57:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:57:33.317+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:57:33.744 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:57:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:57:33.317+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:57:33.744 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:57:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:57:33.317+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:57:33.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:57:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:57:33.364+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:57:33.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:57:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:57:33.364+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:57:33.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:57:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:57:33.364+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:57:34.202 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:57:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:57:33.797+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:57:34.202 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:57:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:57:33.797+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:57:34.202 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:57:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:57:33.797+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:57:34.202 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:57:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:57:33.797+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:57:34.473 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:57:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:57:34.349+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 49698 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:57:34.473 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:57:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:57:34.269+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:57:34.473 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:57:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:57:34.269+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:57:34.473 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:57:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:57:34.269+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:57:34.474 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:57:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:57:34.314+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:57:34.474 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:57:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:57:34.314+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:57:34.474 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:57:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:57:34.314+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:57:35.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:57:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:57:34.815+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:57:35.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:57:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:57:34.815+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:57:35.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:57:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:57:34.815+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:57:35.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:57:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:57:34.815+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:57:35.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:57:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:57:35.308+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:57:35.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:57:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:57:35.308+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:57:35.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:57:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:57:35.308+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:57:35.744 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:57:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:57:35.240+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:57:35.744 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:57:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:57:35.240+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:57:35.744 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:57:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:57:35.240+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:57:36.226 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:57:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:57:35.858+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:57:36.226 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:57:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:57:35.858+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:57:36.226 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:57:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:57:35.858+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:57:36.226 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:57:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:57:35.858+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:57:36.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:57:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:57:36.358+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:57:36.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:57:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:57:36.358+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:57:36.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:57:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:57:36.358+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:57:36.494 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:57:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:57:36.222+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:57:36.494 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:57:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:57:36.222+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:57:36.494 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:57:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:57:36.222+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:57:37.190 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:57:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:57:36.842+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:57:37.190 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:57:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:57:36.842+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:57:37.190 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:57:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:57:36.842+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:57:37.191 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:57:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:57:36.842+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:57:37.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:57:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:57:37.355+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:57:37.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:57:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:57:37.355+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:57:37.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:57:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:57:37.355+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:57:37.494 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:57:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:57:37.186+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:57:37.494 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:57:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:57:37.186+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:57:37.494 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:57:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:57:37.186+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:57:38.145 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:57:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:57:37.883+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:57:38.145 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:57:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:57:37.883+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:57:38.145 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:57:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:57:37.883+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:57:38.145 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:57:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:57:37.883+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:57:38.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:57:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:57:38.349+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:57:38.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:57:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:57:38.350+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:57:38.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:57:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:57:38.350+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:57:38.494 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:57:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:57:38.142+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:57:38.494 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:57:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:57:38.142+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:57:38.494 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:57:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:57:38.142+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:57:39.192 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:57:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:57:38.844+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:57:39.192 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:57:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:57:38.844+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:57:39.192 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:57:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:57:38.844+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:57:39.192 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:57:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:57:38.844+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:57:39.480 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:57:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:57:39.357+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 49771 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:57:39.481 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:57:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:57:39.188+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:57:39.481 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:57:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:57:39.188+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:57:39.481 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:57:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:57:39.188+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:57:39.481 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:57:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:57:39.303+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:57:39.481 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:57:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:57:39.303+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:57:39.481 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:57:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:57:39.303+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:57:40.195 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:57:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:57:39.886+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:57:40.195 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:57:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:57:39.886+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:57:40.195 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:57:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:57:39.886+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:57:40.195 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:57:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:57:39.886+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:57:40.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:57:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:57:40.192+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:57:40.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:57:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:57:40.192+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:57:40.494 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:57:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:57:40.192+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:57:40.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:57:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:57:40.302+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:57:40.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:57:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:57:40.302+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:57:40.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:57:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:57:40.302+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:57:41.205 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:57:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:57:40.887+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:57:41.205 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:57:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:57:40.887+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:57:41.206 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:57:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:57:40.887+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:57:41.206 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:57:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:57:40.887+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:57:41.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:57:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:57:41.301+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:57:41.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:57:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:57:41.301+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:57:41.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:57:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:57:41.301+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:57:41.494 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:57:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:57:41.201+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:57:41.494 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:57:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:57:41.201+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:57:41.494 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:57:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:57:41.201+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:57:42.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:57:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:57:41.906+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:57:42.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:57:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:57:41.906+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:57:42.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:57:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:57:41.906+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:57:42.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:57:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:57:41.906+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:57:42.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:57:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:57:42.277+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:57:42.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:57:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:57:42.277+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:57:42.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:57:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:57:42.277+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:57:42.744 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:57:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:57:42.243+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:57:42.744 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:57:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:57:42.243+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:57:42.744 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:57:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:57:42.243+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:57:43.214 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:57:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:57:42.925+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:57:43.215 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:57:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:57:42.925+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:57:43.215 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:57:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:57:42.925+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:57:43.215 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:57:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:57:42.925+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:57:43.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:57:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:57:43.211+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:57:43.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:57:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:57:43.211+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:57:43.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:57:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:57:43.211+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:57:43.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:57:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:57:43.325+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:57:43.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:57:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:57:43.325+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:57:43.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:57:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:57:43.325+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:57:44.209 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:57:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:57:43.887+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:57:44.210 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:57:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:57:43.887+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:57:44.210 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:57:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:57:43.887+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:57:44.210 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:57:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:57:43.887+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:57:44.489 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:57:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:57:44.365+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 49842 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:57:44.489 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:57:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:57:44.206+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:57:44.489 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:57:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:57:44.206+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:57:44.490 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:57:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:57:44.206+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:57:44.490 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:57:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:57:44.321+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:57:44.490 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:57:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:57:44.321+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:57:44.490 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:57:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:57:44.321+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:57:45.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:57:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:57:44.885+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:57:45.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:57:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:57:44.885+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:57:45.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:57:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:57:44.885+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:57:45.247 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:57:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:57:44.885+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:57:45.744 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:57:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:57:45.243+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:57:45.744 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:57:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:57:45.243+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:57:45.744 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:57:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:57:45.243+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:57:45.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:57:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:57:45.352+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:57:45.745 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:57:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:57:45.352+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:57:45.745 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:57:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:57:45.352+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:57:46.229 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:57:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:57:45.842+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:57:46.229 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:57:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:57:45.842+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:57:46.230 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:57:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:57:45.842+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:57:46.230 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:57:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:57:45.842+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:57:46.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:57:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:57:46.226+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:57:46.494 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:57:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:57:46.226+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:57:46.494 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:57:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:57:46.226+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:57:46.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:57:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:57:46.360+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:57:46.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:57:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:57:46.360+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:57:46.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:57:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:57:46.360+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:57:47.238 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:57:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:57:46.884+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:57:47.238 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:57:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:57:46.884+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:57:47.238 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:57:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:57:46.884+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:57:47.238 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:57:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:57:46.884+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:57:47.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:57:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:57:47.235+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:57:47.494 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:57:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:57:47.235+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:57:47.494 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:57:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:57:47.235+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:57:47.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:57:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:57:47.325+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:57:47.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:57:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:57:47.325+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:57:47.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:57:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:57:47.325+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:57:48.229 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:57:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:57:47.858+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:57:48.229 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:57:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:57:47.858+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:57:48.229 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:57:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:57:47.858+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:57:48.230 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:57:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:57:47.858+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:57:48.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:57:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:57:48.226+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:57:48.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:57:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:57:48.226+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:57:48.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:57:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:57:48.226+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:57:48.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:57:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:57:48.342+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:57:48.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:57:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:57:48.342+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:57:48.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:57:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:57:48.342+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:57:49.219 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:57:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:57:48.896+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:57:49.219 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:57:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:57:48.897+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:57:49.219 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:57:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:57:48.897+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:57:49.219 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:57:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:57:48.897+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:57:49.493 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:57:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:57:49.373+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 49915 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:57:49.494 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:57:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:57:49.216+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:57:49.494 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:57:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:57:49.216+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:57:49.494 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:57:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:57:49.216+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:57:49.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:57:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:57:49.349+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:57:49.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:57:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:57:49.349+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:57:49.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:57:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:57:49.349+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:57:50.228 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:57:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:57:49.925+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:57:50.229 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:57:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:57:49.925+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:57:50.229 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:57:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:57:49.925+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:57:50.229 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:57:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:57:49.925+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:57:50.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:57:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:57:50.225+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:57:50.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:57:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:57:50.225+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:57:50.494 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:57:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:57:50.225+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:57:50.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:57:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:57:50.359+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:57:50.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:57:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:57:50.359+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:57:50.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:57:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:57:50.359+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:57:51.224 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:57:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:57:50.921+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:57:51.224 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:57:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:57:50.921+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:57:51.225 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:57:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:57:50.921+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:57:51.225 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:57:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:57:50.921+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:57:51.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:57:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:57:51.221+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:57:51.494 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:57:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:57:51.221+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:57:51.494 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:57:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:57:51.221+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:57:51.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:57:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:57:51.314+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:57:51.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:57:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:57:51.314+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:57:51.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:57:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:57:51.314+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:57:52.234 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:57:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:57:51.887+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:57:52.234 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:57:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:57:51.887+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:57:52.234 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:57:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:57:51.887+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:57:52.235 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:57:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:57:51.887+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:57:52.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:57:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:57:52.231+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:57:52.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:57:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:57:52.231+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:57:52.494 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:57:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:57:52.231+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:57:52.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:57:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:57:52.302+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:57:52.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:57:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:57:52.302+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:57:52.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:57:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:57:52.302+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:57:52.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:57:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:57:52.863+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:57:52.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:57:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:57:52.863+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:57:52.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:57:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:57:52.863+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:57:52.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:57:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:57:52.863+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:57:53.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:57:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:57:53.226+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:57:53.494 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:57:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:57:53.226+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:57:53.494 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:57:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:57:53.226+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:57:53.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:57:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:57:53.302+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:57:53.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:57:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:57:53.302+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:57:53.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:57:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:57:53.302+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:57:54.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:57:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:57:53.902+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:57:54.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:57:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:57:53.902+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:57:54.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:57:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:57:53.902+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:57:54.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:57:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:57:53.902+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:57:54.505 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:57:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:57:54.380+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 49982 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:57:54.505 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:57:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:57:54.300+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:57:54.506 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:57:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:57:54.300+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:57:54.506 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:57:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:57:54.300+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:57:54.506 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:57:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:57:54.246+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:57:54.506 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:57:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:57:54.246+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:57:54.506 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:57:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:57:54.246+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:57:55.215 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:57:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:57:54.915+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:57:55.215 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:57:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:57:54.915+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:57:55.215 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:57:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:57:54.915+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:57:55.215 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:57:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:57:54.915+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:57:55.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:57:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:57:55.267+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:57:55.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:57:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:57:55.267+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:57:55.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:57:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:57:55.267+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:57:55.494 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:57:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:57:55.212+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:57:55.494 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:57:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:57:55.212+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:57:55.495 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:57:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:57:55.212+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:57:56.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:57:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:57:55.962+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:57:56.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:57:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:57:55.962+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:57:56.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:57:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:57:55.962+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:57:56.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:57:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:57:55.962+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:57:56.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:57:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:57:56.200+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:57:56.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:57:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:57:56.200+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:57:56.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:57:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:57:56.200+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:57:56.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:57:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:57:56.272+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:57:56.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:57:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:57:56.272+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:57:56.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:57:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:57:56.272+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:57:57.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:57:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:57:56.944+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:57:57.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:57:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:57:56.944+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:57:57.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:57:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:57:56.944+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:57:57.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:57:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:57:56.944+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:57:57.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:57:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:57:57.158+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:57:57.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:57:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:57:57.158+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:57:57.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:57:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:57:57.158+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:57:57.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:57:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:57:57.261+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:57:57.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:57:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:57:57.261+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:57:57.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:57:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:57:57.261+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:57:58.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:57:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:57:57.899+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:57:58.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:57:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:57:57.899+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:57:58.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:57:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:57:57.899+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:57:58.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:57:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:57:57.899+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:57:58.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:57:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:57:58.130+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:57:58.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:57:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:57:58.130+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:57:58.245 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:57:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:57:58.130+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:57:58.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:57:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:57:58.309+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:57:58.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:57:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:57:58.309+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:57:58.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:57:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:57:58.309+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:57:59.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:57:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:57:58.924+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:57:59.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:57:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:57:58.924+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:57:59.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:57:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:57:58.924+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:57:59.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:57:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:57:58.924+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:57:59.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:57:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:57:59.168+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:57:59.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:57:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:57:59.168+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:57:59.245 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:57:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:57:59.168+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:57:59.743 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:57:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:57:59.387+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 50055 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:57:59.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:57:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:57:59.310+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:57:59.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:57:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:57:59.310+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:57:59.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:57:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:57:59.310+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:58:00.174 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:57:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:57:59.938+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:58:00.175 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:57:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:57:59.938+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:58:00.175 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:57:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:57:59.938+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:58:00.175 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:57:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:57:59.938+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:58:00.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:58:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:58:00.279+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:58:00.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:58:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:58:00.279+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:58:00.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:58:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:58:00.279+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:58:00.494 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:58:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:58:00.172+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:58:00.494 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:58:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:58:00.172+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:58:00.494 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:58:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:58:00.172+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:58:01.218 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:58:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:58:00.945+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:58:01.218 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:58:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:58:00.945+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:58:01.219 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:58:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:58:00.945+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:58:01.219 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:58:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:58:00.945+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:58:01.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:58:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:58:01.323+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:58:01.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:58:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:58:01.323+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:58:01.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:58:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:58:01.323+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:58:01.494 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:58:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:58:01.215+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:58:01.494 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:58:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:58:01.215+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:58:01.494 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:58:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:58:01.215+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:58:02.186 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:58:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:58:01.992+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:58:02.186 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:58:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:58:01.992+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:58:02.186 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:58:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:58:01.992+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:58:02.186 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:58:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:58:01.992+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:58:02.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:58:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:58:02.301+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:58:02.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:58:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:58:02.301+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:58:02.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:58:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:58:02.301+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:58:02.494 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:58:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:58:02.183+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:58:02.494 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:58:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:58:02.183+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:58:02.494 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:58:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:58:02.183+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:58:03.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:58:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:58:02.976+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:58:03.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:58:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:58:02.976+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:58:03.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:58:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:58:02.976+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:58:03.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:58:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:58:02.976+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:58:03.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:58:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:58:03.136+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:58:03.245 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:58:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:58:03.136+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:58:03.245 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:58:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:58:03.136+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:58:03.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:58:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:58:03.253+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:58:03.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:58:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:58:03.253+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:58:03.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:58:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:58:03.253+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:58:04.277 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:58:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:58:04.021+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:58:04.277 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:58:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:58:04.021+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:58:04.277 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:58:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:58:04.021+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:58:04.277 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:58:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:58:04.021+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:58:04.277 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:58:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:58:04.094+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:58:04.278 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:58:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:58:04.094+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:58:04.278 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:58:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:58:04.094+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:58:04.743 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:58:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:58:04.396+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 50126 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:58:04.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:58:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:58:04.274+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:58:04.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:58:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:58:04.274+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:58:04.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:58:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:58:04.274+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:58:05.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:58:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:58:05.016+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:58:05.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:58:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:58:05.016+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:58:05.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:58:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:58:05.016+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:58:05.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:58:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:58:05.016+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:58:05.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:58:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:58:05.071+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:58:05.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:58:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:58:05.071+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:58:05.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:58:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:58:05.071+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:58:05.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:58:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:58:05.247+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:58:05.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:58:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:58:05.247+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:58:05.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:58:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:58:05.247+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:58:06.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:58:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:58:06.117+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:58:06.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:58:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:58:06.117+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:58:06.494 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:58:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:58:06.117+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:58:06.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:58:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:58:06.034+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:58:06.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:58:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:58:06.034+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:58:06.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:58:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:58:06.034+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:58:06.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:58:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:58:06.034+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:58:06.495 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:58:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:58:06.264+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:58:06.495 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:58:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:58:06.264+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:58:06.495 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:58:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:58:06.264+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:58:07.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:58:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:58:07.074+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:58:07.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:58:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:58:07.074+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:58:07.494 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:58:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:58:07.074+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:58:07.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:58:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:58:07.050+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:58:07.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:58:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:58:07.050+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:58:07.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:58:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:58:07.050+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:58:07.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:58:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:58:07.051+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:58:07.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:58:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:58:07.276+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:58:07.495 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:58:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:58:07.276+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:58:07.495 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:58:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:58:07.276+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:58:08.280 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:58:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:58:08.001+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:58:08.281 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:58:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:58:08.001+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:58:08.281 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:58:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:58:08.001+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:58:08.281 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:58:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:58:08.001+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:58:08.281 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:58:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:58:08.085+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:58:08.281 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:58:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:58:08.085+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:58:08.281 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:58:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:58:08.085+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:58:08.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:58:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:58:08.278+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:58:08.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:58:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:58:08.278+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:58:08.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:58:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:58:08.278+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:58:09.318 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:58:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:58:09.046+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:58:09.318 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:58:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:58:09.046+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:58:09.318 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:58:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:58:09.046+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:58:09.319 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:58:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:58:08.996+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:58:09.319 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:58:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:58:08.996+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:58:09.319 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:58:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:58:08.996+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:58:09.319 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:58:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:58:08.996+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:58:09.743 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:58:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:58:09.403+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 50199 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:58:09.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:58:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:58:09.315+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:58:09.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:58:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:58:09.315+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:58:09.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:58:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:58:09.315+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:58:10.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:58:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:58:09.958+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:58:10.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:58:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:58:09.958+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:58:10.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:58:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:58:09.958+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:58:10.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:58:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:58:09.959+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:58:10.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:58:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:58:10.094+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:58:10.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:58:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:58:10.094+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:58:10.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:58:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:58:10.094+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:58:10.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:58:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:58:10.335+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:58:10.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:58:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:58:10.335+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:58:10.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:58:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:58:10.335+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:58:11.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:58:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:58:10.943+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:58:11.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:58:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:58:10.943+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:58:11.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:58:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:58:10.943+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:58:11.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:58:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:58:10.943+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:58:11.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:58:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:58:11.098+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:58:11.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:58:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:58:11.098+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:58:11.245 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:58:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:58:11.098+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:58:11.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:58:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:58:11.290+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:58:11.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:58:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:58:11.290+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:58:11.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:58:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:58:11.290+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:58:12.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:58:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:58:12.106+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:58:12.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:58:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:58:12.106+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:58:12.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:58:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:58:12.106+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:58:12.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:58:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:58:11.943+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:58:12.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:58:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:58:11.943+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:58:12.245 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:58:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:58:11.943+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:58:12.245 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:58:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:58:11.944+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:58:12.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:58:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:58:12.330+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:58:12.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:58:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:58:12.330+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:58:12.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:58:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:58:12.330+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:58:13.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:58:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:58:13.090+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:58:13.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:58:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:58:13.090+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:58:13.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:58:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:58:13.090+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:58:13.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:58:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:58:12.949+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:58:13.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:58:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:58:12.949+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:58:13.245 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:58:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:58:12.949+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:58:13.245 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:58:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:58:12.949+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:58:13.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:58:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:58:13.311+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:58:13.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:58:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:58:13.311+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:58:13.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:58:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:58:13.311+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:58:14.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:58:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:58:14.085+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:58:14.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:58:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:58:14.085+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:58:14.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:58:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:58:14.085+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:58:14.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:58:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:58:13.930+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:58:14.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:58:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:58:13.930+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:58:14.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:58:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:58:13.930+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:58:14.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:58:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:58:13.930+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:58:14.743 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:58:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:58:14.411+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 50270 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:58:14.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:58:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:58:14.312+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:58:14.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:58:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:58:14.312+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:58:14.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:58:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:58:14.312+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:58:15.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:58:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:58:15.050+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:58:15.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:58:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:58:15.050+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:58:15.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:58:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:58:15.050+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:58:15.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:58:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:58:14.897+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:58:15.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:58:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:58:14.897+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:58:15.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:58:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:58:14.897+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:58:15.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:58:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:58:14.898+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:58:15.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:58:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:58:15.358+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:58:15.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:58:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:58:15.358+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:58:15.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:58:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:58:15.358+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:58:16.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:58:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:58:16.079+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:58:16.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:58:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:58:16.079+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:58:16.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:58:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:58:16.079+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:58:16.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:58:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:58:15.848+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:58:16.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:58:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:58:15.848+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:58:16.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:58:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:58:15.848+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:58:16.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:58:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:58:15.848+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:58:16.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:58:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:58:16.389+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:58:16.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:58:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:58:16.389+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:58:16.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:58:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:58:16.389+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:58:17.093 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:58:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:58:16.840+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:58:17.093 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:58:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:58:16.840+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:58:17.093 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:58:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:58:16.840+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:58:17.093 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:58:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:58:16.840+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:58:17.359 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:58:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:58:17.090+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:58:17.360 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:58:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:58:17.090+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:58:17.360 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:58:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:58:17.090+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:58:17.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:58:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:58:17.357+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:58:17.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:58:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:58:17.357+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:58:17.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:58:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:58:17.357+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:58:18.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:58:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:58:18.047+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:58:18.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:58:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:58:18.047+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:58:18.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:58:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:58:18.047+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:58:18.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:58:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:58:17.813+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:58:18.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:58:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:58:17.813+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:58:18.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:58:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:58:17.813+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:58:18.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:58:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:58:17.813+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:58:18.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:58:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:58:18.317+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:58:18.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:58:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:58:18.317+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:58:18.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:58:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:58:18.317+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:58:19.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:58:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:58:19.034+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:58:19.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:58:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:58:19.034+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:58:19.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:58:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:58:19.034+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:58:19.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:58:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:58:18.861+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:58:19.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:58:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:58:18.861+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:58:19.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:58:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:58:18.861+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:58:19.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:58:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:58:18.861+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:58:19.743 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:58:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:58:19.419+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 50343 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:58:19.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:58:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:58:19.350+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:58:19.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:58:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:58:19.350+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:58:19.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:58:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:58:19.350+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:58:20.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:58:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:58:20.020+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:58:20.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:58:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:58:20.020+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:58:20.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:58:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:58:20.020+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:58:20.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:58:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:58:19.867+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:58:20.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:58:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:58:19.867+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:58:20.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:58:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:58:19.867+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:58:20.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:58:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:58:19.868+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:58:20.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:58:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:58:20.311+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:58:20.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:58:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:58:20.311+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:58:20.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:58:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:58:20.311+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:58:21.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:58:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:58:20.998+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:58:21.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:58:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:58:20.998+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:58:21.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:58:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:58:20.998+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:58:21.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:58:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:58:20.847+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:58:21.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:58:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:58:20.847+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:58:21.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:58:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:58:20.847+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:58:21.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:58:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:58:20.847+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:58:21.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:58:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:58:21.317+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:58:21.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:58:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:58:21.317+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:58:21.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:58:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:58:21.317+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:58:22.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:58:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:58:22.035+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:58:22.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:58:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:58:22.035+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:58:22.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:58:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:58:22.035+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:58:22.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:58:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:58:21.871+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:58:22.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:58:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:58:21.871+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:58:22.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:58:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:58:21.871+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:58:22.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:58:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:58:21.871+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:58:22.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:58:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:58:22.353+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:58:22.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:58:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:58:22.353+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:58:22.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:58:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:58:22.353+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:58:23.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:58:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:58:23.068+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:58:23.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:58:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:58:23.068+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:58:23.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:58:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:58:23.068+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:58:23.245 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:58:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:58:22.842+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:58:23.245 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:58:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:58:22.842+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:58:23.245 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:58:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:58:22.842+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:58:23.245 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:58:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:58:22.842+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:58:23.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:58:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:58:23.355+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:58:23.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:58:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:58:23.355+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:58:23.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:58:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:58:23.355+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:58:24.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:58:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:58:24.072+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:58:24.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:58:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:58:24.072+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:58:24.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:58:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:58:24.072+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:58:24.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:58:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:58:23.882+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:58:24.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:58:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:58:23.882+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:58:24.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:58:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:58:23.882+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:58:24.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:58:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:58:23.882+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:58:24.743 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:58:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:58:24.426+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 50414 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:58:24.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:58:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:58:24.393+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:58:24.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:58:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:58:24.393+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:58:24.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:58:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:58:24.393+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:58:25.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:58:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:58:25.069+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:58:25.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:58:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:58:25.069+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:58:25.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:58:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:58:25.069+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:58:25.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:58:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:58:24.900+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:58:25.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:58:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:58:24.900+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:58:25.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:58:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:58:24.900+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:58:25.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:58:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:58:24.900+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:58:25.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:58:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:58:25.443+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:58:25.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:58:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:58:25.443+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:58:25.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:58:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:58:25.443+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:58:26.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:58:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:58:26.099+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:58:26.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:58:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:58:26.099+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:58:26.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:58:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:58:26.099+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:58:26.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:58:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:58:25.853+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:58:26.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:58:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:58:25.853+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:58:26.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:58:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:58:25.853+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:58:26.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:58:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:58:25.853+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:58:26.864 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:58:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:58:26.493+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:58:26.865 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:58:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:58:26.493+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:58:26.865 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:58:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:58:26.493+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:58:27.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:58:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:58:27.102+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:58:27.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:58:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:58:27.102+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:58:27.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:58:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:58:27.102+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:58:27.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:58:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:58:26.861+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:58:27.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:58:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:58:26.861+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:58:27.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:58:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:58:26.861+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:58:27.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:58:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:58:26.861+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:58:27.877 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:58:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:58:27.515+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:58:27.877 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:58:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:58:27.515+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:58:27.877 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:58:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:58:27.515+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:58:28.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:58:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:58:28.058+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:58:28.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:58:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:58:28.058+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:58:28.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:58:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:58:28.058+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:58:28.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:58:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:58:27.873+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:58:28.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:58:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:58:27.873+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:58:28.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:58:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:58:27.873+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:58:28.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:58:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:58:27.873+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:58:28.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:58:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:58:28.477+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:58:28.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:58:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:58:28.477+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:58:28.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:58:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:58:28.477+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:58:29.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:58:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:58:29.106+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:58:29.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:58:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:58:29.106+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:58:29.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:58:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:58:29.106+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:58:29.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:58:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:58:28.866+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:58:29.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:58:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:58:28.866+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:58:29.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:58:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:58:28.866+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:58:29.245 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:58:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:58:28.866+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:58:29.743 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:58:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:58:29.434+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 50487 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:58:29.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:58:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:58:29.444+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:58:29.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:58:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:58:29.444+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:58:29.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:58:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:58:29.444+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:58:30.149 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:58:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:58:29.888+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:58:30.149 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:58:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:58:29.888+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:58:30.149 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:58:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:58:29.888+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:58:30.149 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:58:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:58:29.888+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:58:30.471 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:58:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:58:30.145+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:58:30.471 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:58:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:58:30.145+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:58:30.471 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:58:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:58:30.145+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:58:30.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:58:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:58:30.467+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:58:30.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:58:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:58:30.467+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:58:30.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:58:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:58:30.467+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:58:31.175 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:58:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:58:30.913+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:58:31.175 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:58:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:58:30.913+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:58:31.175 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:58:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:58:30.913+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:58:31.176 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:58:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:58:30.913+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:58:31.433 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:58:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:58:31.172+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:58:31.433 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:58:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:58:31.172+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:58:31.433 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:58:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:58:31.172+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:58:31.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:58:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:58:31.430+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:58:31.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:58:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:58:31.430+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:58:31.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:58:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:58:31.430+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:58:32.225 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:58:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:58:31.930+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:58:32.225 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:58:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:58:31.930+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:58:32.225 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:58:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:58:31.930+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:58:32.225 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:58:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:58:31.930+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:58:32.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:58:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:58:32.405+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:58:32.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:58:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:58:32.405+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:58:32.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:58:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:58:32.405+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:58:32.494 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:58:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:58:32.221+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:58:32.494 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:58:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:58:32.221+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:58:32.494 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:58:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:58:32.221+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:58:33.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:58:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:58:33.172+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:58:33.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:58:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:58:33.172+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:58:33.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:58:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:58:33.172+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:58:33.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:58:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:58:32.980+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:58:33.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:58:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:58:32.980+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:58:33.245 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:58:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:58:32.980+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:58:33.245 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:58:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:58:32.980+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:58:33.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:58:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:58:33.369+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:58:33.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:58:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:58:33.369+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:58:33.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:58:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:58:33.369+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:58:34.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:58:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:58:34.193+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:58:34.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:58:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:58:34.193+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:58:34.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:58:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:58:34.193+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:58:34.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:58:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:58:33.943+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:58:34.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:58:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:58:33.943+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:58:34.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:58:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:58:33.943+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:58:34.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:58:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:58:33.943+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:58:34.743 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:58:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:58:34.441+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 50558 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:58:34.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:58:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:58:34.371+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:58:34.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:58:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:58:34.371+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:58:34.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:58:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:58:34.371+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:58:35.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:58:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:58:35.154+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:58:35.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:58:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:58:35.154+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:58:35.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:58:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:58:35.154+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:58:35.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:58:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:58:34.967+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:58:35.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:58:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:58:34.967+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:58:35.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:58:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:58:34.967+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:58:35.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:58:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:58:34.967+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:58:35.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:58:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:58:35.326+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:58:35.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:58:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:58:35.326+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:58:35.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:58:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:58:35.326+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:58:36.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:58:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:58:36.181+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:58:36.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:58:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:58:36.181+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:58:36.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:58:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:58:36.181+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:58:36.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:58:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:58:35.947+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:58:36.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:58:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:58:35.947+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:58:36.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:58:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:58:35.947+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:58:36.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:58:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:58:35.947+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:58:36.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:58:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:58:36.352+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:58:36.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:58:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:58:36.352+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:58:36.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:58:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:58:36.352+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:58:37.205 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:58:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:58:36.939+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:58:37.206 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:58:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:58:36.939+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:58:37.206 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:58:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:58:36.939+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:58:37.206 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:58:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:58:36.939+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:58:37.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:58:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:58:37.202+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:58:37.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:58:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:58:37.202+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:58:37.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:58:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:58:37.202+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:58:37.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:58:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:58:37.400+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:58:37.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:58:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:58:37.400+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:58:37.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:58:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:58:37.400+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:58:38.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:58:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:58:38.211+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:58:38.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:58:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:58:38.211+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:58:38.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:58:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:58:38.211+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:58:38.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:58:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:58:37.988+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:58:38.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:58:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:58:37.988+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:58:38.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:58:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:58:37.988+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:58:38.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:58:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:58:37.988+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:58:38.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:58:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:58:38.405+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:58:38.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:58:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:58:38.405+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:58:38.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:58:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:58:38.405+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:58:39.384 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:58:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:58:39.223+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:58:39.384 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:58:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:58:39.223+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:58:39.384 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:58:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:58:39.223+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:58:39.384 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:58:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:58:39.017+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:58:39.384 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:58:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:58:39.017+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:58:39.385 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:58:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:58:39.017+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:58:39.385 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:58:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:58:39.017+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:58:39.743 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:58:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:58:39.449+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 50631 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:58:39.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:58:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:58:39.380+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:58:39.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:58:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:58:39.380+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:58:39.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:58:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:58:39.380+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:58:40.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:58:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:58:40.003+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:58:40.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:58:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:58:40.003+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:58:40.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:58:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:58:40.003+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:58:40.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:58:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:58:40.003+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:58:40.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:58:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:58:40.258+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:58:40.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:58:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:58:40.258+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:58:40.744 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:58:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:58:40.258+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:58:40.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:58:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:58:40.339+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:58:40.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:58:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:58:40.339+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:58:40.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:58:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:58:40.339+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:58:41.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:58:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:58:41.216+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:58:41.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:58:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:58:41.216+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:58:41.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:58:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:58:41.216+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:58:41.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:58:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:58:40.977+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:58:41.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:58:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:58:40.977+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:58:41.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:58:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:58:40.977+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:58:41.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:58:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:58:40.977+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:58:41.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:58:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:58:41.317+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:58:41.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:58:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:58:41.317+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:58:41.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:58:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:58:41.317+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:58:42.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:58:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:58:42.027+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:58:42.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:58:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:58:42.027+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:58:42.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:58:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:58:42.027+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:58:42.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:58:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:58:42.027+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:58:42.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:58:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:58:42.247+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:58:42.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:58:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:58:42.247+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:58:42.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:58:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:58:42.247+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:58:42.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:58:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:58:42.362+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:58:42.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:58:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:58:42.362+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:58:42.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:58:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:58:42.362+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:58:43.335 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:58:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:58:43.287+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:58:43.336 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:58:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:58:43.287+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:58:43.336 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:58:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:58:43.287+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:58:43.336 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:58:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:58:43.040+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:58:43.336 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:58:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:58:43.040+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:58:43.336 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:58:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:58:43.040+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:58:43.336 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:58:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:58:43.041+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:58:43.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:58:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:58:43.333+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:58:43.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:58:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:58:43.333+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:58:43.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:58:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:58:43.333+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:58:44.253 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:58:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:58:43.996+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:58:44.253 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:58:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:58:43.996+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:58:44.254 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:58:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:58:43.996+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:58:44.254 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:58:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:58:43.996+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:58:44.578 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:58:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:58:44.456+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 50702 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:58:44.579 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:58:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:58:44.310+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:58:44.579 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:58:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:58:44.310+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:58:44.579 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:58:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:58:44.310+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:58:44.579 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:58:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:58:44.250+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:58:44.580 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:58:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:58:44.250+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:58:44.580 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:58:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:58:44.250+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:58:45.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:58:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:58:44.977+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:58:45.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:58:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:58:44.977+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:58:45.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:58:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:58:44.977+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:58:45.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:58:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:58:44.977+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:58:45.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:58:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:58:45.272+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:58:45.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:58:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:58:45.272+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:58:45.744 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:58:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:58:45.272+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:58:45.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:58:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:58:45.347+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:58:45.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:58:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:58:45.347+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:58:45.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:58:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:58:45.347+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:58:46.239 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:58:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:58:45.963+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:58:46.239 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:58:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:58:45.963+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:58:46.240 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:58:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:58:45.963+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:58:46.240 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:58:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:58:45.963+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:58:46.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:58:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:58:46.236+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:58:46.494 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:58:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:58:46.236+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:58:46.494 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:58:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:58:46.236+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:58:46.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:58:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:58:46.389+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:58:46.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:58:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:58:46.389+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:58:46.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:58:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:58:46.389+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:58:47.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:58:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:58:46.982+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:58:47.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:58:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:58:46.982+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:58:47.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:58:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:58:46.982+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:58:47.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:58:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:58:46.982+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:58:47.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:58:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:58:47.259+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:58:47.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:58:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:58:47.259+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:58:47.744 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:58:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:58:47.259+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:58:47.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:58:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:58:47.356+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:58:47.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:58:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:58:47.356+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:58:47.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:58:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:58:47.356+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:58:48.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:58:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:58:47.954+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:58:48.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:58:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:58:47.954+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:58:48.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:58:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:58:47.954+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:58:48.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:58:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:58:47.954+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:58:48.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:58:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:58:48.403+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:58:48.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:58:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:58:48.403+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:58:48.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:58:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:58:48.403+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:58:48.744 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:58:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:58:48.259+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:58:48.744 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:58:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:58:48.259+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:58:48.744 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:58:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:58:48.259+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:58:49.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:58:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:58:48.985+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:58:49.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:58:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:58:48.985+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:58:49.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:58:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:58:48.985+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:58:49.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:58:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:58:48.985+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:58:49.587 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:58:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:58:49.464+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 50775 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:58:49.587 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:58:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:58:49.291+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:58:49.587 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:58:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:58:49.291+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:58:49.587 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:58:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:58:49.291+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:58:49.588 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:58:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:58:49.441+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:58:49.588 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:58:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:58:49.441+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:58:49.588 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:58:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:58:49.441+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:58:50.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:58:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:58:50.032+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:58:50.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:58:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:58:50.032+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:58:50.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:58:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:58:50.032+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:58:50.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:58:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:58:50.032+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:58:50.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:58:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:58:50.470+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:58:50.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:58:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:58:50.470+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:58:50.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:58:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:58:50.470+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:58:50.744 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:58:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:58:50.267+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:58:50.744 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:58:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:58:50.267+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:58:50.744 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:58:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:58:50.267+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:58:51.295 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:58:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:58:51.050+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:58:51.295 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:58:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:58:51.050+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:58:51.295 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:58:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:58:51.050+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:58:51.295 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:58:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:58:51.050+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:58:51.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:58:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:58:51.292+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:58:51.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:58:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:58:51.292+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:58:51.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:58:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:58:51.292+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:58:51.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:58:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:58:51.496+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:58:51.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:58:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:58:51.496+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:58:51.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:58:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:58:51.496+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:58:52.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:58:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:58:52.032+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:58:52.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:58:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:58:52.032+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:58:52.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:58:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:58:52.032+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:58:52.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:58:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:58:52.032+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:58:52.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:58:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:58:52.294+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:58:52.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:58:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:58:52.294+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:58:52.744 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:58:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:58:52.294+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:58:52.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:58:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:58:52.538+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:58:52.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:58:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:58:52.538+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:58:52.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:58:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:58:52.538+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:58:53.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:58:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:58:53.252+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:58:53.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:58:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:58:53.252+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:58:53.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:58:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:58:53.252+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:58:53.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:58:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:58:53.045+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:58:53.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:58:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:58:53.045+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:58:53.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:58:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:58:53.045+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:58:53.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:58:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:58:53.045+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:58:53.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:58:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:58:53.537+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:58:53.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:58:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:58:53.537+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:58:53.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:58:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:58:53.537+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:58:54.474 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:58:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:58:54.225+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:58:54.475 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:58:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:58:54.225+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:58:54.475 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:58:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:58:54.225+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:58:54.475 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:58:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:58:54.072+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:58:54.475 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:58:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:58:54.073+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:58:54.475 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:58:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:58:54.073+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:58:54.475 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:58:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:58:54.073+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:58:54.743 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:58:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:58:54.471+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 50846 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:58:54.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:58:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:58:54.565+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:58:54.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:58:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:58:54.565+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:58:54.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:58:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:58:54.565+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:58:55.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:58:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:58:55.228+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:58:55.494 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:58:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:58:55.228+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:58:55.494 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:58:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:58:55.228+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:58:55.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:58:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:58:55.116+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:58:55.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:58:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:58:55.116+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:58:55.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:58:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:58:55.116+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:58:55.495 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:58:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:58:55.116+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:58:55.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:58:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:58:55.559+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:58:55.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:58:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:58:55.559+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:58:55.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:58:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:58:55.559+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:58:56.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:58:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:58:56.270+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:58:56.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:58:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:58:56.270+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:58:56.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:58:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:58:56.271+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:58:56.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:58:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:58:56.150+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:58:56.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:58:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:58:56.150+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:58:56.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:58:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:58:56.150+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:58:56.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:58:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:58:56.150+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:58:56.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:58:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:58:56.568+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:58:56.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:58:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:58:56.568+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:58:56.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:58:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:58:56.568+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:58:57.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:58:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:58:57.227+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:58:57.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:58:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:58:57.227+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:58:57.494 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:58:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:58:57.227+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:58:57.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:58:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:58:57.125+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:58:57.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:58:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:58:57.125+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:58:57.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:58:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:58:57.125+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:58:57.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:58:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:58:57.126+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:58:57.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:58:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:58:57.606+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:58:57.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:58:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:58:57.606+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:58:57.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:58:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:58:57.606+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:58:58.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:58:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:58:58.238+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:58:58.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:58:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:58:58.238+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:58:58.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:58:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:58:58.238+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:58:58.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:58:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:58:58.098+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:58:58.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:58:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:58:58.098+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:58:58.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:58:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:58:58.098+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:58:58.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:58:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:58:58.098+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:58:58.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:58:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:58:58.618+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:58:58.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:58:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:58:58.618+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:58:58.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:58:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:58:58.618+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:58:59.482 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:58:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:58:59.277+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:58:59.482 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:58:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:58:59.277+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:58:59.482 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:58:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:58:59.277+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:58:59.482 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:58:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:58:59.053+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:58:59.483 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:58:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:58:59.053+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:58:59.483 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:58:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:58:59.053+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:58:59.483 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:58:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:58:59.053+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:58:59.743 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:58:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:58:59.479+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 50919 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:58:59.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:58:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:58:59.609+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:58:59.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:58:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:58:59.609+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:58:59.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:58:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:58:59.609+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:59:00.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:59:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:59:00.099+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:59:00.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:59:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:59:00.099+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:59:00.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:59:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:59:00.099+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:59:00.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:59:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:59:00.099+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:59:00.603 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:59:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:59:00.282+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:59:00.604 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:59:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:59:00.282+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:59:00.604 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:59:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:59:00.282+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:59:00.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:59:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:59:00.601+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:59:00.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:59:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:59:00.601+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:59:00.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:59:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:59:00.601+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:59:01.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:59:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:59:01.245+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:59:01.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:59:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:59:01.245+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:59:01.494 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:59:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:59:01.245+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:59:01.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:59:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:59:01.136+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:59:01.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:59:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:59:01.136+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:59:01.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:59:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:59:01.136+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:59:01.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:59:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:59:01.137+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:59:01.883 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:59:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:59:01.619+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:59:01.884 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:59:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:59:01.619+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:59:01.884 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:59:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:59:01.619+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:59:02.236 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:59:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:59:02.110+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:59:02.236 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:59:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:59:02.110+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:59:02.237 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:59:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:59:02.110+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:59:02.237 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:59:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:59:02.110+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:59:02.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:59:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:59:02.234+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:59:02.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:59:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:59:02.234+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:59:02.494 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:59:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:59:02.234+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:59:02.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:59:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:59:02.652+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:59:02.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:59:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:59:02.652+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:59:02.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:59:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:59:02.652+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:59:03.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:59:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:59:03.099+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:59:03.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:59:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:59:03.099+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:59:03.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:59:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:59:03.099+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:59:03.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:59:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:59:03.099+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:59:03.494 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:59:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:59:03.195+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:59:03.494 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:59:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:59:03.195+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:59:03.494 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:59:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:59:03.195+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:59:03.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:59:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:59:03.659+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:59:03.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:59:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:59:03.659+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:59:03.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:59:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:59:03.659+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:59:04.489 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:59:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:59:04.236+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:59:04.489 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:59:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:59:04.236+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:59:04.489 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:59:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:59:04.236+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:59:04.490 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:59:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:59:04.052+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:59:04.490 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:59:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:59:04.052+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:59:04.490 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:59:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:59:04.052+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:59:04.490 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:59:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:59:04.052+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:59:04.743 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:59:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:59:04.486+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 50990 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:59:04.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:59:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:59:04.651+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:59:04.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:59:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:59:04.651+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:59:04.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:59:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:59:04.651+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:59:05.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:59:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:59:05.072+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:59:05.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:59:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:59:05.072+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:59:05.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:59:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:59:05.072+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:59:05.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:59:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:59:05.072+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:59:05.494 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:59:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:59:05.234+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:59:05.494 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:59:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:59:05.234+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:59:05.494 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:59:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:59:05.234+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:59:05.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:59:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:59:05.609+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:59:05.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:59:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:59:05.609+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:59:05.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:59:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:59:05.609+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:59:06.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:59:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:59:06.064+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:59:06.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:59:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:59:06.064+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:59:06.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:59:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:59:06.064+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:59:06.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:59:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:59:06.064+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:59:06.494 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:59:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:59:06.254+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:59:06.494 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:59:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:59:06.254+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:59:06.494 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:59:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:59:06.254+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:59:06.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:59:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:59:06.636+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:59:06.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:59:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:59:06.636+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:59:06.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:59:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:59:06.636+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:59:07.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:59:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:59:07.096+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:59:07.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:59:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:59:07.096+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:59:07.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:59:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:59:07.096+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:59:07.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:59:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:59:07.096+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:59:07.494 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:59:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:59:07.221+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:59:07.494 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:59:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:59:07.221+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:59:07.494 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:59:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:59:07.221+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:59:07.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:59:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:59:07.658+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:59:07.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:59:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:59:07.658+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:59:07.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:59:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:59:07.658+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:59:08.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:59:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:59:08.081+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:59:08.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:59:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:59:08.081+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:59:08.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:59:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:59:08.081+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:59:08.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:59:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:59:08.081+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:59:08.494 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:59:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:59:08.229+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:59:08.494 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:59:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:59:08.229+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:59:08.494 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:59:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:59:08.229+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:59:08.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:59:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:59:08.621+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:59:08.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:59:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:59:08.621+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:59:08.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:59:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:59:08.621+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:59:09.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:59:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:59:09.183+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:59:09.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:59:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:59:09.183+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:59:09.494 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:59:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:59:09.183+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:59:09.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:59:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:59:09.100+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:59:09.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:59:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:59:09.100+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:59:09.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:59:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:59:09.100+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:59:09.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:59:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:59:09.100+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:59:09.887 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:59:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:59:09.494+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 51063 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:59:09.887 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:59:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:59:09.625+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:59:09.887 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:59:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:59:09.625+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:59:09.887 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:59:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:59:09.625+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:59:10.211 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:59:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:59:10.063+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:59:10.211 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:59:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:59:10.063+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:59:10.211 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:59:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:59:10.064+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:59:10.211 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:59:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:59:10.064+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:59:10.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:59:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:59:10.208+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:59:10.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:59:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:59:10.208+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:59:10.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:59:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:59:10.208+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:59:10.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:59:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:59:10.641+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:59:10.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:59:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:59:10.641+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:59:10.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:59:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:59:10.641+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:59:11.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:59:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:59:11.255+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:59:11.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:59:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:59:11.255+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:59:11.494 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:59:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:59:11.255+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:59:11.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:59:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:59:11.109+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:59:11.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:59:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:59:11.109+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:59:11.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:59:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:59:11.109+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:59:11.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:59:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:59:11.109+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:59:11.884 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:59:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:59:11.622+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:59:11.884 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:59:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:59:11.622+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:59:11.884 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:59:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:59:11.622+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:59:12.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:59:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:59:12.288+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:59:12.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:59:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:59:12.288+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:59:12.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:59:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:59:12.288+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:59:12.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:59:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:59:12.155+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:59:12.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:59:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:59:12.155+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:59:12.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:59:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:59:12.155+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:59:12.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:59:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:59:12.155+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:59:12.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:59:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:59:12.617+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:59:12.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:59:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:59:12.617+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:59:12.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:59:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:59:12.617+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:59:13.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:59:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:59:13.299+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:59:13.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:59:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:59:13.299+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:59:13.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:59:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:59:13.299+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:59:13.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:59:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:59:13.121+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:59:13.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:59:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:59:13.121+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:59:13.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:59:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:59:13.121+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:59:13.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:59:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:59:13.121+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:59:13.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:59:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:59:13.579+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:59:13.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:59:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:59:13.579+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:59:13.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:59:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:59:13.579+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:59:14.344 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:59:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:59:14.091+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:59:14.345 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:59:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:59:14.091+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:59:14.345 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:59:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:59:14.091+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:59:14.345 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:59:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:59:14.091+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:59:14.626 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:59:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:59:14.503+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 51134 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:59:14.626 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:59:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:59:14.589+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:59:14.627 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:59:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:59:14.589+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:59:14.627 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:59:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:59:14.589+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:59:14.627 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:59:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:59:14.342+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:59:14.627 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:59:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:59:14.342+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:59:14.627 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:59:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:59:14.342+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:59:15.381 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:59:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:59:15.075+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:59:15.382 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:59:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:59:15.075+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:59:15.382 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:59:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:59:15.075+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:59:15.382 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:59:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:59:15.075+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:59:15.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:59:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:59:15.621+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:59:15.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:59:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:59:15.621+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:59:15.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:59:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:59:15.621+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:59:15.744 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:59:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:59:15.379+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:59:15.744 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:59:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:59:15.379+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:59:15.744 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:59:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:59:15.379+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:59:16.392 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:59:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:59:16.026+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:59:16.392 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:59:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:59:16.026+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:59:16.393 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:59:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:59:16.026+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:59:16.393 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:59:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:59:16.026+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:59:16.661 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:59:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:59:16.389+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:59:16.661 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:59:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:59:16.390+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:59:16.661 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:59:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:59:16.390+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:59:16.990 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:59:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:59:16.659+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:59:16.990 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:59:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:59:16.659+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:59:16.990 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:59:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:59:16.659+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:59:17.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:59:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:59:16.987+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:59:17.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:59:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:59:16.987+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:59:17.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:59:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:59:16.987+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:59:17.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:59:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:59:16.987+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:59:17.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:59:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:59:17.661+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:59:17.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:59:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:59:17.661+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:59:17.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:59:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:59:17.661+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:59:17.744 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:59:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:59:17.429+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:59:17.744 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:59:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:59:17.429+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:59:17.744 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:59:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:59:17.429+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:59:18.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:59:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:59:17.956+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:59:18.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:59:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:59:17.956+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:59:18.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:59:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:59:17.956+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:59:18.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:59:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:59:17.956+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:59:18.703 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:59:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:59:18.393+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:59:18.703 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:59:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:59:18.393+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:59:18.703 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:59:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:59:18.393+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:59:18.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:59:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:59:18.699+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:59:18.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:59:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:59:18.699+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:59:18.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:59:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:59:18.699+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:59:18.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:59:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:59:18.931+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:59:18.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:59:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:59:18.931+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:59:18.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:59:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:59:18.931+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:59:18.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:59:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:59:18.931+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:59:19.633 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:59:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:59:19.509+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 51207 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:59:19.634 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:59:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:59:19.373+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:59:19.634 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:59:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:59:19.373+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:59:19.634 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:59:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:59:19.373+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:59:19.886 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:59:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:59:19.711+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:59:19.887 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:59:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:59:19.711+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:59:19.887 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:59:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:59:19.711+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:59:20.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:59:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:59:19.893+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:59:20.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:59:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:59:19.893+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:59:20.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:59:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:59:19.893+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:59:20.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:59:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:59:19.893+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:59:20.723 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:59:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:59:20.341+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:59:20.723 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:59:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:59:20.341+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:59:20.723 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:59:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:59:20.341+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:59:20.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:59:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:59:20.892+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:59:20.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:59:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:59:20.892+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:59:20.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:59:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:59:20.892+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:59:20.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:59:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:59:20.892+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:59:20.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:59:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:59:20.719+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:59:20.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:59:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:59:20.719+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:59:20.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:59:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:59:20.719+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:59:21.722 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:59:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:59:21.294+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:59:21.722 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:59:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:59:21.294+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:59:21.722 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:59:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:59:21.294+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:59:21.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:59:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:59:21.733+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:59:21.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:59:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:59:21.733+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:59:21.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:59:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:59:21.733+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:59:21.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:59:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:59:21.847+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:59:21.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:59:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:59:21.847+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:59:21.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:59:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:59:21.847+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:59:21.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:59:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:59:21.847+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:59:22.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:59:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:59:22.286+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:59:22.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:59:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:59:22.286+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:59:22.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:59:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:59:22.286+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:59:23.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:59:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:59:22.768+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:59:23.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:59:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:59:22.768+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:59:23.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:59:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:59:22.768+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:59:23.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:59:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:59:22.866+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:59:23.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:59:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:59:22.866+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:59:23.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:59:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:59:22.866+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:59:23.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:59:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:59:22.866+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:59:23.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:59:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:59:23.248+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:59:23.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:59:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:59:23.248+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:59:23.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:59:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:59:23.248+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:59:24.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:59:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:59:23.795+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:59:24.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:59:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:59:23.795+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:59:24.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:59:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:59:23.795+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:59:24.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:59:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:59:23.862+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:59:24.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:59:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:59:23.862+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:59:24.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:59:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:59:23.862+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:59:24.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:59:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:59:23.862+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:59:24.520 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:59:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:59:24.245+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:59:24.520 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:59:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:59:24.245+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:59:24.520 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:59:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:59:24.245+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:59:24.793 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:59:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:59:24.517+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 51278 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:59:25.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:59:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:59:24.823+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:59:25.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:59:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:59:24.823+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:59:25.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:59:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:59:24.823+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:59:25.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:59:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:59:24.823+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:59:25.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:59:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:59:24.799+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:59:25.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:59:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:59:24.799+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:59:25.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:59:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:59:24.799+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:59:25.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:59:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:59:25.242+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:59:25.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:59:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:59:25.242+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:59:25.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:59:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:59:25.242+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:59:26.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:59:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:59:25.825+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:59:26.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:59:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:59:25.825+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:59:26.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:59:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:59:25.825+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:59:26.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:59:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:59:25.825+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:59:26.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:59:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:59:25.825+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:59:26.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:59:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:59:25.825+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:59:26.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:59:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:59:25.825+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:59:26.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:59:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:59:26.289+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:59:26.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:59:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:59:26.289+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:59:26.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:59:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:59:26.289+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:59:27.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:59:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:59:26.845+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:59:27.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:59:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:59:26.845+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:59:27.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:59:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:59:26.845+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:59:27.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:59:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:59:26.845+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:59:27.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:59:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:59:26.806+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:59:27.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:59:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:59:26.806+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:59:27.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:59:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:59:26.806+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:59:27.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:59:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:59:27.279+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:59:27.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:59:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:59:27.279+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:59:27.744 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:59:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:59:27.279+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:59:28.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:59:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:59:27.802+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:59:28.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:59:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:59:27.802+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:59:28.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:59:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:59:27.802+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:59:28.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:59:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:59:27.802+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:59:28.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:59:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:59:27.783+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:59:28.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:59:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:59:27.783+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:59:28.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:59:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:59:27.783+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:59:28.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:59:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:59:28.326+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:59:28.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:59:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:59:28.326+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:59:28.744 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:59:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:59:28.326+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:59:29.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:59:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:59:28.772+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:59:29.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:59:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:59:28.772+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:59:29.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:59:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:59:28.772+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:59:29.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:59:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:59:28.772+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:59:29.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:59:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:59:28.827+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:59:29.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:59:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:59:28.827+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:59:29.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:59:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:59:28.827+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:59:29.549 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:59:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:59:29.524+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 51351 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:59:29.549 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:59:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:59:29.284+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:59:29.549 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:59:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:59:29.284+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:59:29.550 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:59:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:59:29.284+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:59:29.818 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:59:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:59:29.739+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:59:29.818 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:59:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:59:29.739+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:59:29.818 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:59:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:59:29.739+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:59:29.819 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:59:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:59:29.740+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:59:30.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:59:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:59:29.814+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:59:30.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:59:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:59:29.814+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:59:30.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:59:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:59:29.814+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:59:30.703 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:59:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:59:30.255+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:59:30.704 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:59:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:59:30.255+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:59:30.704 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:59:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:59:30.255+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:59:30.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:59:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:59:30.784+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:59:30.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:59:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:59:30.784+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:59:30.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:59:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:59:30.784+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:59:30.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:59:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:59:30.700+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:59:30.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:59:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:59:30.700+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:59:30.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:59:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:59:30.700+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:59:30.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:59:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:59:30.700+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:59:31.661 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:59:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:59:31.301+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:59:31.661 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:59:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:59:31.301+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:59:31.661 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:59:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:59:31.301+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:59:31.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:59:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:59:31.658+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:59:31.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:59:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:59:31.658+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:59:31.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:59:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:59:31.658+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:59:31.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:59:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:59:31.658+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:59:31.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:59:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:59:31.744+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:59:31.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:59:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:59:31.744+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:59:31.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:59:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:59:31.744+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:59:32.662 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:59:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:59:32.288+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:59:32.662 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:59:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:59:32.288+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:59:32.662 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:59:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:59:32.288+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:59:32.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:59:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:59:32.771+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:59:32.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:59:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:59:32.771+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:59:32.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:59:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:59:32.771+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:59:32.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:59:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:59:32.659+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:59:32.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:59:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:59:32.659+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:59:32.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:59:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:59:32.659+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:59:32.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:59:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:59:32.659+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:59:33.627 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:59:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:59:33.281+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:59:33.628 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:59:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:59:33.281+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:59:33.628 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:59:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:59:33.281+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:59:33.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:59:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:59:33.765+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:59:33.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:59:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:59:33.765+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:59:33.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:59:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:59:33.765+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:59:33.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:59:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:59:33.625+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:59:33.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:59:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:59:33.625+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:59:33.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:59:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:59:33.625+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:59:33.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:59:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:59:33.625+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:59:34.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:59:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:59:34.244+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:59:34.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:59:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:59:34.244+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:59:34.494 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:59:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:59:34.244+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:59:34.993 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:59:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:59:34.531+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 51423 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:59:34.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:59:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:59:34.781+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:59:34.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:59:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:59:34.781+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:59:34.995 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:59:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:59:34.781+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:59:34.995 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:59:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:59:34.631+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:59:34.995 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:59:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:59:34.631+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:59:34.996 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:59:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:59:34.631+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:59:34.996 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:59:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:59:34.631+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:59:35.651 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:59:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:59:35.267+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:59:35.651 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:59:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:59:35.267+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:59:35.651 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:59:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:59:35.267+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:59:35.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:59:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:59:35.785+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:59:35.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:59:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:59:35.785+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:59:35.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:59:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:59:35.785+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:59:35.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:59:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:59:35.648+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:59:35.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:59:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:59:35.648+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:59:35.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:59:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:59:35.648+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:59:35.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:59:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:59:35.648+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:59:36.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:59:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:59:36.236+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:59:36.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:59:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:59:36.236+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:59:36.494 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:59:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:59:36.236+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:59:36.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:59:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:59:36.737+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:59:36.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:59:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:59:36.737+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:59:36.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:59:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:59:36.737+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:59:36.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:59:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:59:36.612+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:59:36.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:59:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:59:36.612+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:59:36.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:59:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:59:36.612+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:59:36.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:59:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:59:36.612+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:59:37.624 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:59:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:59:37.279+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:59:37.624 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:59:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:59:37.279+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:59:37.624 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:59:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:59:37.279+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:59:37.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:59:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:59:37.696+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:59:37.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:59:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:59:37.696+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:59:37.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:59:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:59:37.696+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:59:37.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:59:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:59:37.620+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:59:37.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:59:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:59:37.620+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:59:37.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:59:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:59:37.620+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:59:37.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:59:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:59:37.621+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:59:38.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:59:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:59:38.239+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:59:38.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:59:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:59:38.239+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:59:38.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:59:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:59:38.239+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:59:38.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:59:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:59:38.727+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:59:38.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:59:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:59:38.727+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:59:38.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:59:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:59:38.727+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:59:38.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:59:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:59:38.667+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:59:38.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:59:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:59:38.667+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:59:38.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:59:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:59:38.667+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:59:38.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:59:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:59:38.667+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:59:39.542 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:59:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:59:39.252+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:59:39.542 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:59:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:59:39.252+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:59:39.542 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:59:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:59:39.252+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:59:39.887 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:59:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:59:39.538+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 51495 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:59:39.887 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:59:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:59:39.681+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:59:39.887 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:59:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:59:39.681+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:59:39.888 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:59:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:59:39.681+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:59:39.888 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:59:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:59:39.683+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:59:39.888 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:59:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:59:39.683+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:59:39.888 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:59:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:59:39.683+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:59:39.888 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:59:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:59:39.683+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:59:40.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:59:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:59:40.214+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:59:40.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:59:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:59:40.214+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:59:40.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:59:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:59:40.214+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:59:40.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:59:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:59:40.700+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:59:40.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:59:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:59:40.700+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:59:40.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:59:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:59:40.700+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:59:40.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:59:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:59:40.668+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:59:40.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:59:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:59:40.668+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:59:40.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:59:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:59:40.668+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:59:40.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:59:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:59:40.668+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:59:41.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:59:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:59:41.221+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:59:41.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:59:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:59:41.221+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:59:41.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:59:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:59:41.221+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:59:41.887 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:59:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:59:41.736+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:59:41.887 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:59:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:59:41.736+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:59:41.887 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:59:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:59:41.736+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:59:41.888 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:59:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:59:41.623+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:59:41.888 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:59:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:59:41.623+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:59:41.888 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:59:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:59:41.623+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:59:41.888 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:59:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:59:41.623+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:59:42.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:59:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:59:42.187+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:59:42.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:59:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:59:42.187+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:59:42.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:59:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:59:42.187+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:59:42.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:59:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:59:42.729+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:59:42.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:59:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:59:42.729+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:59:42.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:59:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:59:42.729+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:59:42.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:59:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:59:42.642+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:59:42.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:59:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:59:42.642+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:59:42.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:59:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:59:42.642+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:59:42.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:59:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:59:42.642+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:59:43.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:59:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:59:43.190+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:59:43.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:59:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:59:43.190+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:59:43.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:59:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:59:43.190+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:59:43.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:59:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:59:43.732+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:59:43.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:59:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:59:43.732+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:59:43.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:59:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:59:43.732+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:59:43.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:59:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:59:43.627+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:59:43.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:59:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:59:43.627+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:59:43.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:59:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:59:43.627+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:59:43.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:59:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:59:43.627+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:59:44.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:59:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:59:44.165+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:59:44.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:59:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:59:44.165+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:59:44.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:59:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:59:44.165+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:59:44.993 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:59:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:59:44.546+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 51567 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:59:44.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:59:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:59:44.706+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:59:44.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:59:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:59:44.706+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:59:44.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:59:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:59:44.706+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:59:44.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:59:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:59:44.583+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:59:44.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:59:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:59:44.583+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:59:44.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:59:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:59:44.583+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:59:44.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:59:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:59:44.583+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:59:45.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:59:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:59:45.182+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:59:45.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:59:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:59:45.182+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:59:45.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:59:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:59:45.182+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:59:45.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:59:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:59:45.664+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:59:45.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:59:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:59:45.664+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:59:45.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:59:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:59:45.664+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:59:45.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:59:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:59:45.571+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:59:45.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:59:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:59:45.571+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:59:45.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:59:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:59:45.571+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:59:45.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:59:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:59:45.571+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:59:46.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:59:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:59:46.210+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:59:46.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:59:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:59:46.210+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:59:46.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:59:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:59:46.210+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:59:46.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:59:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:59:46.668+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:59:46.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:59:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:59:46.668+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:59:46.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:59:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:59:46.668+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:59:46.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:59:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:59:46.621+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:59:46.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:59:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:59:46.621+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:59:46.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:59:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:59:46.621+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:59:46.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:59:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:59:46.621+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:59:47.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:59:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:59:47.214+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:59:47.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:59:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:59:47.214+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:59:47.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:59:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:59:47.214+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:59:47.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:59:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:59:47.636+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:59:47.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:59:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:59:47.636+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:59:47.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:59:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:59:47.636+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:59:47.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:59:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:59:47.636+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:59:47.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:59:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:59:47.663+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:59:47.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:59:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:59:47.663+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:59:47.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:59:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:59:47.663+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:59:48.664 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:59:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:59:48.250+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:59:48.664 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:59:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:59:48.250+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:59:48.665 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:59:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:59:48.250+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:59:48.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:59:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:59:48.679+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:59:48.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:59:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:59:48.679+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:59:48.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:59:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:59:48.679+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:59:48.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:59:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:59:48.679+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:59:48.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:59:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:59:48.661+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:59:48.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:59:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:59:48.661+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:59:48.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:59:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:59:48.661+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:59:49.557 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:59:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:59:49.242+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:59:49.557 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:59:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:59:49.242+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:59:49.557 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:59:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:59:49.242+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:59:49.886 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:59:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:59:49.554+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 51639 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:59:49.887 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:59:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:59:49.688+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:59:49.887 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:59:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:59:49.688+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:59:49.887 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:59:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:59:49.688+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:59:49.887 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:59:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:59:49.725+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:59:49.888 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:59:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:59:49.725+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:59:49.888 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:59:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:59:49.725+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:59:49.888 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:59:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:59:49.725+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:59:50.681 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:59:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:59:50.245+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:59:50.681 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:59:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:59:50.245+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:59:50.682 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:59:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:59:50.245+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:59:50.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:59:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:59:50.735+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:59:50.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:59:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:59:50.735+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:59:50.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:59:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:59:50.735+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:59:50.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:59:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:59:50.679+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:59:50.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:59:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:59:50.679+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:59:50.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:59:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:59:50.679+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:59:50.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:59:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:59:50.679+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:59:51.689 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:59:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:59:51.260+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:59:51.689 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:59:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:59:51.260+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:59:51.689 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:59:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:59:51.260+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:59:51.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:59:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:59:51.732+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:59:51.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:59:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:59:51.732+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:59:51.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:59:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:59:51.732+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:59:51.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:59:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:59:51.687+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:59:51.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:59:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:59:51.687+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:59:51.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:59:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:59:51.687+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:59:51.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:59:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:59:51.687+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:59:52.641 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:59:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:59:52.255+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:59:52.641 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:59:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:59:52.255+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:59:52.641 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:59:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:59:52.255+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:59:52.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:59:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:59:52.731+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:59:52.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:59:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:59:52.731+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:59:52.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:59:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:59:52.731+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:59:52.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:59:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:59:52.639+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:59:52.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:59:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:59:52.639+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:59:52.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:59:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:59:52.639+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:59:52.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:59:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:59:52.639+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:59:53.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:59:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:59:53.235+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:59:53.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:59:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:59:53.235+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:59:53.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:59:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:59:53.235+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:59:53.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:59:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:59:53.728+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:59:53.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:59:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:59:53.728+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:59:53.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:59:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:59:53.728+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:59:53.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:59:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:59:53.641+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:59:53.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:59:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:59:53.641+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:59:53.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:59:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:59:53.641+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:59:53.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:59:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:59:53.642+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:59:54.564 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:59:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:59:54.263+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:59:54.565 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:59:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:59:54.263+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:59:54.565 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:59:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:59:54.263+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:59:54.993 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:59:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:59:54.562+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 51711 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:59:54.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:59:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:59:54.595+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:59:54.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:59:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:59:54.595+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:59:54.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:59:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:59:54.595+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:59:54.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:59:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:59:54.595+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:59:54.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:59:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:59:54.760+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:59:54.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:59:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:59:54.760+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:59:54.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:59:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:59:54.760+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:59:55.563 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:59:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:59:55.277+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:59:55.564 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:59:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:59:55.277+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:59:55.564 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:59:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:59:55.277+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:59:55.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:59:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:59:55.780+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:59:55.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:59:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:59:55.780+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:59:55.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:59:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:59:55.780+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:59:55.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:59:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:59:55.561+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:59:55.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:59:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:59:55.561+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:59:55.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:59:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:59:55.561+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:59:55.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:59:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:59:55.561+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:59:56.601 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:59:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:59:56.261+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:59:56.601 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:59:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:59:56.261+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:59:56.601 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:59:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:59:56.261+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:59:56.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:59:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:59:56.746+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:59:56.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:59:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:59:56.746+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:59:56.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:59:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:59:56.746+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:59:56.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:59:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:59:56.598+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:59:56.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:59:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:59:56.598+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:59:56.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:59:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:59:56.598+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:59:56.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:59:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:59:56.598+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:59:57.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:59:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:59:57.221+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:59:57.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:59:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:59:57.221+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:59:57.494 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:59:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:59:57.221+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:59:57.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:59:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:59:57.793+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:59:57.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:59:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:59:57.793+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:59:57.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:59:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:59:57.793+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:59:57.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:59:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:59:57.568+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:59:57.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:59:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:59:57.568+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:59:57.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:59:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:59:57.568+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:59:57.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:59:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:59:57.568+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:59:58.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:59:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:59:58.198+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:59:58.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:59:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:59:58.198+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:59:58.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:59:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:59:58.198+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:59:58.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:59:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:59:58.829+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:59:58.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:59:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:59:58.829+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:59:58.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:59:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:59:58.829+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T08:59:58.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:59:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:59:58.602+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:59:58.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:59:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:59:58.602+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:59:58.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:59:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:59:58.602+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:59:58.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:59:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:59:58.602+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T08:59:59.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:59:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:59:59.168+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:59:59.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:59:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:59:59.168+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:59:59.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 08:59:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T08:59:59.168+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T08:59:59.861 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 08:59:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T08:59:59.569+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 51783 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T08:59:59.861 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:59:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:59:59.553+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:59:59.861 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:59:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:59:59.553+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:59:59.861 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:59:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:59:59.553+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T08:59:59.861 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 08:59:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T08:59:59.553+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:00:00.190 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:59:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:59:59.858+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T09:00:00.190 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:59:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:59:59.858+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T09:00:00.190 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 08:59:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T08:59:59.858+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T09:00:00.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:00:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:00:00.187+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T09:00:00.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:00:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:00:00.187+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T09:00:00.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:00:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:00:00.187+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T09:00:00.886 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:00:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:00:00.584+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T09:00:00.886 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:00:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:00:00.584+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T09:00:00.886 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:00:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:00:00.584+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T09:00:00.886 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:00:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:00:00.584+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:00:01.143 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:00:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:00:00.883+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T09:00:01.143 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:00:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:00:00.883+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T09:00:01.143 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:00:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:00:00.883+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T09:00:01.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:00:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:00:01.141+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T09:00:01.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:00:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:00:01.141+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T09:00:01.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:00:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:00:01.141+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:25.699562+0000 (oldest deadline 2024-02-20T08:49:45.699562+0000) 2024-02-20T09:00:01.858 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:00:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:00:01.596+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T09:00:01.858 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:00:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:00:01.596+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T09:00:01.858 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:00:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:00:01.596+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T09:00:01.858 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:00:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:00:01.596+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:00:02.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:00:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:00:01.855+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T09:00:02.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:00:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:00:01.855+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T09:00:02.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:00:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:00:01.855+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T09:00:02.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:00:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:00:02.807+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T09:00:02.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:00:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:00:02.807+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T09:00:02.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:00:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:00:02.807+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T09:00:02.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:00:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:00:02.644+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T09:00:02.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:00:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:00:02.644+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T09:00:02.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:00:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:00:02.644+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T09:00:02.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:00:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:00:02.644+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:00:03.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:00:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:00:03.795+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T09:00:03.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:00:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:00:03.795+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T09:00:03.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:00:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:00:03.795+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T09:00:03.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:00:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:00:03.661+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T09:00:03.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:00:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:00:03.661+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T09:00:03.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:00:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:00:03.661+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T09:00:03.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:00:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:00:03.661+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:00:04.993 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:00:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:00:04.577+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 51855 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:00:04.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:00:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:00:04.755+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T09:00:04.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:00:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:00:04.755+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T09:00:04.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:00:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:00:04.755+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T09:00:04.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:00:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:00:04.646+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T09:00:04.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:00:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:00:04.646+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T09:00:04.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:00:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:00:04.646+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T09:00:04.995 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:00:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:00:04.646+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:00:05.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:00:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:00:05.786+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T09:00:05.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:00:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:00:05.786+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T09:00:05.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:00:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:00:05.786+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T09:00:05.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:00:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:00:05.638+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T09:00:05.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:00:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:00:05.638+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T09:00:05.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:00:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:00:05.638+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T09:00:05.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:00:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:00:05.638+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:00:06.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:00:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:00:06.792+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T09:00:06.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:00:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:00:06.792+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T09:00:06.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:00:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:00:06.792+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T09:00:06.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:00:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:00:06.672+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T09:00:06.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:00:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:00:06.672+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T09:00:06.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:00:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:00:06.672+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T09:00:06.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:00:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:00:06.672+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:00:07.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:00:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:00:07.809+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T09:00:07.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:00:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:00:07.809+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T09:00:07.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:00:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:00:07.809+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T09:00:07.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:00:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:00:07.651+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T09:00:07.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:00:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:00:07.651+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T09:00:07.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:00:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:00:07.651+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T09:00:07.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:00:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:00:07.651+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:00:08.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:00:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:00:08.844+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T09:00:08.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:00:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:00:08.844+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T09:00:08.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:00:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:00:08.844+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T09:00:08.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:00:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:00:08.640+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T09:00:08.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:00:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:00:08.640+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T09:00:08.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:00:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:00:08.640+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T09:00:08.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:00:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:00:08.640+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:00:09.886 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:00:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:00:09.584+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 51927 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:00:09.886 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:00:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:00:09.625+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T09:00:09.886 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:00:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:00:09.625+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T09:00:09.887 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:00:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:00:09.625+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T09:00:09.887 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:00:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:00:09.626+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:00:10.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:00:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:00:09.890+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T09:00:10.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:00:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:00:09.890+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T09:00:10.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:00:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:00:09.890+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T09:00:10.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:00:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:00:10.606+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T09:00:10.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:00:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:00:10.606+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T09:00:10.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:00:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:00:10.606+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T09:00:10.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:00:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:00:10.606+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:00:11.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:00:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:00:10.842+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T09:00:11.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:00:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:00:10.842+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T09:00:11.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:00:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:00:10.842+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:40.215034+0000 (oldest deadline 2024-02-20T08:50:00.215034+0000) 2024-02-20T09:00:11.889 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:00:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:00:11.621+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T09:00:11.890 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:00:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:00:11.621+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T09:00:11.890 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:00:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:00:11.621+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T09:00:11.890 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:00:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:00:11.621+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:00:12.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:00:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:00:12.582+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T09:00:12.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:00:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:00:12.582+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T09:00:12.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:00:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:00:12.582+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T09:00:12.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:00:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:00:12.582+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:00:13.868 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:00:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:00:13.597+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T09:00:13.868 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:00:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:00:13.597+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T09:00:13.868 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:00:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:00:13.597+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T09:00:13.868 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:00:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:00:13.597+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:00:13.868 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:00:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:00:13.626+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:00:13.627897+0000) 2024-02-20T09:00:13.869 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:00:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:00:13.746+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:00:13.747701+0000) 2024-02-20T09:00:14.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:00:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:00:13.864+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:00:13.866016+0000) 2024-02-20T09:00:14.868 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:00:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:00:14.591+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 51999 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:00:14.868 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:00:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:00:14.585+0000 7f8a6006f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:00:14.587232+0000) 2024-02-20T09:00:14.868 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:00:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:00:14.746+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:00:14.747854+0000) 2024-02-20T09:00:14.869 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:00:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:00:14.585+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:00:14.586976+0000) 2024-02-20T09:00:14.869 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:00:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:00:14.865+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:00:14.866166+0000) 2024-02-20T09:00:14.869 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:00:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:00:14.588+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:00:14.590023+0000) 2024-02-20T09:00:14.869 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:00:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:00:14.622+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T09:00:14.869 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:00:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:00:14.622+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T09:00:14.869 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:00:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:00:14.622+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T09:00:14.870 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:00:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:00:14.622+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:00:14.870 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:00:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:00:14.626+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:00:14.628217+0000) 2024-02-20T09:00:15.871 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:00:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:00:15.747+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:00:15.748013+0000) 2024-02-20T09:00:15.871 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:00:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:00:15.604+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T09:00:15.872 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:00:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:00:15.604+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T09:00:15.872 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:00:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:00:15.604+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T09:00:15.872 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:00:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:00:15.604+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:00:15.872 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:00:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:00:15.629+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:00:15.630739+0000) 2024-02-20T09:00:16.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:00:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:00:15.864+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:00:15.866304+0000) 2024-02-20T09:00:16.868 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:00:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:00:16.747+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:00:16.748206+0000) 2024-02-20T09:00:16.869 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:00:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:00:16.565+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T09:00:16.869 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:00:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:00:16.565+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T09:00:16.869 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:00:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:00:16.565+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T09:00:16.869 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:00:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:00:16.565+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:00:16.869 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:00:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:00:16.632+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:00:16.633507+0000) 2024-02-20T09:00:17.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:00:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:00:16.865+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:00:16.866485+0000) 2024-02-20T09:00:17.868 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:00:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:00:17.747+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:00:17.748364+0000) 2024-02-20T09:00:17.868 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:00:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:00:17.568+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T09:00:17.869 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:00:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:00:17.568+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T09:00:17.869 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:00:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:00:17.568+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T09:00:17.869 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:00:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:00:17.568+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:00:17.869 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:00:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:00:17.634+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:00:17.635355+0000) 2024-02-20T09:00:18.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:00:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:00:17.865+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:00:17.866639+0000) 2024-02-20T09:00:18.868 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:00:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:00:18.747+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:00:18.748517+0000) 2024-02-20T09:00:18.869 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:00:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:00:18.579+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T09:00:18.869 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:00:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:00:18.579+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T09:00:18.869 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:00:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:00:18.579+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T09:00:18.869 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:00:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:00:18.579+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:00:18.869 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:00:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:00:18.634+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:00:18.635688+0000) 2024-02-20T09:00:19.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:00:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:00:18.865+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:00:18.866753+0000) 2024-02-20T09:00:19.868 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:00:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:00:19.600+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 52071 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:00:19.869 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:00:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:00:19.593+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:00:19.595173+0000) 2024-02-20T09:00:19.869 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:00:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:00:19.594+0000 7f8a6006f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:00:19.595539+0000) 2024-02-20T09:00:19.869 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:00:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:00:19.747+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:00:19.748700+0000) 2024-02-20T09:00:19.870 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:00:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:00:19.596+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:00:19.598249+0000) 2024-02-20T09:00:19.870 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:00:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:00:19.626+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T09:00:19.870 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:00:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:00:19.626+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T09:00:19.870 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:00:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:00:19.626+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T09:00:19.870 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:00:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:00:19.626+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:00:19.870 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:00:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:00:19.634+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:00:19.635994+0000) 2024-02-20T09:00:20.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:00:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:00:19.866+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:00:19.866946+0000) 2024-02-20T09:00:20.988 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:00:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:00:20.747+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:00:20.748843+0000) 2024-02-20T09:00:20.988 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:00:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:00:20.865+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:00:20.867094+0000) 2024-02-20T09:00:20.989 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:00:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:00:20.637+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:00:20.638623+0000) 2024-02-20T09:00:20.989 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:00:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:00:20.656+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T09:00:20.989 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:00:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:00:20.656+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T09:00:20.989 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:00:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:00:20.656+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T09:00:20.989 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:00:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:00:20.656+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:00:21.890 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:00:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:00:21.748+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:00:21.748942+0000) 2024-02-20T09:00:21.891 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:00:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:00:21.865+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:00:21.867217+0000) 2024-02-20T09:00:21.891 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:00:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:00:21.632+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T09:00:21.891 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:00:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:00:21.632+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T09:00:21.891 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:00:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:00:21.632+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T09:00:21.891 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:00:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:00:21.632+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:00:21.892 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:00:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:00:21.640+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:00:21.641515+0000) 2024-02-20T09:00:22.989 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:00:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:00:22.748+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:00:22.749107+0000) 2024-02-20T09:00:22.989 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:00:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:00:22.866+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:00:22.867379+0000) 2024-02-20T09:00:22.989 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:00:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:00:22.642+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:00:22.643304+0000) 2024-02-20T09:00:22.989 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:00:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:00:22.655+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T09:00:22.990 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:00:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:00:22.655+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T09:00:22.990 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:00:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:00:22.655+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T09:00:22.990 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:00:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:00:22.655+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:00:23.989 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:00:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:00:23.748+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:00:23.749264+0000) 2024-02-20T09:00:23.989 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:00:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:00:23.866+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:00:23.867553+0000) 2024-02-20T09:00:23.989 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:00:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:00:23.642+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:00:23.643491+0000) 2024-02-20T09:00:23.990 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:00:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:00:23.669+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T09:00:23.990 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:00:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:00:23.669+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T09:00:23.990 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:00:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:00:23.669+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T09:00:23.990 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:00:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:00:23.669+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:00:24.605 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:00:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:00:24.249+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:00:24.606 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:00:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:00:24.249+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:00:24.606 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:00:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:00:24.249+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:00:24.869 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:00:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:00:24.608+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 52143 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:00:24.870 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:00:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:00:24.602+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:00:24.603407+0000) 2024-02-20T09:00:24.870 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:00:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:00:24.602+0000 7f8a5f86e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:00:24.603666+0000) 2024-02-20T09:00:24.870 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:00:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:00:24.748+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:00:24.749476+0000) 2024-02-20T09:00:24.870 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:00:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:00:24.605+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:00:24.606431+0000) 2024-02-20T09:00:24.871 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:00:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:00:24.642+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:00:24.643769+0000) 2024-02-20T09:00:24.871 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:00:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:00:24.708+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T09:00:24.871 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:00:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:00:24.708+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T09:00:24.871 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:00:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:00:24.708+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T09:00:24.871 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:00:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:00:24.708+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:00:25.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:00:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:00:24.866+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:00:24.867699+0000) 2024-02-20T09:00:25.646 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:00:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:00:25.246+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:00:25.646 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:00:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:00:25.246+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:00:25.646 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:00:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:00:25.246+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:00:25.989 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:00:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:00:25.748+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:00:25.749582+0000) 2024-02-20T09:00:25.989 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:00:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:00:25.867+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:00:25.867866+0000) 2024-02-20T09:00:25.990 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:00:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:00:25.644+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:00:25.646189+0000) 2024-02-20T09:00:25.990 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:00:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:00:25.695+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T09:00:25.990 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:00:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:00:25.695+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T09:00:25.990 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:00:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:00:25.695+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T08:49:53.311634+0000 (oldest deadline 2024-02-20T08:50:13.311634+0000) 2024-02-20T09:00:25.990 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:00:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:00:25.695+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:00:26.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:00:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:00:26.237+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:00:26.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:00:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:00:26.237+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:00:26.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:00:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:00:26.237+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:00:26.989 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:00:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:00:26.749+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:00:26.749767+0000) 2024-02-20T09:00:26.990 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:00:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:00:26.867+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:00:26.868048+0000) 2024-02-20T09:00:26.990 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:00:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:00:26.647+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:00:26.648974+0000) 2024-02-20T09:00:26.990 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:00:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:00:26.665+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:00:27.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:00:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:00:27.201+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:00:27.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:00:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:00:27.201+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:00:27.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:00:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:00:27.201+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:00:27.989 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:00:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:00:27.749+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:00:27.749952+0000) 2024-02-20T09:00:27.990 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:00:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:00:27.867+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:00:27.868230+0000) 2024-02-20T09:00:27.990 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:00:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:00:27.649+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:00:27.650943+0000) 2024-02-20T09:00:27.990 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:00:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:00:27.691+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:00:28.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:00:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:00:28.205+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:00:28.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:00:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:00:28.205+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:00:28.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:00:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:00:28.205+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:00:28.990 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:00:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:00:28.749+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:00:28.750109+0000) 2024-02-20T09:00:28.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:00:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:00:28.867+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:00:28.868444+0000) 2024-02-20T09:00:28.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:00:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:00:28.649+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:00:28.651253+0000) 2024-02-20T09:00:28.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:00:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:00:28.662+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:00:29.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:00:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:00:29.191+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:00:29.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:00:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:00:29.191+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:00:29.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:00:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:00:29.191+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:00:29.870 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:00:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:00:29.610+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:00:29.611644+0000) 2024-02-20T09:00:29.870 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:00:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:00:29.615+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 52215 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:00:29.871 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:00:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:00:29.611+0000 7f8a6006f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:00:29.612509+0000) 2024-02-20T09:00:29.871 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:00:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:00:29.748+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:00:29.750314+0000) 2024-02-20T09:00:29.871 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:00:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:00:29.614+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:00:29.615702+0000) 2024-02-20T09:00:29.871 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:00:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:00:29.650+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:00:29.651565+0000) 2024-02-20T09:00:29.871 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:00:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:00:29.692+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:00:30.228 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:00:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:00:29.867+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:00:29.868539+0000) 2024-02-20T09:00:30.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:00:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:00:30.225+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:00:30.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:00:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:00:30.225+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:00:30.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:00:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:00:30.225+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:00:30.990 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:00:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:00:30.749+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:00:30.750423+0000) 2024-02-20T09:00:30.990 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:00:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:00:30.648+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:00:30.990 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:00:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:00:30.652+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:00:30.653855+0000) 2024-02-20T09:00:30.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:00:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:00:30.867+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:00:30.868674+0000) 2024-02-20T09:00:31.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:00:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:00:31.203+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:00:31.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:00:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:00:31.203+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:00:31.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:00:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:00:31.203+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:00:31.990 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:00:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:00:31.749+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:00:31.750588+0000) 2024-02-20T09:00:31.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:00:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:00:31.648+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:00:31.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:00:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:00:31.655+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:00:31.656720+0000) 2024-02-20T09:00:31.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:00:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:00:31.867+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:00:31.868872+0000) 2024-02-20T09:00:31.992 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:00:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:00:31.880+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:00:31.992 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:00:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:00:31.880+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:00:31.992 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:00:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:00:31.880+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:00:32.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:00:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:00:32.175+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:00:32.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:00:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:00:32.175+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:00:32.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:00:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:00:32.175+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:00:32.990 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:00:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:00:32.749+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:00:32.750750+0000) 2024-02-20T09:00:32.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:00:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:00:32.657+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:00:32.658753+0000) 2024-02-20T09:00:32.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:00:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:00:32.681+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:00:32.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:00:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:00:32.848+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:00:32.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:00:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:00:32.848+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:00:32.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:00:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:00:32.848+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:00:32.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:00:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:00:32.868+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:00:32.869054+0000) 2024-02-20T09:00:33.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:00:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:00:33.205+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:00:33.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:00:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:00:33.205+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:00:33.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:00:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:00:33.205+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:00:33.990 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:00:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:00:33.750+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:00:33.750959+0000) 2024-02-20T09:00:33.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:00:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:00:33.657+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:00:33.658898+0000) 2024-02-20T09:00:33.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:00:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:00:33.669+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:00:33.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:00:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:00:33.868+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:00:33.869211+0000) 2024-02-20T09:00:33.992 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:00:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:00:33.875+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:00:33.992 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:00:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:00:33.876+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:00:33.992 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:00:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:00:33.876+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:00:34.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:00:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:00:34.204+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:00:34.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:00:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:00:34.204+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:00:34.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:00:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:00:34.204+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:00:34.871 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:00:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:00:34.623+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 52287 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:00:34.871 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:00:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:00:34.619+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:00:34.621345+0000) 2024-02-20T09:00:34.871 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:00:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:00:34.657+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:00:34.659223+0000) 2024-02-20T09:00:34.872 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:00:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:00:34.663+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:00:34.872 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:00:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:00:34.617+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:00:34.618524+0000) 2024-02-20T09:00:34.872 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:00:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:00:34.616+0000 7f8a6006f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:00:34.618264+0000) 2024-02-20T09:00:34.872 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:00:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:00:34.749+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:00:34.751133+0000) 2024-02-20T09:00:35.171 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:00:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:00:34.868+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:00:34.869370+0000) 2024-02-20T09:00:35.172 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:00:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:00:34.889+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:00:35.172 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:00:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:00:34.889+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:00:35.172 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:00:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:00:34.889+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:00:35.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:00:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:00:35.168+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:00:35.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:00:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:00:35.168+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:00:35.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:00:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:00:35.168+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:00:35.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:00:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:00:35.749+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:00:35.751292+0000) 2024-02-20T09:00:35.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:00:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:00:35.661+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:00:35.661789+0000) 2024-02-20T09:00:35.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:00:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:00:35.707+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:00:35.992 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:00:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:00:35.845+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:00:35.992 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:00:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:00:35.845+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:00:35.992 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:00:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:00:35.845+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:00:35.992 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:00:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:00:35.868+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:00:35.869524+0000) 2024-02-20T09:00:36.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:00:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:00:36.162+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:00:36.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:00:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:00:36.162+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:00:36.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:00:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:00:36.162+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:00:36.898 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:00:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:00:36.750+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:00:36.751463+0000) 2024-02-20T09:00:36.898 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:00:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:00:36.663+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:00:36.664607+0000) 2024-02-20T09:00:36.899 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:00:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:00:36.712+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:00:36.899 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:00:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:00:36.869+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:00:36.869694+0000) 2024-02-20T09:00:36.899 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:00:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:00:36.895+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:00:36.899 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:00:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:00:36.895+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:00:36.899 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:00:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:00:36.895+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:00:37.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:00:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:00:37.114+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:00:37.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:00:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:00:37.114+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:00:37.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:00:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:00:37.114+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:00:37.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:00:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:00:37.750+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:00:37.751651+0000) 2024-02-20T09:00:37.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:00:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:00:37.665+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:00:37.666700+0000) 2024-02-20T09:00:37.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:00:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:00:37.677+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:00:37.992 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:00:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:00:37.868+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:00:37.869786+0000) 2024-02-20T09:00:37.992 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:00:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:00:37.890+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:00:37.992 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:00:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:00:37.890+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:00:37.992 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:00:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:00:37.890+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:00:38.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:00:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:00:38.094+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:00:38.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:00:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:00:38.094+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:00:38.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:00:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:00:38.094+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:00:38.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:00:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:00:38.751+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:00:38.751866+0000) 2024-02-20T09:00:38.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:00:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:00:38.632+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:00:38.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:00:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:00:38.665+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:00:38.666921+0000) 2024-02-20T09:00:38.992 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:00:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:00:38.858+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:00:38.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:00:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:00:38.858+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:00:38.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:00:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:00:38.858+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:00:38.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:00:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:00:38.868+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:00:38.869907+0000) 2024-02-20T09:00:39.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:00:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:00:39.078+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:00:39.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:00:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:00:39.078+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:00:39.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:00:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:00:39.078+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:00:39.886 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:00:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:00:39.631+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 52359 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:00:39.887 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:00:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:00:39.624+0000 7f8a5f86e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:00:39.626287+0000) 2024-02-20T09:00:39.887 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:00:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:00:39.751+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:00:39.752039+0000) 2024-02-20T09:00:39.887 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:00:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:00:39.627+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:00:39.628972+0000) 2024-02-20T09:00:39.887 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:00:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:00:39.665+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:00:39.667244+0000) 2024-02-20T09:00:39.888 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:00:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:00:39.680+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:00:39.888 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:00:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:00:39.625+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:00:39.626697+0000) 2024-02-20T09:00:39.888 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:00:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:00:39.834+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:00:39.888 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:00:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:00:39.834+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:00:39.888 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:00:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:00:39.834+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:00:39.888 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:00:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:00:39.869+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:00:39.870121+0000) 2024-02-20T09:00:40.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:00:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:00:40.062+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:00:40.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:00:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:00:40.062+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:00:40.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:00:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:00:40.062+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:00:40.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:00:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:00:40.750+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:00:40.752147+0000) 2024-02-20T09:00:40.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:00:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:00:40.663+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:00:40.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:00:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:00:40.668+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:00:40.669823+0000) 2024-02-20T09:00:40.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:00:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:00:40.869+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:00:40.870277+0000) 2024-02-20T09:00:40.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:00:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:00:40.871+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:00:40.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:00:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:00:40.871+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:00:40.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:00:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:00:40.871+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:00:41.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:00:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:00:41.109+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:00:41.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:00:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:00:41.109+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:00:41.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:00:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:00:41.109+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:00:41.872 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:00:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:00:41.751+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:00:41.752335+0000) 2024-02-20T09:00:41.873 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:00:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:00:41.645+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:00:41.873 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:00:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:00:41.672+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:00:41.672549+0000) 2024-02-20T09:00:42.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:00:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:00:42.104+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:00:42.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:00:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:00:42.104+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:00:42.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:00:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:00:42.104+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:00:42.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:00:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:00:41.870+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:00:41.870498+0000) 2024-02-20T09:00:42.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:00:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:00:41.890+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:00:42.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:00:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:00:41.890+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:00:42.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:00:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:00:41.890+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:00:42.872 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:00:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:00:42.752+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:00:42.752581+0000) 2024-02-20T09:00:42.873 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:00:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:00:42.603+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:00:42.873 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:00:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:00:42.673+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:00:42.674416+0000) 2024-02-20T09:00:43.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:00:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:00:43.113+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:00:43.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:00:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:00:43.113+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:00:43.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:00:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:00:43.113+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:00:43.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:00:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:00:42.870+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:00:42.870642+0000) 2024-02-20T09:00:43.245 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:00:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:00:42.901+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:00:43.245 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:00:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:00:42.901+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:00:43.245 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:00:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:00:42.901+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:00:43.896 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:00:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:00:43.752+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:00:43.752769+0000) 2024-02-20T09:00:43.897 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:00:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:00:43.870+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:00:43.870820+0000) 2024-02-20T09:00:43.897 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:00:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:00:43.894+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:00:43.897 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:00:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:00:43.626+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:00:43.897 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:00:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:00:43.673+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:00:43.674671+0000) 2024-02-20T09:00:44.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:00:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:00:43.894+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:00:44.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:00:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:00:43.894+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:00:44.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:00:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:00:44.137+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:00:44.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:00:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:00:44.137+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:00:44.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:00:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:00:44.137+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:00:44.937 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:00:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:00:44.639+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 52431 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:00:44.938 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:00:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:00:44.633+0000 7f8a5f86e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:00:44.634320+0000) 2024-02-20T09:00:44.938 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:00:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:00:44.752+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:00:44.752903+0000) 2024-02-20T09:00:44.938 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:00:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:00:44.637+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:00:44.638130+0000) 2024-02-20T09:00:44.939 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:00:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:00:44.656+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:00:44.939 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:00:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:00:44.674+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:00:44.674916+0000) 2024-02-20T09:00:44.939 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:00:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:00:44.633+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:00:44.634221+0000) 2024-02-20T09:00:44.939 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:00:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:00:44.870+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:00:44.870993+0000) 2024-02-20T09:00:44.939 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:00:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:00:44.935+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:00:44.939 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:00:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:00:44.935+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:00:44.940 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:00:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:00:44.935+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:00:45.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:00:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:00:45.171+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:00:45.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:00:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:00:45.171+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:00:45.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:00:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:00:45.171+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:00:45.938 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:00:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:00:45.752+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:00:45.753074+0000) 2024-02-20T09:00:45.938 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:00:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:00:45.676+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:00:45.677270+0000) 2024-02-20T09:00:45.938 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:00:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:00:45.704+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:00:45.939 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:00:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:00:45.870+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:00:45.871129+0000) 2024-02-20T09:00:45.939 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:00:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:00:45.935+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:00:45.939 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:00:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:00:45.935+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:00:46.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:00:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:00:45.935+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:00:46.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:00:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:00:46.183+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:00:46.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:00:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:00:46.183+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:00:46.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:00:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:00:46.183+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:00:46.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:00:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:00:46.752+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:00:46.753284+0000) 2024-02-20T09:00:46.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:00:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:00:46.679+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:00:46.680193+0000) 2024-02-20T09:00:46.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:00:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:00:46.695+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:00:46.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:00:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:00:46.870+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:00:46.871335+0000) 2024-02-20T09:00:46.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:00:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:00:46.895+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:00:46.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:00:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:00:46.895+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:00:46.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:00:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:00:46.895+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:00:47.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:00:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:00:47.228+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:00:47.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:00:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:00:47.228+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:00:47.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:00:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:00:47.228+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:00:47.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:00:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:00:47.753+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:00:47.753485+0000) 2024-02-20T09:00:47.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:00:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:00:47.681+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:00:47.682340+0000) 2024-02-20T09:00:47.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:00:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:00:47.741+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:00:47.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:00:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:00:47.851+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:00:47.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:00:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:00:47.851+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:00:47.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:00:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:00:47.851+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:00:47.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:00:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:00:47.871+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:00:47.871536+0000) 2024-02-20T09:00:48.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:00:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:00:48.201+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:00:48.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:00:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:00:48.201+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:00:48.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:00:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:00:48.201+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:00:48.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:00:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:00:48.753+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:00:48.753675+0000) 2024-02-20T09:00:48.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:00:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:00:48.871+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:00:48.871743+0000) 2024-02-20T09:00:48.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:00:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:00:48.896+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:00:48.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:00:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:00:48.896+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:00:48.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:00:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:00:48.896+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:00:48.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:00:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:00:48.682+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:00:48.682696+0000) 2024-02-20T09:00:48.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:00:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:00:48.744+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:00:48.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:00:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:00:48.744+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:00:48.995 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:00:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:00:48.744+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:00:48.995 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:00:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:00:48.745+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:00:49.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:00:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:00:49.204+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:00:49.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:00:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:00:49.204+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:00:49.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:00:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:00:49.204+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:00:49.993 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:00:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:00:49.647+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 52502 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:00:49.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:00:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:00:49.641+0000 7f8a6006f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:00:49.641861+0000) 2024-02-20T09:00:49.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:00:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:00:49.753+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:00:49.753826+0000) 2024-02-20T09:00:49.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:00:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:00:49.644+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:00:49.644862+0000) 2024-02-20T09:00:49.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:00:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:00:49.682+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:00:49.683009+0000) 2024-02-20T09:00:49.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:00:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:00:49.764+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:00:49.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:00:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:00:49.764+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:00:49.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:00:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:00:49.764+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:00:49.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:00:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:00:49.764+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:00:49.995 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:00:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:00:49.641+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:00:49.642111+0000) 2024-02-20T09:00:49.995 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:00:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:00:49.865+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:00:49.995 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:00:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:00:49.865+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:00:49.995 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:00:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:00:49.865+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:00:49.995 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:00:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:00:49.871+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:00:49.871948+0000) 2024-02-20T09:00:50.365 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:00:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:00:50.193+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:00:50.365 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:00:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:00:50.193+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:00:50.365 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:00:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:00:50.193+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:00:50.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:00:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:00:50.753+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:00:50.754002+0000) 2024-02-20T09:00:50.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:00:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:00:50.818+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:00:50.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:00:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:00:50.818+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:00:50.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:00:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:00:50.818+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:00:50.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:00:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:00:50.871+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:00:50.872114+0000) 2024-02-20T09:00:50.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:00:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:00:50.686+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:00:50.686670+0000) 2024-02-20T09:00:50.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:00:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:00:50.750+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:00:50.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:00:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:00:50.750+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:00:50.995 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:00:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:00:50.750+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:00:50.995 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:00:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:00:50.750+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:00:51.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:00:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:00:51.145+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:00:51.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:00:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:00:51.145+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:00:51.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:00:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:00:51.145+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:00:51.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:00:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:00:51.753+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:00:51.754207+0000) 2024-02-20T09:00:51.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:00:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:00:51.866+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:00:51.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:00:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:00:51.866+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:00:51.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:00:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:00:51.866+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:00:51.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:00:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:00:51.871+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:00:51.872326+0000) 2024-02-20T09:00:51.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:00:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:00:51.689+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:00:51.689675+0000) 2024-02-20T09:00:51.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:00:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:00:51.711+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:00:51.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:00:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:00:51.711+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:00:51.995 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:00:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:00:51.711+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:00:51.995 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:00:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:00:51.711+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:00:52.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:00:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:00:52.171+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:00:52.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:00:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:00:52.171+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:00:52.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:00:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:00:52.171+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:00:52.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:00:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:00:52.754+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:00:52.754365+0000) 2024-02-20T09:00:52.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:00:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:00:52.838+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:00:52.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:00:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:00:52.838+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:00:52.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:00:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:00:52.838+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:00:52.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:00:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:00:52.872+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:00:52.872498+0000) 2024-02-20T09:00:52.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:00:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:00:52.689+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:00:52.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:00:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:00:52.689+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:00:52.995 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:00:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:00:52.689+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:00:52.995 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:00:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:00:52.690+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:00:52.995 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:00:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:00:52.691+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:00:52.691668+0000) 2024-02-20T09:00:53.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:00:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:00:53.208+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:00:53.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:00:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:00:53.208+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:00:53.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:00:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:00:53.208+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:00:53.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:00:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:00:53.754+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:00:53.754467+0000) 2024-02-20T09:00:53.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:00:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:00:53.838+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:00:53.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:00:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:00:53.838+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:00:53.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:00:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:00:53.838+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:00:53.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:00:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:00:53.872+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:00:53.872655+0000) 2024-02-20T09:00:53.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:00:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:00:53.641+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:00:53.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:00:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:00:53.641+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:00:53.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:00:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:00:53.641+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:00:53.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:00:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:00:53.641+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:00:53.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:00:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:00:53.691+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:00:53.691950+0000) 2024-02-20T09:00:54.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:00:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:00:54.191+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:00:54.494 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:00:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:00:54.191+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:00:54.494 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:00:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:00:54.191+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:00:54.993 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:00:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:00:54.655+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 52575 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:00:54.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:00:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:00:54.649+0000 7f8a6006f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:00:54.650233+0000) 2024-02-20T09:00:54.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:00:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:00:54.754+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:00:54.754664+0000) 2024-02-20T09:00:54.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:00:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:00:54.637+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:00:54.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:00:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:00:54.637+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:00:54.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:00:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:00:54.637+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:00:54.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:00:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:00:54.637+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:00:54.995 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:00:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:00:54.651+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:00:54.652361+0000) 2024-02-20T09:00:54.995 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:00:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:00:54.691+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:00:54.692263+0000) 2024-02-20T09:00:54.995 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:00:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:00:54.649+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:00:54.649475+0000) 2024-02-20T09:00:54.995 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:00:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:00:54.791+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:00:54.995 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:00:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:00:54.791+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:00:54.995 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:00:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:00:54.791+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:00:54.995 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:00:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:00:54.872+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:00:54.872819+0000) 2024-02-20T09:00:55.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:00:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:00:55.207+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:00:55.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:00:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:00:55.207+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:00:55.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:00:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:00:55.207+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:00:55.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:00:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:00:55.754+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:00:55.754813+0000) 2024-02-20T09:00:55.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:00:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:00:55.653+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:00:55.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:00:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:00:55.653+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:00:55.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:00:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:00:55.653+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:00:55.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:00:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:00:55.653+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:00:55.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:00:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:00:55.694+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:00:55.694883+0000) 2024-02-20T09:00:55.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:00:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:00:55.827+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:00:55.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:00:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:00:55.827+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:00:55.995 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:00:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:00:55.827+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:00:55.995 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:00:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:00:55.872+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:00:55.872951+0000) 2024-02-20T09:00:56.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:00:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:00:56.167+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:00:56.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:00:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:00:56.167+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:00:56.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:00:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:00:56.167+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:00:56.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:00:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:00:56.754+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:00:56.754978+0000) 2024-02-20T09:00:56.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:00:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:00:56.808+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:00:56.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:00:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:00:56.808+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:00:56.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:00:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:00:56.808+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:00:56.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:00:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:00:56.872+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:00:56.873149+0000) 2024-02-20T09:00:56.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:00:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:00:56.697+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:00:56.697782+0000) 2024-02-20T09:00:56.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:00:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:00:56.700+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:00:56.995 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:00:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:00:56.700+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:00:56.995 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:00:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:00:56.700+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:00:56.995 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:00:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:00:56.700+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:00:57.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:00:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:00:57.172+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:00:57.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:00:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:00:57.172+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:00:57.494 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:00:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:00:57.172+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:00:57.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:00:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:00:57.754+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:00:57.755088+0000) 2024-02-20T09:00:57.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:00:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:00:57.700+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:00:57.700278+0000) 2024-02-20T09:00:57.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:00:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:00:57.700+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:00:57.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:00:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:00:57.700+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:00:57.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:00:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:00:57.700+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:00:57.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:00:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:00:57.700+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:00:57.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:00:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:00:57.842+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:00:57.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:00:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:00:57.842+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:00:57.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:00:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:00:57.842+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:00:57.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:00:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:00:57.872+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:00:57.873303+0000) 2024-02-20T09:00:58.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:00:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:00:58.143+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:00:58.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:00:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:00:58.143+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:00:58.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:00:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:00:58.143+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:00:58.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:00:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:00:58.754+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:00:58.755268+0000) 2024-02-20T09:00:58.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:00:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:00:58.700+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:00:58.700559+0000) 2024-02-20T09:00:58.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:00:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:00:58.708+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:00:58.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:00:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:00:58.708+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:00:58.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:00:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:00:58.708+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:00:58.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:00:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:00:58.708+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:00:58.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:00:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:00:58.848+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:00:58.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:00:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:00:58.848+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:00:58.995 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:00:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:00:58.849+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:00:58.995 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:00:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:00:58.873+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:00:58.873478+0000) 2024-02-20T09:00:59.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:00:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:00:59.109+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:00:59.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:00:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:00:59.109+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:00:59.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:00:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:00:59.109+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:00:59.993 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:00:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:00:59.662+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 52646 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:00:59.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:00:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:00:59.657+0000 7f8a5f86e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:00:59.657697+0000) 2024-02-20T09:00:59.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:00:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:00:59.755+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:00:59.755453+0000) 2024-02-20T09:00:59.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:00:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:00:59.660+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:00:59.660685+0000) 2024-02-20T09:00:59.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:00:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:00:59.700+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:00:59.700845+0000) 2024-02-20T09:00:59.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:00:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:00:59.734+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:00:59.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:00:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:00:59.734+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:00:59.995 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:00:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:00:59.734+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:00:59.995 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:00:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:00:59.734+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:00:59.995 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:00:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:00:59.657+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:00:59.657490+0000) 2024-02-20T09:00:59.995 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:00:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:00:59.868+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:00:59.995 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:00:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:00:59.868+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:00:59.995 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:00:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:00:59.868+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:00:59.995 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:00:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:00:59.873+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:00:59.873632+0000) 2024-02-20T09:01:00.366 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:01:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:01:00.089+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:01:00.366 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:01:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:01:00.089+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:01:00.366 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:01:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:01:00.089+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:01:00.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:01:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:01:00.755+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:01:00.755556+0000) 2024-02-20T09:01:00.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:01:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:01:00.704+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:01:00.704601+0000) 2024-02-20T09:01:00.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:01:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:01:00.747+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:01:00.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:01:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:01:00.747+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:01:00.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:01:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:01:00.747+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:01:00.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:01:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:01:00.747+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:01:00.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:01:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:01:00.835+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:01:00.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:01:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:01:00.835+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:01:00.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:01:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:01:00.835+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:01:00.995 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:01:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:01:00.873+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:01:00.873738+0000) 2024-02-20T09:01:01.401 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:01:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:01:01.059+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:01:01.401 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:01:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:01:01.059+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:01:01.401 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:01:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:01:01.059+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:01:01.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:01:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:01:01.755+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:01:01.755774+0000) 2024-02-20T09:01:01.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:01:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:01:01.707+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:01:01.707360+0000) 2024-02-20T09:01:01.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:01:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:01:01.797+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:01:01.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:01:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:01:01.797+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:01:01.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:01:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:01:01.797+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:01:01.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:01:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:01:01.797+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:01:01.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:01:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:01:01.817+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:01:01.995 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:01:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:01:01.817+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:01:01.995 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:01:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:01:01.817+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:01:01.995 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:01:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:01:01.873+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:01:01.873897+0000) 2024-02-20T09:01:02.401 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:01:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:01:02.100+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:01:02.401 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:01:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:01:02.100+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:01:02.402 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:01:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:01:02.100+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:01:02.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:01:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:01:02.755+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:01:02.755926+0000) 2024-02-20T09:01:02.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:01:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:01:02.709+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:01:02.709576+0000) 2024-02-20T09:01:02.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:01:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:01:02.775+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:01:02.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:01:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:01:02.775+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:01:02.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:01:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:01:02.775+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:01:02.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:01:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:01:02.775+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:01:02.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:01:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:01:02.814+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:01:02.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:01:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:01:02.814+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:01:02.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:01:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:01:02.814+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:01:02.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:01:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:01:02.873+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:01:02.874069+0000) 2024-02-20T09:01:03.401 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:01:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:01:03.104+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:01:03.402 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:01:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:01:03.104+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:01:03.402 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:01:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:01:03.104+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:01:03.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:01:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:01:03.755+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:01:03.756136+0000) 2024-02-20T09:01:03.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:01:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:01:03.709+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:01:03.709873+0000) 2024-02-20T09:01:03.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:01:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:01:03.763+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:01:03.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:01:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:01:03.763+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:01:03.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:01:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:01:03.763+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:01:03.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:01:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:01:03.763+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:01:03.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:01:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:01:03.800+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:01:03.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:01:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:01:03.800+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:01:03.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:01:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:01:03.800+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:01:03.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:01:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:01:03.873+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:01:03.874194+0000) 2024-02-20T09:01:04.402 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:01:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:01:04.090+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:01:04.402 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:01:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:01:04.090+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:01:04.402 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:01:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:01:04.090+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:01:04.993 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:01:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:01:04.670+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 52719 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:01:04.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:01:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:01:04.664+0000 7f8a5f86e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:01:04.664798+0000) 2024-02-20T09:01:04.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:01:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:01:04.756+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:01:04.756340+0000) 2024-02-20T09:01:04.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:01:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:01:04.664+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:01:04.664986+0000) 2024-02-20T09:01:04.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:01:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:01:04.842+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:01:04.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:01:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:01:04.842+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:01:04.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:01:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:01:04.842+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:01:04.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:01:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:01:04.873+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:01:04.874346+0000) 2024-02-20T09:01:04.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:01:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:01:04.668+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:01:04.668647+0000) 2024-02-20T09:01:04.995 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:01:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:01:04.709+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:01:04.710169+0000) 2024-02-20T09:01:04.995 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:01:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:01:04.761+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:01:04.995 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:01:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:01:04.761+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:01:04.995 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:01:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:01:04.761+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:01:04.995 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:01:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:01:04.761+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:01:05.402 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:01:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:01:05.109+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:01:05.402 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:01:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:01:05.109+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:01:05.402 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:01:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:01:05.109+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:01:05.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:01:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:01:05.756+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:01:05.756479+0000) 2024-02-20T09:01:05.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:01:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:01:05.873+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:01:05.874538+0000) 2024-02-20T09:01:05.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:01:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:01:05.874+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:01:05.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:01:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:01:05.874+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:01:05.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:01:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:01:05.874+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:01:05.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:01:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:01:05.712+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:01:05.712836+0000) 2024-02-20T09:01:05.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:01:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:01:05.777+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:01:05.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:01:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:01:05.777+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:01:05.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:01:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:01:05.777+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:01:05.995 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:01:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:01:05.777+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:01:06.402 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:01:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:01:06.135+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:01:06.402 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:01:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:01:06.135+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:01:06.402 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:01:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:01:06.135+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:01:06.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:01:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:01:06.755+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:01:06.756691+0000) 2024-02-20T09:01:06.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:01:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:01:06.873+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:01:06.874754+0000) 2024-02-20T09:01:06.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:01:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:01:06.887+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:01:06.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:01:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:01:06.887+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:01:06.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:01:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:01:06.887+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:01:06.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:01:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:01:06.714+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:01:06.715718+0000) 2024-02-20T09:01:06.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:01:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:01:06.766+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:01:06.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:01:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:01:06.766+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:01:06.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:01:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:01:06.766+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:01:06.995 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:01:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:01:06.766+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:01:07.402 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:01:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:01:07.091+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:01:07.402 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:01:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:01:07.091+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:01:07.403 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:01:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:01:07.091+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:01:07.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:01:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:01:07.755+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:01:07.756812+0000) 2024-02-20T09:01:07.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:01:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:01:07.717+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:01:07.718326+0000) 2024-02-20T09:01:07.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:01:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:01:07.735+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:01:07.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:01:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:01:07.735+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:01:07.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:01:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:01:07.735+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:01:07.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:01:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:01:07.735+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:01:07.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:01:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:01:07.849+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:01:07.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:01:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:01:07.849+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:01:07.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:01:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:01:07.849+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:01:07.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:01:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:01:07.873+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:01:07.874919+0000) 2024-02-20T09:01:08.402 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:01:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:01:08.117+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:01:08.402 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:01:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:01:08.117+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:01:08.403 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:01:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:01:08.117+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:01:08.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:01:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:01:08.756+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:01:08.757011+0000) 2024-02-20T09:01:08.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:01:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:01:08.717+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:01:08.718621+0000) 2024-02-20T09:01:08.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:01:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:01:08.765+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:01:08.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:01:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:01:08.765+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:01:08.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:01:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:01:08.765+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:01:08.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:01:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:01:08.765+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:01:08.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:01:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:01:08.845+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:01:08.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:01:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:01:08.845+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:01:08.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:01:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:01:08.845+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:01:08.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:01:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:01:08.873+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:01:08.875094+0000) 2024-02-20T09:01:09.403 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:01:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:01:09.096+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:01:09.403 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:01:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:01:09.096+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:01:09.403 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:01:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:01:09.096+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:01:09.993 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:01:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:01:09.677+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 52790 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:01:09.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:01:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:01:09.671+0000 7f8a5f86e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:01:09.672791+0000) 2024-02-20T09:01:09.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:01:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:01:09.756+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:01:09.757197+0000) 2024-02-20T09:01:09.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:01:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:01:09.671+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:01:09.672699+0000) 2024-02-20T09:01:09.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:01:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:01:09.800+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:01:09.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:01:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:01:09.800+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:01:09.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:01:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:01:09.800+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:01:09.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:01:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:01:09.873+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:01:09.875291+0000) 2024-02-20T09:01:09.995 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:01:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:01:09.674+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:01:09.675593+0000) 2024-02-20T09:01:09.995 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:01:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:01:09.717+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:01:09.718855+0000) 2024-02-20T09:01:09.995 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:01:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:01:09.804+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:01:09.995 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:01:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:01:09.804+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:01:09.995 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:01:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:01:09.804+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:01:09.995 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:01:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:01:09.804+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:01:10.366 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:01:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:01:10.061+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:01:10.367 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:01:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:01:10.061+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:01:10.367 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:01:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:01:10.061+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:01:10.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:01:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:01:10.756+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:01:10.757350+0000) 2024-02-20T09:01:10.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:01:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:01:10.849+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:01:10.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:01:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:01:10.849+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:01:10.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:01:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:01:10.849+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:01:10.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:01:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:01:10.874+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:01:10.875446+0000) 2024-02-20T09:01:10.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:01:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:01:10.721+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:01:10.722434+0000) 2024-02-20T09:01:10.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:01:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:01:10.798+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:01:10.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:01:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:01:10.798+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:01:10.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:01:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:01:10.798+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:01:10.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:01:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:01:10.798+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:01:11.403 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:01:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:01:11.076+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:01:11.403 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:01:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:01:11.076+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:01:11.403 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:01:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:01:11.076+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:01:11.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:01:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:01:11.756+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:01:11.757536+0000) 2024-02-20T09:01:11.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:01:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:01:11.724+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:01:11.725443+0000) 2024-02-20T09:01:11.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:01:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:01:11.755+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:01:11.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:01:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:01:11.755+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:01:11.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:01:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:01:11.755+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:01:11.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:01:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:01:11.755+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:01:11.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:01:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:01:11.874+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:01:11.875675+0000) 2024-02-20T09:01:11.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:01:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:01:11.884+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:01:11.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:01:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:01:11.884+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:01:11.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:01:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:01:11.884+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:01:12.313 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:01:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:01:12.114+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:01:12.314 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:01:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:01:12.114+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:01:12.314 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:01:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:01:12.114+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:01:12.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:01:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:01:12.756+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:01:12.757733+0000) 2024-02-20T09:01:12.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:01:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:01:12.726+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:01:12.727394+0000) 2024-02-20T09:01:12.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:01:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:01:12.729+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:01:12.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:01:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:01:12.729+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:01:12.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:01:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:01:12.729+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:01:12.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:01:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:01:12.730+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:01:12.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:01:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:01:12.874+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:01:12.875829+0000) 2024-02-20T09:01:12.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:01:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:01:12.919+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:01:12.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:01:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:01:12.919+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:01:12.995 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:01:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:01:12.919+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:01:13.403 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:01:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:01:13.117+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:01:13.403 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:01:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:01:13.117+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:01:13.404 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:01:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:01:13.117+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:01:13.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:01:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:01:13.756+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:01:13.757908+0000) 2024-02-20T09:01:13.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:01:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:01:13.874+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:01:13.875967+0000) 2024-02-20T09:01:13.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:01:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:01:13.908+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:01:13.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:01:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:01:13.908+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:01:13.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:01:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:01:13.908+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:01:13.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:01:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:01:13.724+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:01:13.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:01:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:01:13.724+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:01:13.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:01:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:01:13.724+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:01:13.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:01:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:01:13.724+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:01:13.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:01:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:01:13.726+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:01:13.727694+0000) 2024-02-20T09:01:14.404 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:01:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:01:14.153+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:01:14.404 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:01:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:01:14.153+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:01:14.404 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:01:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:01:14.153+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:01:14.993 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:01:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:01:14.685+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 52863 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:01:14.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:01:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:01:14.679+0000 7f8a6006f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:01:14.680440+0000) 2024-02-20T09:01:14.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:01:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:01:14.756+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:01:14.758107+0000) 2024-02-20T09:01:14.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:01:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:01:14.681+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:01:14.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:01:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:01:14.681+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:01:14.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:01:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:01:14.681+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:01:14.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:01:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:01:14.682+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:01:14.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:01:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:01:14.683+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:01:14.684608+0000) 2024-02-20T09:01:14.995 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:01:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:01:14.726+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:01:14.727915+0000) 2024-02-20T09:01:14.995 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:01:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:01:14.678+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:01:14.680104+0000) 2024-02-20T09:01:14.995 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:01:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:01:14.874+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:01:14.876142+0000) 2024-02-20T09:01:14.995 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:01:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:01:14.893+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:01:14.995 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:01:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:01:14.893+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:01:14.995 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:01:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:01:14.893+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:01:15.404 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:01:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:01:15.125+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:01:15.404 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:01:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:01:15.125+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:01:15.404 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:01:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:01:15.125+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:01:15.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:01:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:01:15.757+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:01:15.758241+0000) 2024-02-20T09:01:15.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:01:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:01:15.874+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:01:15.876251+0000) 2024-02-20T09:01:15.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:01:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:01:15.914+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:01:15.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:01:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:01:15.914+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:01:15.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:01:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:01:15.914+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:01:15.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:01:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:01:15.677+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:01:15.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:01:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:01:15.677+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:01:15.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:01:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:01:15.677+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:01:15.995 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:01:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:01:15.677+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:01:15.995 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:01:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:01:15.729+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:01:15.730557+0000) 2024-02-20T09:01:16.404 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:01:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:01:16.101+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:01:16.404 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:01:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:01:16.101+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:01:16.404 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:01:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:01:16.101+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:01:16.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:01:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:01:16.757+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:01:16.758413+0000) 2024-02-20T09:01:16.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:01:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:01:16.872+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:01:16.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:01:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:01:16.872+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:01:16.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:01:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:01:16.872+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:01:16.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:01:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:01:16.875+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:01:16.876434+0000) 2024-02-20T09:01:16.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:01:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:01:16.724+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:01:16.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:01:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:01:16.724+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:01:16.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:01:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:01:16.724+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:01:16.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:01:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:01:16.724+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:01:16.995 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:01:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:01:16.731+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:01:16.733271+0000) 2024-02-20T09:01:17.404 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:01:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:01:17.140+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:01:17.404 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:01:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:01:17.140+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:01:17.404 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:01:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:01:17.140+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:01:17.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:01:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:01:17.757+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:01:17.758515+0000) 2024-02-20T09:01:17.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:01:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:01:17.733+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:01:17.735271+0000) 2024-02-20T09:01:17.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:01:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:01:17.752+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:01:17.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:01:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:01:17.752+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:01:17.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:01:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:01:17.752+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:01:17.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:01:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:01:17.752+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:01:17.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:01:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:01:17.851+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:01:17.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:01:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:01:17.851+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:01:17.995 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:01:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:01:17.851+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:01:17.995 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:01:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:01:17.875+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:01:17.876607+0000) 2024-02-20T09:01:18.404 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:01:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:01:18.126+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:01:18.405 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:01:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:01:18.126+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:01:18.405 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:01:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:01:18.126+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:01:18.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:01:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:01:18.757+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:01:18.758613+0000) 2024-02-20T09:01:18.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:01:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:01:18.852+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:01:18.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:01:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:01:18.852+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:01:18.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:01:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:01:18.852+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:01:18.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:01:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:01:18.875+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:01:18.876789+0000) 2024-02-20T09:01:18.995 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:01:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:01:18.725+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:01:18.995 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:01:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:01:18.725+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:01:18.995 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:01:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:01:18.725+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:01:18.995 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:01:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:01:18.725+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:01:18.995 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:01:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:01:18.734+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:01:18.735481+0000) 2024-02-20T09:01:19.404 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:01:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:01:19.108+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:01:19.405 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:01:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:01:19.108+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:01:19.405 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:01:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:01:19.108+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:01:19.993 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:01:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:01:19.692+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 52934 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:01:19.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:01:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:01:19.687+0000 7f8a5f86e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:01:19.688963+0000) 2024-02-20T09:01:19.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:01:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:01:19.757+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:01:19.758734+0000) 2024-02-20T09:01:19.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:01:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:01:19.688+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:01:19.689909+0000) 2024-02-20T09:01:19.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:01:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:01:19.858+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:01:19.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:01:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:01:19.858+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:01:19.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:01:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:01:19.858+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:01:19.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:01:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:01:19.876+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:01:19.876967+0000) 2024-02-20T09:01:19.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:01:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:01:19.689+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:01:19.691237+0000) 2024-02-20T09:01:19.995 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:01:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:01:19.720+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:01:19.995 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:01:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:01:19.720+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:01:19.995 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:01:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:01:19.720+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:01:19.995 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:01:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:01:19.721+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:01:19.995 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:01:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:01:19.734+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:01:19.735704+0000) 2024-02-20T09:01:20.367 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:01:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:01:20.158+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:01:20.367 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:01:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:01:20.158+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:01:20.368 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:01:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:01:20.158+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:01:20.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:01:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:01:20.758+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:01:20.758870+0000) 2024-02-20T09:01:20.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:01:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:01:20.868+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:01:20.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:01:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:01:20.868+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:01:20.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:01:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:01:20.868+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:01:20.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:01:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:01:20.875+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:01:20.877135+0000) 2024-02-20T09:01:20.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:01:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:01:20.738+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:01:20.739316+0000) 2024-02-20T09:01:20.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:01:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:01:20.742+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:01:20.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:01:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:01:20.742+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:01:20.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:01:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:01:20.742+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:01:20.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:01:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:01:20.742+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:01:21.405 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:01:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:01:21.195+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:01:21.405 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:01:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:01:21.195+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:01:21.405 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:01:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:01:21.195+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:01:21.997 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:01:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:01:21.757+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:01:21.759049+0000) 2024-02-20T09:01:21.998 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:01:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:01:21.876+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:01:21.877300+0000) 2024-02-20T09:01:21.998 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:01:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:01:21.917+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:01:21.998 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:01:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:01:21.917+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:01:21.998 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:01:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:01:21.917+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:01:21.999 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:01:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:01:21.741+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:01:21.742910+0000) 2024-02-20T09:01:21.999 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:01:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:01:21.771+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:01:21.999 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:01:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:01:21.771+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:01:21.999 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:01:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:01:21.771+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:01:21.999 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:01:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:01:21.771+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:01:22.405 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:01:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:01:22.227+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:01:22.405 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:01:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:01:22.227+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:01:22.405 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:01:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:01:22.227+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:01:22.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:01:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:01:22.757+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:01:22.759268+0000) 2024-02-20T09:01:22.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:01:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:01:22.876+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:01:22.877491+0000) 2024-02-20T09:01:22.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:01:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:01:22.912+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:01:22.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:01:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:01:22.912+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:01:22.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:01:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:01:22.912+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:01:22.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:01:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:01:22.728+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:01:22.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:01:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:01:22.728+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:01:22.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:01:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:01:22.728+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:01:22.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:01:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:01:22.728+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:01:22.995 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:01:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:01:22.744+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:01:22.745529+0000) 2024-02-20T09:01:23.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:01:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:01:23.260+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:01:23.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:01:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:01:23.260+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:01:23.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:01:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:01:23.260+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:01:23.998 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:01:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:01:23.758+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:01:23.759427+0000) 2024-02-20T09:01:23.998 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:01:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:01:23.876+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:01:23.877585+0000) 2024-02-20T09:01:23.998 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:01:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:01:23.926+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:01:23.998 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:01:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:01:23.926+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:01:23.998 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:01:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:01:23.926+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:01:23.999 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:01:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:01:23.740+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:01:23.999 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:01:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:01:23.740+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:01:23.999 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:01:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:01:23.740+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:01:23.999 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:01:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:01:23.740+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:01:23.999 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:01:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:01:23.744+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:01:23.745769+0000) 2024-02-20T09:01:24.405 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:01:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:01:24.211+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:01:24.406 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:01:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:01:24.211+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:01:24.406 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:01:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:01:24.211+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:01:24.993 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:01:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:01:24.701+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 53007 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:01:24.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:01:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:01:24.695+0000 7f8a6006f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:01:24.697071+0000) 2024-02-20T09:01:24.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:01:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:01:24.758+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:01:24.759525+0000) 2024-02-20T09:01:24.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:01:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:01:24.694+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:01:24.696138+0000) 2024-02-20T09:01:24.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:01:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:01:24.875+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:01:24.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:01:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:01:24.875+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:01:24.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:01:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:01:24.875+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:01:24.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:01:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:01:24.876+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:01:24.877692+0000) 2024-02-20T09:01:24.995 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:01:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:01:24.698+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:01:24.700021+0000) 2024-02-20T09:01:24.995 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:01:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:01:24.716+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:01:24.995 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:01:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:01:24.716+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:01:24.995 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:01:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:01:24.716+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:01:24.995 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:01:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:01:24.717+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:01:24.995 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:01:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:01:24.744+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:01:24.746064+0000) 2024-02-20T09:01:25.405 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:01:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:01:25.237+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:01:25.406 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:01:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:01:25.237+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:01:25.406 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:01:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:01:25.237+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:01:25.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:01:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:01:25.758+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:01:25.759638+0000) 2024-02-20T09:01:25.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:01:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:01:25.685+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:01:25.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:01:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:01:25.685+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:01:25.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:01:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:01:25.685+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:01:25.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:01:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:01:25.685+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:01:25.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:01:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:01:25.749+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:01:25.749865+0000) 2024-02-20T09:01:25.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:01:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:01:25.877+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:01:25.877835+0000) 2024-02-20T09:01:25.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:01:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:01:25.888+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:01:25.995 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:01:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:01:25.888+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:01:25.995 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:01:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:01:25.888+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:01:26.714 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:01:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:01:26.249+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:01:26.714 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:01:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:01:26.249+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:01:26.714 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:01:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:01:26.249+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:01:26.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:01:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:01:26.759+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:01:26.759833+0000) 2024-02-20T09:01:26.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:01:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:01:26.872+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:01:26.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:01:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:01:26.872+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:01:26.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:01:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:01:26.872+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:01:26.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:01:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:01:26.876+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:01:26.877959+0000) 2024-02-20T09:01:26.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:01:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:01:26.711+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:01:26.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:01:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:01:26.711+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:01:26.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:01:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:01:26.711+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:01:26.995 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:01:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:01:26.711+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:01:26.995 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:01:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:01:26.752+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:01:26.752673+0000) 2024-02-20T09:01:27.406 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:01:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:01:27.229+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:01:27.406 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:01:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:01:27.229+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:01:27.406 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:01:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:01:27.229+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:01:27.998 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:01:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:01:27.758+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:01:27.759983+0000) 2024-02-20T09:01:27.999 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:01:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:01:27.839+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:01:27.999 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:01:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:01:27.839+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:01:27.999 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:01:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:01:27.839+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:01:27.999 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:01:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:01:27.877+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:01:27.878120+0000) 2024-02-20T09:01:28.000 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:01:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:01:27.745+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:01:28.000 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:01:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:01:27.745+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:01:28.000 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:01:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:01:27.745+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:01:28.000 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:01:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:01:27.745+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:01:28.000 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:01:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:01:27.753+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:01:27.754693+0000) 2024-02-20T09:01:28.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:01:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:01:28.270+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:01:28.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:01:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:01:28.270+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:01:28.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:01:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:01:28.270+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:01:29.122 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:01:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:01:28.758+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:01:28.760112+0000) 2024-02-20T09:01:29.122 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:01:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:01:28.874+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:01:29.122 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:01:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:01:28.874+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:01:29.122 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:01:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:01:28.874+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:01:29.122 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:01:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:01:28.876+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:01:28.878259+0000) 2024-02-20T09:01:29.123 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:01:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:01:28.754+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:01:28.754934+0000) 2024-02-20T09:01:29.123 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:01:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:01:28.774+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:01:29.123 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:01:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:01:28.774+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:01:29.123 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:01:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:01:28.774+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:01:29.123 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:01:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:01:28.774+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:01:29.406 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:01:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:01:29.318+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:01:29.406 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:01:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:01:29.318+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:01:29.406 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:01:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:01:29.318+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:01:29.993 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:01:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:01:29.709+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 53078 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:01:29.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:01:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:01:29.703+0000 7f8a6006f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:01:29.704915+0000) 2024-02-20T09:01:29.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:01:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:01:29.758+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:01:29.760263+0000) 2024-02-20T09:01:29.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:01:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:01:29.703+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:01:29.704684+0000) 2024-02-20T09:01:29.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:01:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:01:29.827+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:01:29.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:01:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:01:29.827+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:01:29.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:01:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:01:29.827+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:01:29.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:01:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:01:29.877+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:01:29.878442+0000) 2024-02-20T09:01:29.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:01:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:01:29.706+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:01:29.707725+0000) 2024-02-20T09:01:29.995 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:01:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:01:29.725+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:01:29.995 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:01:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:01:29.725+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:01:29.995 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:01:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:01:29.725+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:01:29.995 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:01:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:01:29.725+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:01:29.995 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:01:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:01:29.753+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:01:29.755123+0000) 2024-02-20T09:01:30.727 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:01:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:01:30.345+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:01:30.727 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:01:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:01:30.345+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:01:30.727 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:01:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:01:30.345+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:01:30.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:01:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:01:30.758+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:01:30.760350+0000) 2024-02-20T09:01:30.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:01:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:01:30.782+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:01:30.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:01:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:01:30.782+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:01:30.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:01:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:01:30.782+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:01:30.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:01:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:01:30.878+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:01:30.878606+0000) 2024-02-20T09:01:30.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:01:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:01:30.724+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:01:30.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:01:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:01:30.724+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:01:30.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:01:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:01:30.724+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:01:30.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:01:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:01:30.724+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:01:30.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:01:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:01:30.756+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:01:30.757543+0000) 2024-02-20T09:01:31.687 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:01:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:01:31.338+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:01:31.687 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:01:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:01:31.338+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:01:31.687 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:01:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:01:31.338+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:01:31.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:01:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:01:31.759+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:01:31.760519+0000) 2024-02-20T09:01:31.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:01:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:01:31.740+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:01:31.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:01:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:01:31.740+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:01:31.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:01:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:01:31.740+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:01:31.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:01:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:01:31.878+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:01:31.878778+0000) 2024-02-20T09:01:31.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:01:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:01:31.685+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:01:31.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:01:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:01:31.685+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:01:31.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:01:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:01:31.685+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:01:31.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:01:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:01:31.685+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:01:31.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:01:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:01:31.759+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:01:31.760656+0000) 2024-02-20T09:01:32.714 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:01:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:01:32.386+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:01:32.714 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:01:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:01:32.386+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:01:32.714 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:01:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:01:32.386+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:01:32.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:01:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:01:32.760+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:01:32.760695+0000) 2024-02-20T09:01:32.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:01:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:01:32.715+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:01:32.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:01:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:01:32.715+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:01:32.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:01:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:01:32.715+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:01:32.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:01:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:01:32.877+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:01:32.878962+0000) 2024-02-20T09:01:32.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:01:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:01:32.712+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:01:32.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:01:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:01:32.712+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:01:32.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:01:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:01:32.712+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:01:32.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:01:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:01:32.712+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:01:32.995 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:01:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:01:32.761+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:01:32.762464+0000) 2024-02-20T09:01:33.705 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:01:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:01:33.431+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:01:33.705 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:01:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:01:33.431+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:01:33.706 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:01:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:01:33.431+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:01:33.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:01:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:01:33.760+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:01:33.760853+0000) 2024-02-20T09:01:33.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:01:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:01:33.745+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:01:33.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:01:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:01:33.745+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:01:33.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:01:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:01:33.745+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:01:33.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:01:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:01:33.877+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:01:33.879161+0000) 2024-02-20T09:01:33.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:01:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:01:33.702+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:01:33.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:01:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:01:33.702+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:01:33.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:01:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:01:33.702+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:01:33.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:01:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:01:33.702+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:01:33.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:01:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:01:33.761+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:01:33.762683+0000) 2024-02-20T09:01:34.682 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:01:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:01:34.409+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:01:34.683 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:01:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:01:34.410+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:01:34.683 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:01:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:01:34.410+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:01:34.993 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:01:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:01:34.717+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 53151 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:01:34.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:01:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:01:34.711+0000 7f8a5f86e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:01:34.712637+0000) 2024-02-20T09:01:34.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:01:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:01:34.759+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:01:34.761000+0000) 2024-02-20T09:01:34.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:01:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:01:34.711+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:01:34.712626+0000) 2024-02-20T09:01:34.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:01:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:01:34.743+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:01:34.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:01:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:01:34.743+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:01:34.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:01:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:01:34.743+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:01:34.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:01:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:01:34.878+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:01:34.879316+0000) 2024-02-20T09:01:34.995 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:01:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:01:34.680+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:01:34.995 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:01:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:01:34.680+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:01:34.995 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:01:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:01:34.680+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:01:34.995 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:01:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:01:34.680+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:01:34.995 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:01:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:01:34.714+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:01:34.715668+0000) 2024-02-20T09:01:34.995 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:01:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:01:34.761+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:01:34.762885+0000) 2024-02-20T09:01:35.717 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:01:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:01:35.406+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:01:35.717 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:01:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:01:35.406+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:01:35.718 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:01:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:01:35.406+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:01:35.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:01:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:01:35.760+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:01:35.761144+0000) 2024-02-20T09:01:35.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:01:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:01:35.773+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:01:35.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:01:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:01:35.773+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:01:35.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:01:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:01:35.773+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:01:35.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:01:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:01:35.878+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:01:35.879470+0000) 2024-02-20T09:01:35.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:01:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:01:35.715+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:01:35.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:01:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:01:35.715+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:01:35.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:01:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:01:35.715+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:01:35.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:01:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:01:35.715+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:01:35.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:01:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:01:35.764+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:01:35.765569+0000) 2024-02-20T09:01:36.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:01:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:01:36.395+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:01:36.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:01:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:01:36.395+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:01:36.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:01:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:01:36.395+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:01:37.128 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:01:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:01:36.760+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:01:36.761313+0000) 2024-02-20T09:01:37.129 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:01:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:01:36.785+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:01:37.129 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:01:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:01:36.785+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:01:37.129 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:01:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:01:36.785+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:01:37.129 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:01:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:01:36.879+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:01:36.879647+0000) 2024-02-20T09:01:37.129 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:01:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:01:36.760+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:01:37.130 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:01:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:01:36.760+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:01:37.130 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:01:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:01:36.760+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:01:37.130 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:01:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:01:36.760+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:01:37.130 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:01:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:01:36.767+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:01:36.769254+0000) 2024-02-20T09:01:37.742 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:01:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:01:37.431+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:01:37.742 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:01:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:01:37.431+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:01:37.742 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:01:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:01:37.431+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:01:37.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:01:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:01:37.761+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:01:37.761510+0000) 2024-02-20T09:01:37.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:01:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:01:37.740+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:01:37.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:01:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:01:37.740+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:01:37.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:01:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:01:37.740+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:01:37.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:01:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:01:37.740+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:01:37.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:01:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:01:37.771+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:01:37.771847+0000) 2024-02-20T09:01:37.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:01:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:01:37.751+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:01:37.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:01:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:01:37.751+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:01:37.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:01:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:01:37.751+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:01:37.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:01:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:01:37.879+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:01:37.879773+0000) 2024-02-20T09:01:38.703 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:01:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:01:38.477+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:01:38.703 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:01:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:01:38.477+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:01:38.704 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:01:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:01:38.477+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:01:38.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:01:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:01:38.761+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:01:38.761727+0000) 2024-02-20T09:01:38.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:01:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:01:38.755+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:01:38.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:01:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:01:38.755+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:01:38.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:01:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:01:38.755+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:01:38.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:01:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:01:38.879+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:01:38.879983+0000) 2024-02-20T09:01:38.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:01:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:01:38.701+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:01:38.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:01:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:01:38.701+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:01:38.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:01:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:01:38.701+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:01:38.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:01:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:01:38.701+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:01:38.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:01:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:01:38.771+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:01:38.772111+0000) 2024-02-20T09:01:39.675 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:01:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:01:39.445+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:01:39.675 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:01:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:01:39.445+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:01:39.675 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:01:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:01:39.445+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:01:39.993 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:01:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:01:39.726+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 53222 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:01:39.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:01:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:01:39.720+0000 7f8a6006f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:01:39.721312+0000) 2024-02-20T09:01:39.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:01:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:01:39.761+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:01:39.761933+0000) 2024-02-20T09:01:39.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:01:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:01:39.673+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:01:39.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:01:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:01:39.673+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:01:39.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:01:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:01:39.673+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:01:39.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:01:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:01:39.673+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:01:39.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:01:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:01:39.723+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:01:39.723544+0000) 2024-02-20T09:01:39.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:01:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:01:39.771+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:01:39.772365+0000) 2024-02-20T09:01:39.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:01:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:01:39.720+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:01:39.720469+0000) 2024-02-20T09:01:39.995 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:01:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:01:39.787+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:01:39.995 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:01:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:01:39.787+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:01:39.995 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:01:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:01:39.787+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:01:39.995 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:01:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:01:39.879+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:01:39.880185+0000) 2024-02-20T09:01:40.685 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:01:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:01:40.450+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:01:40.686 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:01:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:01:40.450+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:01:40.686 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:01:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:01:40.450+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:01:40.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:01:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:01:40.761+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:01:40.762088+0000) 2024-02-20T09:01:40.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:01:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:01:40.755+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:01:40.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:01:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:01:40.755+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:01:40.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:01:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:01:40.755+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:01:40.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:01:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:01:40.879+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:01:40.880280+0000) 2024-02-20T09:01:40.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:01:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:01:40.683+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:01:40.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:01:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:01:40.683+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:01:40.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:01:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:01:40.683+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:01:40.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:01:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:01:40.683+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:01:40.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:01:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:01:40.774+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:01:40.774885+0000) 2024-02-20T09:01:41.687 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:01:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:01:41.482+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:01:41.688 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:01:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:01:41.482+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:01:41.688 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:01:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:01:41.482+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:01:41.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:01:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:01:41.761+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:01:41.762221+0000) 2024-02-20T09:01:41.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:01:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:01:41.805+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:01:41.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:01:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:01:41.805+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:01:41.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:01:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:01:41.805+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:01:41.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:01:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:01:41.880+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:01:41.880453+0000) 2024-02-20T09:01:41.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:01:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:01:41.685+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:01:41.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:01:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:01:41.685+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:01:41.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:01:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:01:41.685+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:01:41.995 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:01:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:01:41.685+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:01:41.995 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:01:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:01:41.777+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:01:41.777825+0000) 2024-02-20T09:01:42.727 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:01:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:01:42.488+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:01:42.727 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:01:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:01:42.488+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:01:42.727 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:01:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:01:42.488+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:01:42.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:01:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:01:42.761+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:01:42.762375+0000) 2024-02-20T09:01:42.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:01:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:01:42.843+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:01:42.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:01:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:01:42.843+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:01:42.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:01:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:01:42.843+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:01:42.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:01:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:01:42.880+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:01:42.880582+0000) 2024-02-20T09:01:42.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:01:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:01:42.725+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:01:42.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:01:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:01:42.725+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:01:42.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:01:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:01:42.725+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:01:42.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:01:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:01:42.725+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:01:42.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:01:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:01:42.779+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:01:42.779898+0000) 2024-02-20T09:01:43.713 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:01:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:01:43.475+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:01:43.713 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:01:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:01:43.475+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:01:43.713 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:01:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:01:43.475+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:01:43.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:01:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:01:43.762+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:01:43.762543+0000) 2024-02-20T09:01:43.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:01:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:01:43.876+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:01:43.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:01:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:01:43.876+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:01:43.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:01:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:01:43.876+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:01:43.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:01:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:01:43.880+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:01:43.880698+0000) 2024-02-20T09:01:43.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:01:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:01:43.710+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:01:43.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:01:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:01:43.710+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:01:43.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:01:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:01:43.710+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:01:43.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:01:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:01:43.711+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:01:43.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:01:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:01:43.779+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:01:43.780227+0000) 2024-02-20T09:01:44.730 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:01:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:01:44.501+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:01:44.730 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:01:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:01:44.501+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:01:44.730 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:01:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:01:44.501+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:01:44.993 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:01:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:01:44.734+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 53295 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:01:44.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:01:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:01:44.728+0000 7f8a5f86e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:01:44.728696+0000) 2024-02-20T09:01:44.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:01:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:01:44.762+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:01:44.762667+0000) 2024-02-20T09:01:44.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:01:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:01:44.732+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:01:44.732964+0000) 2024-02-20T09:01:44.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:01:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:01:44.740+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:01:44.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:01:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:01:44.740+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:01:44.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:01:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:01:44.740+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:01:44.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:01:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:01:44.741+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:01:44.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:01:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:01:44.780+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:01:44.780505+0000) 2024-02-20T09:01:44.995 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:01:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:01:44.728+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:01:44.728592+0000) 2024-02-20T09:01:44.995 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:01:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:01:44.880+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:01:44.880857+0000) 2024-02-20T09:01:44.995 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:01:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:01:44.897+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:01:44.995 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:01:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:01:44.897+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:01:44.995 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:01:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:01:44.897+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:01:45.716 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:01:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:01:45.549+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:01:45.717 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:01:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:01:45.549+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:01:45.717 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:01:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:01:45.549+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:01:45.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:01:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:01:45.762+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:01:45.762830+0000) 2024-02-20T09:01:45.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:01:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:01:45.880+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:01:45.881028+0000) 2024-02-20T09:01:45.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:01:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:01:45.893+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:01:45.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:01:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:01:45.893+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:01:45.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:01:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:01:45.893+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:01:45.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:01:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:01:45.714+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:01:45.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:01:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:01:45.714+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:01:45.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:01:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:01:45.714+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:01:45.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:01:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:01:45.714+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:01:45.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:01:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:01:45.783+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:01:45.784149+0000) 2024-02-20T09:01:46.727 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:01:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:01:46.559+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:01:46.728 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:01:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:01:46.559+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:01:46.728 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:01:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:01:46.559+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:01:46.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:01:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:01:46.762+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:01:46.763039+0000) 2024-02-20T09:01:46.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:01:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:01:46.847+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:01:46.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:01:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:01:46.847+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:01:46.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:01:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:01:46.847+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:01:46.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:01:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:01:46.880+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:01:46.881145+0000) 2024-02-20T09:01:46.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:01:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:01:46.725+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:01:46.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:01:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:01:46.725+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:01:46.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:01:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:01:46.725+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:01:46.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:01:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:01:46.725+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:01:46.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:01:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:01:46.786+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:01:46.787001+0000) 2024-02-20T09:01:47.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:01:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:01:47.580+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:01:47.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:01:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:01:47.580+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:01:47.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:01:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:01:47.580+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:01:48.002 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:01:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:01:47.762+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:01:47.763265+0000) 2024-02-20T09:01:48.003 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:01:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:01:47.881+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:01:47.881348+0000) 2024-02-20T09:01:48.003 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:01:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:01:47.887+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:01:48.003 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:01:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:01:47.887+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:01:48.003 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:01:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:01:47.887+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:01:48.003 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:01:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:01:47.747+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:01:48.003 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:01:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:01:47.747+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:01:48.004 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:01:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:01:47.747+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:01:48.004 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:01:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:01:47.747+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:01:48.004 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:01:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:01:47.788+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:01:47.789157+0000) 2024-02-20T09:01:48.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:01:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:01:48.607+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:01:48.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:01:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:01:48.607+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:01:48.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:01:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:01:48.607+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:01:49.136 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:01:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:01:48.763+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:01:48.763424+0000) 2024-02-20T09:01:49.137 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:01:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:01:48.881+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:01:48.881572+0000) 2024-02-20T09:01:49.137 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:01:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:01:48.924+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:01:49.137 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:01:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:01:48.924+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:01:49.137 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:01:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:01:48.924+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:01:49.138 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:01:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:01:48.774+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:01:49.138 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:01:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:01:48.774+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:01:49.138 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:01:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:01:48.774+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:01:49.138 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:01:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:01:48.775+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:01:49.138 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:01:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:01:48.789+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:01:48.789394+0000) 2024-02-20T09:01:49.738 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:01:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:01:49.633+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:01:49.738 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:01:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:01:49.633+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:01:49.738 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:01:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:01:49.633+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:01:49.993 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:01:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:01:49.743+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 53366 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:01:49.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:01:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:01:49.736+0000 7f8a6006f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:01:49.737100+0000) 2024-02-20T09:01:49.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:01:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:01:49.763+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:01:49.763570+0000) 2024-02-20T09:01:49.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:01:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:01:49.736+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:01:49.736741+0000) 2024-02-20T09:01:49.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:01:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:01:49.881+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:01:49.881738+0000) 2024-02-20T09:01:49.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:01:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:01:49.963+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:01:49.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:01:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:01:49.963+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:01:49.995 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:01:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:01:49.963+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:01:49.995 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:01:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:01:49.739+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:01:49.739806+0000) 2024-02-20T09:01:49.995 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:01:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:01:49.789+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:01:49.789666+0000) 2024-02-20T09:01:49.995 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:01:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:01:49.824+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:01:49.995 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:01:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:01:49.824+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:01:49.995 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:01:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:01:49.824+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:01:49.996 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:01:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:01:49.824+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:01:50.945 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:01:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:01:50.793+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:01:50.793418+0000) 2024-02-20T09:01:50.946 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:01:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:01:50.826+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:01:50.946 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:01:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:01:50.826+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:01:50.946 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:01:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:01:50.826+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:01:50.946 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:01:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:01:50.826+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:01:50.946 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:01:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:01:50.668+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:01:50.946 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:01:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:01:50.668+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:01:50.947 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:01:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:01:50.668+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:01:50.947 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:01:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:01:50.763+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:01:50.763748+0000) 2024-02-20T09:01:50.947 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:01:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:01:50.881+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:01:50.881855+0000) 2024-02-20T09:01:50.947 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:01:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:01:50.943+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:01:50.947 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:01:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:01:50.943+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:01:50.947 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:01:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:01:50.943+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:01:51.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:01:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:01:51.622+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:01:51.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:01:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:01:51.622+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:01:51.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:01:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:01:51.622+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:01:52.139 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:01:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:01:51.763+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:01:51.763917+0000) 2024-02-20T09:01:52.140 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:01:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:01:51.881+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:01:51.882039+0000) 2024-02-20T09:01:52.140 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:01:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:01:51.965+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:01:52.140 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:01:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:01:51.965+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:01:52.140 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:01:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:01:51.965+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:01:52.141 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:01:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:01:51.795+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:01:51.796171+0000) 2024-02-20T09:01:52.141 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:01:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:01:51.850+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:01:52.141 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:01:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:01:51.850+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:01:52.141 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:01:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:01:51.850+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:01:52.141 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:01:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:01:51.850+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:01:52.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:01:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:01:52.632+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:01:52.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:01:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:01:52.632+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:01:52.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:01:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:01:52.632+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:01:53.139 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:01:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:01:52.763+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:01:52.764129+0000) 2024-02-20T09:01:53.140 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:01:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:01:52.798+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:01:52.798363+0000) 2024-02-20T09:01:53.140 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:01:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:01:52.849+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:01:53.140 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:01:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:01:52.849+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:01:53.140 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:01:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:01:52.849+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:01:53.141 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:01:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:01:52.849+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:01:53.141 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:01:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:01:52.881+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:01:52.882256+0000) 2024-02-20T09:01:53.141 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:01:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:01:52.991+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:01:53.141 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:01:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:01:52.991+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:01:53.141 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:01:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:01:52.991+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:01:53.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:01:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:01:53.600+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:01:53.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:01:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:01:53.600+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:01:53.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:01:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:01:53.600+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:01:54.038 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:01:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:01:53.763+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:01:53.764312+0000) 2024-02-20T09:01:54.038 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:01:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:01:53.882+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:01:53.882439+0000) 2024-02-20T09:01:54.038 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:01:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:01:54.035+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:01:54.039 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:01:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:01:53.798+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:01:53.798727+0000) 2024-02-20T09:01:54.039 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:01:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:01:53.818+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:01:54.039 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:01:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:01:53.818+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:01:54.039 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:01:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:01:53.818+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:01:54.039 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:01:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:01:53.818+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:01:54.410 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:01:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:01:54.036+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:01:54.410 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:01:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:01:54.036+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:01:54.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:01:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:01:54.612+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:01:54.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:01:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:01:54.612+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:01:54.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:01:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:01:54.612+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:01:55.003 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:01:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:01:54.751+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 53438 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:01:55.004 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:01:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:01:54.744+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:01:54.745023+0000) 2024-02-20T09:01:55.004 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:01:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:01:54.882+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:01:54.882591+0000) 2024-02-20T09:01:55.004 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:01:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:01:54.744+0000 7f8a6006f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:01:54.745141+0000) 2024-02-20T09:01:55.004 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:01:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:01:54.764+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:01:54.764450+0000) 2024-02-20T09:01:55.005 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:01:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:01:54.748+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:01:54.749276+0000) 2024-02-20T09:01:55.005 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:01:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:01:54.798+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:01:54.798965+0000) 2024-02-20T09:01:55.005 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:01:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:01:54.816+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:01:55.005 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:01:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:01:54.816+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:01:55.005 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:01:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:01:54.816+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:01:55.005 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:01:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:01:54.816+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:01:55.411 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:01:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:01:55.057+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:01:55.411 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:01:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:01:55.057+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:01:55.411 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:01:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:01:55.057+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:01:55.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:01:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:01:55.646+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:01:55.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:01:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:01:55.646+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:01:55.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:01:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:01:55.646+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:01:56.078 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:01:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:01:55.764+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:01:55.764612+0000) 2024-02-20T09:01:56.078 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:01:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:01:55.801+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:01:55.801487+0000) 2024-02-20T09:01:56.078 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:01:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:01:55.810+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:01:56.079 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:01:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:01:55.810+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:01:56.079 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:01:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:01:55.810+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:01:56.079 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:01:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:01:55.810+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:01:56.079 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:01:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:01:55.882+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:01:55.882736+0000) 2024-02-20T09:01:56.079 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:01:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:01:56.075+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:01:56.079 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:01:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:01:56.075+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:01:56.080 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:01:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:01:56.075+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:01:56.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:01:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:01:56.624+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:01:56.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:01:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:01:56.624+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:01:56.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:01:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:01:56.624+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:01:57.044 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:01:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:01:56.764+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:01:56.764773+0000) 2024-02-20T09:01:57.045 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:01:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:01:56.882+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:01:56.882895+0000) 2024-02-20T09:01:57.045 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:01:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:01:57.042+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:01:57.045 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:01:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:01:57.042+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:01:57.045 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:01:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:01:57.042+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:01:57.046 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:01:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:01:56.796+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:01:57.046 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:01:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:01:56.796+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:01:57.046 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:01:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:01:56.796+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:01:57.046 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:01:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:01:56.796+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:01:57.046 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:01:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:01:56.804+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:01:56.804357+0000) 2024-02-20T09:01:57.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:01:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:01:57.618+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:01:57.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:01:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:01:57.618+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:01:57.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:01:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:01:57.618+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:01:58.074 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:01:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:01:57.764+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:01:57.764953+0000) 2024-02-20T09:01:58.074 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:01:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:01:57.882+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:01:57.883045+0000) 2024-02-20T09:01:58.074 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:01:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:01:58.071+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:01:58.074 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:01:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:01:58.071+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:01:58.074 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:01:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:01:58.071+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:01:58.075 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:01:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:01:57.776+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:01:58.075 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:01:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:01:57.776+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:01:58.075 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:01:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:01:57.776+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:01:58.075 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:01:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:01:57.776+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:01:58.075 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:01:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:01:57.806+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:01:57.806576+0000) 2024-02-20T09:01:58.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:01:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:01:58.643+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:01:58.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:01:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:01:58.643+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:01:58.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:01:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:01:58.643+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:01:59.047 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:01:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:01:58.765+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:01:58.765159+0000) 2024-02-20T09:01:59.048 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:01:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:01:58.882+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:01:58.883157+0000) 2024-02-20T09:01:59.048 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:01:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:01:59.045+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:01:59.048 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:01:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:01:59.045+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:01:59.048 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:01:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:01:58.804+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:01:59.048 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:01:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:01:58.804+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:01:59.048 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:01:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:01:58.804+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:01:59.049 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:01:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:01:58.804+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:01:59.049 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:01:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:01:58.806+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:01:58.806907+0000) 2024-02-20T09:01:59.411 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:01:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:01:59.045+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:01:59.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:01:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:01:59.604+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:01:59.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:01:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:01:59.604+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:01:59.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:01:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:01:59.604+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:02:00.059 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:01:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:01:59.759+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 53509 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:02:00.059 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:01:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:01:59.753+0000 7f8a5f86e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:01:59.753869+0000) 2024-02-20T09:02:00.059 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:01:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:01:59.764+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:01:59.765289+0000) 2024-02-20T09:02:00.060 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:01:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:01:59.756+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:01:59.756847+0000) 2024-02-20T09:02:00.060 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:01:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:01:59.765+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:02:00.060 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:01:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:01:59.765+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:02:00.060 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:01:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:01:59.765+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:02:00.061 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:01:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:01:59.765+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:02:00.061 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:01:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:01:59.806+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:01:59.807197+0000) 2024-02-20T09:02:00.061 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:01:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:01:59.753+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:01:59.753611+0000) 2024-02-20T09:02:00.061 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:01:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:01:59.881+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:01:59.883244+0000) 2024-02-20T09:02:00.061 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:02:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:02:00.056+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:02:00.061 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:02:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:02:00.056+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:02:00.061 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:02:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:02:00.056+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:02:00.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:02:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:02:00.567+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:02:00.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:02:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:02:00.567+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:02:00.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:02:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:02:00.567+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:02:01.081 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:02:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:02:00.764+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:02:00.765452+0000) 2024-02-20T09:02:01.081 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:02:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:02:00.882+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:02:00.883393+0000) 2024-02-20T09:02:01.081 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:02:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:02:01.077+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:02:01.081 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:02:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:02:01.077+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:02:01.081 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:02:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:02:00.773+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:02:01.082 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:02:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:02:00.773+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:02:01.082 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:02:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:02:00.773+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:02:01.082 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:02:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:02:00.773+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:02:01.082 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:02:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:02:00.808+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:02:00.809797+0000) 2024-02-20T09:02:01.411 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:02:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:02:01.077+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:02:01.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:02:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:02:01.575+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:02:01.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:02:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:02:01.575+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:02:01.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:02:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:02:01.575+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:02:02.070 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:02:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:02:01.764+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:02:01.765655+0000) 2024-02-20T09:02:02.071 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:02:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:02:01.767+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:02:02.071 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:02:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:02:01.767+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:02:02.071 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:02:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:02:01.767+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:02:02.071 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:02:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:02:01.768+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:02:02.071 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:02:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:02:01.811+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:02:01.812800+0000) 2024-02-20T09:02:02.071 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:02:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:02:01.882+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:02:01.883604+0000) 2024-02-20T09:02:02.071 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:02:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:02:02.067+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:02:02.072 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:02:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:02:02.067+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:02:02.072 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:02:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:02:02.067+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:02:02.738 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:02:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:02:02.580+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:02:02.738 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:02:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:02:02.580+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:02:02.738 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:02:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:02:02.580+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:02:02.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:02:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:02:02.764+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:02:02.765826+0000) 2024-02-20T09:02:02.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:02:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:02:02.882+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:02:02.883809+0000) 2024-02-20T09:02:02.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:02:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:02:02.734+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:02:02.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:02:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:02:02.734+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:02:02.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:02:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:02:02.734+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:02:02.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:02:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:02:02.734+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:02:02.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:02:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:02:02.813+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:02:02.815024+0000) 2024-02-20T09:02:03.412 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:02:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:02:03.103+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:02:03.412 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:02:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:02:03.103+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:02:03.412 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:02:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:02:03.103+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:02:03.721 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:02:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:02:03.601+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:02:03.722 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:02:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:02:03.601+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:02:03.722 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:02:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:02:03.601+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:02:03.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:02:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:02:03.765+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:02:03.765990+0000) 2024-02-20T09:02:03.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:02:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:02:03.883+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:02:03.883984+0000) 2024-02-20T09:02:03.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:02:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:02:03.718+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:02:03.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:02:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:02:03.718+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:02:03.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:02:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:02:03.718+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:02:03.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:02:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:02:03.719+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:02:03.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:02:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:02:03.813+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:02:03.815334+0000) 2024-02-20T09:02:04.412 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:02:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:02:04.079+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:02:04.412 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:02:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:02:04.079+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:02:04.412 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:02:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:02:04.079+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:02:04.691 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:02:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:02:04.578+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:02:04.692 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:02:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:02:04.578+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:02:04.692 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:02:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:02:04.578+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:02:04.993 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:02:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:02:04.766+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 53578 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:02:04.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:02:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:02:04.760+0000 7f8a6006f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:02:04.761528+0000) 2024-02-20T09:02:04.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:02:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:02:04.764+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:02:04.766134+0000) 2024-02-20T09:02:04.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:02:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:02:04.760+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:02:04.761785+0000) 2024-02-20T09:02:04.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:02:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:02:04.883+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:02:04.884141+0000) 2024-02-20T09:02:04.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:02:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:02:04.689+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:02:04.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:02:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:02:04.689+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:02:04.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:02:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:02:04.689+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:02:04.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:02:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:02:04.689+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:02:04.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:02:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:02:04.765+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:02:04.766484+0000) 2024-02-20T09:02:04.995 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:02:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:02:04.814+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:02:04.815597+0000) 2024-02-20T09:02:05.412 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:02:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:02:05.056+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:02:05.412 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:02:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:02:05.056+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:02:05.412 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:02:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:02:05.056+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:02:05.730 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:02:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:02:05.609+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:02:05.730 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:02:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:02:05.609+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:02:05.730 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:02:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:02:05.609+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:02:05.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:02:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:02:05.765+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:02:05.766304+0000) 2024-02-20T09:02:05.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:02:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:02:05.882+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:02:05.884252+0000) 2024-02-20T09:02:05.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:02:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:02:05.727+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:02:05.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:02:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:02:05.727+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:02:05.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:02:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:02:05.727+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:02:05.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:02:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:02:05.727+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:02:05.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:02:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:02:05.816+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:02:05.818151+0000) 2024-02-20T09:02:06.413 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:02:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:02:06.105+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:02:06.413 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:02:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:02:06.105+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:02:06.413 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:02:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:02:06.105+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:02:06.734 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:02:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:02:06.606+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:02:06.734 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:02:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:02:06.606+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:02:06.734 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:02:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:02:06.606+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:02:06.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:02:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:02:06.765+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:02:06.766510+0000) 2024-02-20T09:02:06.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:02:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:02:06.883+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:02:06.884431+0000) 2024-02-20T09:02:06.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:02:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:02:06.731+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:02:06.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:02:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:02:06.731+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:02:06.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:02:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:02:06.731+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:02:06.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:02:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:02:06.731+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:02:06.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:02:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:02:06.820+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:02:06.821926+0000) 2024-02-20T09:02:07.308 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:02:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:02:07.102+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:02:07.308 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:02:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:02:07.102+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:02:07.308 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:02:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:02:07.102+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:02:07.886 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:02:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:02:07.590+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:02:07.886 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:02:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:02:07.590+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:02:07.886 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:02:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:02:07.590+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:02:07.886 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:02:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:02:07.765+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:02:07.766707+0000) 2024-02-20T09:02:07.887 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:02:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:02:07.689+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:02:07.887 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:02:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:02:07.689+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:02:07.887 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:02:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:02:07.689+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:02:07.888 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:02:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:02:07.689+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:02:07.888 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:02:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:02:07.823+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:02:07.824168+0000) 2024-02-20T09:02:08.151 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:02:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:02:07.883+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:02:07.884611+0000) 2024-02-20T09:02:08.151 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:02:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:02:08.066+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:02:08.152 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:02:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:02:08.066+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:02:08.152 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:02:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:02:08.066+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:02:08.686 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:02:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:02:08.569+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:02:08.686 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:02:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:02:08.569+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:02:08.686 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:02:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:02:08.569+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:02:08.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:02:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:02:08.765+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:02:08.766879+0000) 2024-02-20T09:02:08.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:02:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:02:08.883+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:02:08.884794+0000) 2024-02-20T09:02:08.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:02:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:02:08.684+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:02:08.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:02:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:02:08.684+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:02:08.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:02:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:02:08.684+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:02:08.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:02:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:02:08.684+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:02:08.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:02:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:02:08.823+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:02:08.824466+0000) 2024-02-20T09:02:09.413 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:02:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:02:09.025+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:02:09.413 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:02:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:02:09.025+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:02:09.413 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:02:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:02:09.025+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:02:09.711 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:02:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:02:09.542+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:02:09.711 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:02:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:02:09.542+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:02:09.711 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:02:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:02:09.542+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:02:09.984 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:02:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:02:09.773+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 53646 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:02:09.985 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:02:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:02:09.766+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:02:09.767064+0000) 2024-02-20T09:02:09.985 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:02:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:02:09.767+0000 7f8a5f86e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:02:09.769162+0000) 2024-02-20T09:02:09.986 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:02:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:02:09.767+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:02:09.769033+0000) 2024-02-20T09:02:09.986 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:02:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:02:09.883+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:02:09.884946+0000) 2024-02-20T09:02:09.986 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:02:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:02:09.981+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:02:09.986 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:02:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:02:09.707+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:02:09.986 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:02:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:02:09.708+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:02:09.986 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:02:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:02:09.708+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:02:09.987 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:02:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:02:09.708+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:02:09.987 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:02:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:02:09.770+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:02:09.772114+0000) 2024-02-20T09:02:09.987 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:02:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:02:09.823+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:02:09.824771+0000) 2024-02-20T09:02:10.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:02:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:02:09.981+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:02:10.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:02:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:02:09.981+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:02:10.681 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:02:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:02:10.523+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:02:10.681 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:02:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:02:10.523+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:02:10.681 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:02:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:02:10.523+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:02:10.964 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:02:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:02:10.766+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:02:10.767269+0000) 2024-02-20T09:02:10.964 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:02:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:02:10.745+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:02:10.964 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:02:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:02:10.745+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:02:10.965 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:02:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:02:10.745+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:02:10.965 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:02:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:02:10.745+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:02:10.965 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:02:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:02:10.826+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:02:10.827324+0000) 2024-02-20T09:02:10.965 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:02:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:02:10.883+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:02:10.885107+0000) 2024-02-20T09:02:10.965 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:02:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:02:10.961+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:02:10.965 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:02:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:02:10.961+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:02:10.965 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:02:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:02:10.961+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:02:11.731 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:02:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:02:11.535+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:02:11.731 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:02:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:02:11.535+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:02:11.731 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:02:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:02:11.535+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:02:11.983 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:02:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:02:11.766+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:02:11.767418+0000) 2024-02-20T09:02:11.983 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:02:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:02:11.884+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:02:11.885298+0000) 2024-02-20T09:02:11.983 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:02:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:02:11.980+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:02:11.984 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:02:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:02:11.980+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:02:11.984 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:02:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:02:11.980+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:02:11.984 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:02:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:02:11.728+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:02:11.984 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:02:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:02:11.728+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:02:11.984 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:02:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:02:11.728+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:02:11.984 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:02:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:02:11.728+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:02:11.984 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:02:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:02:11.829+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:02:11.830351+0000) 2024-02-20T09:02:12.713 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:02:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:02:12.513+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:02:12.714 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:02:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:02:12.513+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:02:12.714 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:02:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:02:12.513+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:02:12.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:02:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:02:12.766+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:02:12.767593+0000) 2024-02-20T09:02:12.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:02:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:02:12.884+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:02:12.885455+0000) 2024-02-20T09:02:12.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:02:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:02:12.948+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:02:12.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:02:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:02:12.948+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:02:12.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:02:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:02:12.948+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:02:12.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:02:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:02:12.711+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:02:12.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:02:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:02:12.711+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:02:12.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:02:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:02:12.711+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:02:12.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:02:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:02:12.711+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:02:12.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:02:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:02:12.832+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:02:12.832828+0000) 2024-02-20T09:02:13.710 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:02:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:02:13.501+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:02:13.710 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:02:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:02:13.501+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:02:13.710 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:02:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:02:13.501+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:02:13.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:02:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:02:13.766+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:02:13.767773+0000) 2024-02-20T09:02:13.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:02:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:02:13.884+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:02:13.885617+0000) 2024-02-20T09:02:13.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:02:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:02:13.707+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:02:13.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:02:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:02:13.707+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:02:13.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:02:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:02:13.707+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:02:13.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:02:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:02:13.707+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:02:13.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:02:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:02:13.832+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:02:13.833121+0000) 2024-02-20T09:02:14.414 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:02:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:02:13.993+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:02:14.414 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:02:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:02:13.993+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:02:14.414 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:02:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:02:13.993+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:02:14.708 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:02:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:02:14.497+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:02:14.708 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:02:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:02:14.497+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:02:14.708 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:02:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:02:14.497+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:02:14.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:02:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:02:14.766+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:02:14.767943+0000) 2024-02-20T09:02:14.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:02:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:02:14.775+0000 7f8a5f86e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:02:14.776816+0000) 2024-02-20T09:02:14.993 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:02:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:02:14.782+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 53713 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:02:14.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:02:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:02:14.775+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:02:14.777232+0000) 2024-02-20T09:02:14.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:02:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:02:14.885+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:02:14.885825+0000) 2024-02-20T09:02:14.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:02:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:02:14.705+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:02:14.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:02:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:02:14.705+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:02:14.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:02:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:02:14.705+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:02:14.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:02:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:02:14.705+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:02:14.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:02:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:02:14.778+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:02:14.779970+0000) 2024-02-20T09:02:14.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:02:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:02:14.832+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:02:14.833424+0000) 2024-02-20T09:02:15.414 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:02:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:02:15.038+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:02:15.414 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:02:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:02:15.038+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:02:15.414 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:02:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:02:15.038+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:02:15.716 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:02:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:02:15.449+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:02:15.717 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:02:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:02:15.449+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:02:15.717 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:02:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:02:15.449+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:02:15.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:02:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:02:15.766+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:02:15.768114+0000) 2024-02-20T09:02:15.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:02:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:02:15.885+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:02:15.885923+0000) 2024-02-20T09:02:15.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:02:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:02:15.714+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:02:15.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:02:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:02:15.714+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:02:15.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:02:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:02:15.714+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:02:15.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:02:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:02:15.714+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:02:15.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:02:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:02:15.834+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:02:15.835768+0000) 2024-02-20T09:02:16.414 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:02:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:02:16.041+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:02:16.414 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:02:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:02:16.041+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:02:16.415 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:02:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:02:16.041+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:02:16.677 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:02:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:02:16.489+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:02:16.677 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:02:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:02:16.489+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:02:16.677 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:02:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:02:16.489+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:02:16.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:02:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:02:16.766+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:02:16.768284+0000) 2024-02-20T09:02:16.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:02:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:02:16.885+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:02:16.886124+0000) 2024-02-20T09:02:16.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:02:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:02:16.674+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:02:16.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:02:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:02:16.674+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:02:16.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:02:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:02:16.674+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:02:16.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:02:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:02:16.674+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:02:16.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:02:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:02:16.837+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:02:16.838463+0000) 2024-02-20T09:02:17.414 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:02:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:02:17.062+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:02:17.415 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:02:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:02:17.062+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:02:17.415 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:02:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:02:17.062+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:02:17.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:02:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:02:17.526+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:02:17.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:02:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:02:17.526+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:02:17.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:02:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:02:17.526+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:02:17.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:02:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:02:17.637+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:02:17.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:02:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:02:17.637+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:02:17.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:02:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:02:17.637+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:02:17.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:02:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:02:17.637+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:02:18.093 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:02:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:02:17.767+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:02:17.768501+0000) 2024-02-20T09:02:18.093 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:02:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:02:17.885+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:02:17.886316+0000) 2024-02-20T09:02:18.093 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:02:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:02:17.839+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:02:17.840701+0000) 2024-02-20T09:02:18.414 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:02:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:02:18.090+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:02:18.415 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:02:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:02:18.090+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:02:18.415 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:02:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:02:18.090+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:02:18.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:02:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:02:18.504+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:02:18.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:02:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:02:18.504+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:02:18.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:02:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:02:18.504+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:02:18.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:02:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:02:18.616+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:02:18.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:02:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:02:18.616+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:02:18.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:02:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:02:18.616+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:02:18.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:02:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:02:18.616+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:02:19.140 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:02:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:02:18.768+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:02:18.768712+0000) 2024-02-20T09:02:19.141 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:02:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:02:18.840+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:02:18.840923+0000) 2024-02-20T09:02:19.141 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:02:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:02:18.885+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:02:18.886504+0000) 2024-02-20T09:02:19.141 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:02:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:02:19.138+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:02:19.141 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:02:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:02:19.138+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:02:19.142 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:02:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:02:19.138+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:02:19.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:02:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:02:19.503+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:02:19.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:02:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:02:19.503+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:02:19.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:02:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:02:19.503+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:02:19.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:02:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:02:19.581+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:02:19.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:02:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:02:19.581+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:02:19.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:02:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:02:19.581+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:02:19.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:02:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:02:19.581+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:02:20.158 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:02:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:02:19.790+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 53778 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:02:20.159 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:02:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:02:19.768+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:02:19.768927+0000) 2024-02-20T09:02:20.159 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:02:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:02:19.783+0000 7f8a5f86e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:02:19.785161+0000) 2024-02-20T09:02:20.159 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:02:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:02:19.786+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:02:19.787850+0000) 2024-02-20T09:02:20.160 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:02:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:02:19.840+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:02:19.841251+0000) 2024-02-20T09:02:20.160 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:02:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:02:19.784+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:02:19.785429+0000) 2024-02-20T09:02:20.160 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:02:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:02:19.885+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:02:19.886630+0000) 2024-02-20T09:02:20.415 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:02:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:02:20.156+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:02:20.415 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:02:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:02:20.156+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:02:20.415 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:02:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:02:20.156+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:02:20.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:02:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:02:20.518+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:02:20.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:02:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:02:20.518+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:02:20.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:02:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:02:20.518+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:02:20.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:02:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:02:20.588+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:02:20.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:02:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:02:20.588+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:02:20.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:02:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:02:20.588+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:02:20.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:02:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:02:20.588+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:02:21.161 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:02:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:02:20.767+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:02:20.769105+0000) 2024-02-20T09:02:21.161 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:02:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:02:20.843+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:02:20.844834+0000) 2024-02-20T09:02:21.162 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:02:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:02:20.886+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:02:20.886791+0000) 2024-02-20T09:02:21.415 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:02:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:02:21.180+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:02:21.415 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:02:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:02:21.180+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:02:21.415 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:02:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:02:21.180+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:02:21.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:02:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:02:21.556+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:02:21.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:02:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:02:21.556+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:02:21.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:02:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:02:21.556+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:02:21.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:02:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:02:21.558+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:02:21.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:02:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:02:21.558+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:02:21.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:02:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:02:21.558+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:02:21.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:02:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:02:21.558+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:02:22.143 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:02:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:02:21.767+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:02:21.769295+0000) 2024-02-20T09:02:22.144 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:02:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:02:21.847+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:02:21.848537+0000) 2024-02-20T09:02:22.144 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:02:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:02:21.886+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:02:21.886998+0000) 2024-02-20T09:02:22.144 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:02:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:02:22.140+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:02:22.144 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:02:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:02:22.140+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:02:22.144 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:02:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:02:22.140+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:02:22.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:02:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:02:22.562+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:02:22.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:02:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:02:22.562+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:02:22.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:02:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:02:22.562+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:02:22.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:02:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:02:22.561+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:02:22.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:02:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:02:22.561+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:02:22.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:02:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:02:22.561+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:02:22.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:02:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:02:22.561+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:02:23.162 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:02:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:02:22.850+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:02:22.850744+0000) 2024-02-20T09:02:23.162 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:02:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:02:22.768+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:02:22.769492+0000) 2024-02-20T09:02:23.162 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:02:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:02:22.886+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:02:22.887137+0000) 2024-02-20T09:02:23.415 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:02:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:02:23.185+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:02:23.415 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:02:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:02:23.185+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:02:23.416 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:02:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:02:23.185+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:02:23.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:02:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:02:23.543+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:02:23.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:02:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:02:23.543+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:02:23.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:02:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:02:23.543+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:02:23.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:02:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:02:23.599+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:02:23.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:02:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:02:23.599+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:02:23.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:02:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:02:23.599+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:02:23.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:02:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:02:23.599+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:02:24.162 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:02:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:02:23.768+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:02:23.769652+0000) 2024-02-20T09:02:24.162 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:02:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:02:23.850+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:02:23.851036+0000) 2024-02-20T09:02:24.163 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:02:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:02:23.886+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:02:23.887320+0000) 2024-02-20T09:02:24.415 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:02:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:02:24.183+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:02:24.416 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:02:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:02:24.183+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:02:24.416 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:02:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:02:24.183+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:02:24.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:02:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:02:24.570+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:02:24.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:02:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:02:24.570+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:02:24.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:02:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:02:24.570+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:02:24.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:02:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:02:24.585+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:02:24.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:02:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:02:24.585+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:02:24.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:02:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:02:24.585+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:02:24.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:02:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:02:24.585+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:02:25.145 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:02:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:02:24.798+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 53847 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:02:25.146 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:02:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:02:24.769+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:02:24.769841+0000) 2024-02-20T09:02:25.146 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:02:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:02:24.792+0000 7f8a5f86e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:02:24.793451+0000) 2024-02-20T09:02:25.146 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:02:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:02:24.795+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:02:24.797274+0000) 2024-02-20T09:02:25.147 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:02:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:02:24.850+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:02:24.851338+0000) 2024-02-20T09:02:25.147 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:02:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:02:24.792+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:02:24.793841+0000) 2024-02-20T09:02:25.147 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:02:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:02:24.886+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:02:24.887548+0000) 2024-02-20T09:02:25.147 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:02:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:02:25.143+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:02:25.147 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:02:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:02:25.143+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:02:25.147 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:02:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:02:25.143+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:02:25.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:02:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:02:25.573+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:02:25.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:02:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:02:25.573+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:02:25.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:02:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:02:25.573+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:02:25.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:02:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:02:25.633+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:02:25.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:02:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:02:25.633+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:02:25.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:02:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:02:25.633+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:02:25.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:02:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:02:25.633+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:02:26.165 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:02:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:02:25.769+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:02:25.770013+0000) 2024-02-20T09:02:26.165 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:02:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:02:25.853+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:02:25.854122+0000) 2024-02-20T09:02:26.166 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:02:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:02:25.887+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:02:25.887744+0000) 2024-02-20T09:02:26.416 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:02:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:02:26.187+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:02:26.416 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:02:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:02:26.187+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:02:26.416 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:02:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:02:26.187+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:02:26.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:02:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:02:26.540+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:02:26.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:02:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:02:26.540+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:02:26.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:02:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:02:26.540+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:02:26.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:02:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:02:26.590+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:02:26.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:02:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:02:26.590+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:02:26.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:02:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:02:26.590+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:02:26.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:02:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:02:26.590+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:02:27.165 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:02:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:02:26.769+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:02:26.770126+0000) 2024-02-20T09:02:27.166 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:02:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:02:26.856+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:02:26.857051+0000) 2024-02-20T09:02:27.166 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:02:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:02:26.887+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:02:26.887903+0000) 2024-02-20T09:02:27.416 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:02:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:02:27.212+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:02:27.416 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:02:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:02:27.212+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:02:27.417 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:02:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:02:27.212+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:02:27.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:02:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:02:27.501+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:02:27.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:02:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:02:27.501+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:02:27.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:02:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:02:27.501+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:02:27.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:02:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:02:27.604+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:02:27.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:02:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:02:27.604+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:02:27.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:02:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:02:27.604+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:02:27.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:02:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:02:27.604+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:02:28.165 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:02:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:02:27.768+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:02:27.770302+0000) 2024-02-20T09:02:28.166 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:02:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:02:27.886+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:02:27.888101+0000) 2024-02-20T09:02:28.166 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:02:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:02:27.858+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:02:27.859256+0000) 2024-02-20T09:02:28.416 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:02:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:02:28.195+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:02:28.416 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:02:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:02:28.195+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:02:28.417 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:02:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:02:28.195+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:02:28.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:02:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:02:28.499+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:02:28.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:02:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:02:28.499+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:02:28.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:02:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:02:28.499+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:02:28.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:02:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:02:28.630+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:02:28.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:02:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:02:28.630+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:02:28.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:02:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:02:28.630+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:02:28.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:02:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:02:28.630+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:02:29.166 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:02:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:02:28.770+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:02:28.770515+0000) 2024-02-20T09:02:29.166 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:02:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:02:28.858+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:02:28.859522+0000) 2024-02-20T09:02:29.166 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:02:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:02:28.887+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:02:28.888255+0000) 2024-02-20T09:02:29.416 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:02:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:02:29.220+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:02:29.417 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:02:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:02:29.220+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:02:29.417 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:02:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:02:29.220+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:02:29.669 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:02:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:02:29.516+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:02:29.669 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:02:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:02:29.516+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:02:29.669 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:02:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:02:29.516+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:02:29.929 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:02:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:02:29.806+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 53912 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:02:29.930 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:02:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:02:29.770+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:02:29.770727+0000) 2024-02-20T09:02:29.930 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:02:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:02:29.799+0000 7f8a6006f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:02:29.801266+0000) 2024-02-20T09:02:29.930 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:02:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:02:29.799+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:02:29.801049+0000) 2024-02-20T09:02:29.930 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:02:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:02:29.886+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:02:29.888360+0000) 2024-02-20T09:02:29.931 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:02:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:02:29.667+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:02:29.931 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:02:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:02:29.667+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:02:29.931 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:02:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:02:29.667+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:02:29.931 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:02:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:02:29.667+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:02:29.931 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:02:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:02:29.802+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:02:29.804284+0000) 2024-02-20T09:02:29.931 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:02:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:02:29.859+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:02:29.860002+0000) 2024-02-20T09:02:30.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:02:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:02:30.235+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:02:30.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:02:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:02:30.235+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:02:30.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:02:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:02:30.235+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:02:30.864 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:02:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:02:30.561+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:02:30.864 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:02:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:02:30.561+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:02:30.864 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:02:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:02:30.561+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:02:30.864 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:02:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:02:30.770+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:02:30.770949+0000) 2024-02-20T09:02:30.865 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:02:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:02:30.716+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:02:30.865 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:02:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:02:30.716+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:02:30.865 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:02:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:02:30.716+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:02:30.865 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:02:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:02:30.716+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:02:31.168 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:02:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:02:30.862+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:02:30.862488+0000) 2024-02-20T09:02:31.169 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:02:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:02:30.888+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:02:30.888507+0000) 2024-02-20T09:02:31.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:02:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:02:31.217+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:02:31.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:02:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:02:31.217+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:02:31.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:02:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:02:31.217+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:02:31.867 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:02:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:02:31.705+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:02:31.867 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:02:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:02:31.705+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:02:31.867 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:02:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:02:31.705+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:02:31.867 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:02:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:02:31.705+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:02:31.867 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:02:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:02:31.575+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:02:31.867 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:02:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:02:31.575+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:02:31.868 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:02:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:02:31.575+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:02:31.868 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:02:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:02:31.770+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:02:31.771119+0000) 2024-02-20T09:02:32.169 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:02:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:02:31.863+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:02:31.865260+0000) 2024-02-20T09:02:32.169 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:02:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:02:31.887+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:02:31.888642+0000) 2024-02-20T09:02:32.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:02:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:02:32.197+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:02:32.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:02:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:02:32.197+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:02:32.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:02:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:02:32.197+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:02:32.869 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:02:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:02:32.597+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:02:32.869 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:02:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:02:32.597+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:02:32.869 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:02:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:02:32.597+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:02:32.870 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:02:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:02:32.770+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:02:32.771344+0000) 2024-02-20T09:02:32.870 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:02:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:02:32.679+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:02:32.870 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:02:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:02:32.679+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:02:32.870 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:02:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:02:32.679+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:02:32.870 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:02:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:02:32.679+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:02:33.169 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:02:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:02:32.867+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:02:32.867662+0000) 2024-02-20T09:02:33.170 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:02:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:02:32.888+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:02:32.888746+0000) 2024-02-20T09:02:33.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:02:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:02:33.189+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:02:33.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:02:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:02:33.189+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:02:33.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:02:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:02:33.189+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:02:33.870 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:02:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:02:33.590+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:02:33.870 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:02:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:02:33.590+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:02:33.870 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:02:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:02:33.590+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:02:33.870 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:02:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:02:33.771+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:02:33.771571+0000) 2024-02-20T09:02:33.870 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:02:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:02:33.728+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:02:33.870 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:02:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:02:33.728+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:02:33.871 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:02:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:02:33.728+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:02:33.871 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:02:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:02:33.728+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:02:34.166 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:02:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:02:33.867+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:02:33.867953+0000) 2024-02-20T09:02:34.166 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:02:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:02:33.888+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:02:33.888899+0000) 2024-02-20T09:02:34.166 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:02:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:02:34.163+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:02:34.167 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:02:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:02:34.163+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:02:34.417 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:02:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:02:34.163+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:02:34.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:02:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:02:34.567+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:02:34.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:02:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:02:34.567+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:02:34.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:02:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:02:34.567+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:02:35.170 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:02:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:02:34.815+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 53981 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:02:35.170 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:02:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:02:34.771+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:02:34.771746+0000) 2024-02-20T09:02:35.170 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:02:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:02:34.808+0000 7f8a5f86e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:02:34.808876+0000) 2024-02-20T09:02:35.171 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:02:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:02:34.808+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:02:34.809144+0000) 2024-02-20T09:02:35.171 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:02:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:02:34.888+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:02:34.889086+0000) 2024-02-20T09:02:35.171 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:02:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:02:34.765+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:02:35.171 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:02:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:02:34.765+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:02:35.171 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:02:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:02:34.765+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:02:35.172 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:02:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:02:34.765+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:02:35.172 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:02:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:02:34.812+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:02:34.812582+0000) 2024-02-20T09:02:35.172 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:02:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:02:34.867+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:02:34.868255+0000) 2024-02-20T09:02:35.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:02:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:02:35.200+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:02:35.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:02:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:02:35.200+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:02:35.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:02:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:02:35.200+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:02:35.872 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:02:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:02:35.535+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:02:35.872 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:02:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:02:35.535+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:02:35.873 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:02:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:02:35.535+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:02:35.873 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:02:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:02:35.771+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:02:35.771909+0000) 2024-02-20T09:02:35.873 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:02:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:02:35.747+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:02:35.873 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:02:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:02:35.747+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:02:35.873 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:02:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:02:35.747+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:02:35.873 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:02:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:02:35.747+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:02:35.873 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:02:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:02:35.870+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:02:35.870731+0000) 2024-02-20T09:02:36.173 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:02:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:02:35.888+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:02:35.889260+0000) 2024-02-20T09:02:36.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:02:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:02:36.225+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:02:36.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:02:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:02:36.225+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:02:36.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:02:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:02:36.225+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:02:36.875 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:02:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:02:36.575+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:02:36.876 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:02:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:02:36.575+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:02:36.876 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:02:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:02:36.575+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:02:36.876 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:02:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:02:36.771+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:02:36.772066+0000) 2024-02-20T09:02:36.876 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:02:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:02:36.702+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:02:36.876 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:02:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:02:36.702+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:02:36.876 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:02:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:02:36.702+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:02:36.876 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:02:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:02:36.702+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:02:37.173 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:02:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:02:36.873+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:02:36.873963+0000) 2024-02-20T09:02:37.173 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:02:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:02:36.889+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:02:36.889451+0000) 2024-02-20T09:02:37.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:02:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:02:37.183+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:02:37.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:02:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:02:37.183+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:02:37.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:02:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:02:37.183+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:02:37.878 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:02:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:02:37.751+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:02:37.878 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:02:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:02:37.751+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:02:37.878 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:02:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:02:37.751+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:02:37.878 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:02:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:02:37.751+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:02:37.879 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:02:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:02:37.538+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:02:37.879 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:02:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:02:37.538+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:02:37.879 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:02:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:02:37.538+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:02:37.879 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:02:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:02:37.771+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:02:37.772251+0000) 2024-02-20T09:02:38.173 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:02:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:02:37.876+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:02:37.876544+0000) 2024-02-20T09:02:38.173 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:02:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:02:37.889+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:02:37.889564+0000) 2024-02-20T09:02:38.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:02:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:02:38.222+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:02:38.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:02:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:02:38.222+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:02:38.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:02:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:02:38.222+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:02:38.800 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:02:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:02:38.532+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:02:38.800 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:02:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:02:38.532+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:02:38.801 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:02:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:02:38.532+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:02:38.801 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:02:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:02:38.772+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:02:38.772431+0000) 2024-02-20T09:02:39.173 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:02:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:02:38.889+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:02:38.889752+0000) 2024-02-20T09:02:39.174 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:02:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:02:38.798+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:02:39.174 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:02:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:02:38.798+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:02:39.174 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:02:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:02:38.798+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:02:39.174 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:02:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:02:38.798+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:02:39.174 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:02:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:02:38.876+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:02:38.876822+0000) 2024-02-20T09:02:39.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:02:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:02:39.203+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:02:39.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:02:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:02:39.203+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:02:39.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:02:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:02:39.203+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:02:39.774 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:02:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:02:39.512+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:02:39.774 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:02:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:02:39.512+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:02:39.774 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:02:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:02:39.512+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:02:39.774 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:02:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:02:39.752+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:02:39.775 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:02:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:02:39.752+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:02:39.775 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:02:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:02:39.752+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:02:39.775 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:02:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:02:39.753+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:02:40.174 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:02:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:02:39.772+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:02:39.772626+0000) 2024-02-20T09:02:40.174 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:02:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:02:39.816+0000 7f8a6006f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:02:39.817289+0000) 2024-02-20T09:02:40.174 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:02:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:02:39.820+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:02:39.820520+0000) 2024-02-20T09:02:40.174 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:02:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:02:39.876+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:02:39.877069+0000) 2024-02-20T09:02:40.175 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:02:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:02:39.817+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:02:39.817573+0000) 2024-02-20T09:02:40.175 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:02:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:02:39.889+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:02:39.889881+0000) 2024-02-20T09:02:40.175 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:02:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:02:39.823+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 54049 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:02:40.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:02:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:02:40.171+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:02:40.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:02:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:02:40.171+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:02:40.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:02:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:02:40.171+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:02:40.781 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:02:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:02:40.527+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:02:40.781 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:02:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:02:40.527+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:02:40.781 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:02:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:02:40.527+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:02:40.781 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:02:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:02:40.772+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:02:40.772838+0000) 2024-02-20T09:02:40.781 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:02:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:02:40.737+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:02:40.782 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:02:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:02:40.737+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:02:40.782 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:02:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:02:40.737+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:02:40.782 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:02:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:02:40.738+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:02:41.177 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:02:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:02:40.879+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:02:40.879670+0000) 2024-02-20T09:02:41.178 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:02:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:02:40.889+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:02:40.890075+0000) 2024-02-20T09:02:41.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:02:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:02:41.177+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:02:41.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:02:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:02:41.177+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:02:41.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:02:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:02:41.177+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:02:41.774 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:02:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:02:41.501+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:02:41.774 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:02:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:02:41.501+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:02:41.775 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:02:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:02:41.501+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:02:42.178 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:02:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:02:41.772+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:02:41.772952+0000) 2024-02-20T09:02:42.178 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:02:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:02:41.889+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:02:41.890270+0000) 2024-02-20T09:02:42.178 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:02:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:02:41.781+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:02:42.179 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:02:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:02:41.781+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:02:42.179 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:02:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:02:41.781+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:02:42.179 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:02:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:02:41.781+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:02:42.179 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:02:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:02:41.882+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:02:41.882430+0000) 2024-02-20T09:02:42.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:02:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:02:42.219+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:02:42.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:02:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:02:42.219+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:02:42.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:02:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:02:42.219+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:02:42.770 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:02:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:02:42.500+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:02:42.822 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:02:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:02:42.500+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:02:42.822 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:02:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:02:42.500+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:02:43.178 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:02:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:02:42.772+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:02:42.773157+0000) 2024-02-20T09:02:43.178 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:02:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:02:42.890+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:02:42.890493+0000) 2024-02-20T09:02:43.178 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:02:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:02:42.768+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:02:43.178 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:02:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:02:42.768+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:02:43.179 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:02:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:02:42.768+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:02:43.179 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:02:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:02:42.768+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:02:43.179 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:02:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:02:42.884+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:02:42.884654+0000) 2024-02-20T09:02:43.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:02:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:02:43.244+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:02:43.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:02:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:02:43.244+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:02:43.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:02:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:02:43.244+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:02:43.886 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:02:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:02:43.549+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:02:43.887 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:02:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:02:43.549+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:02:43.887 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:02:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:02:43.549+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:02:43.887 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:02:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:02:43.772+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:02:43.773307+0000) 2024-02-20T09:02:43.887 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:02:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:02:43.751+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:02:43.887 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:02:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:02:43.751+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:02:43.887 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:02:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:02:43.751+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:02:43.888 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:02:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:02:43.751+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:02:44.178 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:02:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:02:43.884+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:02:43.884944+0000) 2024-02-20T09:02:44.178 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:02:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:02:43.890+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:02:43.890637+0000) 2024-02-20T09:02:44.473 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:02:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:02:44.263+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:02:44.473 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:02:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:02:44.263+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:02:44.473 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:02:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:02:44.263+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:02:44.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:02:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:02:44.586+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:02:44.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:02:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:02:44.586+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:02:44.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:02:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:02:44.586+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:02:45.178 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:02:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:02:44.831+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 54119 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:02:45.179 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:02:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:02:44.773+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:02:44.773457+0000) 2024-02-20T09:02:45.179 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:02:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:02:44.824+0000 7f8a6006f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:02:44.825374+0000) 2024-02-20T09:02:45.179 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:02:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:02:44.825+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:02:44.825770+0000) 2024-02-20T09:02:45.179 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:02:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:02:44.890+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:02:44.890787+0000) 2024-02-20T09:02:45.180 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:02:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:02:44.789+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:02:45.180 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:02:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:02:44.789+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:02:45.180 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:02:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:02:44.789+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:02:45.180 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:02:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:02:44.789+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:02:45.183 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:02:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:02:44.828+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:02:44.829289+0000) 2024-02-20T09:02:45.183 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:02:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:02:44.884+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:02:44.885218+0000) 2024-02-20T09:02:45.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:02:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:02:45.276+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:02:45.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:02:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:02:45.276+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:02:45.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:02:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:02:45.276+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:02:45.890 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:02:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:02:45.599+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:02:45.890 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:02:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:02:45.599+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:02:45.890 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:02:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:02:45.599+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:02:45.890 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:02:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:02:45.773+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:02:45.773647+0000) 2024-02-20T09:02:45.890 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:02:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:02:45.813+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:02:45.891 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:02:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:02:45.813+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:02:45.891 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:02:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:02:45.813+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:02:45.891 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:02:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:02:45.813+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:02:46.181 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:02:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:02:45.887+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:02:45.887851+0000) 2024-02-20T09:02:46.182 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:02:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:02:45.890+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:02:45.890883+0000) 2024-02-20T09:02:46.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:02:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:02:46.283+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:02:46.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:02:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:02:46.283+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:02:46.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:02:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:02:46.283+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:02:46.892 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:02:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:02:46.602+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:02:46.893 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:02:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:02:46.602+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:02:46.893 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:02:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:02:46.602+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:02:46.893 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:02:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:02:46.773+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:02:46.773841+0000) 2024-02-20T09:02:46.893 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:02:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:02:46.839+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:02:46.893 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:02:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:02:46.839+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:02:46.894 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:02:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:02:46.839+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:02:46.894 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:02:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:02:46.839+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:02:47.182 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:02:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:02:46.890+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:02:46.890666+0000) 2024-02-20T09:02:47.182 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:02:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:02:46.890+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:02:46.891011+0000) 2024-02-20T09:02:47.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:02:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:02:47.302+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:02:47.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:02:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:02:47.302+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:02:47.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:02:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:02:47.302+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:02:47.892 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:02:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:02:47.588+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:02:47.893 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:02:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:02:47.588+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:02:47.893 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:02:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:02:47.588+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:02:47.893 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:02:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:02:47.773+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:02:47.774030+0000) 2024-02-20T09:02:47.893 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:02:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:02:47.808+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:02:47.893 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:02:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:02:47.808+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:02:47.893 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:02:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:02:47.808+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:02:47.893 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:02:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:02:47.808+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:02:48.182 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:02:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:02:47.892+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:02:47.892842+0000) 2024-02-20T09:02:48.182 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:02:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:02:47.891+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:02:47.891174+0000) 2024-02-20T09:02:48.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:02:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:02:48.352+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:02:48.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:02:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:02:48.352+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:02:48.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:02:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:02:48.352+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:02:48.893 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:02:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:02:48.592+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:02:48.893 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:02:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:02:48.592+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:02:48.893 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:02:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:02:48.592+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:02:48.893 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:02:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:02:48.773+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:02:48.774193+0000) 2024-02-20T09:02:48.894 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:02:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:02:48.773+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:02:48.894 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:02:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:02:48.773+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:02:48.894 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:02:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:02:48.773+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:02:48.894 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:02:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:02:48.773+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:02:49.182 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:02:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:02:48.892+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:02:48.893101+0000) 2024-02-20T09:02:49.182 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:02:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:02:48.891+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:02:48.891326+0000) 2024-02-20T09:02:49.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:02:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:02:49.349+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:02:49.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:02:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:02:49.349+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:02:49.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:02:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:02:49.349+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:02:49.835 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:02:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:02:49.777+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:02:49.835 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:02:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:02:49.777+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:02:49.835 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:02:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:02:49.777+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:02:49.835 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:02:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:02:49.777+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:02:49.835 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:02:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:02:49.566+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:02:49.835 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:02:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:02:49.566+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:02:49.836 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:02:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:02:49.566+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:02:49.836 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:02:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:02:49.773+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:02:49.774364+0000) 2024-02-20T09:02:50.182 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:02:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:02:49.839+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 54190 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:02:50.183 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:02:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:02:49.833+0000 7f8a5f86e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:02:49.834354+0000) 2024-02-20T09:02:50.183 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:02:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:02:49.837+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:02:49.838021+0000) 2024-02-20T09:02:50.183 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:02:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:02:49.892+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:02:49.893343+0000) 2024-02-20T09:02:50.183 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:02:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:02:49.833+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:02:49.833435+0000) 2024-02-20T09:02:50.184 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:02:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:02:49.891+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:02:49.891438+0000) 2024-02-20T09:02:50.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:02:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:02:50.300+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:02:50.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:02:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:02:50.300+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:02:50.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:02:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:02:50.300+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:02:50.893 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:02:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:02:50.771+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:02:50.893 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:02:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:02:50.771+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:02:50.893 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:02:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:02:50.771+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:02:50.893 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:02:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:02:50.771+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:02:50.894 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:02:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:02:50.578+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:02:50.894 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:02:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:02:50.578+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:02:50.894 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:02:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:02:50.578+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:02:50.894 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:02:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:02:50.774+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:02:50.774534+0000) 2024-02-20T09:02:51.186 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:02:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:02:50.895+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:02:50.895607+0000) 2024-02-20T09:02:51.186 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:02:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:02:50.891+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:02:50.891591+0000) 2024-02-20T09:02:51.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:02:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:02:51.257+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:02:51.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:02:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:02:51.257+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:02:51.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:02:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:02:51.257+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:02:51.893 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:02:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:02:51.762+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:02:51.893 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:02:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:02:51.762+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:02:51.893 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:02:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:02:51.762+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:02:51.894 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:02:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:02:51.762+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:02:51.894 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:02:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:02:51.610+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:02:51.894 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:02:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:02:51.610+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:02:51.894 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:02:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:02:51.610+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:02:51.894 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:02:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:02:51.774+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:02:51.774682+0000) 2024-02-20T09:02:52.186 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:02:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:02:51.898+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:02:51.898446+0000) 2024-02-20T09:02:52.186 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:02:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:02:51.891+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:02:51.891765+0000) 2024-02-20T09:02:52.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:02:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:02:52.292+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:02:52.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:02:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:02:52.292+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:02:52.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:02:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:02:52.292+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:02:52.893 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:02:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:02:52.625+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:02:52.893 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:02:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:02:52.625+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:02:52.893 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:02:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:02:52.625+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:02:52.894 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:02:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:02:52.774+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:02:52.774869+0000) 2024-02-20T09:02:52.894 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:02:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:02:52.778+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:02:52.894 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:02:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:02:52.778+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:02:52.894 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:02:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:02:52.778+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:02:52.894 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:02:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:02:52.778+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:02:53.186 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:02:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:02:52.900+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:02:52.900872+0000) 2024-02-20T09:02:53.187 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:02:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:02:52.891+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:02:52.891901+0000) 2024-02-20T09:02:53.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:02:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:02:53.319+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:02:53.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:02:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:02:53.319+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:02:53.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:02:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:02:53.319+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:02:53.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:02:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:02:53.890+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:02:53.892100+0000) 2024-02-20T09:02:53.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:02:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:02:53.652+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:02:53.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:02:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:02:53.652+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:02:53.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:02:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:02:53.652+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:02:53.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:02:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:02:53.773+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:02:53.775040+0000) 2024-02-20T09:02:53.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:02:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:02:53.807+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:02:53.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:02:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:02:53.807+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:02:53.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:02:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:02:53.807+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:02:53.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:02:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:02:53.807+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:02:53.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:02:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:02:53.899+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:02:53.901133+0000) 2024-02-20T09:02:54.668 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:02:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:02:54.333+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:02:54.669 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:02:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:02:54.333+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:02:54.669 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:02:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:02:54.333+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:02:54.971 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:02:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:02:54.846+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 54263 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:02:54.972 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:02:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:02:54.840+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:02:54.842332+0000) 2024-02-20T09:02:54.972 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:02:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:02:54.890+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:02:54.892270+0000) 2024-02-20T09:02:54.972 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:02:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:02:54.665+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:02:54.972 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:02:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:02:54.665+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:02:54.972 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:02:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:02:54.665+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:02:54.973 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:02:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:02:54.773+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:02:54.775234+0000) 2024-02-20T09:02:54.973 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:02:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:02:54.840+0000 7f8a6006f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:02:54.841743+0000) 2024-02-20T09:02:54.973 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:02:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:02:54.841+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:02:54.973 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:02:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:02:54.841+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:02:54.973 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:02:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:02:54.841+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:02:54.973 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:02:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:02:54.841+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:02:54.973 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:02:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:02:54.843+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:02:54.844898+0000) 2024-02-20T09:02:54.974 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:02:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:02:54.900+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:02:54.901445+0000) 2024-02-20T09:02:55.652 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:02:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:02:55.299+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:02:55.653 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:02:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:02:55.299+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:02:55.653 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:02:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:02:55.299+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:02:55.905 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:02:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:02:55.891+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:02:55.892454+0000) 2024-02-20T09:02:55.905 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:02:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:02:55.650+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:02:55.906 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:02:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:02:55.650+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:02:55.906 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:02:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:02:55.650+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:02:55.906 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:02:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:02:55.774+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:02:55.775415+0000) 2024-02-20T09:02:55.906 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:02:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:02:55.839+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:02:55.906 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:02:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:02:55.839+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:02:55.906 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:02:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:02:55.839+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:02:55.906 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:02:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:02:55.839+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:02:56.189 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:02:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:02:55.902+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:02:55.903719+0000) 2024-02-20T09:02:56.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:02:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:02:56.270+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:02:56.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:02:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:02:56.270+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:02:56.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:02:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:02:56.270+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:02:56.908 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:02:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:02:56.891+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:02:56.892617+0000) 2024-02-20T09:02:56.908 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:02:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:02:56.812+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:02:56.909 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:02:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:02:56.812+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:02:56.909 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:02:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:02:56.812+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:02:56.909 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:02:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:02:56.812+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:02:56.909 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:02:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:02:56.649+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:02:56.909 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:02:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:02:56.649+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:02:56.909 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:02:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:02:56.649+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:02:56.910 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:02:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:02:56.774+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:02:56.775576+0000) 2024-02-20T09:02:57.190 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:02:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:02:56.905+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:02:56.906579+0000) 2024-02-20T09:02:57.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:02:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:02:57.264+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:02:57.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:02:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:02:57.264+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:02:57.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:02:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:02:57.264+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:02:57.894 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:02:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:02:57.762+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:02:57.894 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:02:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:02:57.762+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:02:57.895 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:02:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:02:57.762+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:02:57.895 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:02:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:02:57.762+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:02:57.895 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:02:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:02:57.607+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:02:57.895 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:02:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:02:57.607+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:02:57.895 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:02:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:02:57.607+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:02:57.895 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:02:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:02:57.774+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:02:57.775797+0000) 2024-02-20T09:02:58.190 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:02:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:02:57.891+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:02:57.892800+0000) 2024-02-20T09:02:58.190 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:02:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:02:57.907+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:02:57.908895+0000) 2024-02-20T09:02:58.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:02:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:02:58.228+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:02:58.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:02:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:02:58.228+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:02:58.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:02:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:02:58.228+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:02:58.911 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:02:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:02:58.892+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:02:58.892983+0000) 2024-02-20T09:02:58.911 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:02:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:02:58.800+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:02:58.911 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:02:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:02:58.800+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:02:58.912 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:02:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:02:58.800+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:02:58.912 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:02:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:02:58.800+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:02:58.912 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:02:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:02:58.644+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:02:58.912 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:02:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:02:58.644+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:02:58.912 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:02:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:02:58.644+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:02:58.912 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:02:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:02:58.775+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:02:58.775984+0000) 2024-02-20T09:02:59.190 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:02:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:02:58.908+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:02:58.909268+0000) 2024-02-20T09:02:59.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:02:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:02:59.196+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:02:59.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:02:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:02:59.196+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:02:59.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:02:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:02:59.196+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:02:59.894 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:02:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:02:59.854+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 54330 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:02:59.895 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:02:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:02:59.848+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:02:59.850009+0000) 2024-02-20T09:02:59.895 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:02:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:02:59.639+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:02:59.895 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:02:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:02:59.639+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:02:59.895 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:02:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:02:59.639+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:02:59.896 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:02:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:02:59.774+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:02:59.776155+0000) 2024-02-20T09:02:59.896 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:02:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:02:59.847+0000 7f8a5f86e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:02:59.849131+0000) 2024-02-20T09:02:59.896 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:02:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:02:59.770+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:02:59.896 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:02:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:02:59.770+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:02:59.896 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:02:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:02:59.770+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:02:59.896 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:02:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:02:59.770+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:02:59.896 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:02:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:02:59.850+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:02:59.852098+0000) 2024-02-20T09:03:00.184 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:02:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:02:59.908+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:02:59.909617+0000) 2024-02-20T09:03:00.184 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:02:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:02:59.891+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:02:59.893134+0000) 2024-02-20T09:03:00.184 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:03:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:03:00.181+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:03:00.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:03:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:03:00.181+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:03:00.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:03:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:03:00.181+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:03:00.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:03:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:03:00.892+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:00.893291+0000) 2024-02-20T09:03:00.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:03:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:03:00.668+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:03:00.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:03:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:03:00.668+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:03:00.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:03:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:03:00.668+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:03:00.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:03:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:03:00.774+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:00.776300+0000) 2024-02-20T09:03:00.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:03:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:03:00.767+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:03:00.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:03:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:03:00.767+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:03:00.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:03:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:03:00.767+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:03:00.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:03:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:03:00.767+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:03:00.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:03:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:03:00.910+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:00.911905+0000) 2024-02-20T09:03:01.422 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:03:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:03:01.184+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:03:01.422 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:03:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:03:01.184+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:03:01.422 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:03:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:03:01.184+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:03:01.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:03:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:03:01.650+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:03:01.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:03:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:03:01.650+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:03:01.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:03:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:03:01.650+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:03:02.193 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:03:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:03:01.775+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:01.776462+0000) 2024-02-20T09:03:02.193 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:03:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:03:01.892+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:01.893464+0000) 2024-02-20T09:03:02.194 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:03:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:03:01.782+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:03:02.194 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:03:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:03:01.782+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:03:02.194 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:03:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:03:01.782+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:03:02.194 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:03:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:03:01.782+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:03:02.194 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:03:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:03:01.913+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:01.914788+0000) 2024-02-20T09:03:02.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:03:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:03:02.214+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:03:02.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:03:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:03:02.214+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:03:02.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:03:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:03:02.214+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:03:02.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:03:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:03:02.892+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:02.893597+0000) 2024-02-20T09:03:02.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:03:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:03:02.756+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:03:02.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:03:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:03:02.756+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:03:02.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:03:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:03:02.756+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:03:02.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:03:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:03:02.756+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:03:02.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:03:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:03:02.915+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:02.917027+0000) 2024-02-20T09:03:02.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:03:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:03:02.673+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:03:02.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:03:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:03:02.673+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:03:02.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:03:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:03:02.673+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:03:02.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:03:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:03:02.775+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:02.776600+0000) 2024-02-20T09:03:03.422 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:03:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:03:03.236+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:03:03.422 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:03:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:03:03.236+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:03:03.423 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:03:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:03:03.236+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:03:03.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:03:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:03:03.666+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:03:03.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:03:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:03:03.666+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:03:03.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:03:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:03:03.666+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:03:04.016 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:03:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:03:03.775+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:03.776780+0000) 2024-02-20T09:03:04.017 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:03:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:03:03.892+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:03.893784+0000) 2024-02-20T09:03:04.017 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:03:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:03:03.743+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:03:04.018 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:03:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:03:03.743+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:03:04.018 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:03:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:03:03.743+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:03:04.018 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:03:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:03:03.743+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:03:04.018 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:03:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:03:03.915+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:03.917348+0000) 2024-02-20T09:03:04.423 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:03:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:03:04.199+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:03:04.423 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:03:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:03:04.199+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:03:04.423 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:03:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:03:04.199+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:03:04.986 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:03:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:03:04.862+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 54403 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:03:04.987 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:03:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:03:04.856+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:04.858027+0000) 2024-02-20T09:03:04.987 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:03:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:03:04.892+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:04.893905+0000) 2024-02-20T09:03:04.987 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:03:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:03:04.676+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:03:04.988 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:03:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:03:04.676+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:03:04.988 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:03:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:03:04.676+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:03:04.988 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:03:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:03:04.776+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:04.776966+0000) 2024-02-20T09:03:04.988 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:03:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:03:04.856+0000 7f8a6006f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:04.857642+0000) 2024-02-20T09:03:04.988 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:03:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:03:04.735+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:03:04.989 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:03:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:03:04.735+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:03:04.989 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:03:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:03:04.735+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:03:04.989 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:03:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:03:04.735+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:03:04.989 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:03:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:03:04.859+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:04.860546+0000) 2024-02-20T09:03:04.989 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:03:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:03:04.916+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:04.917635+0000) 2024-02-20T09:03:05.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:03:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:03:05.232+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:03:05.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:03:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:03:05.232+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:03:05.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:03:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:03:05.232+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:03:05.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:03:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:03:05.892+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:05.894063+0000) 2024-02-20T09:03:05.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:03:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:03:05.704+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:03:05.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:03:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:03:05.704+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:03:05.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:03:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:03:05.704+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:03:05.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:03:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:03:05.776+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:05.777155+0000) 2024-02-20T09:03:05.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:03:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:03:05.753+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:03:05.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:03:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:03:05.753+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:03:05.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:03:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:03:05.753+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:03:05.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:03:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:03:05.753+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:03:05.995 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:03:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:03:05.920+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:05.921342+0000) 2024-02-20T09:03:06.423 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:03:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:03:06.260+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:03:06.423 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:03:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:03:06.260+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:03:06.423 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:03:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:03:06.260+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:03:06.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:03:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:03:06.893+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:06.894245+0000) 2024-02-20T09:03:06.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:03:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:03:06.733+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:03:06.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:03:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:03:06.733+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:03:06.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:03:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:03:06.733+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:03:06.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:03:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:03:06.776+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:06.777308+0000) 2024-02-20T09:03:06.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:03:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:03:06.800+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:03:06.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:03:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:03:06.800+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:03:06.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:03:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:03:06.800+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:03:06.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:03:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:03:06.800+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:03:06.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:03:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:03:06.922+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:06.924282+0000) 2024-02-20T09:03:07.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:03:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:03:07.307+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:03:07.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:03:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:03:07.307+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:03:07.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:03:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:03:07.307+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:03:08.017 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:03:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:03:07.893+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:07.894433+0000) 2024-02-20T09:03:08.017 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:03:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:03:07.752+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:03:08.017 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:03:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:03:07.752+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:03:08.018 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:03:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:03:07.752+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:03:08.018 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:03:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:03:07.776+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:07.777499+0000) 2024-02-20T09:03:08.018 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:03:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:03:07.825+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:03:08.018 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:03:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:03:07.825+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:03:08.018 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:03:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:03:07.825+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:03:08.018 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:03:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:03:07.825+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:03:08.018 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:03:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:03:07.926+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:07.927299+0000) 2024-02-20T09:03:08.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:03:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:03:08.311+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:03:08.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:03:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:03:08.311+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:03:08.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:03:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:03:08.311+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:03:09.197 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:03:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:03:08.893+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:08.894610+0000) 2024-02-20T09:03:09.198 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:03:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:03:08.776+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:08.777679+0000) 2024-02-20T09:03:09.198 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:03:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:03:08.797+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:03:09.198 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:03:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:03:08.797+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:03:09.198 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:03:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:03:08.797+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:03:09.199 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:03:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:03:08.788+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:03:09.199 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:03:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:03:08.788+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:03:09.199 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:03:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:03:08.788+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:03:09.199 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:03:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:03:08.789+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:03:09.199 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:03:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:03:08.926+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:08.927588+0000) 2024-02-20T09:03:09.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:03:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:03:09.307+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:03:09.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:03:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:03:09.307+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:03:09.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:03:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:03:09.307+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:03:10.017 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:03:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:03:09.870+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 54474 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:03:10.018 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:03:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:03:09.864+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:09.865507+0000) 2024-02-20T09:03:10.018 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:03:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:03:09.893+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:09.894734+0000) 2024-02-20T09:03:10.018 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:03:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:03:09.742+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:03:10.019 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:03:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:03:09.742+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:03:10.019 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:03:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:03:09.742+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:03:10.019 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:03:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:03:09.742+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:03:10.019 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:03:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:03:09.867+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:09.868748+0000) 2024-02-20T09:03:10.019 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:03:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:03:09.926+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:09.927808+0000) 2024-02-20T09:03:10.019 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:03:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:03:09.777+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:09.777874+0000) 2024-02-20T09:03:10.020 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:03:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:03:09.838+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:03:10.020 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:03:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:03:09.838+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:03:10.020 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:03:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:03:09.838+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:03:10.020 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:03:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:03:09.864+0000 7f8a5f86e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:09.865776+0000) 2024-02-20T09:03:10.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:03:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:03:10.328+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:03:10.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:03:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:03:10.328+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:03:10.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:03:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:03:10.328+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:03:11.201 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:03:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:03:10.894+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:10.894878+0000) 2024-02-20T09:03:11.202 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:03:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:03:10.776+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:10.778094+0000) 2024-02-20T09:03:11.202 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:03:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:03:10.792+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:03:11.202 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:03:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:03:10.792+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:03:11.202 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:03:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:03:10.792+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:03:11.203 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:03:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:03:10.779+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:03:11.203 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:03:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:03:10.779+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:03:11.203 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:03:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:03:10.779+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:03:11.203 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:03:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:03:10.779+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:03:11.204 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:03:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:03:10.929+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:10.930457+0000) 2024-02-20T09:03:11.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:03:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:03:11.319+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:03:11.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:03:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:03:11.319+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:03:11.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:03:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:03:11.319+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:03:12.121 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:03:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:03:11.894+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:11.895105+0000) 2024-02-20T09:03:12.121 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:03:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:03:11.776+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:11.778214+0000) 2024-02-20T09:03:12.122 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:03:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:03:11.812+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:03:12.122 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:03:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:03:11.812+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:03:12.122 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:03:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:03:11.812+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:03:12.122 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:03:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:03:11.776+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:03:12.122 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:03:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:03:11.776+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:03:12.122 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:03:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:03:11.776+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:03:12.122 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:03:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:03:11.776+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:03:12.123 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:03:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:03:11.932+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:11.933439+0000) 2024-02-20T09:03:12.424 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:03:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:03:12.357+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:03:12.424 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:03:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:03:12.357+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:03:12.424 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:03:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:03:12.357+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:03:13.201 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:03:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:03:12.894+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:12.895298+0000) 2024-02-20T09:03:13.202 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:03:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:03:12.777+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:12.778414+0000) 2024-02-20T09:03:13.202 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:03:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:03:12.857+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:03:13.202 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:03:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:03:12.857+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:03:13.202 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:03:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:03:12.857+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:03:13.202 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:03:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:03:12.791+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:03:13.202 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:03:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:03:12.791+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:03:13.202 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:03:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:03:12.791+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:03:13.203 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:03:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:03:12.791+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:03:13.203 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:03:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:03:12.935+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:12.935840+0000) 2024-02-20T09:03:13.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:03:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:03:13.380+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:03:13.514 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:03:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:03:13.380+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:03:13.515 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:03:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:03:13.380+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:03:14.203 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:03:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:03:13.894+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:13.895533+0000) 2024-02-20T09:03:14.204 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:03:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:03:13.777+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:13.778592+0000) 2024-02-20T09:03:14.204 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:03:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:03:13.821+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:03:14.204 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:03:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:03:13.821+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:03:14.204 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:03:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:03:13.821+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:03:14.205 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:03:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:03:13.813+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:03:14.205 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:03:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:03:13.813+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:03:14.205 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:03:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:03:13.813+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:03:14.205 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:03:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:03:13.813+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:03:14.205 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:03:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:03:13.934+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:13.936110+0000) 2024-02-20T09:03:14.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:03:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:03:14.388+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:03:14.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:03:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:03:14.388+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:03:14.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:03:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:03:14.388+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:03:15.202 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:03:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:03:14.878+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 54547 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:03:15.202 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:03:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:03:14.871+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:14.873120+0000) 2024-02-20T09:03:15.202 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:03:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:03:14.894+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:14.895711+0000) 2024-02-20T09:03:15.203 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:03:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:03:14.777+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:14.778746+0000) 2024-02-20T09:03:15.203 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:03:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:03:14.837+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:03:15.203 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:03:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:03:14.837+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:03:15.203 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:03:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:03:14.837+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:03:15.203 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:03:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:03:14.871+0000 7f8a5f86e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:14.873369+0000) 2024-02-20T09:03:15.203 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:03:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:03:14.792+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:03:15.203 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:03:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:03:14.792+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:03:15.204 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:03:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:03:14.792+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:03:15.204 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:03:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:03:14.792+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:03:15.204 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:03:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:03:14.875+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:14.876732+0000) 2024-02-20T09:03:15.204 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:03:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:03:14.935+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:14.936393+0000) 2024-02-20T09:03:15.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:03:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:03:15.390+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:03:15.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:03:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:03:15.390+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:03:15.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:03:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:03:15.390+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:03:16.206 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:03:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:03:15.894+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:15.895805+0000) 2024-02-20T09:03:16.207 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:03:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:03:15.841+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:03:16.207 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:03:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:03:15.841+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:03:16.207 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:03:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:03:15.841+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:03:16.207 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:03:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:03:15.841+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:03:16.207 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:03:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:03:15.938+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:15.939098+0000) 2024-02-20T09:03:16.207 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:03:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:03:15.778+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:15.778913+0000) 2024-02-20T09:03:16.207 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:03:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:03:15.797+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:03:16.208 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:03:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:03:15.797+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:03:16.208 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:03:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:03:15.797+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:03:16.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:03:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:03:16.368+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:03:16.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:03:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:03:16.368+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:03:16.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:03:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:03:16.368+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:03:17.206 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:03:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:03:16.894+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:16.895983+0000) 2024-02-20T09:03:17.207 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:03:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:03:16.889+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:03:17.207 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:03:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:03:16.889+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:03:17.207 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:03:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:03:16.889+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:03:17.207 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:03:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:03:16.889+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:03:17.207 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:03:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:03:16.941+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:16.941982+0000) 2024-02-20T09:03:17.207 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:03:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:03:16.778+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:16.779019+0000) 2024-02-20T09:03:17.208 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:03:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:03:16.785+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:03:17.208 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:03:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:03:16.785+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:03:17.208 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:03:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:03:16.785+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:03:17.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:03:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:03:17.390+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:03:17.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:03:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:03:17.390+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:03:17.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:03:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:03:17.390+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:03:18.206 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:03:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:03:17.894+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:17.896164+0000) 2024-02-20T09:03:18.207 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:03:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:03:17.928+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:03:18.207 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:03:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:03:17.928+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:03:18.207 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:03:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:03:17.928+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:03:18.207 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:03:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:03:17.928+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:03:18.207 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:03:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:03:17.943+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:17.944604+0000) 2024-02-20T09:03:18.208 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:03:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:03:17.777+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:17.779208+0000) 2024-02-20T09:03:18.208 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:03:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:03:17.801+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:03:18.208 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:03:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:03:17.801+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:03:18.208 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:03:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:03:17.801+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:03:18.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:03:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:03:18.416+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:03:18.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:03:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:03:18.416+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:03:18.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:03:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:03:18.416+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:03:19.206 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:03:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:03:18.895+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:18.896321+0000) 2024-02-20T09:03:19.207 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:03:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:03:18.916+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:03:19.207 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:03:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:03:18.916+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:03:19.207 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:03:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:03:18.916+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:03:19.207 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:03:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:03:18.916+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:03:19.208 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:03:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:03:18.943+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:18.944854+0000) 2024-02-20T09:03:19.208 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:03:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:03:18.778+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:18.779369+0000) 2024-02-20T09:03:19.208 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:03:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:03:18.801+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:03:19.208 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:03:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:03:18.801+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:03:19.208 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:03:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:03:18.801+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:03:19.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:03:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:03:19.392+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:03:19.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:03:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:03:19.392+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:03:19.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:03:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:03:19.392+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:03:20.012 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:03:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:03:19.886+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 54617 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:03:20.013 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:03:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:03:19.879+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:19.881079+0000) 2024-02-20T09:03:20.013 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:03:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:03:19.895+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:19.896499+0000) 2024-02-20T09:03:20.013 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:03:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:03:19.756+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:03:20.014 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:03:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:03:19.756+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:03:20.014 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:03:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:03:19.756+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:03:20.014 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:03:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:03:19.779+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:19.779589+0000) 2024-02-20T09:03:20.014 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:03:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:03:19.880+0000 7f8a5f86e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:19.881652+0000) 2024-02-20T09:03:20.014 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:03:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:03:19.884+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:19.885469+0000) 2024-02-20T09:03:20.014 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:03:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:03:19.944+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:19.945196+0000) 2024-02-20T09:03:20.015 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:03:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:03:19.966+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:03:20.015 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:03:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:03:19.966+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:03:20.015 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:03:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:03:19.966+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:03:20.015 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:03:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:03:19.966+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:03:20.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:03:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:03:20.438+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:03:20.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:03:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:03:20.438+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:03:20.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:03:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:03:20.438+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:03:21.210 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:03:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:03:20.896+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:20.896637+0000) 2024-02-20T09:03:21.210 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:03:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:03:20.779+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:20.779729+0000) 2024-02-20T09:03:21.211 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:03:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:03:20.795+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:03:21.211 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:03:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:03:20.795+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:03:21.211 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:03:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:03:20.795+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:03:21.211 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:03:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:03:20.947+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:20.947759+0000) 2024-02-20T09:03:21.211 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:03:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:03:20.982+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:03:21.211 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:03:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:03:20.982+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:03:21.211 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:03:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:03:20.982+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:03:21.212 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:03:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:03:20.982+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:03:21.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:03:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:03:21.473+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:03:21.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:03:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:03:21.473+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:03:21.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:03:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:03:21.473+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:03:22.210 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:03:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:03:21.895+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:21.896791+0000) 2024-02-20T09:03:22.211 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:03:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:03:21.950+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:21.950586+0000) 2024-02-20T09:03:22.211 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:03:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:03:21.985+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:03:22.211 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:03:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:03:21.985+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:03:22.211 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:03:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:03:21.985+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:03:22.211 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:03:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:03:21.985+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:03:22.211 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:03:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:03:21.778+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:21.779820+0000) 2024-02-20T09:03:22.212 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:03:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:03:21.804+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:03:22.212 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:03:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:03:21.804+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:03:22.212 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:03:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:03:21.804+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:03:22.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:03:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:03:22.476+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:03:22.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:03:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:03:22.476+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:03:22.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:03:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:03:22.476+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:03:23.210 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:03:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:03:22.896+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:22.896934+0000) 2024-02-20T09:03:23.211 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:03:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:03:22.779+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:22.780018+0000) 2024-02-20T09:03:23.211 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:03:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:03:22.803+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:03:23.211 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:03:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:03:22.803+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:03:23.211 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:03:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:03:22.803+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:03:23.211 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:03:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:03:22.953+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:22.953816+0000) 2024-02-20T09:03:23.211 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:03:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:03:23.021+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:03:23.212 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:03:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:03:23.021+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:03:23.212 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:03:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:03:23.021+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:03:23.212 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:03:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:03:23.022+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:03:23.781 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:03:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:03:23.769+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:03:23.782 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:03:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:03:23.769+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:03:23.782 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:03:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:03:23.769+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:03:23.782 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:03:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:03:23.525+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:03:23.782 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:03:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:03:23.525+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:03:23.782 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:03:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:03:23.525+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:03:24.050 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:03:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:03:23.779+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:23.780210+0000) 2024-02-20T09:03:24.050 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:03:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:03:23.896+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:23.897122+0000) 2024-02-20T09:03:24.051 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:03:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:03:23.953+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:23.954145+0000) 2024-02-20T09:03:24.051 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:03:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:03:24.047+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:03:24.051 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:03:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:03:24.047+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:03:24.051 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:03:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:03:24.047+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:03:24.051 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:03:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:03:24.047+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:03:24.729 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:03:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:03:24.534+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:03:24.729 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:03:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:03:24.534+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:03:24.730 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:03:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:03:24.534+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:03:24.993 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:03:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:03:24.894+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 54690 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:03:24.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:03:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:03:24.887+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:24.889234+0000) 2024-02-20T09:03:24.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:03:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:03:24.895+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:24.897274+0000) 2024-02-20T09:03:24.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:03:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:03:24.891+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:24.892681+0000) 2024-02-20T09:03:24.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:03:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:03:24.953+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:24.954381+0000) 2024-02-20T09:03:24.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:03:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:03:24.727+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:03:24.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:03:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:03:24.727+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:03:24.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:03:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:03:24.727+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:03:24.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:03:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:03:24.778+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:24.780373+0000) 2024-02-20T09:03:24.995 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:03:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:03:24.888+0000 7f8a6006f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:24.889932+0000) 2024-02-20T09:03:25.427 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:03:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:03:25.031+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:03:25.427 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:03:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:03:25.031+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:03:25.427 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:03:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:03:25.031+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:03:25.428 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:03:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:03:25.031+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:03:25.700 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:03:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:03:25.559+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:03:25.701 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:03:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:03:25.559+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:03:25.701 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:03:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:03:25.559+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:03:25.958 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:03:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:03:25.897+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:25.897392+0000) 2024-02-20T09:03:25.958 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:03:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:03:25.698+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:03:25.958 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:03:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:03:25.698+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:03:25.959 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:03:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:03:25.698+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:03:25.959 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:03:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:03:25.780+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:25.780568+0000) 2024-02-20T09:03:26.214 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:03:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:03:25.956+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:25.956782+0000) 2024-02-20T09:03:26.214 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:03:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:03:26.056+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:03:26.214 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:03:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:03:26.057+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:03:26.214 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:03:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:03:26.057+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:03:26.214 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:03:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:03:26.057+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:03:26.899 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:03:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:03:26.722+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:03:26.899 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:03:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:03:26.722+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:03:26.899 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:03:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:03:26.722+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:03:26.899 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:03:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:03:26.780+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:26.780714+0000) 2024-02-20T09:03:26.899 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:03:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:03:26.603+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:03:26.900 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:03:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:03:26.603+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:03:26.900 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:03:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:03:26.603+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:03:27.214 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:03:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:03:26.897+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:26.897527+0000) 2024-02-20T09:03:27.215 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:03:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:03:26.959+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:26.959609+0000) 2024-02-20T09:03:27.215 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:03:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:03:27.039+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:03:27.215 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:03:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:03:27.039+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:03:27.215 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:03:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:03:27.039+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:03:27.215 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:03:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:03:27.039+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:03:27.899 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:03:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:03:27.691+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:03:27.899 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:03:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:03:27.691+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:03:27.900 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:03:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:03:27.691+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:03:27.900 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:03:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:03:27.780+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:27.780842+0000) 2024-02-20T09:03:27.900 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:03:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:03:27.595+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:03:27.900 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:03:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:03:27.595+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:03:27.900 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:03:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:03:27.595+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:03:28.214 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:03:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:03:27.897+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:27.897644+0000) 2024-02-20T09:03:28.215 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:03:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:03:27.961+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:27.962008+0000) 2024-02-20T09:03:28.215 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:03:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:03:28.027+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:03:28.215 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:03:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:03:28.027+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:03:28.215 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:03:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:03:28.027+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:03:28.215 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:03:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:03:28.028+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:03:28.899 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:03:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:03:28.692+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:03:28.899 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:03:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:03:28.692+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:03:28.900 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:03:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:03:28.692+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:03:28.900 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:03:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:03:28.780+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:28.781007+0000) 2024-02-20T09:03:28.900 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:03:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:03:28.589+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:03:28.900 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:03:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:03:28.589+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:03:28.900 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:03:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:03:28.589+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:03:29.214 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:03:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:03:28.897+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:28.897812+0000) 2024-02-20T09:03:29.215 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:03:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:03:28.961+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:28.962379+0000) 2024-02-20T09:03:29.215 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:03:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:03:29.073+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:03:29.215 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:03:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:03:29.073+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:03:29.215 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:03:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:03:29.073+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:03:29.215 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:03:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:03:29.073+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:03:29.886 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:03:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:03:29.575+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:03:29.886 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:03:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:03:29.575+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:03:29.886 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:03:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:03:29.575+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:03:29.886 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:03:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:03:29.733+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:03:29.887 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:03:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:03:29.733+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:03:29.887 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:03:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:03:29.733+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:03:29.887 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:03:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:03:29.780+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:29.781198+0000) 2024-02-20T09:03:30.215 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:03:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:03:29.904+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 54760 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:03:30.215 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:03:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:03:29.897+0000 7f8a5f86e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:29.897640+0000) 2024-02-20T09:03:30.216 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:03:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:03:29.897+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:29.897745+0000) 2024-02-20T09:03:30.216 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:03:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:03:29.897+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:29.897956+0000) 2024-02-20T09:03:30.216 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:03:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:03:29.900+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:29.901060+0000) 2024-02-20T09:03:30.216 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:03:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:03:29.962+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:29.962681+0000) 2024-02-20T09:03:30.216 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:03:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:03:30.032+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:03:30.216 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:03:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:03:30.032+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:03:30.217 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:03:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:03:30.032+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:03:30.217 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:03:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:03:30.032+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:03:30.783 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:03:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:03:30.699+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:03:30.783 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:03:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:03:30.699+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:03:30.783 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:03:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:03:30.699+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:03:30.783 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:03:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:03:30.528+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:03:30.783 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:03:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:03:30.528+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:03:30.783 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:03:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:03:30.528+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:03:31.219 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:03:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:03:30.781+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:30.781350+0000) 2024-02-20T09:03:31.219 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:03:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:03:30.897+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:30.898115+0000) 2024-02-20T09:03:31.219 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:03:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:03:30.964+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:30.965267+0000) 2024-02-20T09:03:31.219 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:03:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:03:30.998+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:03:31.220 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:03:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:03:30.998+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:03:31.220 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:03:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:03:30.998+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:03:31.220 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:03:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:03:30.999+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:03:31.783 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:03:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:03:31.657+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:03:31.783 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:03:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:03:31.657+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:03:31.783 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:03:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:03:31.657+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:03:31.783 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:03:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:03:31.505+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:03:31.784 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:03:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:03:31.505+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:03:31.784 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:03:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:03:31.505+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:03:32.219 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:03:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:03:31.781+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:31.781506+0000) 2024-02-20T09:03:32.219 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:03:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:03:31.897+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:31.898273+0000) 2024-02-20T09:03:32.220 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:03:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:03:31.967+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:31.968227+0000) 2024-02-20T09:03:32.220 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:03:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:03:31.999+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:03:32.220 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:03:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:03:31.999+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:03:32.220 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:03:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:03:31.999+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:03:32.220 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:03:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:03:32.000+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:03:32.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:03:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:03:32.486+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:03:32.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:03:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:03:32.486+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:03:32.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:03:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:03:32.486+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:03:32.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:03:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:03:32.635+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:03:32.744 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:03:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:03:32.635+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:03:32.744 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:03:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:03:32.635+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:03:33.219 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:03:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:03:32.781+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:32.781721+0000) 2024-02-20T09:03:33.220 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:03:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:03:32.898+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:32.898424+0000) 2024-02-20T09:03:33.220 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:03:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:03:32.970+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:32.970658+0000) 2024-02-20T09:03:33.220 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:03:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:03:32.982+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:03:33.220 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:03:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:03:32.982+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:03:33.220 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:03:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:03:32.982+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:03:33.220 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:03:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:03:32.982+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:03:33.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:03:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:03:33.487+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:03:33.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:03:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:03:33.487+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:03:33.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:03:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:03:33.487+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:03:33.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:03:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:03:33.646+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:03:33.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:03:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:03:33.646+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:03:33.744 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:03:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:03:33.646+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:03:34.219 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:03:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:03:33.781+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:33.781881+0000) 2024-02-20T09:03:34.220 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:03:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:03:33.898+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:33.898591+0000) 2024-02-20T09:03:34.220 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:03:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:03:33.970+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:33.971005+0000) 2024-02-20T09:03:34.220 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:03:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:03:33.992+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:03:34.220 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:03:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:03:33.992+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:03:34.221 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:03:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:03:33.992+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:03:34.221 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:03:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:03:33.992+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:03:34.783 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:03:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:03:34.502+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:03:34.784 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:03:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:03:34.502+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:03:34.784 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:03:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:03:34.502+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:03:34.784 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:03:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:03:34.632+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:03:34.784 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:03:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:03:34.632+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:03:34.784 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:03:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:03:34.632+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:03:35.039 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:03:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:03:34.912+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 54833 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:03:35.040 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:03:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:03:34.781+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:34.782042+0000) 2024-02-20T09:03:35.040 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:03:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:03:34.906+0000 7f8a5f86e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:34.906719+0000) 2024-02-20T09:03:35.040 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:03:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:03:34.898+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:34.898766+0000) 2024-02-20T09:03:35.040 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:03:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:03:34.905+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:34.906063+0000) 2024-02-20T09:03:35.041 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:03:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:03:34.909+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:34.909995+0000) 2024-02-20T09:03:35.041 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:03:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:03:34.970+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:34.971340+0000) 2024-02-20T09:03:35.041 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:03:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:03:35.037+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:03:35.041 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:03:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:03:35.037+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:03:35.041 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:03:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:03:35.037+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:03:35.041 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:03:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:03:35.037+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:03:35.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:03:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:03:35.660+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:03:35.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:03:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:03:35.660+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:03:35.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:03:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:03:35.660+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:03:35.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:03:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:03:35.522+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:03:35.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:03:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:03:35.522+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:03:35.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:03:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:03:35.522+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:03:36.223 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:03:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:03:35.781+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:35.782187+0000) 2024-02-20T09:03:36.223 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:03:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:03:35.898+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:35.898977+0000) 2024-02-20T09:03:36.223 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:03:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:03:35.973+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:35.973894+0000) 2024-02-20T09:03:36.224 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:03:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:03:36.016+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:03:36.224 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:03:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:03:36.016+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:03:36.224 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:03:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:03:36.016+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:03:36.224 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:03:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:03:36.016+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:03:36.900 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:03:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:03:36.565+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:03:36.901 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:03:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:03:36.565+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:03:36.901 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:03:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:03:36.565+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:03:36.901 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:03:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:03:36.681+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:03:36.901 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:03:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:03:36.681+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:03:36.901 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:03:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:03:36.681+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:03:36.901 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:03:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:03:36.782+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:36.782337+0000) 2024-02-20T09:03:37.223 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:03:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:03:36.898+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:36.899136+0000) 2024-02-20T09:03:37.223 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:03:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:03:36.976+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:36.976722+0000) 2024-02-20T09:03:37.223 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:03:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:03:37.008+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:03:37.223 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:03:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:03:37.008+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:03:37.224 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:03:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:03:37.008+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:03:37.224 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:03:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:03:37.008+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:03:37.784 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:03:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:03:37.693+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:03:37.784 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:03:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:03:37.693+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:03:37.784 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:03:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:03:37.693+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:03:37.785 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:03:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:03:37.527+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:03:37.785 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:03:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:03:37.527+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:03:37.785 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:03:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:03:37.527+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:03:38.223 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:03:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:03:37.782+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:37.782587+0000) 2024-02-20T09:03:38.224 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:03:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:03:37.898+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:37.899284+0000) 2024-02-20T09:03:38.224 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:03:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:03:37.978+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:37.979165+0000) 2024-02-20T09:03:38.224 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:03:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:03:38.025+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:03:38.224 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:03:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:03:38.025+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:03:38.224 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:03:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:03:38.025+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:03:38.224 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:03:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:03:38.026+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:03:38.784 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:03:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:03:38.499+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:03:38.784 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:03:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:03:38.499+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:03:38.784 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:03:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:03:38.499+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:03:38.784 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:03:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:03:38.667+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:03:38.785 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:03:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:03:38.667+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:03:38.785 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:03:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:03:38.667+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:03:39.223 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:03:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:03:38.782+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:38.782734+0000) 2024-02-20T09:03:39.224 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:03:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:03:38.899+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:38.899439+0000) 2024-02-20T09:03:39.224 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:03:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:03:38.979+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:38.979490+0000) 2024-02-20T09:03:39.224 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:03:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:03:39.008+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:03:39.224 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:03:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:03:39.008+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:03:39.225 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:03:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:03:39.008+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:03:39.225 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:03:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:03:39.008+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:03:39.784 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:03:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:03:39.521+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:03:39.784 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:03:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:03:39.521+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:03:39.784 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:03:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:03:39.521+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:03:39.785 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:03:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:03:39.673+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:03:39.785 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:03:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:03:39.673+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:03:39.785 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:03:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:03:39.673+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:03:40.043 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:03:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:03:39.920+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 54904 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:03:40.043 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:03:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:03:39.782+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:39.782895+0000) 2024-02-20T09:03:40.043 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:03:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:03:39.913+0000 7f8a5f86e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:39.914200+0000) 2024-02-20T09:03:40.044 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:03:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:03:39.899+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:39.899599+0000) 2024-02-20T09:03:40.044 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:03:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:03:39.913+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:39.914253+0000) 2024-02-20T09:03:40.044 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:03:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:03:39.916+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:39.917268+0000) 2024-02-20T09:03:40.044 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:03:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:03:39.960+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:03:40.045 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:03:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:03:39.960+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:03:40.045 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:03:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:03:39.960+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:03:40.045 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:03:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:03:39.960+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:03:40.045 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:03:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:03:39.979+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:39.979740+0000) 2024-02-20T09:03:40.670 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:03:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:03:40.483+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:03:40.671 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:03:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:03:40.483+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:03:40.671 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:03:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:03:40.483+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:03:40.954 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:03:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:03:40.899+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:40.899760+0000) 2024-02-20T09:03:40.955 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:03:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:03:40.668+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:03:40.955 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:03:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:03:40.668+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:03:40.955 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:03:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:03:40.668+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:03:40.955 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:03:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:03:40.782+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:40.783060+0000) 2024-02-20T09:03:41.227 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:03:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:03:40.952+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:03:41.227 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:03:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:03:40.952+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:03:41.227 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:03:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:03:40.952+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:03:41.227 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:03:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:03:40.952+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:03:41.227 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:03:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:03:40.982+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:40.982327+0000) 2024-02-20T09:03:41.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:03:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:03:41.450+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:03:41.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:03:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:03:41.450+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:03:41.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:03:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:03:41.450+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:03:41.901 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:03:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:03:41.637+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:03:41.901 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:03:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:03:41.637+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:03:41.902 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:03:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:03:41.637+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:03:41.902 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:03:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:03:41.782+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:41.783230+0000) 2024-02-20T09:03:42.227 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:03:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:03:41.899+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:41.899883+0000) 2024-02-20T09:03:42.228 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:03:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:03:41.928+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:03:42.228 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:03:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:03:41.928+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:03:42.228 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:03:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:03:41.928+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:03:42.228 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:03:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:03:41.928+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:03:42.228 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:03:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:03:41.985+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:41.985148+0000) 2024-02-20T09:03:42.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:03:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:03:42.409+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:03:42.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:03:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:03:42.409+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:03:42.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:03:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:03:42.409+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:03:42.978 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:03:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:03:42.899+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:42.900028+0000) 2024-02-20T09:03:42.978 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:03:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:03:42.662+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:03:42.978 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:03:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:03:42.662+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:03:42.978 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:03:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:03:42.662+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:03:42.978 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:03:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:03:42.783+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:42.783450+0000) 2024-02-20T09:03:43.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:03:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:03:42.976+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:03:43.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:03:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:03:42.976+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:03:43.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:03:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:03:42.976+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:03:43.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:03:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:03:42.976+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:03:43.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:03:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:03:42.987+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:42.987598+0000) 2024-02-20T09:03:43.714 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:03:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:03:43.451+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:03:43.714 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:03:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:03:43.451+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:03:43.714 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:03:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:03:43.451+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:03:43.989 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:03:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:03:43.900+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:43.900209+0000) 2024-02-20T09:03:43.989 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:03:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:03:43.712+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:03:43.989 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:03:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:03:43.712+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:03:43.990 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:03:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:03:43.712+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:03:43.990 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:03:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:03:43.783+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:43.783657+0000) 2024-02-20T09:03:44.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:03:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:03:43.987+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:43.987867+0000) 2024-02-20T09:03:44.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:03:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:03:44.017+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:03:44.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:03:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:03:44.017+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:03:44.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:03:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:03:44.017+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:03:44.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:03:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:03:44.017+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:03:44.688 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:03:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:03:44.455+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:03:44.688 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:03:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:03:44.455+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:03:44.689 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:03:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:03:44.455+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:03:44.947 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:03:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:03:44.927+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 54977 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:03:44.948 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:03:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:03:44.925+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:44.925538+0000) 2024-02-20T09:03:44.948 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:03:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:03:44.900+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:44.900425+0000) 2024-02-20T09:03:44.948 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:03:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:03:44.921+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:44.922152+0000) 2024-02-20T09:03:44.949 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:03:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:03:44.686+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:03:44.949 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:03:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:03:44.686+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:03:44.949 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:03:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:03:44.686+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:03:44.949 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:03:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:03:44.783+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:44.783798+0000) 2024-02-20T09:03:44.949 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:03:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:03:44.922+0000 7f8a5f86e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:44.922562+0000) 2024-02-20T09:03:45.228 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:03:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:03:44.987+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:44.987983+0000) 2024-02-20T09:03:45.228 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:03:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:03:45.052+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:03:45.228 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:03:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:03:45.052+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:03:45.228 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:03:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:03:45.052+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:03:45.228 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:03:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:03:45.052+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:03:45.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:03:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:03:45.406+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:03:45.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:03:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:03:45.406+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:03:45.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:03:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:03:45.406+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:03:45.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:03:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:03:45.900+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:45.900620+0000) 2024-02-20T09:03:45.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:03:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:03:45.708+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:03:45.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:03:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:03:45.708+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:03:45.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:03:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:03:45.708+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:03:45.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:03:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:03:45.783+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:45.783958+0000) 2024-02-20T09:03:46.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:03:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:03:45.989+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:45.990251+0000) 2024-02-20T09:03:46.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:03:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:03:46.005+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:03:46.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:03:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:03:46.005+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:03:46.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:03:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:03:46.005+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:03:46.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:03:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:03:46.005+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:03:46.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:03:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:03:46.406+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:03:46.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:03:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:03:46.406+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:03:46.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:03:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:03:46.406+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:03:47.024 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:03:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:03:46.900+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:46.900762+0000) 2024-02-20T09:03:47.025 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:03:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:03:46.992+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:46.993264+0000) 2024-02-20T09:03:47.025 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:03:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:03:46.743+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:03:47.025 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:03:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:03:46.743+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:03:47.025 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:03:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:03:46.743+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:03:47.025 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:03:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:03:46.783+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:46.784055+0000) 2024-02-20T09:03:47.429 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:03:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:03:47.050+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:03:47.430 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:03:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:03:47.050+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:03:47.430 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:03:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:03:47.050+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:03:47.430 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:03:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:03:47.050+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:03:47.727 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:03:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:03:47.448+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:03:47.727 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:03:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:03:47.448+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:03:47.727 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:03:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:03:47.448+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:03:47.992 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:03:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:03:47.899+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:47.900970+0000) 2024-02-20T09:03:47.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:03:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:03:47.724+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:03:47.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:03:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:03:47.724+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:03:47.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:03:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:03:47.724+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:03:47.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:03:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:03:47.782+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:47.784242+0000) 2024-02-20T09:03:48.430 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:03:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:03:47.994+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:47.995626+0000) 2024-02-20T09:03:48.430 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:03:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:03:48.072+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:03:48.430 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:03:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:03:48.072+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:03:48.431 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:03:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:03:48.072+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:03:48.431 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:03:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:03:48.072+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:03:48.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:03:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:03:48.472+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:03:48.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:03:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:03:48.472+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:03:48.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:03:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:03:48.472+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:03:49.024 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:03:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:03:48.994+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:48.995844+0000) 2024-02-20T09:03:49.025 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:03:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:03:48.899+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:48.901145+0000) 2024-02-20T09:03:49.025 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:03:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:03:48.755+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:03:49.025 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:03:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:03:48.755+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:03:49.026 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:03:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:03:48.755+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:03:49.026 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:03:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:03:48.783+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:48.784432+0000) 2024-02-20T09:03:49.430 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:03:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:03:49.036+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:03:49.430 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:03:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:03:49.036+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:03:49.430 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:03:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:03:49.036+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:03:49.431 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:03:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:03:49.036+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:03:49.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:03:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:03:49.517+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:03:49.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:03:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:03:49.517+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:03:49.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:03:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:03:49.517+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:03:50.025 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:03:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:03:49.935+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 55048 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:03:50.025 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:03:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:03:49.899+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:49.901327+0000) 2024-02-20T09:03:50.025 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:03:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:03:49.928+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:49.929672+0000) 2024-02-20T09:03:50.026 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:03:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:03:49.754+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:03:50.026 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:03:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:03:49.754+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:03:50.026 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:03:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:03:49.754+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:03:50.026 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:03:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:03:49.783+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:49.784556+0000) 2024-02-20T09:03:50.026 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:03:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:03:49.928+0000 7f8a5f86e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:49.929593+0000) 2024-02-20T09:03:50.027 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:03:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:03:49.931+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:49.933223+0000) 2024-02-20T09:03:50.027 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:03:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:03:49.995+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:49.996086+0000) 2024-02-20T09:03:50.027 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:03:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:03:49.997+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:03:50.027 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:03:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:03:49.997+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:03:50.027 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:03:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:03:49.997+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:03:50.027 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:03:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:03:49.997+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:03:50.733 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:03:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:03:50.548+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:03:50.733 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:03:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:03:50.548+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:03:50.733 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:03:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:03:50.548+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:03:50.992 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:03:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:03:50.900+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:50.901456+0000) 2024-02-20T09:03:50.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:03:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:03:50.730+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:03:50.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:03:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:03:50.730+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:03:50.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:03:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:03:50.730+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:03:50.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:03:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:03:50.783+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:50.784759+0000) 2024-02-20T09:03:51.431 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:03:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:03:50.998+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:50.999708+0000) 2024-02-20T09:03:51.431 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:03:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:03:51.024+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:03:51.431 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:03:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:03:51.024+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:03:51.431 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:03:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:03:51.024+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:03:51.431 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:03:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:03:51.024+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:03:51.736 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:03:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:03:51.512+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:03:51.736 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:03:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:03:51.512+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:03:51.736 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:03:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:03:51.512+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:03:51.992 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:03:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:03:51.900+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:51.901589+0000) 2024-02-20T09:03:51.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:03:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:03:51.733+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:03:51.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:03:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:03:51.733+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:03:51.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:03:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:03:51.733+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:03:51.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:03:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:03:51.783+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:51.784912+0000) 2024-02-20T09:03:52.431 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:03:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:03:52.002+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:52.003467+0000) 2024-02-20T09:03:52.431 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:03:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:03:52.020+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:03:52.431 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:03:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:03:52.020+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:03:52.431 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:03:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:03:52.020+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:03:52.431 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:03:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:03:52.020+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:03:52.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:03:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:03:52.536+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:03:52.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:03:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:03:52.536+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:03:52.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:03:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:03:52.536+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:03:53.028 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:03:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:03:52.900+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:52.901813+0000) 2024-02-20T09:03:53.028 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:03:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:03:52.776+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:03:53.029 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:03:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:03:52.776+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:03:53.029 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:03:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:03:52.776+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:03:53.029 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:03:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:03:52.783+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:52.785049+0000) 2024-02-20T09:03:53.029 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:03:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:03:53.004+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:53.006026+0000) 2024-02-20T09:03:53.029 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:03:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:03:53.025+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:03:53.029 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:03:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:03:53.025+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:03:53.029 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:03:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:03:53.025+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:03:53.030 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:03:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:03:53.025+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:03:53.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:03:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:03:53.547+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:03:53.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:03:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:03:53.547+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:03:53.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:03:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:03:53.547+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:03:54.069 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:03:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:03:53.901+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:53.902005+0000) 2024-02-20T09:03:54.069 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:03:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:03:54.005+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:54.006365+0000) 2024-02-20T09:03:54.069 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:03:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:03:54.066+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:03:54.070 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:03:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:03:53.784+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:53.785219+0000) 2024-02-20T09:03:54.070 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:03:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:03:53.810+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:03:54.070 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:03:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:03:53.810+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:03:54.070 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:03:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:03:53.810+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:03:54.431 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:03:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:03:54.066+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:03:54.479 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:03:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:03:54.066+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:03:54.479 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:03:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:03:54.066+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:03:54.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:03:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:03:54.532+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:03:54.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:03:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:03:54.532+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:03:54.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:03:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:03:54.532+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:03:55.053 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:03:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:03:54.943+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 55121 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:03:55.054 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:03:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:03:54.901+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:54.902180+0000) 2024-02-20T09:03:55.054 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:03:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:03:54.937+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:54.938910+0000) 2024-02-20T09:03:55.054 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:03:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:03:54.784+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:54.785418+0000) 2024-02-20T09:03:55.054 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:03:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:03:54.822+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:03:55.055 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:03:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:03:54.823+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:03:55.055 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:03:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:03:54.823+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:03:55.055 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:03:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:03:54.936+0000 7f8a5f86e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:54.938260+0000) 2024-02-20T09:03:55.055 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:03:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:03:54.941+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:54.942445+0000) 2024-02-20T09:03:55.055 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:03:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:03:55.005+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:55.006755+0000) 2024-02-20T09:03:55.055 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:03:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:03:55.051+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:03:55.056 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:03:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:03:55.051+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:03:55.056 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:03:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:03:55.051+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:03:55.056 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:03:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:03:55.051+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:03:55.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:03:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:03:55.574+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:03:55.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:03:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:03:55.574+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:03:55.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:03:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:03:55.574+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:03:56.094 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:03:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:03:55.901+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:55.902337+0000) 2024-02-20T09:03:56.094 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:03:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:03:56.008+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:56.009174+0000) 2024-02-20T09:03:56.094 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:03:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:03:56.091+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:03:56.095 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:03:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:03:55.784+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:55.785551+0000) 2024-02-20T09:03:56.095 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:03:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:03:55.795+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:03:56.095 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:03:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:03:55.795+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:03:56.095 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:03:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:03:55.795+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:03:56.431 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:03:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:03:56.091+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:03:56.432 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:03:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:03:56.091+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:03:56.432 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:03:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:03:56.091+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:03:56.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:03:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:03:56.562+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:03:56.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:03:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:03:56.562+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:03:56.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:03:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:03:56.562+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:03:57.084 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:03:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:03:56.901+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:56.902489+0000) 2024-02-20T09:03:57.085 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:03:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:03:57.010+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:57.012037+0000) 2024-02-20T09:03:57.085 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:03:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:03:57.081+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:03:57.085 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:03:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:03:57.081+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:03:57.085 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:03:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:03:56.784+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:56.785667+0000) 2024-02-20T09:03:57.085 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:03:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:03:56.844+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:03:57.085 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:03:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:03:56.844+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:03:57.085 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:03:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:03:56.844+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:03:57.431 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:03:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:03:57.081+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:03:57.432 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:03:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:03:57.081+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:03:57.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:03:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:03:57.552+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:03:57.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:03:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:03:57.552+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:03:57.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:03:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:03:57.552+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:03:58.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:03:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:03:57.901+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:57.902662+0000) 2024-02-20T09:03:58.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:03:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:03:57.784+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:57.785768+0000) 2024-02-20T09:03:58.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:03:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:03:57.876+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:03:58.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:03:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:03:57.876+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:03:58.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:03:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:03:57.876+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:03:58.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:03:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:03:58.013+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:58.014763+0000) 2024-02-20T09:03:58.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:03:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:03:58.032+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:03:58.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:03:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:03:58.032+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:03:58.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:03:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:03:58.032+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:03:58.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:03:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:03:58.032+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:03:58.787 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:03:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:03:58.526+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:03:58.787 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:03:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:03:58.526+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:03:58.787 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:03:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:03:58.526+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:03:59.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:03:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:03:58.901+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:58.902777+0000) 2024-02-20T09:03:59.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:03:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:03:58.784+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:58.785869+0000) 2024-02-20T09:03:59.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:03:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:03:58.848+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:03:59.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:03:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:03:58.848+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:03:59.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:03:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:03:58.848+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:03:59.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:03:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:03:58.996+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:03:59.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:03:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:03:58.996+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:03:59.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:03:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:03:58.996+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:03:59.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:03:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:03:58.996+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:03:59.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:03:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:03:59.014+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:59.015087+0000) 2024-02-20T09:03:59.787 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:03:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:03:59.529+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:03:59.788 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:03:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:03:59.529+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:03:59.788 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:03:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:03:59.529+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:04:00.043 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:03:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:03:59.950+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 55192 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:04:00.043 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:03:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:03:59.902+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:59.902925+0000) 2024-02-20T09:04:00.044 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:03:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:03:59.944+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:59.946093+0000) 2024-02-20T09:04:00.044 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:03:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:03:59.785+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:59.786023+0000) 2024-02-20T09:04:00.044 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:03:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:03:59.852+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:04:00.044 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:03:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:03:59.852+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:04:00.044 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:03:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:03:59.852+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:04:00.044 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:03:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:03:59.945+0000 7f8a5f86e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:59.946930+0000) 2024-02-20T09:04:00.045 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:03:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:03:59.949+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:03:59.950436+0000) 2024-02-20T09:04:00.045 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:04:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:04:00.014+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:00.015367+0000) 2024-02-20T09:04:00.045 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:04:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:04:00.040+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:04:00.045 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:04:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:04:00.040+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:04:00.045 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:04:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:04:00.040+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:04:00.045 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:04:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:04:00.040+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:04:00.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:04:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:04:00.490+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:04:00.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:04:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:04:00.490+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:04:00.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:04:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:04:00.490+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:04:01.088 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:04:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:04:00.902+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:00.903126+0000) 2024-02-20T09:04:01.089 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:04:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:04:00.785+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:00.786233+0000) 2024-02-20T09:04:01.089 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:04:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:04:00.821+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:04:01.089 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:04:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:04:00.821+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:04:01.089 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:04:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:04:00.821+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:04:01.089 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:04:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:04:01.016+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:01.017691+0000) 2024-02-20T09:04:01.090 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:04:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:04:01.086+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:04:01.090 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:04:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:04:01.086+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:04:01.090 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:04:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:04:01.086+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:04:01.090 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:04:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:04:01.086+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:04:01.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:04:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:04:01.485+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:04:01.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:04:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:04:01.485+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:04:01.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:04:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:04:01.485+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:04:02.099 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:04:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:04:01.902+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:01.903304+0000) 2024-02-20T09:04:02.099 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:04:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:04:02.019+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:02.020673+0000) 2024-02-20T09:04:02.100 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:04:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:04:02.096+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:04:02.100 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:04:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:04:01.785+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:01.786388+0000) 2024-02-20T09:04:02.100 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:04:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:04:01.869+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:04:02.100 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:04:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:04:01.869+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:04:02.100 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:04:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:04:01.869+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:04:02.432 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:04:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:04:02.096+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:04:02.432 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:04:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:04:02.096+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:04:02.432 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:04:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:04:02.096+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:04:02.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:04:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:04:02.444+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:04:02.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:04:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:04:02.444+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:04:02.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:04:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:04:02.444+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:04:03.110 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:04:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:04:02.902+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:02.903489+0000) 2024-02-20T09:04:03.110 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:04:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:04:02.785+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:02.786548+0000) 2024-02-20T09:04:03.111 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:04:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:04:02.876+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:04:03.111 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:04:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:04:02.876+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:04:03.111 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:04:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:04:02.876+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:04:03.111 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:04:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:04:03.022+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:03.023370+0000) 2024-02-20T09:04:03.111 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:04:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:04:03.108+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:04:03.111 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:04:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:04:03.108+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:04:03.111 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:04:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:04:03.108+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:04:03.112 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:04:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:04:03.108+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:04:03.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:04:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:04:03.416+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:04:03.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:04:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:04:03.416+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:04:03.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:04:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:04:03.416+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:04:04.096 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:04:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:04:03.902+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:03.903649+0000) 2024-02-20T09:04:04.096 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:04:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:04:04.022+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:04.023725+0000) 2024-02-20T09:04:04.097 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:04:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:04:04.093+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:04:04.097 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:04:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:04:04.093+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:04:04.097 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:04:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:04:03.785+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:03.786702+0000) 2024-02-20T09:04:04.097 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:04:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:04:03.904+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:04:04.097 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:04:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:04:03.904+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:04:04.098 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:04:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:04:03.904+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:04:04.415 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:04:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:04:04.093+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:04:04.415 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:04:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:04:04.093+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:04:04.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:04:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:04:04.412+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:04:04.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:04:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:04:04.412+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:04:04.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:04:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:04:04.412+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:04:05.080 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:04:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:04:04.958+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 55265 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:04:05.081 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:04:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:04:04.955+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:04.956834+0000) 2024-02-20T09:04:05.081 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:04:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:04:05.023+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:05.024034+0000) 2024-02-20T09:04:05.081 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:04:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:04:04.902+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:04.903761+0000) 2024-02-20T09:04:05.081 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:04:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:04:04.952+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:04.953457+0000) 2024-02-20T09:04:05.082 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:04:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:04:04.786+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:04.786883+0000) 2024-02-20T09:04:05.082 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:04:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:04:04.944+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:04:05.082 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:04:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:04:04.944+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:04:05.082 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:04:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:04:04.944+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:04:05.082 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:04:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:04:04.952+0000 7f8a5f86e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:04.953807+0000) 2024-02-20T09:04:05.391 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:04:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:04:05.119+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:04:05.392 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:04:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:04:05.119+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:04:05.392 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:04:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:04:05.119+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:04:05.392 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:04:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:04:05.119+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:04:05.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:04:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:04:05.389+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:04:05.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:04:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:04:05.389+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:04:05.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:04:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:04:05.389+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:04:06.111 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:04:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:04:05.903+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:05.903946+0000) 2024-02-20T09:04:06.112 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:04:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:04:05.786+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:05.787074+0000) 2024-02-20T09:04:06.112 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:04:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:04:05.967+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:04:06.112 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:04:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:04:05.967+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:04:06.112 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:04:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:04:05.967+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:04:06.113 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:04:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:04:06.024+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:06.026311+0000) 2024-02-20T09:04:06.113 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:04:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:04:06.109+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:04:06.113 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:04:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:04:06.109+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:04:06.113 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:04:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:04:06.109+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:04:06.113 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:04:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:04:06.109+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:04:06.433 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:04:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:04:06.351+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:04:06.433 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:04:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:04:06.351+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:04:06.433 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:04:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:04:06.351+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:04:07.071 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:04:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:04:06.902+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:06.904039+0000) 2024-02-20T09:04:07.071 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:04:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:04:07.027+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:07.029149+0000) 2024-02-20T09:04:07.072 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:04:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:04:07.069+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:04:07.072 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:04:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:04:07.069+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:04:07.072 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:04:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:04:06.786+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:06.787227+0000) 2024-02-20T09:04:07.072 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:04:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:04:07.011+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:04:07.072 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:04:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:04:07.011+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:04:07.072 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:04:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:04:07.011+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:04:07.360 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:04:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:04:07.069+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:04:07.360 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:04:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:04:07.069+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:04:07.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:04:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:04:07.357+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:04:07.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:04:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:04:07.357+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:04:07.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:04:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:04:07.357+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:04:08.121 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:04:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:04:07.903+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:07.904197+0000) 2024-02-20T09:04:08.121 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:04:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:04:07.785+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:07.787360+0000) 2024-02-20T09:04:08.121 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:04:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:04:08.031+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:04:08.122 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:04:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:04:08.031+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:04:08.122 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:04:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:04:08.031+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:04:08.122 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:04:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:04:08.030+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:08.031778+0000) 2024-02-20T09:04:08.122 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:04:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:04:08.118+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:04:08.122 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:04:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:04:08.118+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:04:08.122 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:04:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:04:08.118+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:04:08.122 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:04:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:04:08.118+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:04:08.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:04:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:04:08.386+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:04:08.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:04:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:04:08.386+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:04:08.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:04:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:04:08.386+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:04:09.043 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:04:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:04:09.030+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:09.032100+0000) 2024-02-20T09:04:09.043 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:04:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:04:08.903+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:08.904392+0000) 2024-02-20T09:04:09.044 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:04:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:04:08.786+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:08.787580+0000) 2024-02-20T09:04:09.044 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:04:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:04:09.040+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:04:09.044 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:04:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:04:09.040+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:04:09.417 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:04:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:04:09.040+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:04:09.418 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:04:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:04:09.164+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:04:09.418 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:04:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:04:09.164+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:04:09.418 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:04:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:04:09.164+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:04:09.418 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:04:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:04:09.164+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:04:09.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:04:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:04:09.414+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:04:09.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:04:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:04:09.414+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:04:09.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:04:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:04:09.414+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:04:10.090 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:04:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:04:09.967+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 55337 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:04:10.091 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:04:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:04:09.962+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:09.964304+0000) 2024-02-20T09:04:10.091 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:04:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:04:10.031+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:10.032435+0000) 2024-02-20T09:04:10.091 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:04:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:04:09.904+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:09.904623+0000) 2024-02-20T09:04:10.091 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:04:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:04:09.960+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:09.961752+0000) 2024-02-20T09:04:10.092 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:04:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:04:09.787+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:09.787745+0000) 2024-02-20T09:04:10.092 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:04:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:04:09.959+0000 7f8a5f86e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:09.961317+0000) 2024-02-20T09:04:10.092 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:04:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:04:10.023+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:04:10.092 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:04:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:04:10.023+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:04:10.092 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:04:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:04:10.023+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:04:10.381 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:04:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:04:10.126+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:04:10.381 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:04:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:04:10.126+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:04:10.381 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:04:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:04:10.126+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:04:10.381 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:04:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:04:10.126+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:04:10.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:04:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:04:10.435+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:04:10.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:04:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:04:10.435+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:04:10.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:04:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:04:10.435+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:04:11.177 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:04:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:04:10.904+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:10.904808+0000) 2024-02-20T09:04:11.177 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:04:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:04:10.786+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:10.787911+0000) 2024-02-20T09:04:11.177 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:04:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:04:11.007+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:04:11.177 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:04:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:04:11.007+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:04:11.178 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:04:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:04:11.007+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:04:11.178 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:04:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:04:11.033+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:11.034779+0000) 2024-02-20T09:04:11.178 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:04:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:04:11.174+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:04:11.178 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:04:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:04:11.174+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:04:11.178 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:04:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:04:11.174+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:04:11.178 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:04:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:04:11.174+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:04:11.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:04:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:04:11.463+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:04:11.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:04:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:04:11.463+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:04:11.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:04:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:04:11.463+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:04:12.191 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:04:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:04:11.904+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:11.904981+0000) 2024-02-20T09:04:12.192 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:04:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:04:12.036+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:12.037456+0000) 2024-02-20T09:04:12.192 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:04:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:04:12.187+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:04:12.192 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:04:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:04:11.787+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:11.788008+0000) 2024-02-20T09:04:12.192 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:04:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:04:12.026+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:04:12.192 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:04:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:04:12.026+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:04:12.193 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:04:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:04:12.026+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:04:12.478 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:04:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:04:12.187+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:04:12.478 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:04:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:04:12.187+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:04:12.478 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:04:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:04:12.187+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:04:12.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:04:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:04:12.475+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:04:12.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:04:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:04:12.475+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:04:12.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:04:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:04:12.475+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:04:13.041 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:04:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:04:12.787+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:12.788197+0000) 2024-02-20T09:04:13.041 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:04:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:04:12.904+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:12.905161+0000) 2024-02-20T09:04:13.434 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:04:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:04:13.059+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:04:13.434 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:04:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:04:13.059+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:04:13.434 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:04:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:04:13.059+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:04:13.435 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:04:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:04:13.039+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:13.039883+0000) 2024-02-20T09:04:13.435 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:04:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:04:13.164+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:04:13.435 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:04:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:04:13.164+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:04:13.435 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:04:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:04:13.164+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:04:13.435 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:04:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:04:13.164+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:04:13.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:04:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:04:13.488+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:04:13.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:04:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:04:13.488+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:04:13.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:04:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:04:13.488+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:04:14.042 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:04:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:04:13.904+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:13.905344+0000) 2024-02-20T09:04:14.042 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:04:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:04:13.786+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:13.788370+0000) 2024-02-20T09:04:14.042 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:04:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:04:14.029+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:04:14.042 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:04:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:04:14.029+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:04:14.042 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:04:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:04:14.029+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:04:14.434 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:04:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:04:14.038+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:14.040196+0000) 2024-02-20T09:04:14.434 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:04:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:04:14.120+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:04:14.434 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:04:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:04:14.120+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:04:14.435 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:04:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:04:14.120+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:04:14.435 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:04:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:04:14.120+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:04:14.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:04:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:04:14.443+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:04:14.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:04:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:04:14.443+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:04:14.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:04:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:04:14.443+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:04:15.042 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:04:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:04:14.975+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 55410 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:04:15.042 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:04:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:04:14.971+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:14.973066+0000) 2024-02-20T09:04:15.042 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:04:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:04:14.905+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:14.905562+0000) 2024-02-20T09:04:15.043 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:04:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:04:14.969+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:14.970847+0000) 2024-02-20T09:04:15.043 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:04:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:04:14.788+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:14.788515+0000) 2024-02-20T09:04:15.043 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:04:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:04:14.968+0000 7f8a5f86e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:14.970311+0000) 2024-02-20T09:04:15.043 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:04:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:04:15.026+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:04:15.043 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:04:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:04:15.026+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:04:15.043 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:04:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:04:15.026+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:04:15.434 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:04:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:04:15.040+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:15.040548+0000) 2024-02-20T09:04:15.435 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:04:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:04:15.083+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:04:15.435 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:04:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:04:15.083+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:04:15.435 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:04:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:04:15.083+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:04:15.435 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:04:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:04:15.083+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:04:15.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:04:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:04:15.434+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:04:15.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:04:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:04:15.434+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:04:15.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:04:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:04:15.434+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:04:16.044 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:04:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:04:15.905+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:15.905731+0000) 2024-02-20T09:04:16.044 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:04:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:04:15.788+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:15.788712+0000) 2024-02-20T09:04:16.425 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:04:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:04:16.047+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:04:16.425 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:04:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:04:16.047+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:04:16.425 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:04:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:04:16.047+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:04:16.425 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:04:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:04:16.041+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:16.043011+0000) 2024-02-20T09:04:16.425 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:04:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:04:16.114+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:04:16.426 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:04:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:04:16.114+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:04:16.426 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:04:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:04:16.114+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:04:16.426 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:04:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:04:16.114+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:04:16.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:04:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:04:16.422+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:04:16.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:04:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:04:16.422+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:04:16.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:04:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:04:16.422+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:04:17.047 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:04:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:04:16.905+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:16.905904+0000) 2024-02-20T09:04:17.048 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:04:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:04:16.788+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:16.788857+0000) 2024-02-20T09:04:17.048 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:04:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:04:17.024+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:04:17.048 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:04:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:04:17.024+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:04:17.048 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:04:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:04:17.024+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:04:17.397 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:04:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:04:17.045+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:17.045939+0000) 2024-02-20T09:04:17.397 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:04:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:04:17.093+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:04:17.397 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:04:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:04:17.093+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:04:17.397 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:04:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:04:17.093+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:04:17.398 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:04:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:04:17.093+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:04:17.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:04:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:04:17.394+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:04:17.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:04:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:04:17.394+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:04:17.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:04:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:04:17.394+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:04:18.050 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:04:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:04:17.905+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:17.906108+0000) 2024-02-20T09:04:18.050 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:04:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:04:17.788+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:17.789017+0000) 2024-02-20T09:04:18.050 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:04:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:04:18.007+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:04:18.050 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:04:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:04:18.007+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:04:18.050 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:04:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:04:18.007+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:04:18.355 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:04:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:04:18.048+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:18.048577+0000) 2024-02-20T09:04:18.355 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:04:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:04:18.135+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:04:18.355 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:04:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:04:18.135+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:04:18.355 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:04:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:04:18.135+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:04:18.355 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:04:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:04:18.135+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:04:18.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:04:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:04:18.353+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:04:18.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:04:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:04:18.353+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:04:18.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:04:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:04:18.353+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:04:19.047 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:04:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:04:18.905+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:18.906289+0000) 2024-02-20T09:04:19.048 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:04:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:04:18.788+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:18.789221+0000) 2024-02-20T09:04:19.048 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:04:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:04:19.045+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:04:19.384 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:04:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:04:19.045+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:04:19.384 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:04:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:04:19.045+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:04:19.385 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:04:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:04:19.048+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:19.048840+0000) 2024-02-20T09:04:19.385 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:04:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:04:19.168+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:04:19.385 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:04:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:04:19.168+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:04:19.385 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:04:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:04:19.168+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:04:19.385 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:04:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:04:19.169+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:04:19.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:04:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:04:19.382+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:04:19.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:04:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:04:19.382+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:04:19.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:04:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:04:19.382+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:04:20.042 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:04:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:04:19.984+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 55481 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:04:20.067 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:04:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:04:19.981+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:19.982139+0000) 2024-02-20T09:04:20.067 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:04:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:04:19.906+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:19.906540+0000) 2024-02-20T09:04:20.067 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:04:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:04:19.978+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:19.978503+0000) 2024-02-20T09:04:20.068 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:04:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:04:19.789+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:19.789442+0000) 2024-02-20T09:04:20.068 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:04:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:04:19.977+0000 7f8a6006f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:19.978376+0000) 2024-02-20T09:04:20.068 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:04:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:04:20.040+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:04:20.068 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:04:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:04:20.040+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:04:20.068 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:04:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:04:20.040+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:04:20.381 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:04:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:04:20.048+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:20.049121+0000) 2024-02-20T09:04:20.381 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:04:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:04:20.181+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:04:20.382 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:04:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:04:20.181+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:04:20.382 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:04:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:04:20.181+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:04:20.382 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:04:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:04:20.181+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:04:20.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:04:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:04:20.393+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:04:20.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:04:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:04:20.393+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:04:20.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:04:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:04:20.393+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:04:21.052 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:04:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:04:20.906+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:20.906664+0000) 2024-02-20T09:04:21.053 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:04:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:04:20.789+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:20.789614+0000) 2024-02-20T09:04:21.421 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:04:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:04:21.079+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:04:21.421 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:04:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:04:21.079+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:04:21.421 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:04:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:04:21.079+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:04:21.421 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:04:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:04:21.050+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:21.051350+0000) 2024-02-20T09:04:21.421 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:04:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:04:21.225+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:04:21.421 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:04:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:04:21.225+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:04:21.422 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:04:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:04:21.225+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:04:21.422 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:04:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:04:21.225+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:04:21.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:04:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:04:21.418+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:04:21.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:04:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:04:21.418+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:04:21.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:04:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:04:21.418+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:04:22.055 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:04:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:04:21.789+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:21.789746+0000) 2024-02-20T09:04:22.056 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:04:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:04:21.906+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:21.906802+0000) 2024-02-20T09:04:22.422 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:04:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:04:22.072+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:04:22.422 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:04:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:04:22.072+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:04:22.422 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:04:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:04:22.072+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:04:22.423 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:04:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:04:22.053+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:22.054228+0000) 2024-02-20T09:04:22.423 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:04:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:04:22.220+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:04:22.423 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:04:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:04:22.220+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:04:22.423 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:04:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:04:22.220+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:04:22.423 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:04:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:04:22.220+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:04:22.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:04:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:04:22.420+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:04:22.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:04:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:04:22.420+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:04:22.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:04:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:04:22.420+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:04:23.030 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:04:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:04:22.789+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:22.789890+0000) 2024-02-20T09:04:23.031 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:04:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:04:22.906+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:22.906983+0000) 2024-02-20T09:04:23.390 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:04:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:04:23.081+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:04:23.391 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:04:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:04:23.081+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:04:23.391 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:04:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:04:23.081+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:04:23.391 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:04:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:04:23.056+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:23.056806+0000) 2024-02-20T09:04:23.391 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:04:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:04:23.207+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:04:23.391 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:04:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:04:23.207+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:04:23.391 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:04:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:04:23.207+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:04:23.392 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:04:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:04:23.208+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:04:23.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:04:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:04:23.388+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:04:23.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:04:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:04:23.388+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:04:23.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:04:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:04:23.388+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:04:24.058 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:04:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:04:23.906+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:23.907120+0000) 2024-02-20T09:04:24.059 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:04:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:04:23.789+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:23.790083+0000) 2024-02-20T09:04:24.059 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:04:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:04:24.032+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:04:24.059 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:04:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:04:24.032+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:04:24.059 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:04:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:04:24.032+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:04:24.374 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:04:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:04:24.056+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:24.057094+0000) 2024-02-20T09:04:24.374 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:04:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:04:24.180+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:04:24.374 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:04:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:04:24.180+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:04:24.374 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:04:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:04:24.180+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:04:24.374 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:04:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:04:24.180+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:04:24.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:04:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:04:24.372+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:04:24.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:04:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:04:24.372+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:04:24.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:04:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:04:24.372+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:04:25.051 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:04:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:04:24.992+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 55554 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:04:25.051 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:04:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:04:24.989+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:24.990004+0000) 2024-02-20T09:04:25.052 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:04:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:04:24.907+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:24.907343+0000) 2024-02-20T09:04:25.052 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:04:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:04:24.985+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:24.986153+0000) 2024-02-20T09:04:25.052 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:04:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:04:24.789+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:24.790274+0000) 2024-02-20T09:04:25.052 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:04:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:04:24.985+0000 7f8a6006f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:24.986386+0000) 2024-02-20T09:04:25.052 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:04:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:04:25.049+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:04:25.052 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:04:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:04:25.049+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:04:25.053 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:04:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:04:25.049+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:04:25.343 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:04:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:04:25.057+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:25.057493+0000) 2024-02-20T09:04:25.343 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:04:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:04:25.176+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:04:25.343 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:04:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:04:25.176+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:04:25.343 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:04:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:04:25.176+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:04:25.343 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:04:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:04:25.176+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:04:25.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:04:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:04:25.341+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:04:25.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:04:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:04:25.341+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:04:25.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:04:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:04:25.341+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:04:26.043 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:04:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:04:25.907+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:25.907566+0000) 2024-02-20T09:04:26.044 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:04:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:04:25.790+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:25.790496+0000) 2024-02-20T09:04:26.044 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:04:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:04:26.041+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:04:26.349 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:04:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:04:26.041+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:04:26.350 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:04:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:04:26.041+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:04:26.350 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:04:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:04:26.059+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:26.059951+0000) 2024-02-20T09:04:26.350 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:04:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:04:26.209+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:04:26.350 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:04:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:04:26.209+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:04:26.350 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:04:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:04:26.209+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:04:26.350 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:04:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:04:26.209+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:04:26.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:04:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:04:26.347+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:04:26.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:04:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:04:26.347+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:04:26.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:04:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:04:26.347+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:04:27.064 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:04:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:04:26.907+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:26.907750+0000) 2024-02-20T09:04:27.064 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:04:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:04:26.790+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:26.790629+0000) 2024-02-20T09:04:27.064 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:04:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:04:27.021+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:04:27.065 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:04:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:04:27.021+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:04:27.065 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:04:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:04:27.021+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:04:27.436 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:04:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:04:27.303+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:04:27.437 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:04:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:04:27.303+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:04:27.437 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:04:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:04:27.303+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:04:27.437 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:04:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:04:27.062+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:27.062756+0000) 2024-02-20T09:04:27.437 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:04:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:04:27.179+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:04:27.437 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:04:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:04:27.179+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:04:27.437 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:04:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:04:27.179+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:04:27.437 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:04:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:04:27.179+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:04:28.067 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:04:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:04:27.907+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:27.907873+0000) 2024-02-20T09:04:28.067 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:04:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:04:27.790+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:27.790855+0000) 2024-02-20T09:04:28.067 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:04:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:04:28.009+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:04:28.067 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:04:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:04:28.009+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:04:28.067 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:04:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:04:28.009+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:04:28.436 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:04:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:04:28.285+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:04:28.436 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:04:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:04:28.285+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:04:28.437 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:04:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:04:28.285+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:04:28.437 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:04:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:04:28.065+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:28.065416+0000) 2024-02-20T09:04:28.437 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:04:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:04:28.132+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:04:28.437 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:04:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:04:28.132+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:04:28.437 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:04:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:04:28.132+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:04:28.437 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:04:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:04:28.132+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:04:29.067 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:04:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:04:28.907+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:28.907986+0000) 2024-02-20T09:04:29.067 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:04:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:04:28.790+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:28.791057+0000) 2024-02-20T09:04:29.067 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:04:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:04:29.002+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:04:29.067 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:04:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:04:29.002+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:04:29.067 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:04:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:04:29.002+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:04:29.437 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:04:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:04:29.249+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:04:29.437 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:04:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:04:29.249+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:04:29.437 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:04:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:04:29.249+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:04:29.437 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:04:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:04:29.065+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:29.065653+0000) 2024-02-20T09:04:29.438 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:04:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:04:29.160+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:04:29.438 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:04:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:04:29.160+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:04:29.438 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:04:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:04:29.160+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:04:29.438 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:04:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:04:29.160+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:04:30.067 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:04:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:04:30.000+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 55625 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:04:30.068 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:04:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:04:29.997+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:29.998021+0000) 2024-02-20T09:04:30.068 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:04:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:04:29.907+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:29.908148+0000) 2024-02-20T09:04:30.069 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:04:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:04:29.994+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:29.995248+0000) 2024-02-20T09:04:30.069 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:04:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:04:29.790+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:29.791257+0000) 2024-02-20T09:04:30.069 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:04:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:04:29.971+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:04:30.069 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:04:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:04:29.971+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:04:30.069 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:04:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:04:29.971+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:04:30.069 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:04:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:04:29.994+0000 7f8a5f86e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:29.994469+0000) 2024-02-20T09:04:30.382 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:04:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:04:30.299+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:04:30.382 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:04:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:04:30.299+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:04:30.382 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:04:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:04:30.299+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:04:30.383 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:04:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:04:30.065+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:30.065932+0000) 2024-02-20T09:04:30.383 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:04:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:04:30.123+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:04:30.383 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:04:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:04:30.123+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:04:30.383 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:04:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:04:30.123+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:04:30.383 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:04:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:04:30.123+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:04:31.070 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:04:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:04:30.907+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:30.908305+0000) 2024-02-20T09:04:31.070 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:04:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:04:30.791+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:30.791486+0000) 2024-02-20T09:04:31.070 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:04:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:04:30.963+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:04:31.071 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:04:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:04:30.963+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:04:31.071 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:04:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:04:30.963+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:04:31.350 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:04:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:04:31.068+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:31.068581+0000) 2024-02-20T09:04:31.350 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:04:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:04:31.157+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:04:31.350 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:04:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:04:31.157+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:04:31.350 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:04:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:04:31.157+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:04:31.350 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:04:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:04:31.157+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:04:31.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:04:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:04:31.347+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:04:31.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:04:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:04:31.347+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:04:31.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:04:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:04:31.347+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:04:32.073 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:04:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:04:31.908+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:31.908424+0000) 2024-02-20T09:04:32.073 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:04:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:04:31.791+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:31.791642+0000) 2024-02-20T09:04:32.073 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:04:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:04:31.944+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:04:32.073 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:04:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:04:31.944+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:04:32.073 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:04:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:04:31.944+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:04:32.437 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:04:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:04:32.317+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:04:32.437 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:04:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:04:32.317+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:04:32.438 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:04:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:04:32.317+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:04:32.438 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:04:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:04:32.071+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:32.071471+0000) 2024-02-20T09:04:32.438 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:04:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:04:32.176+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:04:32.438 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:04:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:04:32.176+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:04:32.438 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:04:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:04:32.176+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:04:32.438 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:04:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:04:32.176+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:04:33.075 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:04:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:04:32.908+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:32.908611+0000) 2024-02-20T09:04:33.076 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:04:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:04:32.791+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:32.791843+0000) 2024-02-20T09:04:33.076 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:04:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:04:32.926+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:04:33.076 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:04:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:04:32.926+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:04:33.076 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:04:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:04:32.926+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:04:33.437 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:04:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:04:33.279+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:04:33.437 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:04:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:04:33.279+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:04:33.438 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:04:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:04:33.279+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:04:33.438 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:04:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:04:33.074+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:33.074247+0000) 2024-02-20T09:04:33.438 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:04:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:04:33.182+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:04:33.438 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:04:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:04:33.182+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:04:33.438 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:04:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:04:33.182+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:04:33.438 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:04:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:04:33.182+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:04:34.076 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:04:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:04:33.908+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:33.908785+0000) 2024-02-20T09:04:34.076 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:04:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:04:33.791+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:33.792046+0000) 2024-02-20T09:04:34.076 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:04:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:04:33.903+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:04:34.076 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:04:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:04:33.903+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:04:34.076 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:04:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:04:33.903+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:04:34.437 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:04:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:04:34.278+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:04:34.438 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:04:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:04:34.278+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:04:34.438 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:04:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:04:34.278+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:04:34.438 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:04:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:04:34.074+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:34.074476+0000) 2024-02-20T09:04:34.438 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:04:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:04:34.221+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:04:34.438 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:04:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:04:34.221+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:04:34.438 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:04:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:04:34.221+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:04:34.439 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:04:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:04:34.221+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:04:35.076 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:04:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:04:35.008+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 55697 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:04:35.077 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:04:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:04:35.005+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:35.006127+0000) 2024-02-20T09:04:35.077 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:04:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:04:34.908+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:34.908973+0000) 2024-02-20T09:04:35.077 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:04:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:04:35.002+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:35.002867+0000) 2024-02-20T09:04:35.077 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:04:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:04:34.791+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:34.792252+0000) 2024-02-20T09:04:35.077 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:04:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:04:34.874+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:04:35.078 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:04:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:04:34.874+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:04:35.078 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:04:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:04:34.874+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:04:35.078 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:04:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:04:35.001+0000 7f8a6006f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:35.002150+0000) 2024-02-20T09:04:35.437 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:04:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:04:35.074+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:35.074815+0000) 2024-02-20T09:04:35.438 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:04:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:04:35.240+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:04:35.438 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:04:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:04:35.240+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:04:35.438 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:04:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:04:35.240+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:04:35.438 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:04:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:04:35.241+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:04:35.438 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:04:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:04:35.293+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:04:35.438 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:04:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:04:35.293+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:04:35.438 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:04:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:04:35.293+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:04:36.079 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:04:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:04:35.909+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:35.909156+0000) 2024-02-20T09:04:36.079 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:04:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:04:35.792+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:35.792443+0000) 2024-02-20T09:04:36.079 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:04:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:04:35.871+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:04:36.079 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:04:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:04:35.871+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:04:36.080 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:04:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:04:35.871+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:04:36.438 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:04:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:04:36.257+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:04:36.438 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:04:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:04:36.257+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:04:36.438 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:04:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:04:36.257+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:04:36.438 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:04:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:04:36.077+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:36.077556+0000) 2024-02-20T09:04:36.438 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:04:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:04:36.248+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:04:36.439 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:04:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:04:36.248+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:04:36.439 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:04:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:04:36.248+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:04:36.439 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:04:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:04:36.249+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:04:37.082 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:04:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:04:36.908+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:36.909325+0000) 2024-02-20T09:04:37.082 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:04:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:04:36.792+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:36.792638+0000) 2024-02-20T09:04:37.082 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:04:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:04:36.913+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:04:37.082 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:04:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:04:36.913+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:04:37.082 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:04:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:04:36.913+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:04:37.438 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:04:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:04:37.294+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:04:37.438 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:04:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:04:37.294+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:04:37.438 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:04:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:04:37.294+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:04:37.438 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:04:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:04:37.079+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:37.080296+0000) 2024-02-20T09:04:37.438 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:04:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:04:37.239+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:04:37.438 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:04:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:04:37.239+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:04:37.439 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:04:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:04:37.239+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:04:37.439 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:04:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:04:37.239+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:04:38.084 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:04:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:04:37.909+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:37.909525+0000) 2024-02-20T09:04:38.085 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:04:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:04:37.792+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:37.792850+0000) 2024-02-20T09:04:38.085 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:04:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:04:37.950+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:04:38.085 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:04:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:04:37.950+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:04:38.085 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:04:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:04:37.950+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:04:38.438 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:04:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:04:38.082+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:38.082957+0000) 2024-02-20T09:04:38.438 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:04:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:04:38.220+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:04:38.438 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:04:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:04:38.220+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:04:38.438 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:04:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:04:38.220+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:04:38.439 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:04:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:04:38.220+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:04:38.439 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:04:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:04:38.249+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:04:38.439 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:04:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:04:38.249+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:04:38.439 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:04:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:04:38.249+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:04:39.084 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:04:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:04:38.909+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:38.909754+0000) 2024-02-20T09:04:39.085 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:04:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:04:38.792+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:38.793032+0000) 2024-02-20T09:04:39.085 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:04:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:04:38.941+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:04:39.085 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:04:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:04:38.941+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:04:39.085 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:04:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:04:38.941+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:04:39.438 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:04:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:04:39.257+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:04:39.438 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:04:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:04:39.257+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:04:39.438 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:04:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:04:39.257+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:04:39.439 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:04:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:04:39.083+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:39.083246+0000) 2024-02-20T09:04:39.439 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:04:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:04:39.235+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:04:39.439 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:04:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:04:39.235+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:04:39.439 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:04:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:04:39.235+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:04:39.439 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:04:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:04:39.235+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:04:40.085 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:04:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:04:40.016+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 55769 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:04:40.085 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:04:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:04:40.013+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:40.014211+0000) 2024-02-20T09:04:40.086 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:04:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:04:39.909+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:39.909934+0000) 2024-02-20T09:04:40.086 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:04:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:04:40.010+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:40.011086+0000) 2024-02-20T09:04:40.086 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:04:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:04:39.793+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:39.793216+0000) 2024-02-20T09:04:40.086 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:04:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:04:39.992+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:04:40.086 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:04:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:04:39.992+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:04:40.086 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:04:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:04:39.992+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:04:40.086 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:04:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:04:40.010+0000 7f8a6006f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:40.010439+0000) 2024-02-20T09:04:40.383 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:04:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:04:40.287+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:04:40.383 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:04:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:04:40.287+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:04:40.383 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:04:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:04:40.287+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:04:40.384 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:04:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:04:40.083+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:40.083589+0000) 2024-02-20T09:04:40.384 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:04:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:04:40.189+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:04:40.384 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:04:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:04:40.189+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:04:40.384 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:04:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:04:40.189+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:04:40.384 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:04:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:04:40.189+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:04:41.087 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:04:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:04:40.908+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:40.910122+0000) 2024-02-20T09:04:41.088 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:04:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:04:40.792+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:40.793425+0000) 2024-02-20T09:04:41.088 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:04:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:04:40.951+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:04:41.088 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:04:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:04:40.951+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:04:41.088 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:04:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:04:40.951+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:04:41.438 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:04:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:04:41.334+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:04:41.438 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:04:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:04:41.334+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:04:41.438 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:04:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:04:41.334+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:04:41.439 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:04:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:04:41.085+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:41.086437+0000) 2024-02-20T09:04:41.439 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:04:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:04:41.189+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:04:41.439 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:04:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:04:41.189+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:04:41.439 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:04:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:04:41.189+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:04:41.439 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:04:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:04:41.189+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:04:42.090 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:04:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:04:41.908+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:41.910286+0000) 2024-02-20T09:04:42.091 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:04:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:04:41.792+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:41.793586+0000) 2024-02-20T09:04:42.091 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:04:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:04:41.998+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:04:42.091 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:04:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:04:41.998+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:04:42.091 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:04:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:04:41.998+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:04:42.382 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:04:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:04:42.087+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:42.089049+0000) 2024-02-20T09:04:42.382 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:04:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:04:42.145+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:04:42.382 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:04:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:04:42.145+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:04:42.382 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:04:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:04:42.145+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:04:42.382 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:04:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:04:42.145+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:04:42.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:04:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:04:42.379+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:04:42.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:04:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:04:42.379+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:04:42.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:04:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:04:42.379+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:04:43.093 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:04:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:04:42.909+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:42.910467+0000) 2024-02-20T09:04:43.093 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:04:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:04:42.792+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:42.793778+0000) 2024-02-20T09:04:43.093 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:04:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:04:42.983+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:04:43.094 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:04:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:04:42.983+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:04:43.094 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:04:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:04:42.983+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:04:43.363 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:04:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:04:43.090+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:43.091653+0000) 2024-02-20T09:04:43.364 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:04:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:04:43.114+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:04:43.364 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:04:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:04:43.114+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:04:43.364 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:04:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:04:43.114+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:04:43.364 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:04:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:04:43.114+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:04:43.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:04:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:04:43.360+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:04:43.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:04:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:04:43.360+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:04:43.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:04:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:04:43.360+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:04:44.093 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:04:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:04:43.909+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:43.910685+0000) 2024-02-20T09:04:44.094 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:04:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:04:43.792+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:43.793967+0000) 2024-02-20T09:04:44.094 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:04:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:04:43.948+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:04:44.094 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:04:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:04:43.948+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:04:44.094 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:04:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:04:43.948+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:04:44.374 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:04:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:04:44.091+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:44.092012+0000) 2024-02-20T09:04:44.374 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:04:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:04:44.125+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:04:44.374 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:04:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:04:44.125+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:04:44.374 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:04:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:04:44.125+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:04:44.374 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:04:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:04:44.125+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:04:44.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:04:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:04:44.370+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:04:44.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:04:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:04:44.370+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:04:44.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:04:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:04:44.370+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:04:45.045 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:04:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:04:45.024+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 55841 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:04:45.046 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:04:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:04:45.021+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:45.022988+0000) 2024-02-20T09:04:45.046 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:04:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:04:44.909+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:44.910860+0000) 2024-02-20T09:04:45.046 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:04:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:04:45.017+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:45.019042+0000) 2024-02-20T09:04:45.047 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:04:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:04:44.793+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:44.794156+0000) 2024-02-20T09:04:45.047 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:04:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:04:44.996+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:04:45.047 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:04:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:04:44.996+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:04:45.047 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:04:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:04:44.996+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:04:45.047 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:04:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:04:45.017+0000 7f8a5f86e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:45.018777+0000) 2024-02-20T09:04:45.333 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:04:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:04:45.090+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:45.092370+0000) 2024-02-20T09:04:45.333 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:04:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:04:45.124+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:04:45.333 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:04:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:04:45.124+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:04:45.334 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:04:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:04:45.124+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:04:45.334 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:04:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:04:45.124+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:04:45.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:04:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:04:45.330+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:04:45.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:04:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:04:45.330+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:04:45.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:04:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:04:45.330+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:04:46.078 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:04:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:04:45.910+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:45.911016+0000) 2024-02-20T09:04:46.078 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:04:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:04:45.793+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:45.794347+0000) 2024-02-20T09:04:46.078 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:04:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:04:45.977+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:04:46.079 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:04:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:04:45.977+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:04:46.079 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:04:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:04:45.977+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:04:46.371 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:04:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:04:46.075+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:04:46.371 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:04:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:04:46.075+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:04:46.371 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:04:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:04:46.075+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:04:46.371 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:04:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:04:46.075+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:04:46.371 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:04:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:04:46.093+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:46.094711+0000) 2024-02-20T09:04:46.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:04:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:04:46.368+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:04:46.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:04:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:04:46.368+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:04:46.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:04:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:04:46.368+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:04:47.100 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:04:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:04:46.909+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:46.911213+0000) 2024-02-20T09:04:47.100 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:04:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:04:46.793+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:46.794501+0000) 2024-02-20T09:04:47.100 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:04:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:04:47.005+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:04:47.100 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:04:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:04:47.005+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:04:47.100 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:04:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:04:47.005+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:04:47.375 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:04:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:04:47.097+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:47.098517+0000) 2024-02-20T09:04:47.375 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:04:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:04:47.113+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:04:47.375 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:04:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:04:47.113+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:04:47.375 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:04:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:04:47.113+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:04:47.376 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:04:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:04:47.113+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:04:47.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:04:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:04:47.372+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:04:47.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:04:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:04:47.372+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:04:47.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:04:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:04:47.372+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:04:48.103 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:04:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:04:47.910+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:47.911428+0000) 2024-02-20T09:04:48.104 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:04:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:04:47.793+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:47.794649+0000) 2024-02-20T09:04:48.104 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:04:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:04:48.004+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:04:48.104 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:04:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:04:48.004+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:04:48.104 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:04:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:04:48.004+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:04:48.407 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:04:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:04:48.101+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:48.102003+0000) 2024-02-20T09:04:48.408 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:04:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:04:48.145+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:04:48.408 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:04:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:04:48.145+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:04:48.408 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:04:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:04:48.145+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:04:48.408 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:04:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:04:48.145+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:04:48.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:04:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:04:48.404+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:04:48.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:04:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:04:48.404+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:04:48.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:04:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:04:48.404+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:04:49.103 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:04:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:04:48.910+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:48.911617+0000) 2024-02-20T09:04:49.104 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:04:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:04:48.793+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:48.794842+0000) 2024-02-20T09:04:49.104 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:04:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:04:48.972+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:04:49.104 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:04:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:04:48.972+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:04:49.104 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:04:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:04:48.972+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:04:49.417 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:04:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:04:49.101+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:49.102300+0000) 2024-02-20T09:04:49.417 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:04:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:04:49.193+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:04:49.418 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:04:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:04:49.193+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:04:49.418 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:04:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:04:49.193+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:04:49.418 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:04:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:04:49.193+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:04:49.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:04:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:04:49.414+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:04:49.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:04:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:04:49.414+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:04:49.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:04:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:04:49.414+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:04:50.054 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:04:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:04:50.032+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 55913 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:04:50.055 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:04:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:04:50.029+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:50.030841+0000) 2024-02-20T09:04:50.055 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:04:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:04:49.910+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:49.911785+0000) 2024-02-20T09:04:50.055 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:04:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:04:50.025+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:50.026901+0000) 2024-02-20T09:04:50.056 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:04:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:04:49.793+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:49.795033+0000) 2024-02-20T09:04:50.056 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:04:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:04:49.977+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:04:50.056 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:04:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:04:49.977+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:04:50.056 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:04:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:04:49.977+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:04:50.056 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:04:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:04:50.025+0000 7f8a6006f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:50.027036+0000) 2024-02-20T09:04:50.383 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:04:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:04:50.101+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:50.102576+0000) 2024-02-20T09:04:50.384 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:04:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:04:50.210+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:04:50.384 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:04:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:04:50.210+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:04:50.384 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:04:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:04:50.210+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:04:50.384 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:04:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:04:50.210+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:04:50.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:04:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:04:50.455+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:04:50.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:04:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:04:50.455+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:04:50.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:04:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:04:50.455+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:04:51.107 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:04:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:04:50.911+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:50.912008+0000) 2024-02-20T09:04:51.108 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:04:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:04:50.794+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:50.795288+0000) 2024-02-20T09:04:51.108 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:04:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:04:50.959+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:04:51.108 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:04:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:04:50.959+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:04:51.108 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:04:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:04:50.959+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:04:51.440 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:04:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:04:51.104+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:51.106177+0000) 2024-02-20T09:04:51.440 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:04:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:04:51.249+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:04:51.440 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:04:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:04:51.249+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:04:51.441 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:04:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:04:51.249+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:04:51.441 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:04:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:04:51.249+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:04:51.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:04:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:04:51.491+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:04:51.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:04:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:04:51.491+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:04:51.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:04:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:04:51.491+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:04:52.110 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:04:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:04:51.910+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:51.912201+0000) 2024-02-20T09:04:52.110 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:04:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:04:51.794+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:51.795436+0000) 2024-02-20T09:04:52.111 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:04:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:04:51.999+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:04:52.111 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:04:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:04:51.999+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:04:52.111 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:04:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:04:51.999+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:04:52.440 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:04:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:04:52.107+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:52.108863+0000) 2024-02-20T09:04:52.440 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:04:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:04:52.207+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:04:52.441 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:04:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:04:52.207+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:04:52.441 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:04:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:04:52.207+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:04:52.441 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:04:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:04:52.208+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:04:52.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:04:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:04:52.497+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:04:52.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:04:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:04:52.497+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:04:52.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:04:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:04:52.497+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:04:53.113 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:04:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:04:52.911+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:52.912353+0000) 2024-02-20T09:04:53.113 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:04:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:04:52.794+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:52.795642+0000) 2024-02-20T09:04:53.113 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:04:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:04:52.999+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:04:53.114 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:04:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:04:52.999+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:04:53.114 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:04:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:04:52.999+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:04:53.440 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:04:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:04:53.110+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:53.111834+0000) 2024-02-20T09:04:53.440 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:04:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:04:53.251+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:04:53.440 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:04:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:04:53.251+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:04:53.440 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:04:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:04:53.251+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:04:53.441 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:04:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:04:53.251+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:04:53.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:04:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:04:53.494+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:04:53.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:04:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:04:53.494+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:04:53.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:04:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:04:53.494+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:04:54.113 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:04:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:04:53.911+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:53.912547+0000) 2024-02-20T09:04:54.114 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:04:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:04:53.795+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:53.795836+0000) 2024-02-20T09:04:54.114 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:04:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:04:53.992+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:04:54.114 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:04:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:04:53.992+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:04:54.114 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:04:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:04:53.992+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:04:54.441 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:04:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:04:54.110+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:54.112140+0000) 2024-02-20T09:04:54.441 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:04:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:04:54.231+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:04:54.441 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:04:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:04:54.231+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:04:54.441 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:04:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:04:54.231+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:04:54.441 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:04:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:04:54.231+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:04:54.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:04:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:04:54.483+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:04:54.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:04:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:04:54.483+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:04:54.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:04:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:04:54.483+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:04:55.053 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:04:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:04:55.040+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 55985 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:04:55.054 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:04:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:04:55.036+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:55.038288+0000) 2024-02-20T09:04:55.054 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:04:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:04:54.911+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:54.912736+0000) 2024-02-20T09:04:55.054 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:04:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:04:55.033+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:55.034921+0000) 2024-02-20T09:04:55.055 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:04:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:04:54.795+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:54.795995+0000) 2024-02-20T09:04:55.055 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:04:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:04:54.975+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:04:55.055 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:04:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:04:54.975+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:04:55.055 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:04:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:04:54.975+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:04:55.055 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:04:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:04:55.033+0000 7f8a5f86e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:55.035109+0000) 2024-02-20T09:04:55.441 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:04:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:04:55.111+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:55.112414+0000) 2024-02-20T09:04:55.441 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:04:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:04:55.215+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:04:55.441 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:04:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:04:55.215+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:04:55.441 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:04:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:04:55.215+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:04:55.441 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:04:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:04:55.215+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:04:55.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:04:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:04:55.478+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:04:55.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:04:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:04:55.478+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:04:55.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:04:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:04:55.478+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:04:56.116 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:04:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:04:55.912+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:55.912892+0000) 2024-02-20T09:04:56.116 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:04:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:04:55.794+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:55.796125+0000) 2024-02-20T09:04:56.116 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:04:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:04:55.933+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:04:56.117 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:04:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:04:55.933+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:04:56.117 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:04:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:04:55.933+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:04:56.441 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:04:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:04:56.114+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:56.114753+0000) 2024-02-20T09:04:56.441 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:04:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:04:56.233+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:04:56.441 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:04:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:04:56.233+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:04:56.441 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:04:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:04:56.233+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:04:56.441 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:04:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:04:56.233+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:04:56.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:04:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:04:56.501+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:04:56.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:04:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:04:56.501+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:04:56.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:04:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:04:56.501+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:04:57.119 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:04:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:04:56.912+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:56.913029+0000) 2024-02-20T09:04:57.120 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:04:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:04:56.794+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:56.796283+0000) 2024-02-20T09:04:57.120 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:04:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:04:56.963+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:04:57.120 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:04:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:04:56.963+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:04:57.120 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:04:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:04:56.963+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:04:57.441 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:04:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:04:57.117+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:57.118481+0000) 2024-02-20T09:04:57.441 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:04:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:04:57.230+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:04:57.441 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:04:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:04:57.231+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:04:57.441 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:04:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:04:57.231+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:04:57.442 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:04:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:04:57.231+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:04:57.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:04:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:04:57.473+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:04:57.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:04:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:04:57.473+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:04:57.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:04:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:04:57.473+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:04:58.123 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:04:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:04:57.911+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:57.913234+0000) 2024-02-20T09:04:58.123 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:04:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:04:57.795+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:57.796475+0000) 2024-02-20T09:04:58.123 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:04:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:04:57.967+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:04:58.124 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:04:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:04:57.967+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:04:58.124 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:04:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:04:57.967+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:04:58.441 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:04:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:04:58.121+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:58.121838+0000) 2024-02-20T09:04:58.441 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:04:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:04:58.269+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:04:58.441 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:04:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:04:58.269+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:04:58.441 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:04:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:04:58.269+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:04:58.442 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:04:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:04:58.269+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:04:58.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:04:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:04:58.458+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:04:58.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:04:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:04:58.458+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:04:58.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:04:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:04:58.458+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:04:59.123 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:04:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:04:58.912+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:58.913439+0000) 2024-02-20T09:04:59.124 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:04:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:04:58.795+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:58.796634+0000) 2024-02-20T09:04:59.124 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:04:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:04:58.987+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:04:59.124 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:04:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:04:58.987+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:04:59.124 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:04:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:04:58.987+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:04:59.441 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:04:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:04:59.120+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:59.122199+0000) 2024-02-20T09:04:59.441 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:04:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:04:59.294+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:04:59.441 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:04:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:04:59.294+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:04:59.442 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:04:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:04:59.294+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:04:59.442 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:04:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:04:59.295+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:04:59.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:04:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:04:59.452+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:04:59.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:04:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:04:59.452+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:04:59.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:04:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:04:59.452+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:05:00.048 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:04:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:04:59.912+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:59.913624+0000) 2024-02-20T09:05:00.048 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:05:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:05:00.041+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:00.043036+0000) 2024-02-20T09:05:00.049 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:04:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:04:59.796+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:04:59.796821+0000) 2024-02-20T09:05:00.049 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:04:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:04:59.963+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:05:00.049 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:04:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:04:59.963+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:05:00.049 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:04:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:04:59.963+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:05:00.049 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:05:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:05:00.041+0000 7f8a6006f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:00.043270+0000) 2024-02-20T09:05:00.384 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:05:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:05:00.048+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 56057 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:05:00.384 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:05:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:05:00.045+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:00.046991+0000) 2024-02-20T09:05:00.385 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:05:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:05:00.121+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:00.122551+0000) 2024-02-20T09:05:00.385 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:05:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:05:00.271+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:05:00.385 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:05:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:05:00.271+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:05:00.385 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:05:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:05:00.271+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:05:00.385 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:05:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:05:00.271+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:05:00.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:05:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:05:00.408+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:05:00.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:05:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:05:00.408+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:05:00.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:05:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:05:00.408+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:05:01.126 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:05:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:05:00.912+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:00.913834+0000) 2024-02-20T09:05:01.126 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:05:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:05:00.796+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:00.797024+0000) 2024-02-20T09:05:01.127 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:05:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:05:00.951+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:05:01.127 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:05:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:05:00.951+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:05:01.127 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:05:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:05:00.951+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:05:01.391 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:05:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:05:01.124+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:01.124933+0000) 2024-02-20T09:05:01.391 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:05:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:05:01.256+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:05:01.391 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:05:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:05:01.256+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:05:01.391 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:05:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:05:01.256+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:05:01.391 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:05:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:05:01.256+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:05:01.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:05:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:05:01.388+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:05:01.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:05:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:05:01.388+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:05:01.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:05:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:05:01.388+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:05:02.129 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:05:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:05:01.912+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:01.913997+0000) 2024-02-20T09:05:02.129 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:05:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:05:01.795+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:01.797202+0000) 2024-02-20T09:05:02.129 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:05:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:05:01.903+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:05:02.130 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:05:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:05:01.903+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:05:02.130 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:05:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:05:01.903+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:05:02.423 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:05:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:05:02.127+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:02.127752+0000) 2024-02-20T09:05:02.423 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:05:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:05:02.287+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:05:02.423 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:05:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:05:02.287+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:05:02.423 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:05:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:05:02.287+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:05:02.423 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:05:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:05:02.287+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:05:02.713 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:05:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:05:02.421+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:05:02.713 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:05:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:05:02.421+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:05:02.714 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:05:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:05:02.421+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:05:02.992 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:05:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:05:02.913+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:02.914178+0000) 2024-02-20T09:05:02.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:05:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:05:02.796+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:02.797423+0000) 2024-02-20T09:05:02.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:05:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:05:02.875+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:05:02.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:05:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:05:02.875+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:05:02.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:05:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:05:02.875+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:05:03.442 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:05:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:05:03.378+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:05:03.442 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:05:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:05:03.378+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:05:03.442 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:05:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:05:03.378+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:05:03.443 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:05:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:05:03.129+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:03.130504+0000) 2024-02-20T09:05:03.443 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:05:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:05:03.306+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:05:03.443 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:05:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:05:03.306+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:05:03.443 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:05:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:05:03.306+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:05:03.443 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:05:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:05:03.306+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:05:04.132 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:05:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:05:03.913+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:03.914302+0000) 2024-02-20T09:05:04.132 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:05:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:05:03.797+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:03.797611+0000) 2024-02-20T09:05:04.133 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:05:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:05:03.859+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:05:04.133 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:05:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:05:03.859+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:05:04.133 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:05:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:05:03.859+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:05:04.405 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:05:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:05:04.129+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:04.130726+0000) 2024-02-20T09:05:04.405 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:05:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:05:04.292+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:05:04.406 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:05:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:05:04.292+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:05:04.406 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:05:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:05:04.292+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:05:04.406 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:05:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:05:04.293+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:05:04.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:05:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:05:04.402+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:05:04.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:05:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:05:04.402+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:05:04.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:05:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:05:04.402+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:05:05.053 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:05:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:05:04.913+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:04.914470+0000) 2024-02-20T09:05:05.053 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:05:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:05:04.797+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:04.797772+0000) 2024-02-20T09:05:05.053 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:05:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:05:04.818+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:05:05.054 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:05:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:05:04.818+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:05:05.054 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:05:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:05:04.818+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:05:05.337 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:05:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:05:05.050+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:05.052196+0000) 2024-02-20T09:05:05.338 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:05:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:05:05.057+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 56129 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:05:05.338 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:05:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:05:05.051+0000 7f8a6006f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:05.053184+0000) 2024-02-20T09:05:05.338 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:05:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:05:05.054+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:05.055729+0000) 2024-02-20T09:05:05.338 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:05:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:05:05.129+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:05.131026+0000) 2024-02-20T09:05:05.339 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:05:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:05:05.335+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:05:05.339 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:05:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:05:05.335+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:05:05.339 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:05:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:05:05.335+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:05:05.339 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:05:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:05:05.335+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:05:05.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:05:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:05:05.406+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:05:05.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:05:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:05:05.406+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:05:05.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:05:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:05:05.406+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:05:06.135 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:05:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:05:05.914+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:05.914657+0000) 2024-02-20T09:05:06.135 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:05:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:05:05.797+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:05.797978+0000) 2024-02-20T09:05:06.135 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:05:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:05:05.866+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:05:06.135 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:05:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:05:05.866+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:05:06.135 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:05:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:05:05.866+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:05:06.443 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:05:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:05:06.132+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:06.133529+0000) 2024-02-20T09:05:06.443 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:05:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:05:06.382+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:05:06.443 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:05:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:05:06.382+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:05:06.443 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:05:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:05:06.382+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:05:06.443 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:05:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:05:06.382+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:05:06.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:05:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:05:06.454+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:05:06.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:05:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:05:06.454+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:05:06.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:05:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:05:06.454+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:05:07.138 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:05:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:05:06.914+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:06.914811+0000) 2024-02-20T09:05:07.139 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:05:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:05:06.797+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:06.798128+0000) 2024-02-20T09:05:07.139 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:05:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:05:06.905+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:05:07.139 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:05:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:05:06.905+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:05:07.139 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:05:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:05:06.905+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:05:07.443 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:05:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:05:07.136+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:07.137109+0000) 2024-02-20T09:05:07.443 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:05:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:05:07.337+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:05:07.443 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:05:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:05:07.337+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:05:07.443 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:05:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:05:07.337+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:05:07.443 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:05:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:05:07.337+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:05:07.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:05:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:05:07.456+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:05:07.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:05:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:05:07.456+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:05:07.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:05:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:05:07.456+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:05:08.142 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:05:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:05:07.913+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:07.914952+0000) 2024-02-20T09:05:08.142 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:05:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:05:07.796+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:07.798356+0000) 2024-02-20T09:05:08.142 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:05:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:05:07.949+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:05:08.142 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:05:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:05:07.949+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:05:08.142 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:05:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:05:07.949+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:05:08.443 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:05:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:05:08.139+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:08.140555+0000) 2024-02-20T09:05:08.443 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:05:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:05:08.338+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:05:08.443 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:05:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:05:08.338+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:05:08.443 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:05:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:05:08.338+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:05:08.444 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:05:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:05:08.338+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:05:08.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:05:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:05:08.443+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:05:08.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:05:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:05:08.443+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:05:08.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:05:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:05:08.443+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:05:09.142 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:05:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:05:08.914+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:08.915153+0000) 2024-02-20T09:05:09.142 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:05:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:05:08.798+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:08.798582+0000) 2024-02-20T09:05:09.143 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:05:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:05:08.950+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:05:09.143 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:05:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:05:08.950+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:05:09.143 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:05:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:05:08.950+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:05:09.443 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:05:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:05:09.139+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:09.140969+0000) 2024-02-20T09:05:09.443 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:05:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:05:09.335+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:05:09.443 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:05:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:05:09.335+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:05:09.443 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:05:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:05:09.335+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:05:09.443 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:05:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:05:09.335+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:05:09.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:05:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:05:09.463+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:05:09.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:05:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:05:09.463+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:05:09.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:05:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:05:09.463+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:05:10.061 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:05:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:05:09.913+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:09.915365+0000) 2024-02-20T09:05:10.062 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:05:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:05:09.798+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:09.798802+0000) 2024-02-20T09:05:10.062 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:05:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:05:09.987+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:05:10.062 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:05:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:05:09.987+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:05:10.062 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:05:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:05:09.987+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:05:10.385 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:05:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:05:10.066+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 56201 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:05:10.385 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:05:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:05:10.059+0000 7f8a6006f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:10.060985+0000) 2024-02-20T09:05:10.386 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:05:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:05:10.058+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:10.060337+0000) 2024-02-20T09:05:10.386 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:05:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:05:10.062+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:10.064128+0000) 2024-02-20T09:05:10.386 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:05:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:05:10.139+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:10.141202+0000) 2024-02-20T09:05:10.386 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:05:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:05:10.298+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:05:10.386 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:05:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:05:10.298+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:05:10.386 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:05:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:05:10.298+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:05:10.387 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:05:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:05:10.298+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:05:10.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:05:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:05:10.458+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:05:10.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:05:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:05:10.458+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:05:10.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:05:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:05:10.458+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:05:11.145 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:05:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:05:10.915+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:10.915577+0000) 2024-02-20T09:05:11.145 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:05:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:05:10.798+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:10.799024+0000) 2024-02-20T09:05:11.145 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:05:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:05:11.022+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:05:11.146 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:05:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:05:11.022+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:05:11.146 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:05:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:05:11.022+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:05:11.443 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:05:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:05:11.143+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:11.143732+0000) 2024-02-20T09:05:11.443 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:05:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:05:11.347+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:05:11.444 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:05:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:05:11.347+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:05:11.444 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:05:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:05:11.347+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:05:11.444 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:05:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:05:11.347+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:05:11.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:05:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:05:11.457+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:05:11.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:05:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:05:11.457+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:05:11.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:05:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:05:11.457+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:05:12.148 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:05:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:05:11.915+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:11.915692+0000) 2024-02-20T09:05:12.148 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:05:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:05:11.798+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:11.799196+0000) 2024-02-20T09:05:12.149 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:05:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:05:11.990+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:05:12.149 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:05:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:05:11.990+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:05:12.149 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:05:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:05:11.990+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:05:12.443 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:05:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:05:12.146+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:12.146941+0000) 2024-02-20T09:05:12.444 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:05:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:05:12.310+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:05:12.444 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:05:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:05:12.310+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:05:12.444 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:05:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:05:12.310+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:05:12.444 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:05:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:05:12.310+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:05:12.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:05:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:05:12.455+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:05:12.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:05:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:05:12.455+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:05:12.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:05:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:05:12.455+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:05:13.151 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:05:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:05:12.915+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:12.915877+0000) 2024-02-20T09:05:13.151 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:05:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:05:12.799+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:12.799405+0000) 2024-02-20T09:05:13.151 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:05:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:05:12.982+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:05:13.151 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:05:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:05:12.982+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:05:13.152 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:05:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:05:12.982+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:05:13.434 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:05:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:05:13.149+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:13.149678+0000) 2024-02-20T09:05:13.435 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:05:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:05:13.273+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:05:13.435 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:05:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:05:13.273+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:05:13.435 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:05:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:05:13.273+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:05:13.435 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:05:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:05:13.273+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:05:13.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:05:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:05:13.432+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:05:13.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:05:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:05:13.432+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:05:13.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:05:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:05:13.432+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:05:14.151 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:05:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:05:13.915+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:13.916072+0000) 2024-02-20T09:05:14.152 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:05:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:05:13.799+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:13.799572+0000) 2024-02-20T09:05:14.152 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:05:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:05:13.997+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:05:14.152 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:05:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:05:13.997+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:05:14.152 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:05:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:05:13.997+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:05:14.444 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:05:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:05:14.149+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:14.149957+0000) 2024-02-20T09:05:14.444 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:05:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:05:14.277+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:05:14.444 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:05:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:05:14.277+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:05:14.444 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:05:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:05:14.277+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:05:14.444 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:05:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:05:14.277+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:05:14.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:05:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:05:14.466+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:05:14.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:05:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:05:14.466+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:05:14.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:05:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:05:14.466+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:05:15.071 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:05:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:05:14.915+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:14.916267+0000) 2024-02-20T09:05:15.071 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:05:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:05:14.799+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:14.799786+0000) 2024-02-20T09:05:15.071 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:05:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:05:14.952+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:05:15.071 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:05:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:05:14.952+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:05:15.072 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:05:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:05:14.952+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:05:15.072 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:05:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:05:15.069+0000 7f8a5f86e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:15.069646+0000) 2024-02-20T09:05:15.444 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:05:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:05:15.069+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:15.069716+0000) 2024-02-20T09:05:15.444 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:05:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:05:15.075+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 56273 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:05:15.445 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:05:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:05:15.071+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:15.072066+0000) 2024-02-20T09:05:15.445 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:05:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:05:15.149+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:15.150272+0000) 2024-02-20T09:05:15.445 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:05:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:05:15.233+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:05:15.445 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:05:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:05:15.233+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:05:15.445 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:05:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:05:15.233+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:05:15.445 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:05:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:05:15.233+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:05:15.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:05:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:05:15.447+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:05:15.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:05:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:05:15.447+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:05:15.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:05:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:05:15.447+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:05:16.155 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:05:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:05:15.916+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:15.916468+0000) 2024-02-20T09:05:16.156 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:05:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:05:15.799+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:15.799972+0000) 2024-02-20T09:05:16.156 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:05:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:05:15.991+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:05:16.156 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:05:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:05:15.991+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:05:16.156 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:05:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:05:15.991+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:05:16.444 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:05:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:05:16.153+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:16.154133+0000) 2024-02-20T09:05:16.444 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:05:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:05:16.229+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:05:16.444 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:05:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:05:16.229+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:05:16.444 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:05:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:05:16.229+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:05:16.445 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:05:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:05:16.229+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:05:16.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:05:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:05:16.496+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:05:16.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:05:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:05:16.496+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:05:16.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:05:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:05:16.496+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:05:17.159 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:05:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:05:16.916+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:16.916651+0000) 2024-02-20T09:05:17.160 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:05:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:05:16.799+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:16.800141+0000) 2024-02-20T09:05:17.160 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:05:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:05:17.028+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:05:17.160 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:05:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:05:17.028+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:05:17.160 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:05:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:05:17.028+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:05:17.444 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:05:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:05:17.157+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:17.157954+0000) 2024-02-20T09:05:17.444 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:05:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:05:17.213+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:05:17.444 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:05:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:05:17.213+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:05:17.445 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:05:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:05:17.213+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:05:17.445 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:05:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:05:17.213+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:05:17.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:05:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:05:17.458+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:05:17.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:05:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:05:17.458+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:05:17.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:05:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:05:17.458+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:05:18.162 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:05:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:05:17.916+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:17.916855+0000) 2024-02-20T09:05:18.162 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:05:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:05:17.799+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:17.800303+0000) 2024-02-20T09:05:18.162 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:05:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:05:18.026+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:05:18.163 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:05:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:05:18.026+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:05:18.163 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:05:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:05:18.026+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:05:18.444 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:05:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:05:18.160+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:18.160718+0000) 2024-02-20T09:05:18.444 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:05:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:05:18.246+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:05:18.445 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:05:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:05:18.246+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:05:18.445 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:05:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:05:18.246+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:05:18.445 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:05:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:05:18.246+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:05:18.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:05:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:05:18.480+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:05:18.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:05:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:05:18.480+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:05:18.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:05:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:05:18.480+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:05:19.162 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:05:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:05:18.916+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:18.917055+0000) 2024-02-20T09:05:19.163 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:05:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:05:18.800+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:18.800459+0000) 2024-02-20T09:05:19.163 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:05:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:05:18.981+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:05:19.163 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:05:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:05:18.981+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:05:19.163 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:05:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:05:18.981+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:05:19.444 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:05:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:05:19.160+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:19.161043+0000) 2024-02-20T09:05:19.445 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:05:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:05:19.217+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:05:19.445 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:05:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:05:19.217+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:05:19.445 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:05:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:05:19.217+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:05:19.445 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:05:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:05:19.217+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:05:19.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:05:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:05:19.488+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:05:19.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:05:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:05:19.488+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:05:19.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:05:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:05:19.488+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:05:20.079 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:05:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:05:19.916+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:19.917262+0000) 2024-02-20T09:05:20.079 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:05:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:05:19.800+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:19.800648+0000) 2024-02-20T09:05:20.079 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:05:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:05:19.947+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:05:20.079 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:05:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:05:19.947+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:05:20.080 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:05:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:05:19.947+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:05:20.385 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:05:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:05:20.077+0000 7f8a6006f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:20.077718+0000) 2024-02-20T09:05:20.386 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:05:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:05:20.077+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:20.078154+0000) 2024-02-20T09:05:20.386 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:05:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:05:20.084+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 56345 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:05:20.386 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:05:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:05:20.080+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:20.081320+0000) 2024-02-20T09:05:20.387 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:05:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:05:20.161+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:20.161334+0000) 2024-02-20T09:05:20.387 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:05:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:05:20.174+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:05:20.387 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:05:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:05:20.174+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:05:20.387 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:05:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:05:20.174+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:05:20.387 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:05:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:05:20.174+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:05:20.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:05:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:05:20.475+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:05:20.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:05:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:05:20.475+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:05:20.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:05:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:05:20.475+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:05:21.145 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:05:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:05:20.917+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:20.917458+0000) 2024-02-20T09:05:21.145 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:05:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:05:20.800+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:20.800808+0000) 2024-02-20T09:05:21.145 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:05:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:05:20.939+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:05:21.146 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:05:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:05:20.939+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:05:21.146 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:05:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:05:20.939+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:05:21.445 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:05:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:05:21.157+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:05:21.445 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:05:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:05:21.157+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:05:21.445 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:05:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:05:21.157+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:05:21.445 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:05:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:05:21.157+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:05:21.445 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:05:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:05:21.163+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:21.164052+0000) 2024-02-20T09:05:21.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:05:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:05:21.491+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:05:21.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:05:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:05:21.491+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:05:21.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:05:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:05:21.491+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:05:22.168 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:05:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:05:21.917+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:21.917579+0000) 2024-02-20T09:05:22.168 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:05:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:05:21.800+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:21.800967+0000) 2024-02-20T09:05:22.169 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:05:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:05:21.912+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:05:22.169 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:05:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:05:21.912+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:05:22.169 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:05:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:05:21.912+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:05:22.445 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:05:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:05:22.166+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:22.166991+0000) 2024-02-20T09:05:22.445 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:05:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:05:22.177+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:05:22.445 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:05:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:05:22.177+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:05:22.445 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:05:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:05:22.177+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:05:22.446 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:05:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:05:22.178+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:05:22.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:05:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:05:22.506+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:05:22.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:05:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:05:22.506+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:05:22.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:05:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:05:22.506+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:05:23.171 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:05:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:05:22.917+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:22.917792+0000) 2024-02-20T09:05:23.171 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:05:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:05:22.800+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:22.801165+0000) 2024-02-20T09:05:23.171 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:05:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:05:22.933+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:05:23.172 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:05:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:05:22.933+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:05:23.172 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:05:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:05:22.933+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:05:23.445 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:05:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:05:23.169+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:23.169758+0000) 2024-02-20T09:05:23.445 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:05:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:05:23.178+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:05:23.445 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:05:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:05:23.178+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:05:23.446 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:05:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:05:23.178+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:05:23.446 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:05:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:05:23.178+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:05:23.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:05:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:05:23.460+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:05:23.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:05:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:05:23.460+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:05:23.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:05:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:05:23.460+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:05:24.171 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:05:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:05:23.917+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:23.917993+0000) 2024-02-20T09:05:24.172 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:05:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:05:23.801+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:23.801356+0000) 2024-02-20T09:05:24.172 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:05:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:05:23.889+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:05:24.172 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:05:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:05:23.889+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:05:24.172 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:05:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:05:23.889+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:05:24.445 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:05:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:05:24.169+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:24.170129+0000) 2024-02-20T09:05:24.446 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:05:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:05:24.198+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:05:24.446 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:05:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:05:24.198+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:05:24.446 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:05:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:05:24.198+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:05:24.446 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:05:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:05:24.198+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:05:24.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:05:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:05:24.507+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:05:24.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:05:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:05:24.507+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:05:24.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:05:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:05:24.507+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:05:25.088 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:05:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:05:24.917+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:24.918150+0000) 2024-02-20T09:05:25.089 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:05:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:05:24.801+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:24.801536+0000) 2024-02-20T09:05:25.089 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:05:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:05:24.853+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:05:25.089 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:05:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:05:24.853+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:05:25.089 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:05:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:05:24.853+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:05:25.445 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:05:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:05:25.093+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 56417 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:05:25.446 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:05:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:05:25.086+0000 7f8a5f86e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:25.086708+0000) 2024-02-20T09:05:25.446 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:05:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:05:25.086+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:25.086881+0000) 2024-02-20T09:05:25.446 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:05:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:05:25.089+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:25.089929+0000) 2024-02-20T09:05:25.447 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:05:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:05:25.170+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:25.170441+0000) 2024-02-20T09:05:25.447 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:05:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:05:25.229+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:05:25.447 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:05:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:05:25.229+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:05:25.447 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:05:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:05:25.229+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:05:25.447 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:05:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:05:25.229+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:05:25.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:05:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:05:25.463+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:05:25.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:05:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:05:25.463+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:05:25.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:05:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:05:25.463+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:05:26.175 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:05:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:05:25.918+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:25.918333+0000) 2024-02-20T09:05:26.176 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:05:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:05:25.801+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:25.801675+0000) 2024-02-20T09:05:26.176 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:05:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:05:25.885+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:05:26.176 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:05:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:05:25.885+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:05:26.176 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:05:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:05:25.885+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:05:26.446 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:05:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:05:26.174+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:26.174228+0000) 2024-02-20T09:05:26.446 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:05:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:05:26.214+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:05:26.446 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:05:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:05:26.214+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:05:26.446 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:05:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:05:26.214+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:05:26.446 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:05:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:05:26.214+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:05:26.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:05:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:05:26.447+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:05:26.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:05:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:05:26.447+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:05:26.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:05:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:05:26.447+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:05:27.179 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:05:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:05:26.918+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:26.918419+0000) 2024-02-20T09:05:27.179 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:05:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:05:26.801+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:26.801783+0000) 2024-02-20T09:05:27.179 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:05:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:05:26.924+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:05:27.179 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:05:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:05:26.924+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:05:27.179 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:05:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:05:26.924+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:05:27.446 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:05:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:05:27.421+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:05:27.446 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:05:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:05:27.421+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:05:27.446 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:05:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:05:27.421+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:05:27.446 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:05:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:05:27.177+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:27.177538+0000) 2024-02-20T09:05:27.446 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:05:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:05:27.198+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:05:27.447 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:05:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:05:27.198+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:05:27.447 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:05:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:05:27.198+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:05:27.447 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:05:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:05:27.198+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:05:28.181 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:05:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:05:27.918+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:27.918572+0000) 2024-02-20T09:05:28.181 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:05:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:05:27.801+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:27.801937+0000) 2024-02-20T09:05:28.181 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:05:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:05:27.957+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:05:28.182 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:05:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:05:27.957+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:05:28.182 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:05:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:05:27.957+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:05:28.446 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:05:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:05:28.377+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:05:28.446 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:05:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:05:28.377+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:05:28.446 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:05:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:05:28.377+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:05:28.447 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:05:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:05:28.179+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:28.179812+0000) 2024-02-20T09:05:28.447 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:05:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:05:28.200+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:05:28.447 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:05:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:05:28.200+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:05:28.447 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:05:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:05:28.200+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:05:28.448 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:05:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:05:28.200+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:05:29.181 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:05:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:05:28.918+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:28.918759+0000) 2024-02-20T09:05:29.182 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:05:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:05:28.801+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:28.802102+0000) 2024-02-20T09:05:29.182 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:05:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:05:28.986+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:05:29.182 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:05:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:05:28.986+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:05:29.182 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:05:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:05:28.986+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:05:29.446 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:05:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:05:29.358+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:05:29.446 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:05:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:05:29.358+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:05:29.446 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:05:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:05:29.358+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:05:29.447 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:05:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:05:29.180+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:29.180191+0000) 2024-02-20T09:05:29.447 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:05:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:05:29.249+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:05:29.447 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:05:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:05:29.249+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:05:29.447 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:05:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:05:29.249+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:05:29.447 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:05:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:05:29.249+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:05:30.097 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:05:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:05:29.918+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:29.918945+0000) 2024-02-20T09:05:30.098 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:05:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:05:29.802+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:29.802292+0000) 2024-02-20T09:05:30.098 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:05:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:05:29.952+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:05:30.098 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:05:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:05:29.952+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:05:30.098 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:05:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:05:29.952+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:05:30.373 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:05:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:05:30.095+0000 7f8a6006f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:30.095968+0000) 2024-02-20T09:05:30.373 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:05:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:05:30.102+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 56489 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:05:30.373 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:05:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:05:30.096+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:30.096659+0000) 2024-02-20T09:05:30.374 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:05:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:05:30.370+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:05:30.374 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:05:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:05:30.370+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:05:30.374 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:05:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:05:30.099+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:30.099540+0000) 2024-02-20T09:05:30.374 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:05:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:05:30.180+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:30.180539+0000) 2024-02-20T09:05:30.374 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:05:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:05:30.202+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:05:30.374 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:05:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:05:30.202+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:05:30.374 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:05:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:05:30.202+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:05:30.374 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:05:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:05:30.202+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:05:30.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:05:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:05:30.370+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:05:31.184 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:05:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:05:30.918+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:30.919176+0000) 2024-02-20T09:05:31.185 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:05:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:05:30.802+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:30.802488+0000) 2024-02-20T09:05:31.185 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:05:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:05:30.982+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:05:31.185 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:05:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:05:30.982+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:05:31.185 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:05:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:05:30.982+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:05:31.446 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:05:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:05:31.371+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:05:31.447 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:05:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:05:31.371+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:05:31.447 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:05:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:05:31.371+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:05:31.447 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:05:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:05:31.182+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:31.182913+0000) 2024-02-20T09:05:31.447 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:05:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:05:31.183+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:05:31.447 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:05:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:05:31.183+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:05:31.447 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:05:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:05:31.183+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:05:31.448 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:05:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:05:31.183+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:05:32.187 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:05:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:05:31.918+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:31.919370+0000) 2024-02-20T09:05:32.188 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:05:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:05:31.802+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:31.802611+0000) 2024-02-20T09:05:32.188 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:05:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:05:32.031+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:05:32.188 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:05:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:05:32.031+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:05:32.188 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:05:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:05:32.031+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:05:32.447 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:05:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:05:32.341+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:05:32.447 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:05:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:05:32.341+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:05:32.447 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:05:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:05:32.341+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:05:32.447 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:05:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:05:32.185+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:32.185737+0000) 2024-02-20T09:05:32.447 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:05:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:05:32.220+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:05:32.447 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:05:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:05:32.220+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:05:32.447 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:05:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:05:32.220+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:05:32.448 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:05:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:05:32.220+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:05:33.191 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:05:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:05:32.919+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:32.919532+0000) 2024-02-20T09:05:33.191 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:05:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:05:32.802+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:32.802808+0000) 2024-02-20T09:05:33.191 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:05:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:05:32.991+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:05:33.191 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:05:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:05:32.991+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:05:33.191 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:05:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:05:32.991+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:05:33.447 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:05:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:05:33.391+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:05:33.447 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:05:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:05:33.391+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:05:33.447 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:05:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:05:33.391+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:05:33.447 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:05:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:05:33.188+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:33.188998+0000) 2024-02-20T09:05:33.447 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:05:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:05:33.189+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:05:33.447 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:05:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:05:33.189+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:05:33.448 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:05:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:05:33.189+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:05:33.448 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:05:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:05:33.190+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:05:34.176 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:05:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:05:33.918+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:33.919651+0000) 2024-02-20T09:05:34.177 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:05:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:05:33.801+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:33.802933+0000) 2024-02-20T09:05:34.177 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:05:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:05:34.003+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:05:34.177 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:05:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:05:34.003+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:05:34.177 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:05:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:05:34.003+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:05:34.447 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:05:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:05:34.357+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:05:34.447 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:05:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:05:34.357+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:05:34.447 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:05:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:05:34.357+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:05:34.448 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:05:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:05:34.174+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:05:34.448 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:05:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:05:34.174+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:05:34.448 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:05:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:05:34.174+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:05:34.448 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:05:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:05:34.174+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:05:34.448 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:05:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:05:34.187+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:34.189355+0000) 2024-02-20T09:05:35.052 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:05:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:05:34.918+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:34.919833+0000) 2024-02-20T09:05:35.053 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:05:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:05:34.802+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:34.803118+0000) 2024-02-20T09:05:35.053 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:05:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:05:35.049+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:05:35.318 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:05:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:05:35.109+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 56562 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:05:35.319 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:05:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:05:35.103+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:35.105275+0000) 2024-02-20T09:05:35.319 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:05:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:05:35.049+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:05:35.319 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:05:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:05:35.049+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:05:35.320 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:05:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:05:35.102+0000 7f8a5f86e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:35.104333+0000) 2024-02-20T09:05:35.320 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:05:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:05:35.107+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:35.108933+0000) 2024-02-20T09:05:35.320 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:05:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:05:35.142+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:05:35.320 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:05:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:05:35.142+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:05:35.320 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:05:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:05:35.142+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:05:35.321 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:05:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:05:35.142+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:05:35.321 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:05:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:05:35.188+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:35.189710+0000) 2024-02-20T09:05:35.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:05:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:05:35.330+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:05:35.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:05:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:05:35.330+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:05:35.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:05:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:05:35.330+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:05:36.060 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:05:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:05:35.918+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:35.919991+0000) 2024-02-20T09:05:36.061 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:05:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:05:35.802+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:35.803328+0000) 2024-02-20T09:05:36.061 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:05:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:05:36.057+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:05:36.321 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:05:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:05:36.057+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:05:36.321 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:05:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:05:36.057+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:05:36.322 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:05:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:05:36.142+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:05:36.322 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:05:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:05:36.142+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:05:36.322 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:05:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:05:36.142+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:05:36.322 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:05:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:05:36.142+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:05:36.322 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:05:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:05:36.191+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:36.192304+0000) 2024-02-20T09:05:36.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:05:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:05:36.363+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:05:36.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:05:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:05:36.363+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:05:36.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:05:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:05:36.363+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:05:37.054 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:05:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:05:36.802+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:36.803444+0000) 2024-02-20T09:05:37.054 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:05:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:05:36.919+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:36.920150+0000) 2024-02-20T09:05:37.322 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:05:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:05:37.051+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:05:37.322 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:05:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:05:37.051+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:05:37.322 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:05:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:05:37.051+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:05:37.322 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:05:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:05:37.100+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:05:37.323 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:05:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:05:37.100+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:05:37.323 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:05:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:05:37.100+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:05:37.323 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:05:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:05:37.100+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:05:37.323 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:05:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:05:37.194+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:37.195236+0000) 2024-02-20T09:05:37.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:05:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:05:37.352+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:05:37.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:05:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:05:37.352+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:05:37.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:05:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:05:37.352+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:05:38.118 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:05:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:05:37.919+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:37.920315+0000) 2024-02-20T09:05:38.119 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:05:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:05:37.802+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:37.803628+0000) 2024-02-20T09:05:38.119 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:05:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:05:38.019+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:05:38.119 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:05:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:05:38.019+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:05:38.119 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:05:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:05:38.019+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:05:38.448 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:05:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:05:38.351+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:05:38.448 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:05:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:05:38.351+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:05:38.448 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:05:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:05:38.351+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:05:38.448 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:05:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:05:38.116+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:05:38.448 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:05:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:05:38.116+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:05:38.448 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:05:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:05:38.116+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:05:38.449 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:05:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:05:38.116+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:05:38.449 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:05:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:05:38.197+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:38.198126+0000) 2024-02-20T09:05:39.054 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:05:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:05:38.919+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:38.920419+0000) 2024-02-20T09:05:39.054 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:05:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:05:38.802+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:38.803844+0000) 2024-02-20T09:05:39.054 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:05:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:05:39.051+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:05:39.322 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:05:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:05:39.051+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:05:39.322 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:05:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:05:39.051+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:05:39.323 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:05:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:05:39.135+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:05:39.323 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:05:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:05:39.135+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:05:39.323 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:05:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:05:39.135+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:05:39.323 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:05:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:05:39.135+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:05:39.323 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:05:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:05:39.197+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:39.198477+0000) 2024-02-20T09:05:39.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:05:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:05:39.325+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:05:39.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:05:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:05:39.325+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:05:39.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:05:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:05:39.325+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:05:40.113 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:05:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:05:39.919+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:39.920565+0000) 2024-02-20T09:05:40.114 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:05:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:05:39.803+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:39.804053+0000) 2024-02-20T09:05:40.114 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:05:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:05:40.006+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:05:40.114 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:05:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:05:40.006+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:05:40.114 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:05:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:05:40.006+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:05:40.114 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:05:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:05:40.110+0000 7f8a6006f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:40.112278+0000) 2024-02-20T09:05:40.387 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:05:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:05:40.116+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 56635 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:05:40.388 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:05:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:05:40.111+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:40.112787+0000) 2024-02-20T09:05:40.388 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:05:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:05:40.315+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:05:40.388 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:05:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:05:40.315+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:05:40.388 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:05:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:05:40.315+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:05:40.388 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:05:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:05:40.114+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:40.115691+0000) 2024-02-20T09:05:40.388 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:05:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:05:40.118+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:05:40.389 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:05:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:05:40.118+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:05:40.389 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:05:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:05:40.118+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:05:40.389 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:05:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:05:40.118+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:05:40.389 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:05:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:05:40.197+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:40.198774+0000) 2024-02-20T09:05:41.090 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:05:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:05:40.919+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:40.920756+0000) 2024-02-20T09:05:41.090 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:05:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:05:40.803+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:40.804237+0000) 2024-02-20T09:05:41.091 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:05:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:05:41.026+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:05:41.091 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:05:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:05:41.026+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:05:41.091 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:05:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:05:41.026+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:05:41.341 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:05:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:05:41.087+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:05:41.341 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:05:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:05:41.087+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:05:41.341 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:05:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:05:41.087+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:05:41.341 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:05:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:05:41.087+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:05:41.341 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:05:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:05:41.200+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:41.200904+0000) 2024-02-20T09:05:41.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:05:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:05:41.338+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:05:41.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:05:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:05:41.338+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:05:41.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:05:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:05:41.338+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:05:42.123 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:05:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:05:41.919+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:41.920934+0000) 2024-02-20T09:05:42.123 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:05:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:05:41.803+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:41.804366+0000) 2024-02-20T09:05:42.124 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:05:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:05:42.021+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:05:42.124 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:05:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:05:42.021+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:05:42.124 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:05:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:05:42.021+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:05:42.448 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:05:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:05:42.120+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:05:42.449 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:05:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:05:42.120+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:05:42.449 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:05:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:05:42.120+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:05:42.449 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:05:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:05:42.120+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:05:42.449 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:05:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:05:42.202+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:42.203836+0000) 2024-02-20T09:05:42.449 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:05:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:05:42.336+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:05:42.450 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:05:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:05:42.336+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:05:42.450 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:05:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:05:42.336+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:05:43.145 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:05:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:05:42.920+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:42.921142+0000) 2024-02-20T09:05:43.146 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:05:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:05:42.803+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:42.804560+0000) 2024-02-20T09:05:43.146 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:05:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:05:42.992+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:05:43.146 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:05:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:05:42.992+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:05:43.146 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:05:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:05:42.992+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:05:43.448 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:05:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:05:43.349+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:05:43.448 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:05:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:05:43.349+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:05:43.449 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:05:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:05:43.349+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:05:43.449 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:05:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:05:43.143+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:05:43.449 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:05:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:05:43.143+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:05:43.449 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:05:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:05:43.143+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:05:43.449 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:05:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:05:43.143+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:05:43.449 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:05:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:05:43.205+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:43.206603+0000) 2024-02-20T09:05:44.124 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:05:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:05:43.920+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:43.921332+0000) 2024-02-20T09:05:44.125 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:05:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:05:43.803+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:43.804749+0000) 2024-02-20T09:05:44.125 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:05:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:05:43.990+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:05:44.125 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:05:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:05:43.990+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:05:44.125 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:05:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:05:43.990+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:05:44.449 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:05:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:05:44.352+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:05:44.449 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:05:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:05:44.352+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:05:44.449 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:05:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:05:44.352+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:05:44.449 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:05:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:05:44.122+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:05:44.450 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:05:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:05:44.122+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:05:44.450 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:05:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:05:44.122+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:05:44.450 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:05:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:05:44.122+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:05:44.450 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:05:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:05:44.205+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:44.206917+0000) 2024-02-20T09:05:45.087 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:05:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:05:44.920+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:44.921507+0000) 2024-02-20T09:05:45.087 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:05:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:05:44.804+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:44.804936+0000) 2024-02-20T09:05:45.087 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:05:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:05:44.987+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:05:45.087 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:05:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:05:44.987+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:05:45.087 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:05:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:05:44.987+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:05:45.393 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:05:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:05:45.125+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 56706 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:05:45.393 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:05:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:05:45.118+0000 7f8a6006f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:45.120358+0000) 2024-02-20T09:05:45.394 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:05:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:05:45.118+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:45.120133+0000) 2024-02-20T09:05:45.394 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:05:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:05:45.390+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:05:45.394 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:05:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:05:45.390+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:05:45.394 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:05:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:05:45.084+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:05:45.394 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:05:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:05:45.084+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:05:45.394 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:05:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:05:45.084+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:05:45.395 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:05:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:05:45.084+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:05:45.395 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:05:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:05:45.121+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:45.123244+0000) 2024-02-20T09:05:45.395 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:05:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:05:45.206+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:45.207185+0000) 2024-02-20T09:05:45.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:05:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:05:45.390+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:05:46.058 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:05:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:05:45.920+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:45.921693+0000) 2024-02-20T09:05:46.058 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:05:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:05:45.803+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:45.805138+0000) 2024-02-20T09:05:46.058 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:05:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:05:46.007+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:05:46.058 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:05:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:05:46.007+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:05:46.058 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:05:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:05:46.007+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:05:46.330 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:05:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:05:46.055+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:05:46.330 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:05:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:05:46.055+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:05:46.330 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:05:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:05:46.055+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:05:46.331 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:05:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:05:46.055+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:05:46.331 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:05:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:05:46.209+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:46.209911+0000) 2024-02-20T09:05:46.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:05:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:05:46.376+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:05:46.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:05:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:05:46.376+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:05:46.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:05:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:05:46.376+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:05:47.044 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:05:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:05:46.921+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:46.921831+0000) 2024-02-20T09:05:47.045 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:05:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:05:46.803+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:46.805293+0000) 2024-02-20T09:05:47.045 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:05:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:05:47.042+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:05:47.331 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:05:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:05:47.042+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:05:47.331 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:05:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:05:47.042+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:05:47.331 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:05:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:05:47.083+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:05:47.331 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:05:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:05:47.083+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:05:47.331 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:05:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:05:47.083+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:05:47.331 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:05:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:05:47.083+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:05:47.332 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:05:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:05:47.211+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:47.212614+0000) 2024-02-20T09:05:47.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:05:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:05:47.330+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:05:47.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:05:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:05:47.330+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:05:47.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:05:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:05:47.330+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:05:48.065 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:05:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:05:47.921+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:47.922009+0000) 2024-02-20T09:05:48.066 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:05:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:05:47.804+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:47.805474+0000) 2024-02-20T09:05:48.066 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:05:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:05:48.063+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:05:48.066 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:05:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:05:48.063+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:05:48.066 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:05:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:05:48.063+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:05:48.331 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:05:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:05:48.288+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:05:48.331 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:05:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:05:48.288+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:05:48.331 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:05:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:05:48.288+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:05:48.331 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:05:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:05:48.110+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:05:48.332 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:05:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:05:48.110+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:05:48.332 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:05:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:05:48.110+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:05:48.332 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:05:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:05:48.111+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:05:48.332 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:05:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:05:48.214+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:48.215347+0000) 2024-02-20T09:05:49.077 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:05:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:05:48.804+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:48.805604+0000) 2024-02-20T09:05:49.078 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:05:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:05:48.921+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:48.922209+0000) 2024-02-20T09:05:49.331 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:05:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:05:49.074+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:05:49.331 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:05:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:05:49.074+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:05:49.331 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:05:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:05:49.074+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:05:49.332 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:05:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:05:49.130+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:05:49.332 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:05:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:05:49.130+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:05:49.332 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:05:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:05:49.130+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:05:49.332 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:05:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:05:49.130+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:05:49.332 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:05:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:05:49.214+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:49.215633+0000) 2024-02-20T09:05:49.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:05:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:05:49.331+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:05:49.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:05:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:05:49.331+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:05:49.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:05:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:05:49.331+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:05:50.072 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:05:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:05:49.921+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:49.922464+0000) 2024-02-20T09:05:50.072 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:05:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:05:49.804+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:49.805737+0000) 2024-02-20T09:05:50.072 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:05:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:05:50.069+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:05:50.331 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:05:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:05:50.133+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 56779 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:05:50.332 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:05:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:05:50.127+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:50.128600+0000) 2024-02-20T09:05:50.332 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:05:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:05:50.069+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:05:50.332 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:05:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:05:50.069+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:05:50.332 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:05:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:05:50.127+0000 7f8a6006f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:50.128518+0000) 2024-02-20T09:05:50.333 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:05:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:05:50.087+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:05:50.333 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:05:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:05:50.087+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:05:50.333 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:05:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:05:50.087+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:05:50.333 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:05:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:05:50.087+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:05:50.333 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:05:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:05:50.131+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:50.132528+0000) 2024-02-20T09:05:50.333 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:05:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:05:50.215+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:50.215963+0000) 2024-02-20T09:05:50.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:05:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:05:50.366+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:05:50.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:05:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:05:50.366+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:05:50.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:05:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:05:50.366+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:05:51.049 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:05:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:05:50.921+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:50.922673+0000) 2024-02-20T09:05:51.049 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:05:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:05:50.805+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:50.805890+0000) 2024-02-20T09:05:51.335 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:05:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:05:51.047+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:05:51.335 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:05:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:05:51.047+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:05:51.335 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:05:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:05:51.047+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:05:51.336 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:05:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:05:51.046+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:05:51.336 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:05:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:05:51.046+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:05:51.336 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:05:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:05:51.046+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:05:51.336 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:05:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:05:51.046+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:05:51.336 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:05:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:05:51.217+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:51.218381+0000) 2024-02-20T09:05:51.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:05:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:05:51.378+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:05:51.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:05:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:05:51.378+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:05:51.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:05:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:05:51.378+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:05:52.222 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:05:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:05:51.922+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:51.922841+0000) 2024-02-20T09:05:52.223 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:05:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:05:52.031+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:05:52.223 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:05:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:05:52.031+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:05:52.223 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:05:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:05:52.031+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:05:52.223 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:05:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:05:52.031+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:05:52.223 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:05:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:05:51.805+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:51.805986+0000) 2024-02-20T09:05:52.223 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:05:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:05:52.012+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:05:52.223 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:05:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:05:52.012+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:05:52.224 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:05:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:05:52.012+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:05:52.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:05:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:05:52.220+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:52.221119+0000) 2024-02-20T09:05:52.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:05:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:05:52.419+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:05:52.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:05:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:05:52.419+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:05:52.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:05:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:05:52.419+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:05:53.225 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:05:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:05:52.922+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:52.923019+0000) 2024-02-20T09:05:53.225 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:05:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:05:53.009+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:05:53.226 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:05:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:05:53.009+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:05:53.226 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:05:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:05:53.009+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:05:53.226 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:05:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:05:53.009+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:05:53.226 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:05:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:05:52.804+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:52.806203+0000) 2024-02-20T09:05:53.226 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:05:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:05:53.034+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:05:53.226 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:05:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:05:53.034+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:05:53.226 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:05:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:05:53.034+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:05:53.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:05:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:05:53.223+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:53.223805+0000) 2024-02-20T09:05:53.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:05:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:05:53.380+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:05:53.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:05:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:05:53.380+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:05:53.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:05:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:05:53.380+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:05:54.051 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:05:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:05:53.921+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:53.923191+0000) 2024-02-20T09:05:54.051 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:05:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:05:53.805+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:53.806370+0000) 2024-02-20T09:05:54.052 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:05:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:05:54.029+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:05:54.052 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:05:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:05:54.029+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:05:54.052 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:05:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:05:54.029+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:05:54.336 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:05:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:05:54.049+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:05:54.336 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:05:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:05:54.049+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:05:54.337 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:05:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:05:54.049+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:05:54.337 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:05:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:05:54.049+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:05:54.337 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:05:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:05:54.222+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:54.224111+0000) 2024-02-20T09:05:54.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:05:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:05:54.344+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:05:54.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:05:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:05:54.344+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:05:54.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:05:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:05:54.344+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:05:55.055 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:05:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:05:54.922+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:54.923392+0000) 2024-02-20T09:05:55.056 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:05:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:05:54.805+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:54.806539+0000) 2024-02-20T09:05:55.056 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:05:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:05:55.052+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:05:55.056 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:05:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:05:55.052+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:05:55.056 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:05:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:05:55.052+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:05:55.056 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:05:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:05:55.038+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:05:55.056 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:05:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:05:55.038+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:05:55.057 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:05:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:05:55.038+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:05:55.057 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:05:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:05:55.039+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:05:55.336 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:05:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:05:55.135+0000 7f8a5f86e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:55.136670+0000) 2024-02-20T09:05:55.337 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:05:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:05:55.136+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:55.137811+0000) 2024-02-20T09:05:55.337 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:05:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:05:55.138+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:55.140352+0000) 2024-02-20T09:05:55.337 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:05:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:05:55.223+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:55.224432+0000) 2024-02-20T09:05:55.337 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:05:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:05:55.141+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 56851 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:05:55.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:05:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:05:55.393+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:05:55.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:05:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:05:55.393+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:05:55.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:05:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:05:55.393+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:05:56.048 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:05:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:05:55.922+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:55.923584+0000) 2024-02-20T09:05:56.048 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:05:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:05:55.806+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:55.806694+0000) 2024-02-20T09:05:56.048 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:05:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:05:56.045+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:05:56.048 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:05:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:05:56.045+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:05:56.048 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:05:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:05:56.045+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:05:56.339 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:05:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:05:56.055+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:05:56.340 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:05:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:05:56.055+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:05:56.340 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:05:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:05:56.055+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:05:56.340 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:05:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:05:56.055+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:05:56.340 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:05:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:05:56.226+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:56.226914+0000) 2024-02-20T09:05:56.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:05:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:05:56.361+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:05:56.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:05:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:05:56.361+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:05:56.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:05:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:05:56.361+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:05:57.057 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:05:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:05:56.806+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:56.806848+0000) 2024-02-20T09:05:57.057 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:05:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:05:56.923+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:56.923756+0000) 2024-02-20T09:05:57.340 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:05:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:05:57.065+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:05:57.340 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:05:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:05:57.065+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:05:57.340 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:05:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:05:57.065+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:05:57.341 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:05:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:05:57.055+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:05:57.341 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:05:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:05:57.055+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:05:57.341 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:05:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:05:57.055+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:05:57.341 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:05:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:05:57.055+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:05:57.341 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:05:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:05:57.229+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:57.229682+0000) 2024-02-20T09:05:57.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:05:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:05:57.385+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:05:57.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:05:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:05:57.385+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:05:57.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:05:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:05:57.385+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:05:58.084 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:05:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:05:57.806+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:57.807034+0000) 2024-02-20T09:05:58.085 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:05:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:05:57.922+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:57.923873+0000) 2024-02-20T09:05:58.340 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:05:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:05:58.084+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:05:58.340 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:05:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:05:58.084+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:05:58.340 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:05:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:05:58.084+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:05:58.341 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:05:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:05:58.082+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:05:58.341 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:05:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:05:58.082+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:05:58.341 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:05:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:05:58.082+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:05:58.341 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:05:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:05:58.082+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:05:58.341 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:05:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:05:58.232+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:58.232631+0000) 2024-02-20T09:05:58.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:05:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:05:58.362+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:05:58.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:05:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:05:58.362+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:05:58.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:05:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:05:58.362+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:05:59.056 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:05:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:05:58.806+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:58.807168+0000) 2024-02-20T09:05:59.056 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:05:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:05:58.922+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:58.924088+0000) 2024-02-20T09:05:59.341 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:05:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:05:59.057+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:05:59.341 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:05:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:05:59.057+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:05:59.341 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:05:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:05:59.057+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:05:59.341 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:05:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:05:59.054+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:05:59.341 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:05:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:05:59.054+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:05:59.342 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:05:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:05:59.054+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:05:59.342 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:05:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:05:59.054+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:05:59.342 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:05:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:05:59.232+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:59.232950+0000) 2024-02-20T09:05:59.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:05:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:05:59.352+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:05:59.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:05:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:05:59.352+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:05:59.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:05:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:05:59.352+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:06:00.109 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:05:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:05:59.922+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:59.924269+0000) 2024-02-20T09:06:00.109 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:05:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:05:59.805+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:05:59.807360+0000) 2024-02-20T09:06:00.109 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:06:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:06:00.105+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:06:00.109 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:06:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:06:00.105+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:06:00.109 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:06:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:06:00.105+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:06:00.110 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:06:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:06:00.027+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:06:00.110 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:06:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:06:00.027+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:06:00.110 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:06:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:06:00.027+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:06:00.110 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:06:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:06:00.027+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:06:00.388 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:06:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:06:00.143+0000 7f8a5f86e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:00.144538+0000) 2024-02-20T09:06:00.389 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:06:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:06:00.146+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:00.148355+0000) 2024-02-20T09:06:00.389 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:06:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:06:00.232+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:00.233281+0000) 2024-02-20T09:06:00.389 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:06:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:06:00.148+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 56924 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:06:00.389 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:06:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:06:00.143+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:00.144682+0000) 2024-02-20T09:06:00.390 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:06:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:06:00.348+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:06:00.390 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:06:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:06:00.348+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:06:00.390 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:06:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:06:00.348+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:06:01.063 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:06:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:06:00.807+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:00.807542+0000) 2024-02-20T09:06:01.063 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:06:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:06:00.923+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:00.924456+0000) 2024-02-20T09:06:01.344 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:06:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:06:01.072+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:06:01.344 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:06:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:06:01.072+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:06:01.344 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:06:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:06:01.072+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:06:01.344 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:06:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:06:01.060+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:06:01.345 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:06:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:06:01.060+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:06:01.345 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:06:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:06:01.060+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:06:01.345 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:06:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:06:01.060+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:06:01.345 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:06:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:06:01.235+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:01.235793+0000) 2024-02-20T09:06:01.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:06:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:06:01.377+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:06:01.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:06:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:06:01.377+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:06:01.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:06:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:06:01.377+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:06:02.072 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:06:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:06:01.924+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:01.924634+0000) 2024-02-20T09:06:02.073 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:06:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:06:01.806+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:01.807678+0000) 2024-02-20T09:06:02.073 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:06:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:06:02.041+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:06:02.073 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:06:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:06:02.041+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:06:02.073 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:06:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:06:02.041+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:06:02.344 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:06:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:06:02.070+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:06:02.345 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:06:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:06:02.070+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:06:02.345 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:06:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:06:02.070+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:06:02.345 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:06:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:06:02.070+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:06:02.345 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:06:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:06:02.237+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:02.238734+0000) 2024-02-20T09:06:02.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:06:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:06:02.413+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:06:02.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:06:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:06:02.413+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:06:02.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:06:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:06:02.413+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:06:03.066 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:06:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:06:02.924+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:02.924791+0000) 2024-02-20T09:06:03.066 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:06:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:06:03.026+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:06:03.066 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:06:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:06:03.026+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:06:03.066 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:06:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:06:03.026+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:06:03.066 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:06:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:06:03.026+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:06:03.067 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:06:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:06:02.807+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:02.807902+0000) 2024-02-20T09:06:03.067 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:06:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:06:03.063+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:06:03.067 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:06:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:06:03.063+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:06:03.067 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:06:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:06:03.063+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:06:03.344 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:06:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:06:03.240+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:03.241463+0000) 2024-02-20T09:06:03.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:06:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:06:03.377+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:06:03.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:06:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:06:03.377+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:06:03.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:06:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:06:03.377+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:06:04.063 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:06:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:06:03.924+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:03.924969+0000) 2024-02-20T09:06:04.064 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:06:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:06:03.807+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:03.808120+0000) 2024-02-20T09:06:04.345 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:06:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:06:04.112+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:06:04.345 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:06:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:06:04.112+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:06:04.345 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:06:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:06:04.112+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:06:04.345 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:06:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:06:04.061+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:06:04.346 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:06:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:06:04.061+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:06:04.346 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:06:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:06:04.061+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:06:04.346 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:06:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:06:04.061+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:06:04.346 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:06:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:06:04.241+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:04.241764+0000) 2024-02-20T09:06:04.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:06:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:06:04.369+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:06:04.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:06:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:06:04.369+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:06:04.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:06:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:06:04.369+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:06:05.069 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:06:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:06:04.807+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:04.808301+0000) 2024-02-20T09:06:05.069 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:06:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:06:04.923+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:04.925170+0000) 2024-02-20T09:06:05.345 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:06:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:06:05.157+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 56996 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:06:05.345 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:06:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:06:05.151+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:05.152699+0000) 2024-02-20T09:06:05.346 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:06:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:06:05.075+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:06:05.346 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:06:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:06:05.075+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:06:05.346 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:06:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:06:05.075+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:06:05.346 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:06:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:06:05.150+0000 7f8a5f86e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:05.152129+0000) 2024-02-20T09:06:05.346 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:06:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:06:05.067+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:06:05.346 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:06:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:06:05.067+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:06:05.346 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:06:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:06:05.067+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:06:05.347 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:06:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:06:05.067+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:06:05.347 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:06:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:06:05.153+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:05.154687+0000) 2024-02-20T09:06:05.347 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:06:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:06:05.240+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:05.241999+0000) 2024-02-20T09:06:05.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:06:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:06:05.403+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:06:05.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:06:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:06:05.403+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:06:05.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:06:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:06:05.403+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:06:06.052 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:06:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:06:05.924+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:05.925326+0000) 2024-02-20T09:06:06.052 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:06:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:06:05.808+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:05.808495+0000) 2024-02-20T09:06:06.052 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:06:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:06:06.050+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:06:06.052 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:06:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:06:06.050+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:06:06.053 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:06:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:06:06.050+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:06:06.348 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:06:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:06:06.058+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:06:06.349 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:06:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:06:06.058+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:06:06.349 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:06:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:06:06.058+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:06:06.349 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:06:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:06:06.058+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:06:06.349 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:06:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:06:06.243+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:06.244376+0000) 2024-02-20T09:06:06.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:06:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:06:06.377+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:06:06.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:06:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:06:06.377+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:06:06.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:06:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:06:06.377+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:06:07.070 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:06:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:06:06.925+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:06.925483+0000) 2024-02-20T09:06:07.070 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:06:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:06:06.808+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:06.808662+0000) 2024-02-20T09:06:07.070 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:06:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:06:07.001+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:06:07.070 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:06:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:06:07.001+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:06:07.070 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:06:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:06:07.001+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:06:07.349 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:06:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:06:07.068+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:06:07.349 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:06:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:06:07.068+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:06:07.349 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:06:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:06:07.068+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:06:07.350 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:06:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:06:07.068+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:06:07.350 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:06:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:06:07.246+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:07.247119+0000) 2024-02-20T09:06:07.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:06:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:06:07.349+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:06:07.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:06:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:06:07.349+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:06:07.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:06:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:06:07.349+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:06:08.100 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:06:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:06:07.925+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:07.925610+0000) 2024-02-20T09:06:08.100 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:06:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:06:07.808+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:07.808827+0000) 2024-02-20T09:06:08.100 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:06:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:06:08.017+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:06:08.100 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:06:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:06:08.017+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:06:08.100 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:06:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:06:08.017+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:06:08.359 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:06:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:06:08.098+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:06:08.359 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:06:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:06:08.098+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:06:08.359 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:06:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:06:08.098+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:06:08.359 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:06:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:06:08.098+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:06:08.359 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:06:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:06:08.249+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:08.249739+0000) 2024-02-20T09:06:08.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:06:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:06:08.357+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:06:08.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:06:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:06:08.357+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:06:08.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:06:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:06:08.357+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:06:09.090 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:06:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:06:08.925+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:08.925789+0000) 2024-02-20T09:06:09.090 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:06:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:06:08.808+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:08.809021+0000) 2024-02-20T09:06:09.090 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:06:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:06:09.040+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:06:09.090 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:06:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:06:09.040+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:06:09.090 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:06:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:06:09.040+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:06:09.349 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:06:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:06:09.088+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:06:09.349 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:06:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:06:09.088+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:06:09.349 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:06:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:06:09.088+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:06:09.350 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:06:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:06:09.088+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:06:09.350 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:06:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:06:09.249+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:09.250047+0000) 2024-02-20T09:06:09.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:06:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:06:09.353+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:06:09.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:06:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:06:09.353+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:06:09.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:06:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:06:09.353+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:06:10.161 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:06:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:06:09.925+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:09.925995+0000) 2024-02-20T09:06:10.162 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:06:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:06:09.808+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:09.809171+0000) 2024-02-20T09:06:10.162 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:06:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:06:09.995+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:06:10.162 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:06:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:06:09.995+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:06:10.162 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:06:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:06:09.995+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:06:10.162 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:06:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:06:10.044+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:06:10.163 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:06:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:06:10.044+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:06:10.163 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:06:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:06:10.044+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:06:10.163 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:06:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:06:10.044+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:06:10.453 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:06:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:06:10.161+0000 7f8a5f86e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:10.161412+0000) 2024-02-20T09:06:10.454 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:06:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:06:10.164+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:10.164781+0000) 2024-02-20T09:06:10.454 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:06:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:06:10.250+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:10.250370+0000) 2024-02-20T09:06:10.454 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:06:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:06:10.166+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 57069 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:06:10.454 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:06:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:06:10.159+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:10.160191+0000) 2024-02-20T09:06:10.454 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:06:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:06:10.314+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:06:10.455 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:06:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:06:10.314+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:06:10.455 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:06:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:06:10.314+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:06:11.067 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:06:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:06:10.925+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:10.926174+0000) 2024-02-20T09:06:11.067 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:06:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:06:10.808+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:10.809353+0000) 2024-02-20T09:06:11.068 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:06:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:06:10.970+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:06:11.068 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:06:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:06:10.970+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:06:11.068 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:06:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:06:10.970+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:06:11.352 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:06:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:06:11.065+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:06:11.352 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:06:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:06:11.065+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:06:11.353 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:06:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:06:11.065+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:06:11.353 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:06:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:06:11.065+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:06:11.353 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:06:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:06:11.252+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:11.252805+0000) 2024-02-20T09:06:11.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:06:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:06:11.360+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:06:11.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:06:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:06:11.360+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:06:11.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:06:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:06:11.360+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:06:12.097 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:06:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:06:11.925+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:11.926371+0000) 2024-02-20T09:06:12.097 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:06:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:06:11.809+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:11.809481+0000) 2024-02-20T09:06:12.097 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:06:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:06:11.980+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:06:12.098 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:06:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:06:11.980+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:06:12.098 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:06:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:06:11.980+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:06:12.353 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:06:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:06:12.312+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:06:12.353 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:06:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:06:12.312+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:06:12.353 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:06:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:06:12.312+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:06:12.353 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:06:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:06:12.095+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:06:12.354 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:06:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:06:12.095+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:06:12.354 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:06:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:06:12.095+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:06:12.354 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:06:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:06:12.095+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:06:12.354 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:06:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:06:12.255+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:12.255608+0000) 2024-02-20T09:06:13.122 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:06:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:06:12.926+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:12.926566+0000) 2024-02-20T09:06:13.123 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:06:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:06:12.809+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:12.809665+0000) 2024-02-20T09:06:13.123 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:06:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:06:12.957+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:06:13.123 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:06:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:06:12.957+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:06:13.123 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:06:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:06:12.957+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:06:13.454 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:06:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:06:13.279+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:06:13.454 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:06:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:06:13.279+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:06:13.454 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:06:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:06:13.279+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:06:13.454 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:06:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:06:13.120+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:06:13.454 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:06:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:06:13.120+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:06:13.455 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:06:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:06:13.120+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:06:13.455 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:06:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:06:13.120+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:06:13.455 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:06:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:06:13.257+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:13.258240+0000) 2024-02-20T09:06:14.102 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:06:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:06:13.926+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:13.926667+0000) 2024-02-20T09:06:14.102 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:06:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:06:13.809+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:13.809894+0000) 2024-02-20T09:06:14.102 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:06:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:06:13.949+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:06:14.103 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:06:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:06:13.949+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:06:14.103 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:06:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:06:13.949+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:06:14.353 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:06:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:06:14.292+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:06:14.354 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:06:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:06:14.292+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:06:14.354 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:06:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:06:14.292+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:06:14.354 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:06:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:06:14.100+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:06:14.354 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:06:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:06:14.100+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:06:14.354 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:06:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:06:14.100+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:06:14.354 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:06:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:06:14.100+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:06:14.354 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:06:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:06:14.258+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:14.258514+0000) 2024-02-20T09:06:15.124 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:06:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:06:14.926+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:14.926869+0000) 2024-02-20T09:06:15.124 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:06:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:06:14.809+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:14.810071+0000) 2024-02-20T09:06:15.124 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:06:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:06:14.923+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:06:15.124 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:06:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:06:14.923+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:06:15.124 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:06:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:06:14.923+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:06:15.454 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:06:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:06:15.174+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 57141 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:06:15.455 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:06:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:06:15.167+0000 7f8a6006f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:15.168243+0000) 2024-02-20T09:06:15.455 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:06:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:06:15.168+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:15.168724+0000) 2024-02-20T09:06:15.455 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:06:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:06:15.322+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:06:15.455 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:06:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:06:15.322+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:06:15.455 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:06:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:06:15.322+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:06:15.456 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:06:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:06:15.122+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:06:15.456 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:06:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:06:15.122+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:06:15.456 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:06:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:06:15.122+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:06:15.456 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:06:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:06:15.122+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:06:15.456 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:06:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:06:15.171+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:15.171696+0000) 2024-02-20T09:06:15.456 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:06:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:06:15.258+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:15.258789+0000) 2024-02-20T09:06:16.105 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:06:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:06:15.926+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:15.927046+0000) 2024-02-20T09:06:16.106 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:06:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:06:15.810+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:15.810286+0000) 2024-02-20T09:06:16.106 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:06:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:06:15.941+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:06:16.106 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:06:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:06:15.941+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:06:16.106 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:06:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:06:15.941+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:06:16.356 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:06:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:06:16.329+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:06:16.356 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:06:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:06:16.329+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:06:16.357 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:06:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:06:16.329+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:06:16.357 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:06:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:06:16.103+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:06:16.357 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:06:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:06:16.103+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:06:16.357 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:06:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:06:16.103+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:06:16.357 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:06:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:06:16.103+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:06:16.357 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:06:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:06:16.262+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:16.262438+0000) 2024-02-20T09:06:17.153 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:06:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:06:16.926+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:16.927218+0000) 2024-02-20T09:06:17.154 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:06:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:06:16.810+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:16.810436+0000) 2024-02-20T09:06:17.154 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:06:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:06:16.942+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:06:17.154 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:06:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:06:16.942+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:06:17.154 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:06:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:06:16.942+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:06:17.454 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:06:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:06:17.304+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:06:17.454 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:06:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:06:17.304+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:06:17.455 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:06:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:06:17.304+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:06:17.455 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:06:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:06:17.152+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:06:17.455 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:06:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:06:17.152+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:06:17.455 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:06:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:06:17.152+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:06:17.455 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:06:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:06:17.152+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:06:17.455 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:06:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:06:17.265+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:17.266197+0000) 2024-02-20T09:06:18.118 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:06:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:06:17.927+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:17.927432+0000) 2024-02-20T09:06:18.119 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:06:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:06:17.810+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:17.810631+0000) 2024-02-20T09:06:18.119 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:06:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:06:17.982+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:06:18.119 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:06:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:06:17.982+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:06:18.119 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:06:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:06:17.982+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:06:18.455 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:06:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:06:18.116+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:06:18.455 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:06:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:06:18.116+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:06:18.455 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:06:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:06:18.116+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:06:18.455 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:06:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:06:18.116+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:06:18.455 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:06:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:06:18.269+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:18.269960+0000) 2024-02-20T09:06:18.456 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:06:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:06:18.346+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:06:18.456 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:06:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:06:18.346+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:06:18.456 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:06:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:06:18.346+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:06:19.079 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:06:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:06:18.927+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:18.927584+0000) 2024-02-20T09:06:19.079 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:06:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:06:18.810+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:18.810829+0000) 2024-02-20T09:06:19.079 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:06:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:06:18.974+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:06:19.079 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:06:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:06:18.974+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:06:19.079 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:06:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:06:18.974+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:06:19.357 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:06:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:06:19.077+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:06:19.358 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:06:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:06:19.077+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:06:19.358 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:06:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:06:19.077+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:06:19.358 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:06:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:06:19.077+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:06:19.358 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:06:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:06:19.270+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:19.270241+0000) 2024-02-20T09:06:19.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:06:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:06:19.359+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:06:19.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:06:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:06:19.359+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:06:19.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:06:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:06:19.359+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:06:20.097 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:06:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:06:19.927+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:19.927772+0000) 2024-02-20T09:06:20.097 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:06:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:06:19.810+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:19.811013+0000) 2024-02-20T09:06:20.098 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:06:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:06:19.938+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:06:20.098 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:06:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:06:19.938+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:06:20.098 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:06:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:06:19.938+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:06:20.357 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:06:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:06:20.175+0000 7f8a6006f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:20.175975+0000) 2024-02-20T09:06:20.357 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:06:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:06:20.183+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 57217 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:06:20.357 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:06:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:06:20.175+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:20.176240+0000) 2024-02-20T09:06:20.357 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:06:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:06:20.355+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:06:20.358 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:06:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:06:20.355+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:06:20.358 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:06:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:06:20.355+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:06:20.358 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:06:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:06:20.095+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:06:20.358 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:06:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:06:20.095+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:06:20.358 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:06:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:06:20.095+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:06:20.358 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:06:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:06:20.095+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:06:20.358 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:06:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:06:20.179+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:20.179953+0000) 2024-02-20T09:06:20.358 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:06:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:06:20.270+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:20.270500+0000) 2024-02-20T09:06:21.065 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:06:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:06:20.927+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:20.927966+0000) 2024-02-20T09:06:21.065 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:06:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:06:20.810+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:20.811113+0000) 2024-02-20T09:06:21.065 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:06:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:06:20.947+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:06:21.065 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:06:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:06:20.947+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:06:21.066 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:06:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:06:20.947+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:06:21.316 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:06:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:06:21.063+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:06:21.316 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:06:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:06:21.063+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:06:21.316 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:06:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:06:21.063+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:06:21.317 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:06:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:06:21.063+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:06:21.317 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:06:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:06:21.272+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:21.273190+0000) 2024-02-20T09:06:21.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:06:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:06:21.314+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:06:21.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:06:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:06:21.314+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:06:21.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:06:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:06:21.314+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:06:22.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:06:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:06:21.927+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:21.928135+0000) 2024-02-20T09:06:22.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:06:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:06:21.811+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:21.811230+0000) 2024-02-20T09:06:22.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:06:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:06:21.923+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:06:22.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:06:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:06:21.923+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:06:22.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:06:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:06:21.923+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:06:22.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:06:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:06:22.046+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:06:22.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:06:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:06:22.046+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:06:22.245 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:06:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:06:22.046+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:06:22.245 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:06:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:06:22.046+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:06:22.585 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:06:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:06:22.275+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:22.275968+0000) 2024-02-20T09:06:22.585 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:06:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:06:22.299+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:06:22.585 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:06:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:06:22.299+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:06:22.585 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:06:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:06:22.299+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:06:22.919 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:06:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:06:22.811+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:22.811447+0000) 2024-02-20T09:06:22.959 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:06:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:06:22.917+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:06:23.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:06:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:06:22.917+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:06:23.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:06:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:06:22.917+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:06:23.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:06:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:06:22.927+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:22.928333+0000) 2024-02-20T09:06:23.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:06:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:06:23.014+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:06:23.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:06:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:06:23.014+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:06:23.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:06:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:06:23.014+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:06:23.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:06:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:06:23.014+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:06:23.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:06:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:06:23.279+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:23.279625+0000) 2024-02-20T09:06:23.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:06:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:06:23.321+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:06:23.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:06:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:06:23.321+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:06:23.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:06:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:06:23.321+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:06:24.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:06:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:06:23.928+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:23.928520+0000) 2024-02-20T09:06:24.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:06:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:06:23.811+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:23.811625+0000) 2024-02-20T09:06:24.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:06:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:06:23.953+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:06:24.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:06:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:06:23.953+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:06:24.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:06:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:06:23.953+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:06:24.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:06:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:06:24.029+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:06:24.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:06:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:06:24.029+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:06:24.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:06:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:06:24.029+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:06:24.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:06:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:06:24.029+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:06:24.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:06:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:06:24.279+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:24.279870+0000) 2024-02-20T09:06:24.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:06:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:06:24.297+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:06:24.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:06:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:06:24.297+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:06:24.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:06:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:06:24.297+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:06:25.187 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:06:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:06:24.993+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:06:25.187 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:06:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:06:24.993+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:06:25.187 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:06:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:06:24.993+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:06:25.187 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:06:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:06:24.993+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:06:25.188 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:06:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:06:24.928+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:24.928686+0000) 2024-02-20T09:06:25.188 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:06:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:06:24.811+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:24.811756+0000) 2024-02-20T09:06:25.188 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:06:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:06:24.929+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:06:25.188 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:06:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:06:24.929+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:06:25.188 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:06:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:06:24.929+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:06:25.456 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:06:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:06:25.191+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 57289 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:06:25.457 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:06:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:06:25.185+0000 7f8a6006f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:25.186030+0000) 2024-02-20T09:06:25.457 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:06:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:06:25.188+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:25.189262+0000) 2024-02-20T09:06:25.457 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:06:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:06:25.280+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:25.280205+0000) 2024-02-20T09:06:25.457 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:06:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:06:25.184+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:25.185247+0000) 2024-02-20T09:06:25.457 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:06:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:06:25.248+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:06:25.458 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:06:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:06:25.248+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:06:25.458 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:06:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:06:25.248+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:06:26.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:06:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:06:25.928+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:25.928895+0000) 2024-02-20T09:06:26.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:06:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:06:26.030+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:06:26.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:06:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:06:26.030+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:06:26.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:06:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:06:26.030+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:06:26.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:06:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:06:26.030+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:06:26.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:06:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:06:25.811+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:25.811969+0000) 2024-02-20T09:06:26.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:06:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:06:25.925+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:06:26.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:06:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:06:25.925+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:06:26.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:06:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:06:25.925+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:06:26.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:06:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:06:26.282+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:26.282718+0000) 2024-02-20T09:06:26.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:06:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:06:26.278+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:06:26.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:06:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:06:26.278+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:06:26.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:06:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:06:26.278+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:06:27.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:06:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:06:26.927+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:26.929048+0000) 2024-02-20T09:06:27.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:06:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:06:27.028+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:06:27.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:06:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:06:27.028+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:06:27.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:06:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:06:27.028+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:06:27.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:06:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:06:27.028+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:06:27.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:06:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:06:26.812+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:26.812128+0000) 2024-02-20T09:06:27.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:06:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:06:26.960+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:06:27.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:06:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:06:26.960+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:06:27.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:06:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:06:26.960+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:06:27.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:06:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:06:27.284+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:27.285539+0000) 2024-02-20T09:06:27.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:06:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:06:27.256+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:06:27.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:06:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:06:27.256+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:06:27.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:06:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:06:27.256+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:06:28.053 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:06:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:06:27.927+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:27.929232+0000) 2024-02-20T09:06:28.054 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:06:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:06:27.811+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:27.812287+0000) 2024-02-20T09:06:28.054 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:06:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:06:27.966+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:06:28.054 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:06:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:06:27.966+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:06:28.054 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:06:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:06:27.966+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:06:28.365 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:06:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:06:28.285+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:06:28.365 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:06:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:06:28.285+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:06:28.365 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:06:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:06:28.285+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:06:28.366 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:06:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:06:28.051+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:06:28.366 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:06:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:06:28.051+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:06:28.366 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:06:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:06:28.051+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:06:28.366 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:06:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:06:28.051+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:06:28.366 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:06:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:06:28.287+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:28.288439+0000) 2024-02-20T09:06:29.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:06:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:06:28.928+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:28.929437+0000) 2024-02-20T09:06:29.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:06:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:06:29.013+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:06:29.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:06:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:06:29.013+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:06:29.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:06:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:06:29.013+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:06:29.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:06:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:06:29.013+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:06:29.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:06:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:06:28.811+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:28.812489+0000) 2024-02-20T09:06:29.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:06:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:06:28.923+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:06:29.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:06:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:06:28.923+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:06:29.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:06:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:06:28.923+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:06:29.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:06:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:06:29.287+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:29.288721+0000) 2024-02-20T09:06:29.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:06:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:06:29.309+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:06:29.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:06:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:06:29.309+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:06:29.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:06:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:06:29.309+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:06:30.195 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:06:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:06:29.811+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:29.812635+0000) 2024-02-20T09:06:30.195 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:06:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:06:29.936+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:06:30.195 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:06:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:06:29.936+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:06:30.195 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:06:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:06:29.936+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:06:30.195 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:06:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:06:30.192+0000 7f8a6006f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:30.193449+0000) 2024-02-20T09:06:30.196 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:06:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:06:30.014+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:06:30.196 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:06:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:06:30.014+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:06:30.196 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:06:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:06:30.014+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:06:30.196 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:06:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:06:30.014+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:06:30.196 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:06:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:06:29.928+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:29.929641+0000) 2024-02-20T09:06:30.456 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:06:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:06:30.198+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 57362 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:06:30.457 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:06:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:06:30.195+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:30.196663+0000) 2024-02-20T09:06:30.457 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:06:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:06:30.287+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:30.289023+0000) 2024-02-20T09:06:30.457 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:06:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:06:30.192+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:30.193960+0000) 2024-02-20T09:06:30.457 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:06:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:06:30.286+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:06:30.457 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:06:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:06:30.286+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:06:30.458 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:06:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:06:30.286+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:06:31.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:06:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:06:30.928+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:30.929786+0000) 2024-02-20T09:06:31.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:06:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:06:30.811+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:30.812806+0000) 2024-02-20T09:06:31.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:06:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:06:30.902+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:06:31.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:06:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:06:30.902+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:06:31.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:06:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:06:30.902+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:06:31.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:06:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:06:31.022+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:06:31.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:06:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:06:31.022+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:06:31.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:06:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:06:31.022+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:06:31.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:06:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:06:31.022+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:06:31.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:06:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:06:31.290+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:31.291581+0000) 2024-02-20T09:06:31.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:06:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:06:31.328+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:06:31.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:06:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:06:31.328+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:06:31.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:06:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:06:31.328+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:06:32.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:06:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:06:31.928+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:31.929984+0000) 2024-02-20T09:06:32.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:06:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:06:31.811+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:31.812907+0000) 2024-02-20T09:06:32.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:06:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:06:31.877+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:06:32.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:06:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:06:31.877+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:06:32.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:06:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:06:31.877+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:06:32.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:06:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:06:31.976+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:06:32.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:06:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:06:31.976+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:06:32.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:06:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:06:31.976+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:06:32.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:06:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:06:31.976+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:06:32.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:06:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:06:32.293+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:32.294251+0000) 2024-02-20T09:06:32.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:06:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:06:32.370+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:06:32.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:06:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:06:32.370+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:06:32.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:06:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:06:32.370+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:06:33.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:06:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:06:32.929+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:32.930141+0000) 2024-02-20T09:06:33.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:06:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:06:33.017+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:06:33.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:06:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:06:33.017+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:06:33.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:06:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:06:33.017+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:06:33.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:06:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:06:33.017+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:06:33.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:06:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:06:32.811+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:32.813105+0000) 2024-02-20T09:06:33.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:06:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:06:32.864+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:06:33.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:06:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:06:32.864+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:06:33.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:06:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:06:32.864+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:06:33.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:06:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:06:33.295+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:33.296971+0000) 2024-02-20T09:06:33.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:06:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:06:33.324+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:06:33.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:06:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:06:33.324+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:06:33.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:06:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:06:33.324+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:06:34.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:06:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:06:33.929+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:33.930327+0000) 2024-02-20T09:06:34.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:06:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:06:33.971+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:06:34.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:06:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:06:33.971+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:06:34.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:06:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:06:33.971+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:06:34.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:06:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:06:33.971+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:06:34.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:06:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:06:33.812+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:33.813323+0000) 2024-02-20T09:06:34.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:06:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:06:33.859+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:06:34.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:06:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:06:33.859+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:06:34.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:06:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:06:33.859+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:06:34.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:06:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:06:34.296+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:34.297249+0000) 2024-02-20T09:06:34.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:06:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:06:34.343+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:06:34.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:06:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:06:34.343+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:06:34.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:06:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:06:34.343+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:06:35.203 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:06:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:06:35.001+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:06:35.203 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:06:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:06:35.001+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:06:35.203 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:06:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:06:35.001+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:06:35.203 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:06:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:06:35.001+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:06:35.204 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:06:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:06:34.929+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:34.930514+0000) 2024-02-20T09:06:35.204 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:06:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:06:34.809+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:06:35.204 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:06:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:06:34.809+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:06:35.204 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:06:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:06:34.809+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:06:35.204 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:06:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:06:34.812+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:34.813497+0000) 2024-02-20T09:06:35.457 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:06:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:06:35.206+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 57434 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:06:35.458 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:06:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:06:35.201+0000 7f8a5f86e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:35.202687+0000) 2024-02-20T09:06:35.458 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:06:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:06:35.203+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:35.205270+0000) 2024-02-20T09:06:35.458 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:06:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:06:35.296+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:35.297536+0000) 2024-02-20T09:06:35.459 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:06:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:06:35.200+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:35.201637+0000) 2024-02-20T09:06:35.459 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:06:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:06:35.357+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:06:35.459 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:06:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:06:35.357+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:06:35.459 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:06:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:06:35.357+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:06:36.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:06:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:06:35.929+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:35.930682+0000) 2024-02-20T09:06:36.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:06:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:06:35.812+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:35.813678+0000) 2024-02-20T09:06:36.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:06:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:06:35.855+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:06:36.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:06:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:06:35.855+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:06:36.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:06:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:06:35.855+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:06:36.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:06:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:06:36.034+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:06:36.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:06:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:06:36.034+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:06:36.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:06:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:06:36.034+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:06:36.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:06:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:06:36.034+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:06:36.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:06:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:06:36.299+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:36.300048+0000) 2024-02-20T09:06:36.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:06:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:06:36.349+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:06:36.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:06:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:06:36.349+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:06:36.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:06:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:06:36.349+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:06:37.057 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:06:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:06:36.929+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:36.930775+0000) 2024-02-20T09:06:37.057 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:06:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:06:36.812+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:36.813814+0000) 2024-02-20T09:06:37.057 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:06:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:06:36.825+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:06:37.057 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:06:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:06:36.825+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:06:37.057 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:06:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:06:36.825+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:06:37.373 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:06:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:06:37.054+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:06:37.373 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:06:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:06:37.054+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:06:37.374 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:06:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:06:37.054+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:06:37.374 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:06:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:06:37.054+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:06:37.374 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:06:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:06:37.302+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:37.302876+0000) 2024-02-20T09:06:37.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:06:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:06:37.382+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:06:37.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:06:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:06:37.382+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:06:37.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:06:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:06:37.382+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:06:38.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:06:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:06:37.930+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:37.930963+0000) 2024-02-20T09:06:38.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:06:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:06:37.809+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:06:38.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:06:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:06:37.809+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:06:38.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:06:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:06:37.809+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:06:38.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:06:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:06:37.812+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:37.814019+0000) 2024-02-20T09:06:38.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:06:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:06:38.005+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:06:38.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:06:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:06:38.005+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:06:38.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:06:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:06:38.005+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:06:38.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:06:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:06:38.005+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:06:38.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:06:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:06:38.304+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:38.305856+0000) 2024-02-20T09:06:38.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:06:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:06:38.404+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:06:38.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:06:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:06:38.404+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:06:38.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:06:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:06:38.404+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:06:39.057 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:06:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:06:38.929+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:38.931134+0000) 2024-02-20T09:06:39.057 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:06:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:06:38.813+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:38.814202+0000) 2024-02-20T09:06:39.057 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:06:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:06:38.817+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:06:39.057 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:06:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:06:38.817+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:06:39.057 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:06:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:06:38.817+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:06:39.307 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:06:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:06:39.054+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:06:39.308 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:06:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:06:39.054+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:06:39.308 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:06:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:06:39.054+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:06:39.308 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:06:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:06:39.054+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:06:39.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:06:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:06:39.305+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:39.306249+0000) 2024-02-20T09:06:39.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:06:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:06:39.364+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:06:39.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:06:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:06:39.364+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:06:39.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:06:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:06:39.364+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:06:40.053 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:06:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:06:39.929+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:39.931344+0000) 2024-02-20T09:06:40.054 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:06:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:06:39.797+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:06:40.054 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:06:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:06:39.797+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:06:40.054 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:06:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:06:39.797+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:06:40.054 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:06:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:06:39.813+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:39.814423+0000) 2024-02-20T09:06:40.308 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:06:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:06:40.215+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 57507 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:06:40.308 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:06:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:06:40.208+0000 7f8a6006f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:40.209866+0000) 2024-02-20T09:06:40.309 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:06:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:06:40.208+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:40.210052+0000) 2024-02-20T09:06:40.309 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:06:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:06:40.051+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:06:40.309 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:06:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:06:40.051+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:06:40.309 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:06:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:06:40.051+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:06:40.309 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:06:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:06:40.051+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:06:40.309 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:06:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:06:40.211+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:40.213319+0000) 2024-02-20T09:06:40.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:06:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:06:40.305+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:40.306522+0000) 2024-02-20T09:06:40.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:06:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:06:40.317+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:06:40.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:06:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:06:40.317+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:06:40.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:06:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:06:40.317+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:06:41.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:06:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:06:40.930+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:40.931479+0000) 2024-02-20T09:06:41.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:06:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:06:41.006+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:06:41.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:06:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:06:41.006+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:06:41.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:06:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:06:41.006+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:06:41.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:06:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:06:41.006+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:06:41.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:06:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:06:40.813+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:40.814602+0000) 2024-02-20T09:06:41.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:06:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:06:40.843+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:06:41.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:06:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:06:40.843+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:06:41.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:06:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:06:40.843+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:06:41.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:06:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:06:41.308+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:41.309150+0000) 2024-02-20T09:06:41.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:06:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:06:41.275+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:06:41.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:06:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:06:41.275+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:06:41.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:06:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:06:41.275+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:06:42.228 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:06:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:06:42.009+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:06:42.228 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:06:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:06:42.009+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:06:42.228 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:06:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:06:42.009+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:06:42.228 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:06:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:06:42.009+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:06:42.228 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:06:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:06:41.930+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:41.931626+0000) 2024-02-20T09:06:42.229 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:06:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:06:42.224+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:06:42.229 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:06:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:06:41.813+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:41.814736+0000) 2024-02-20T09:06:42.229 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:06:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:06:41.844+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:06:42.229 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:06:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:06:41.844+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:06:42.229 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:06:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:06:41.844+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:06:42.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:06:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:06:42.311+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:42.311900+0000) 2024-02-20T09:06:42.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:06:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:06:42.224+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:06:42.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:06:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:06:42.224+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:06:43.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:06:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:06:42.931+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:42.931795+0000) 2024-02-20T09:06:43.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:06:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:06:42.960+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:06:43.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:06:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:06:42.960+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:06:43.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:06:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:06:42.960+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:06:43.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:06:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:06:42.960+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:06:43.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:06:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:06:42.814+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:42.814911+0000) 2024-02-20T09:06:43.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:06:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:06:42.893+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:06:43.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:06:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:06:42.893+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:06:43.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:06:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:06:42.893+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:06:43.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:06:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:06:43.313+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:43.314698+0000) 2024-02-20T09:06:43.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:06:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:06:43.252+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:06:43.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:06:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:06:43.252+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:06:43.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:06:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:06:43.252+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:06:44.089 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:06:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:06:43.931+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:43.931969+0000) 2024-02-20T09:06:44.089 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:06:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:06:43.814+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:43.815070+0000) 2024-02-20T09:06:44.090 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:06:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:06:43.896+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:06:44.090 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:06:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:06:43.896+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:06:44.090 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:06:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:06:43.896+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:06:44.090 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:06:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:06:43.931+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:06:44.090 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:06:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:06:43.931+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:06:44.090 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:06:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:06:43.931+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:06:44.090 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:06:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:06:43.931+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:06:44.379 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:06:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:06:44.313+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:44.314997+0000) 2024-02-20T09:06:44.379 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:06:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:06:44.224+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:06:44.379 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:06:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:06:44.224+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:06:44.380 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:06:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:06:44.224+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:06:45.216 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:06:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:06:44.930+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:44.932154+0000) 2024-02-20T09:06:45.216 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:06:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:06:44.897+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:06:45.216 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:06:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:06:44.897+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:06:45.217 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:06:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:06:44.897+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:06:45.217 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:06:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:06:44.897+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:06:45.217 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:06:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:06:44.814+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:44.815249+0000) 2024-02-20T09:06:45.217 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:06:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:06:44.917+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:06:45.217 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:06:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:06:44.917+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:06:45.217 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:06:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:06:44.917+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:06:45.492 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:06:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:06:45.223+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 57579 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:06:45.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:06:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:06:45.217+0000 7f8a6006f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:45.218506+0000) 2024-02-20T09:06:45.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:06:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:06:45.220+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:45.221786+0000) 2024-02-20T09:06:45.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:06:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:06:45.314+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:45.315287+0000) 2024-02-20T09:06:45.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:06:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:06:45.214+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:06:45.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:06:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:06:45.214+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:06:45.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:06:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:06:45.214+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:06:45.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:06:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:06:45.216+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:45.218146+0000) 2024-02-20T09:06:46.235 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:06:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:06:45.931+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:45.932325+0000) 2024-02-20T09:06:46.235 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:06:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:06:46.232+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:06:46.235 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:06:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:06:45.868+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:06:46.236 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:06:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:06:45.868+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:06:46.236 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:06:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:06:45.868+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:06:46.236 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:06:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:06:45.868+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:06:46.236 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:06:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:06:45.814+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:45.815424+0000) 2024-02-20T09:06:46.236 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:06:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:06:45.880+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:06:46.236 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:06:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:06:45.880+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:06:46.236 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:06:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:06:45.880+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:06:46.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:06:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:06:46.316+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:46.317762+0000) 2024-02-20T09:06:46.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:06:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:06:46.232+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:06:46.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:06:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:06:46.232+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:06:47.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:06:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:06:46.931+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:46.932467+0000) 2024-02-20T09:06:47.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:06:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:06:46.814+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:46.815556+0000) 2024-02-20T09:06:47.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:06:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:06:46.905+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:06:47.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:06:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:06:46.905+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:06:47.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:06:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:06:46.905+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:06:47.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:06:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:06:46.842+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:06:47.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:06:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:06:46.842+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:06:47.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:06:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:06:46.842+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:06:47.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:06:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:06:46.842+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:06:47.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:06:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:06:47.320+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:47.321428+0000) 2024-02-20T09:06:47.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:06:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:06:47.279+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:06:47.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:06:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:06:47.279+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:06:47.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:06:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:06:47.279+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:06:48.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:06:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:06:47.931+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:47.932611+0000) 2024-02-20T09:06:48.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:06:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:06:47.815+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:47.815711+0000) 2024-02-20T09:06:48.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:06:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:06:47.944+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:06:48.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:06:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:06:47.944+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:06:48.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:06:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:06:47.944+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:06:48.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:06:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:06:47.828+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:06:48.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:06:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:06:47.828+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:06:48.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:06:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:06:47.828+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:06:48.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:06:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:06:47.828+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:06:48.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:06:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:06:48.324+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:48.325143+0000) 2024-02-20T09:06:48.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:06:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:06:48.258+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:06:48.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:06:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:06:48.258+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:06:48.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:06:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:06:48.258+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:06:49.232 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:06:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:06:48.932+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:48.932771+0000) 2024-02-20T09:06:49.233 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:06:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:06:49.229+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:06:49.233 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:06:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:06:48.814+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:48.815854+0000) 2024-02-20T09:06:49.233 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:06:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:06:48.896+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:06:49.233 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:06:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:06:48.896+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:06:49.234 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:06:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:06:48.896+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:06:49.234 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:06:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:06:48.793+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:06:49.234 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:06:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:06:48.793+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:06:49.234 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:06:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:06:48.793+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:06:49.234 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:06:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:06:48.793+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:06:49.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:06:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:06:49.324+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:49.325599+0000) 2024-02-20T09:06:49.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:06:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:06:49.229+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:06:49.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:06:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:06:49.229+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:06:50.228 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:06:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:06:49.932+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:49.932959+0000) 2024-02-20T09:06:50.228 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:06:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:06:49.749+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:06:50.228 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:06:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:06:49.749+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:06:50.228 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:06:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:06:49.749+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:06:50.229 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:06:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:06:49.749+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:06:50.229 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:06:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:06:49.815+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:49.816060+0000) 2024-02-20T09:06:50.229 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:06:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:06:49.889+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:06:50.229 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:06:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:06:49.889+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:06:50.229 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:06:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:06:49.889+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:06:50.492 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:06:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:06:50.232+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 57652 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:06:50.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:06:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:06:50.225+0000 7f8a6006f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:50.226582+0000) 2024-02-20T09:06:50.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:06:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:06:50.228+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:50.230308+0000) 2024-02-20T09:06:50.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:06:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:06:50.324+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:50.325888+0000) 2024-02-20T09:06:50.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:06:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:06:50.225+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:50.226687+0000) 2024-02-20T09:06:50.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:06:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:06:50.233+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:06:50.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:06:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:06:50.233+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:06:50.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:06:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:06:50.233+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:06:51.213 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:06:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:06:50.932+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:50.933117+0000) 2024-02-20T09:06:51.214 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:06:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:06:51.211+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:06:51.214 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:06:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:06:51.211+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:06:51.214 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:06:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:06:51.211+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:06:51.214 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:06:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:06:50.815+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:50.816240+0000) 2024-02-20T09:06:51.214 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:06:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:06:50.910+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:06:51.215 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:06:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:06:50.910+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:06:51.215 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:06:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:06:50.910+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:06:51.215 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:06:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:06:50.781+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:06:51.215 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:06:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:06:50.781+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:06:51.215 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:06:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:06:50.781+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:06:51.215 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:06:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:06:50.781+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:06:51.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:06:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:06:51.327+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:51.328129+0000) 2024-02-20T09:06:52.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:06:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:06:51.932+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:51.933274+0000) 2024-02-20T09:06:52.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:06:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:06:51.815+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:51.816395+0000) 2024-02-20T09:06:52.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:06:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:06:51.939+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:06:52.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:06:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:06:51.939+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:06:52.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:06:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:06:51.939+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:06:52.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:06:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:06:51.787+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:06:52.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:06:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:06:51.787+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:06:52.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:06:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:06:51.787+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:06:52.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:06:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:06:51.787+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:06:52.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:06:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:06:52.329+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:52.330917+0000) 2024-02-20T09:06:52.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:06:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:06:52.260+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:06:52.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:06:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:06:52.261+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:06:52.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:06:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:06:52.261+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:06:53.228 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:06:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:06:52.815+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:52.816562+0000) 2024-02-20T09:06:53.229 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:06:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:06:52.910+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:06:53.229 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:06:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:06:52.910+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:06:53.229 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:06:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:06:52.910+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:06:53.229 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:06:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:06:52.774+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:06:53.229 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:06:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:06:52.774+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:06:53.230 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:06:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:06:52.774+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:06:53.230 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:06:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:06:52.774+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:06:53.230 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:06:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:06:52.932+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:52.933455+0000) 2024-02-20T09:06:53.230 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:06:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:06:53.226+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:06:53.230 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:06:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:06:53.226+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:06:53.230 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:06:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:06:53.226+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:06:53.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:06:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:06:53.333+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:53.333736+0000) 2024-02-20T09:06:54.214 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:06:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:06:53.933+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:53.933659+0000) 2024-02-20T09:06:54.214 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:06:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:06:54.212+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:06:54.215 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:06:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:06:53.816+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:53.816770+0000) 2024-02-20T09:06:54.215 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:06:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:06:53.938+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:06:54.215 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:06:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:06:53.938+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:06:54.215 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:06:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:06:53.938+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:06:54.215 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:06:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:06:53.818+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:06:54.215 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:06:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:06:53.818+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:06:54.215 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:06:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:06:53.818+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:06:54.215 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:06:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:06:53.818+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:06:54.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:06:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:06:54.333+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:54.334347+0000) 2024-02-20T09:06:54.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:06:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:06:54.212+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:06:54.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:06:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:06:54.212+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:06:55.175 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:06:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:06:54.816+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:54.816963+0000) 2024-02-20T09:06:55.176 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:06:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:06:54.919+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:06:55.176 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:06:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:06:54.919+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:06:55.176 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:06:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:06:54.919+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:06:55.176 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:06:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:06:54.933+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:54.933815+0000) 2024-02-20T09:06:55.177 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:06:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:06:55.172+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:06:55.177 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:06:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:06:55.172+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:06:55.177 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:06:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:06:55.172+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:06:55.177 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:06:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:06:54.868+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:06:55.177 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:06:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:06:54.868+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:06:55.177 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:06:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:06:54.868+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:06:55.177 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:06:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:06:54.868+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:06:55.461 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:06:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:06:55.234+0000 7f8a5f86e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:55.236259+0000) 2024-02-20T09:06:55.461 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:06:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:06:55.235+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:55.236496+0000) 2024-02-20T09:06:55.462 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:06:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:06:55.241+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 57724 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:06:55.462 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:06:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:06:55.238+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:55.240017+0000) 2024-02-20T09:06:55.462 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:06:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:06:55.333+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:55.334640+0000) 2024-02-20T09:06:56.169 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:06:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:06:55.816+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:55.817186+0000) 2024-02-20T09:06:56.169 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:06:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:06:55.933+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:06:56.169 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:06:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:06:55.933+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:06:56.169 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:06:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:06:55.933+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:06:56.169 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:06:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:06:55.849+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:06:56.170 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:06:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:06:55.849+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:06:56.170 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:06:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:06:55.849+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:06:56.170 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:06:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:06:55.849+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:06:56.170 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:06:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:06:55.933+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:55.933968+0000) 2024-02-20T09:06:56.170 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:06:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:06:56.167+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:06:56.170 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:06:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:06:56.167+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:06:56.170 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:06:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:06:56.167+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:06:56.461 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:06:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:06:56.336+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:56.336836+0000) 2024-02-20T09:06:57.183 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:06:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:06:56.815+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:56.817353+0000) 2024-02-20T09:06:57.183 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:06:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:06:56.944+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:06:57.183 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:06:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:06:56.944+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:06:57.183 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:06:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:06:56.944+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:06:57.184 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:06:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:06:56.805+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:06:57.184 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:06:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:06:56.805+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:06:57.184 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:06:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:06:56.805+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:06:57.184 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:06:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:06:56.805+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:06:57.184 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:06:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:06:56.933+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:56.934147+0000) 2024-02-20T09:06:57.184 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:06:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:06:57.180+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:06:57.185 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:06:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:06:57.180+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:06:57.185 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:06:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:06:57.180+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:06:57.461 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:06:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:06:57.339+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:57.339487+0000) 2024-02-20T09:06:58.164 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:06:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:06:57.933+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:57.934296+0000) 2024-02-20T09:06:58.164 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:06:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:06:58.162+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:06:58.165 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:06:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:06:58.162+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:06:58.165 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:06:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:06:57.817+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:57.817561+0000) 2024-02-20T09:06:58.165 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:06:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:06:57.992+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:06:58.165 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:06:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:06:57.992+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:06:58.165 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:06:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:06:57.992+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:06:58.166 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:06:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:06:57.834+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:06:58.166 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:06:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:06:57.834+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:06:58.166 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:06:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:06:57.834+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:06:58.166 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:06:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:06:57.834+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:06:58.462 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:06:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:06:58.341+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:58.342214+0000) 2024-02-20T09:06:58.462 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:06:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:06:58.162+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:06:59.154 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:06:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:06:58.934+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:58.934489+0000) 2024-02-20T09:06:59.155 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:06:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:06:59.153+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:06:59.155 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:06:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:06:58.817+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:58.817744+0000) 2024-02-20T09:06:59.155 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:06:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:06:58.967+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:06:59.155 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:06:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:06:58.967+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:06:59.155 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:06:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:06:58.967+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:06:59.156 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:06:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:06:58.798+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:06:59.156 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:06:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:06:58.798+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:06:59.156 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:06:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:06:58.798+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:06:59.156 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:06:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:06:58.798+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:06:59.462 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:06:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:06:59.342+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:59.342707+0000) 2024-02-20T09:06:59.462 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:06:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:06:59.153+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:06:59.462 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:06:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:06:59.153+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:07:00.126 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:06:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:06:59.934+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:59.934639+0000) 2024-02-20T09:07:00.127 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:06:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:06:59.780+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:07:00.127 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:06:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:06:59.780+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:07:00.127 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:06:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:06:59.780+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:07:00.127 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:06:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:06:59.780+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:07:00.128 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:06:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:06:59.817+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:06:59.817925+0000) 2024-02-20T09:07:00.128 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:06:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:06:59.964+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:07:00.128 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:06:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:06:59.964+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:07:00.128 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:06:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:06:59.964+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:07:00.393 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:07:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:07:00.251+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 57789 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:07:00.393 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:07:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:07:00.244+0000 7f8a6006f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:00.245281+0000) 2024-02-20T09:07:00.393 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:07:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:07:00.248+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:00.248430+0000) 2024-02-20T09:07:00.394 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:07:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:07:00.342+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:00.343002+0000) 2024-02-20T09:07:00.394 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:07:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:07:00.124+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:07:00.394 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:07:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:07:00.124+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:07:00.394 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:07:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:07:00.124+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:07:00.394 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:07:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:07:00.244+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:00.244571+0000) 2024-02-20T09:07:01.139 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:07:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:07:00.934+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:00.934831+0000) 2024-02-20T09:07:01.139 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:07:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:07:01.137+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:07:01.139 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:07:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:07:01.137+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:07:01.139 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:07:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:07:01.137+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:07:01.140 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:07:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:07:00.817+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:00.818039+0000) 2024-02-20T09:07:01.140 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:07:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:07:00.967+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:07:01.140 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:07:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:07:00.967+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:07:01.140 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:07:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:07:00.967+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:07:01.141 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:07:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:07:00.804+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:07:01.141 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:07:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:07:00.804+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:07:01.141 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:07:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:07:00.804+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:07:01.141 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:07:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:07:00.804+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:07:01.396 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:07:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:07:01.346+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:01.346738+0000) 2024-02-20T09:07:02.154 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:07:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:07:01.820+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:07:02.154 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:07:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:07:01.820+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:07:02.155 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:07:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:07:01.820+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:07:02.155 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:07:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:07:01.820+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:07:02.155 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:07:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:07:01.934+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:01.934973+0000) 2024-02-20T09:07:02.155 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:07:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:07:02.152+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:07:02.155 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:07:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:07:02.152+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:07:02.156 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:07:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:07:01.817+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:01.818192+0000) 2024-02-20T09:07:02.156 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:07:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:07:01.982+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:07:02.156 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:07:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:07:01.982+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:07:02.156 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:07:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:07:01.982+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:07:02.462 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:07:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:07:02.349+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:02.349628+0000) 2024-02-20T09:07:02.462 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:07:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:07:02.152+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:07:03.158 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:07:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:07:02.808+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:07:03.159 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:07:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:07:02.808+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:07:03.159 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:07:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:07:02.808+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:07:03.159 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:07:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:07:02.808+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:07:03.159 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:07:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:07:02.818+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:02.818381+0000) 2024-02-20T09:07:03.159 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:07:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:07:03.024+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:07:03.159 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:07:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:07:03.024+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:07:03.160 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:07:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:07:03.024+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:07:03.160 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:07:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:07:02.934+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:02.935153+0000) 2024-02-20T09:07:03.160 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:07:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:07:03.156+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:07:03.160 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:07:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:07:03.156+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:07:03.462 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:07:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:07:03.352+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:03.352418+0000) 2024-02-20T09:07:03.462 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:07:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:07:03.156+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:07:04.121 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:07:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:07:03.818+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:03.818582+0000) 2024-02-20T09:07:04.121 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:07:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:07:04.006+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:07:04.121 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:07:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:07:04.006+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:07:04.121 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:07:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:07:04.006+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:07:04.122 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:07:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:07:03.934+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:03.935311+0000) 2024-02-20T09:07:04.122 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:07:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:07:04.119+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:07:04.122 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:07:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:07:04.119+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:07:04.122 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:07:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:07:03.768+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:07:04.122 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:07:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:07:03.768+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:07:04.122 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:07:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:07:03.768+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:07:04.123 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:07:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:07:03.768+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:07:04.397 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:07:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:07:04.352+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:04.352802+0000) 2024-02-20T09:07:04.398 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:07:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:07:04.119+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:07:05.024 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:07:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:07:04.935+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:04.935503+0000) 2024-02-20T09:07:05.024 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:07:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:07:04.818+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:04.818737+0000) 2024-02-20T09:07:05.025 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:07:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:07:05.022+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:07:05.025 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:07:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:07:04.744+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:07:05.025 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:07:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:07:04.744+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:07:05.025 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:07:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:07:04.744+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:07:05.025 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:07:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:07:04.744+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:07:05.281 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:07:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:07:05.259+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 57861 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:07:05.282 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:07:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:07:05.255+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:05.256127+0000) 2024-02-20T09:07:05.282 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:07:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:07:05.022+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:07:05.282 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:07:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:07:05.022+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:07:05.282 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:07:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:07:05.253+0000 7f8a6006f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:05.253949+0000) 2024-02-20T09:07:05.282 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:07:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:07:05.087+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:07:05.283 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:07:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:07:05.087+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:07:05.283 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:07:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:07:05.087+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:07:05.283 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:07:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:07:05.253+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:05.253410+0000) 2024-02-20T09:07:05.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:07:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:07:05.352+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:05.353112+0000) 2024-02-20T09:07:06.067 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:07:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:07:05.818+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:05.818896+0000) 2024-02-20T09:07:06.067 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:07:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:07:06.065+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:07:06.068 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:07:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:07:05.935+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:05.935705+0000) 2024-02-20T09:07:06.068 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:07:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:07:05.752+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:07:06.068 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:07:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:07:05.752+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:07:06.068 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:07:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:07:05.752+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:07:06.068 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:07:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:07:05.752+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:07:06.357 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:07:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:07:06.065+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:07:06.357 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:07:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:07:06.065+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:07:06.357 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:07:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:07:06.133+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:07:06.358 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:07:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:07:06.133+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:07:06.358 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:07:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:07:06.133+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:07:06.727 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:07:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:07:06.355+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:06.355603+0000) 2024-02-20T09:07:06.728 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:07:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:07:06.725+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:07:06.728 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:07:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:07:06.725+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:07:06.728 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:07:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:07:06.725+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:07:06.728 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:07:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:07:06.725+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:07:06.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:07:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:07:06.818+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:06.819073+0000) 2024-02-20T09:07:06.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:07:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:07:06.935+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:06.935858+0000) 2024-02-20T09:07:07.360 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:07:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:07:07.182+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:07:07.360 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:07:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:07:07.182+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:07:07.361 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:07:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:07:07.182+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:07:07.361 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:07:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:07:07.065+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:07:07.361 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:07:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:07:07.065+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:07:07.362 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:07:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:07:07.065+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:07:07.698 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:07:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:07:07.358+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:07.358800+0000) 2024-02-20T09:07:07.698 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:07:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:07:07.696+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:07:07.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:07:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:07:07.818+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:07.819229+0000) 2024-02-20T09:07:07.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:07:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:07:07.935+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:07.936038+0000) 2024-02-20T09:07:07.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:07:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:07:07.696+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:07:07.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:07:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:07:07.696+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:07:07.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:07:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:07:07.696+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:07:08.363 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:07:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:07:08.091+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:07:08.363 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:07:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:07:08.091+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:07:08.363 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:07:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:07:08.091+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:07:08.363 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:07:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:07:08.226+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:07:08.363 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:07:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:07:08.226+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:07:08.364 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:07:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:07:08.226+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:07:08.697 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:07:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:07:08.361+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:08.361685+0000) 2024-02-20T09:07:08.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:07:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:07:08.819+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:08.819441+0000) 2024-02-20T09:07:08.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:07:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:07:08.935+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:08.936220+0000) 2024-02-20T09:07:08.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:07:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:07:08.695+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:07:08.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:07:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:07:08.695+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:07:08.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:07:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:07:08.695+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:07:08.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:07:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:07:08.695+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:07:09.401 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:07:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:07:09.361+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:09.362090+0000) 2024-02-20T09:07:09.402 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:07:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:07:09.140+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:07:09.402 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:07:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:07:09.140+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:07:09.402 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:07:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:07:09.140+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:07:09.403 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:07:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:07:09.240+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:07:09.403 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:07:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:07:09.240+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:07:09.403 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:07:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:07:09.240+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:07:09.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:07:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:07:09.646+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:07:09.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:07:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:07:09.646+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:07:09.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:07:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:07:09.646+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:07:09.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:07:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:07:09.646+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:07:10.107 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:07:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:07:09.936+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:09.936436+0000) 2024-02-20T09:07:10.107 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:07:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:07:09.819+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:09.819597+0000) 2024-02-20T09:07:10.107 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:07:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:07:10.105+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:07:10.364 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:07:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:07:10.268+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 57934 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:07:10.364 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:07:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:07:10.265+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:10.265771+0000) 2024-02-20T09:07:10.364 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:07:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:07:10.105+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:07:10.365 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:07:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:07:10.105+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:07:10.365 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:07:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:07:10.261+0000 7f8a5f86e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:10.262221+0000) 2024-02-20T09:07:10.365 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:07:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:07:10.249+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:07:10.365 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:07:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:07:10.249+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:07:10.365 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:07:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:07:10.249+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:07:10.365 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:07:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:07:10.262+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:10.262916+0000) 2024-02-20T09:07:10.677 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:07:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:07:10.362+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:10.362430+0000) 2024-02-20T09:07:10.677 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:07:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:07:10.675+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:07:10.938 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:07:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:07:10.819+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:10.819713+0000) 2024-02-20T09:07:10.938 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:07:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:07:10.675+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:07:10.938 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:07:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:07:10.675+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:07:10.938 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:07:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:07:10.675+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:07:11.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:07:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:07:10.936+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:10.936622+0000) 2024-02-20T09:07:11.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:07:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:07:11.142+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:07:11.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:07:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:07:11.142+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:07:11.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:07:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:07:11.142+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:07:11.727 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:07:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:07:11.293+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:07:11.727 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:07:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:07:11.293+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:07:11.727 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:07:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:07:11.293+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:07:11.727 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:07:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:07:11.364+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:11.365194+0000) 2024-02-20T09:07:11.727 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:07:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:07:11.725+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:07:11.727 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:07:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:07:11.725+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:07:11.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:07:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:07:11.725+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:07:11.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:07:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:07:11.725+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:07:11.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:07:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:07:11.819+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:11.819843+0000) 2024-02-20T09:07:11.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:07:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:07:11.936+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:11.936758+0000) 2024-02-20T09:07:12.464 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:07:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:07:12.367+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:12.368061+0000) 2024-02-20T09:07:12.464 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:07:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:07:12.190+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:07:12.464 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:07:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:07:12.190+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:07:12.465 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:07:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:07:12.190+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:07:12.465 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:07:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:07:12.284+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:07:12.465 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:07:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:07:12.284+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:07:12.465 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:07:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:07:12.284+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:07:13.167 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:07:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:07:12.936+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:12.936969+0000) 2024-02-20T09:07:13.168 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:07:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:07:12.819+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:12.820017+0000) 2024-02-20T09:07:13.168 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:07:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:07:13.165+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:07:13.168 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:07:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:07:12.771+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:07:13.168 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:07:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:07:12.771+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:07:13.168 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:07:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:07:12.771+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:07:13.169 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:07:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:07:12.771+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:07:13.464 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:07:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:07:13.370+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:13.370900+0000) 2024-02-20T09:07:13.464 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:07:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:07:13.165+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:07:13.465 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:07:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:07:13.165+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:07:13.465 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:07:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:07:13.268+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:07:13.465 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:07:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:07:13.268+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:07:13.465 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:07:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:07:13.268+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:07:14.158 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:07:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:07:13.937+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:13.937152+0000) 2024-02-20T09:07:14.159 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:07:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:07:13.764+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:07:14.159 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:07:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:07:13.764+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:07:14.159 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:07:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:07:13.764+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:07:14.159 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:07:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:07:13.764+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:07:14.159 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:07:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:07:13.820+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:13.820228+0000) 2024-02-20T09:07:14.160 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:07:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:07:14.157+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:07:14.160 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:07:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:07:14.157+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:07:14.160 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:07:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:07:14.157+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:07:14.464 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:07:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:07:14.371+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:14.371446+0000) 2024-02-20T09:07:14.465 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:07:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:07:14.281+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:07:14.465 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:07:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:07:14.281+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:07:14.465 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:07:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:07:14.281+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:07:15.122 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:07:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:07:14.936+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:14.937270+0000) 2024-02-20T09:07:15.122 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:07:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:07:14.756+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:07:15.122 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:07:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:07:14.756+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:07:15.123 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:07:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:07:14.756+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:07:15.123 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:07:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:07:14.757+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:07:15.123 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:07:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:07:14.820+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:14.820446+0000) 2024-02-20T09:07:15.123 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:07:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:07:15.120+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:07:15.123 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:07:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:07:15.120+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:07:15.123 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:07:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:07:15.120+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:07:15.373 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:07:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:07:15.277+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 58006 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:07:15.373 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:07:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:07:15.271+0000 7f8a6006f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:15.271750+0000) 2024-02-20T09:07:15.373 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:07:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:07:15.273+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:15.274346+0000) 2024-02-20T09:07:15.374 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:07:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:07:15.270+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:15.271214+0000) 2024-02-20T09:07:15.374 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:07:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:07:15.299+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:07:15.374 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:07:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:07:15.299+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:07:15.374 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:07:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:07:15.299+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:07:15.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:07:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:07:15.371+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:15.371762+0000) 2024-02-20T09:07:16.169 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:07:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:07:15.937+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:15.937475+0000) 2024-02-20T09:07:16.169 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:07:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:07:15.800+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:07:16.170 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:07:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:07:15.800+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:07:16.170 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:07:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:07:15.800+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:07:16.170 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:07:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:07:15.800+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:07:16.170 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:07:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:07:15.820+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:15.820662+0000) 2024-02-20T09:07:16.170 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:07:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:07:16.167+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:07:16.170 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:07:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:07:16.167+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:07:16.170 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:07:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:07:16.167+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:07:16.464 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:07:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:07:16.374+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:16.374390+0000) 2024-02-20T09:07:16.465 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:07:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:07:16.272+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:07:16.465 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:07:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:07:16.272+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:07:16.465 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:07:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:07:16.272+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:07:17.155 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:07:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:07:16.937+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:16.937602+0000) 2024-02-20T09:07:17.155 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:07:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:07:16.833+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:07:17.155 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:07:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:07:16.833+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:07:17.155 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:07:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:07:16.833+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:07:17.155 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:07:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:07:16.833+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:07:17.156 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:07:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:07:16.820+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:16.820799+0000) 2024-02-20T09:07:17.156 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:07:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:07:17.153+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:07:17.156 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:07:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:07:17.153+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:07:17.156 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:07:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:07:17.153+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:07:17.409 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:07:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:07:17.377+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:17.377172+0000) 2024-02-20T09:07:17.409 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:07:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:07:17.263+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:07:17.409 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:07:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:07:17.263+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:07:17.409 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:07:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:07:17.263+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:07:18.196 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:07:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:07:17.937+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:17.937777+0000) 2024-02-20T09:07:18.196 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:07:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:07:17.820+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:17.820956+0000) 2024-02-20T09:07:18.196 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:07:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:07:18.194+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:07:18.197 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:07:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:07:17.845+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:07:18.197 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:07:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:07:17.845+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:07:18.197 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:07:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:07:17.845+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:07:18.197 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:07:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:07:17.845+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:07:18.465 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:07:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:07:18.379+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:18.379953+0000) 2024-02-20T09:07:18.465 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:07:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:07:18.194+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:07:18.465 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:07:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:07:18.194+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:07:18.466 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:07:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:07:18.313+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:07:18.466 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:07:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:07:18.313+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:07:18.466 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:07:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:07:18.313+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:07:19.215 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:07:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:07:18.937+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:18.937983+0000) 2024-02-20T09:07:19.216 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:07:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:07:18.821+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:18.821130+0000) 2024-02-20T09:07:19.216 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:07:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:07:19.214+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:07:19.216 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:07:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:07:19.214+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:07:19.216 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:07:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:07:18.863+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:07:19.216 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:07:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:07:18.863+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:07:19.217 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:07:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:07:18.863+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:07:19.217 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:07:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:07:18.864+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:07:19.492 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:07:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:07:19.214+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:07:19.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:07:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:07:19.380+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:19.380453+0000) 2024-02-20T09:07:19.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:07:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:07:19.338+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:07:19.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:07:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:07:19.339+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:07:19.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:07:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:07:19.339+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:07:20.196 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:07:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:07:19.821+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:19.821349+0000) 2024-02-20T09:07:20.196 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:07:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:07:20.193+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:07:20.197 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:07:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:07:19.937+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:19.938145+0000) 2024-02-20T09:07:20.197 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:07:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:07:19.890+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:07:20.197 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:07:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:07:19.890+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:07:20.197 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:07:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:07:19.890+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:07:20.197 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:07:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:07:19.890+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:07:20.465 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:07:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:07:20.285+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 58079 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:07:20.466 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:07:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:07:20.283+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:20.284657+0000) 2024-02-20T09:07:20.466 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:07:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:07:20.379+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:20.380795+0000) 2024-02-20T09:07:20.466 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:07:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:07:20.193+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:07:20.466 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:07:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:07:20.193+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:07:20.466 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:07:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:07:20.278+0000 7f8a6006f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:20.280061+0000) 2024-02-20T09:07:20.467 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:07:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:07:20.278+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:20.279976+0000) 2024-02-20T09:07:20.467 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:07:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:07:20.315+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:07:20.467 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:07:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:07:20.315+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:07:20.467 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:07:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:07:20.315+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:07:21.206 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:07:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:07:20.936+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:20.938348+0000) 2024-02-20T09:07:21.206 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:07:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:07:20.820+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:20.821495+0000) 2024-02-20T09:07:21.206 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:07:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:07:21.202+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:07:21.207 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:07:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:07:20.843+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:07:21.207 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:07:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:07:20.843+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:07:21.207 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:07:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:07:20.843+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:07:21.207 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:07:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:07:20.843+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:07:21.465 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:07:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:07:21.382+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:21.383613+0000) 2024-02-20T09:07:21.466 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:07:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:07:21.203+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:07:21.466 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:07:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:07:21.203+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:07:21.466 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:07:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:07:21.294+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:07:21.466 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:07:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:07:21.294+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:07:21.466 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:07:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:07:21.294+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:07:22.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:07:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:07:21.820+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:21.821638+0000) 2024-02-20T09:07:22.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:07:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:07:21.937+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:21.938480+0000) 2024-02-20T09:07:22.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:07:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:07:21.841+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:07:22.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:07:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:07:21.841+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:07:22.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:07:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:07:21.841+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:07:22.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:07:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:07:21.841+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:07:22.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:07:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:07:22.385+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:22.386437+0000) 2024-02-20T09:07:22.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:07:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:07:22.253+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:07:22.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:07:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:07:22.253+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:07:22.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:07:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:07:22.253+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:07:22.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:07:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:07:22.246+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:07:22.744 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:07:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:07:22.246+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:07:22.744 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:07:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:07:22.246+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:07:23.207 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:07:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:07:22.937+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:22.938643+0000) 2024-02-20T09:07:23.207 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:07:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:07:22.845+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:07:23.207 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:07:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:07:22.845+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:07:23.207 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:07:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:07:22.845+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:07:23.208 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:07:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:07:22.845+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:07:23.208 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:07:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:07:22.820+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:22.821839+0000) 2024-02-20T09:07:23.208 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:07:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:07:23.204+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:07:23.208 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:07:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:07:23.204+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:07:23.208 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:07:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:07:23.204+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:07:23.465 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:07:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:07:23.387+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:23.389225+0000) 2024-02-20T09:07:23.466 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:07:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:07:23.287+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:07:23.466 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:07:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:07:23.287+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:07:23.466 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:07:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:07:23.287+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:07:24.174 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:07:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:07:23.937+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:23.938800+0000) 2024-02-20T09:07:24.175 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:07:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:07:23.820+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:23.821998+0000) 2024-02-20T09:07:24.175 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:07:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:07:24.171+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:07:24.175 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:07:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:07:23.851+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:07:24.176 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:07:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:07:23.851+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:07:24.176 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:07:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:07:23.851+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:07:24.176 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:07:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:07:23.851+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:07:24.467 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:07:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:07:24.388+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:24.389745+0000) 2024-02-20T09:07:24.468 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:07:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:07:24.171+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:07:24.468 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:07:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:07:24.171+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:07:24.468 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:07:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:07:24.336+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:07:24.468 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:07:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:07:24.336+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:07:24.468 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:07:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:07:24.336+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:07:25.182 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:07:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:07:24.938+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:24.938987+0000) 2024-02-20T09:07:25.183 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:07:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:07:24.821+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:24.822171+0000) 2024-02-20T09:07:25.183 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:07:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:07:25.179+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:07:25.183 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:07:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:07:24.885+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:07:25.184 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:07:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:07:24.885+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:07:25.184 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:07:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:07:24.885+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:07:25.184 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:07:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:07:24.885+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:07:25.466 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:07:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:07:25.294+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 58151 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:07:25.466 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:07:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:07:25.290+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:25.291868+0000) 2024-02-20T09:07:25.466 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:07:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:07:25.389+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:25.390088+0000) 2024-02-20T09:07:25.467 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:07:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:07:25.179+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:07:25.467 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:07:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:07:25.179+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:07:25.467 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:07:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:07:25.287+0000 7f8a6006f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:25.288702+0000) 2024-02-20T09:07:25.467 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:07:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:07:25.287+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:25.289194+0000) 2024-02-20T09:07:25.467 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:07:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:07:25.323+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:07:25.467 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:07:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:07:25.323+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:07:25.468 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:07:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:07:25.323+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:07:26.162 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:07:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:07:25.937+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:25.939165+0000) 2024-02-20T09:07:26.162 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:07:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:07:25.820+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:25.822354+0000) 2024-02-20T09:07:26.162 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:07:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:07:26.159+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:07:26.163 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:07:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:07:26.159+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:07:26.163 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:07:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:07:26.159+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:07:26.163 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:07:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:07:25.934+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:07:26.163 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:07:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:07:25.934+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:07:26.163 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:07:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:07:25.934+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:07:26.163 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:07:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:07:25.934+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:07:26.416 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:07:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:07:26.391+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:26.392707+0000) 2024-02-20T09:07:26.417 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:07:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:07:26.325+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:07:26.417 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:07:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:07:26.325+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:07:26.417 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:07:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:07:26.325+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:07:27.208 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:07:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:07:26.821+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:26.822548+0000) 2024-02-20T09:07:27.208 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:07:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:07:26.938+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:26.939327+0000) 2024-02-20T09:07:27.209 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:07:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:07:26.981+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:07:27.209 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:07:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:07:26.981+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:07:27.209 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:07:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:07:26.981+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:07:27.209 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:07:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:07:26.981+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:07:27.466 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:07:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:07:27.394+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:27.395703+0000) 2024-02-20T09:07:27.467 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:07:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:07:27.358+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:07:27.467 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:07:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:07:27.358+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:07:27.467 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:07:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:07:27.358+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:07:27.467 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:07:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:07:27.205+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:07:27.467 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:07:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:07:27.205+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:07:27.468 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:07:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:07:27.205+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:07:28.162 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:07:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:07:27.938+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:27.939506+0000) 2024-02-20T09:07:28.162 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:07:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:07:28.028+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:07:28.163 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:07:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:07:28.028+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:07:28.163 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:07:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:07:28.028+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:07:28.163 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:07:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:07:28.028+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:07:28.163 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:07:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:07:27.821+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:27.822693+0000) 2024-02-20T09:07:28.163 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:07:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:07:28.159+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:07:28.163 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:07:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:07:28.159+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:07:28.163 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:07:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:07:28.159+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:07:28.417 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:07:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:07:28.397+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:28.398678+0000) 2024-02-20T09:07:28.418 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:07:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:07:28.334+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:07:28.418 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:07:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:07:28.334+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:07:28.418 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:07:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:07:28.334+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:07:29.146 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:07:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:07:28.938+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:28.939686+0000) 2024-02-20T09:07:29.146 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:07:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:07:28.821+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:28.822847+0000) 2024-02-20T09:07:29.147 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:07:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:07:29.143+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:07:29.147 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:07:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:07:29.018+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:07:29.147 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:07:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:07:29.018+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:07:29.147 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:07:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:07:29.018+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:07:29.147 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:07:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:07:29.018+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:07:29.400 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:07:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:07:29.143+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:07:29.400 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:07:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:07:29.143+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:07:29.401 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:07:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:07:29.285+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:07:29.401 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:07:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:07:29.285+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:07:29.401 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:07:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:07:29.285+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:07:29.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:07:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:07:29.398+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:29.399168+0000) 2024-02-20T09:07:30.192 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:07:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:07:29.938+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:29.939861+0000) 2024-02-20T09:07:30.192 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:07:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:07:30.051+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:07:30.192 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:07:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:07:30.051+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:07:30.192 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:07:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:07:30.051+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:07:30.193 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:07:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:07:30.051+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:07:30.193 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:07:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:07:29.822+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:29.823007+0000) 2024-02-20T09:07:30.193 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:07:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:07:30.189+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:07:30.193 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:07:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:07:30.189+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:07:30.193 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:07:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:07:30.189+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:07:30.467 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:07:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:07:30.302+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 58224 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:07:30.467 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:07:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:07:30.295+0000 7f8a6006f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:30.296896+0000) 2024-02-20T09:07:30.467 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:07:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:07:30.299+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:30.300935+0000) 2024-02-20T09:07:30.468 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:07:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:07:30.398+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:30.399491+0000) 2024-02-20T09:07:30.468 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:07:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:07:30.295+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:30.297052+0000) 2024-02-20T09:07:30.468 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:07:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:07:30.324+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:07:30.468 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:07:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:07:30.324+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:07:30.468 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:07:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:07:30.324+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:07:31.066 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:07:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:07:30.822+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:30.823180+0000) 2024-02-20T09:07:31.066 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:07:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:07:30.939+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:30.940058+0000) 2024-02-20T09:07:31.345 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:07:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:07:31.150+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:07:31.346 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:07:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:07:31.150+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:07:31.346 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:07:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:07:31.150+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:07:31.346 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:07:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:07:31.063+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:07:31.346 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:07:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:07:31.063+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:07:31.346 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:07:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:07:31.063+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:07:31.346 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:07:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:07:31.063+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:07:31.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:07:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:07:31.401+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:31.401967+0000) 2024-02-20T09:07:31.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:07:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:07:31.342+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:07:31.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:07:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:07:31.342+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:07:31.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:07:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:07:31.342+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:07:32.085 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:07:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:07:31.821+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:31.823340+0000) 2024-02-20T09:07:32.085 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:07:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:07:31.938+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:31.940245+0000) 2024-02-20T09:07:32.406 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:07:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:07:32.193+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:07:32.406 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:07:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:07:32.193+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:07:32.406 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:07:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:07:32.193+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:07:32.407 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:07:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:07:32.314+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:07:32.407 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:07:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:07:32.315+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:07:32.407 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:07:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:07:32.315+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:07:32.407 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:07:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:07:32.082+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:07:32.407 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:07:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:07:32.082+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:07:32.407 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:07:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:07:32.082+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:07:32.407 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:07:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:07:32.082+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:07:32.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:07:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:07:32.403+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:32.404812+0000) 2024-02-20T09:07:33.084 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:07:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:07:32.822+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:32.823530+0000) 2024-02-20T09:07:33.085 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:07:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:07:32.939+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:32.940434+0000) 2024-02-20T09:07:33.409 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:07:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:07:33.161+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:07:33.409 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:07:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:07:33.161+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:07:33.409 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:07:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:07:33.161+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:07:33.410 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:07:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:07:33.330+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:07:33.410 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:07:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:07:33.330+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:07:33.410 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:07:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:07:33.330+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:07:33.410 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:07:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:07:33.082+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:07:33.410 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:07:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:07:33.082+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:07:33.410 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:07:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:07:33.082+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:07:33.410 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:07:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:07:33.082+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:07:33.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:07:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:07:33.406+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:33.407552+0000) 2024-02-20T09:07:34.093 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:07:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:07:33.939+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:33.940615+0000) 2024-02-20T09:07:34.094 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:07:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:07:33.822+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:33.823731+0000) 2024-02-20T09:07:34.409 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:07:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:07:34.196+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:07:34.409 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:07:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:07:34.196+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:07:34.410 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:07:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:07:34.196+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:07:34.410 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:07:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:07:34.298+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:07:34.410 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:07:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:07:34.298+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:07:34.410 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:07:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:07:34.298+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:07:34.410 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:07:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:07:34.091+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:07:34.411 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:07:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:07:34.091+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:07:34.411 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:07:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:07:34.091+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:07:34.411 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:07:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:07:34.091+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:07:34.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:07:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:07:34.406+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:34.408074+0000) 2024-02-20T09:07:35.140 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:07:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:07:34.822+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:34.823872+0000) 2024-02-20T09:07:35.141 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:07:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:07:34.940+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:34.940817+0000) 2024-02-20T09:07:35.409 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:07:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:07:35.311+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 58296 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:07:35.410 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:07:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:07:35.223+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:07:35.410 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:07:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:07:35.223+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:07:35.410 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:07:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:07:35.223+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:07:35.410 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:07:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:07:35.305+0000 7f8a6006f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:35.306542+0000) 2024-02-20T09:07:35.411 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:07:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:07:35.304+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:35.305957+0000) 2024-02-20T09:07:35.411 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:07:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:07:35.336+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:07:35.411 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:07:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:07:35.336+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:07:35.411 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:07:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:07:35.336+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:07:35.411 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:07:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:07:35.138+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:07:35.411 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:07:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:07:35.138+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:07:35.411 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:07:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:07:35.138+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:07:35.411 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:07:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:07:35.138+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:07:35.412 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:07:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:07:35.307+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:35.309390+0000) 2024-02-20T09:07:35.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:07:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:07:35.406+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:35.408360+0000) 2024-02-20T09:07:36.101 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:07:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:07:35.939+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:35.940966+0000) 2024-02-20T09:07:36.102 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:07:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:07:35.823+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:35.824056+0000) 2024-02-20T09:07:36.363 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:07:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:07:36.099+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:07:36.363 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:07:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:07:36.099+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:07:36.363 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:07:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:07:36.099+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:07:36.363 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:07:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:07:36.099+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:07:36.363 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:07:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:07:36.180+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:07:36.364 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:07:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:07:36.180+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:07:36.364 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:07:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:07:36.180+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:07:36.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:07:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:07:36.410+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:36.411952+0000) 2024-02-20T09:07:36.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:07:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:07:36.360+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:07:36.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:07:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:07:36.360+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:07:36.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:07:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:07:36.360+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:07:37.123 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:07:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:07:36.940+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:36.941104+0000) 2024-02-20T09:07:37.124 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:07:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:07:36.822+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:36.824169+0000) 2024-02-20T09:07:37.416 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:07:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:07:37.330+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:07:37.416 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:07:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:07:37.330+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:07:37.416 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:07:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:07:37.330+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:07:37.417 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:07:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:07:37.217+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:07:37.417 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:07:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:07:37.217+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:07:37.417 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:07:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:07:37.217+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:07:37.417 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:07:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:07:37.119+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:07:37.417 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:07:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:07:37.119+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:07:37.418 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:07:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:07:37.119+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:07:37.418 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:07:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:07:37.119+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:07:37.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:07:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:07:37.414+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:37.414852+0000) 2024-02-20T09:07:38.111 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:07:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:07:37.823+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:37.824333+0000) 2024-02-20T09:07:38.112 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:07:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:07:37.939+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:37.941279+0000) 2024-02-20T09:07:38.370 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:07:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:07:38.261+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:07:38.370 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:07:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:07:38.261+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:07:38.371 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:07:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:07:38.261+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:07:38.371 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:07:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:07:38.109+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:07:38.371 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:07:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:07:38.109+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:07:38.371 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:07:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:07:38.109+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:07:38.371 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:07:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:07:38.109+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:07:38.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:07:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:07:38.416+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:38.417821+0000) 2024-02-20T09:07:38.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:07:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:07:38.368+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:07:38.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:07:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:07:38.368+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:07:38.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:07:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:07:38.368+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:07:39.140 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:07:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:07:38.940+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:38.941489+0000) 2024-02-20T09:07:39.140 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:07:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:07:38.823+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:38.824510+0000) 2024-02-20T09:07:39.401 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:07:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:07:39.300+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:07:39.401 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:07:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:07:39.300+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:07:39.402 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:07:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:07:39.300+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:07:39.402 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:07:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:07:39.137+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:07:39.402 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:07:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:07:39.137+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:07:39.402 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:07:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:07:39.137+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:07:39.402 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:07:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:07:39.137+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:07:39.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:07:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:07:39.416+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:39.418370+0000) 2024-02-20T09:07:39.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:07:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:07:39.399+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:07:39.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:07:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:07:39.399+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:07:39.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:07:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:07:39.399+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:07:40.183 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:07:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:07:39.824+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:39.824715+0000) 2024-02-20T09:07:40.183 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:07:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:07:39.941+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:39.941667+0000) 2024-02-20T09:07:40.443 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:07:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:07:40.319+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 58369 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:07:40.443 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:07:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:07:40.298+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:07:40.443 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:07:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:07:40.298+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:07:40.443 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:07:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:07:40.298+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:07:40.444 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:07:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:07:40.313+0000 7f8a6006f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:40.314540+0000) 2024-02-20T09:07:40.444 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:07:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:07:40.313+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:40.314569+0000) 2024-02-20T09:07:40.444 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:07:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:07:40.364+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:07:40.444 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:07:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:07:40.364+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:07:40.444 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:07:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:07:40.364+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:07:40.445 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:07:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:07:40.180+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:07:40.445 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:07:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:07:40.180+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:07:40.445 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:07:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:07:40.180+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:07:40.445 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:07:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:07:40.180+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:07:40.445 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:07:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:07:40.315+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:40.316918+0000) 2024-02-20T09:07:40.445 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:07:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:07:40.417+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:40.418659+0000) 2024-02-20T09:07:41.150 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:07:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:07:40.940+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:40.941858+0000) 2024-02-20T09:07:41.151 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:07:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:07:40.824+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:40.824899+0000) 2024-02-20T09:07:41.422 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:07:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:07:41.254+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:07:41.423 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:07:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:07:41.254+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:07:41.423 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:07:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:07:41.254+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:07:41.423 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:07:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:07:41.372+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:07:41.423 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:07:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:07:41.372+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:07:41.423 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:07:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:07:41.372+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:07:41.424 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:07:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:07:41.148+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:07:41.424 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:07:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:07:41.148+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:07:41.424 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:07:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:07:41.148+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:07:41.424 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:07:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:07:41.148+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:07:41.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:07:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:07:41.420+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:41.421354+0000) 2024-02-20T09:07:42.187 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:07:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:07:41.823+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:41.824992+0000) 2024-02-20T09:07:42.188 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:07:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:07:41.941+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:41.941973+0000) 2024-02-20T09:07:42.457 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:07:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:07:42.210+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:07:42.457 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:07:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:07:42.210+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:07:42.457 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:07:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:07:42.210+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:07:42.458 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:07:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:07:42.381+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:07:42.458 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:07:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:07:42.381+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:07:42.458 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:07:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:07:42.381+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:07:42.458 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:07:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:07:42.185+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:07:42.458 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:07:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:07:42.185+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:07:42.458 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:07:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:07:42.185+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:07:42.458 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:07:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:07:42.185+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:07:42.459 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:07:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:07:42.423+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:42.424436+0000) 2024-02-20T09:07:43.191 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:07:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:07:42.941+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:42.942133+0000) 2024-02-20T09:07:43.192 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:07:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:07:42.823+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:42.825141+0000) 2024-02-20T09:07:43.469 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:07:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:07:43.239+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:07:43.469 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:07:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:07:43.239+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:07:43.469 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:07:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:07:43.239+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:07:43.470 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:07:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:07:43.429+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:07:43.470 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:07:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:07:43.429+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:07:43.470 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:07:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:07:43.429+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:07:43.470 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:07:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:07:43.188+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:07:43.470 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:07:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:07:43.188+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:07:43.470 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:07:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:07:43.188+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:07:43.471 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:07:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:07:43.188+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:07:43.471 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:07:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:07:43.426+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:43.427213+0000) 2024-02-20T09:07:44.162 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:07:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:07:43.941+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:43.942320+0000) 2024-02-20T09:07:44.162 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:07:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:07:43.823+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:43.825337+0000) 2024-02-20T09:07:44.429 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:07:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:07:44.160+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:07:44.429 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:07:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:07:44.160+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:07:44.429 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:07:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:07:44.160+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:07:44.429 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:07:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:07:44.160+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:07:44.430 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:07:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:07:44.259+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:07:44.430 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:07:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:07:44.259+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:07:44.430 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:07:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:07:44.259+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:07:44.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:07:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:07:44.427+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:44.427675+0000) 2024-02-20T09:07:44.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:07:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:07:44.480+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:07:44.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:07:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:07:44.480+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:07:44.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:07:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:07:44.480+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:07:45.153 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:07:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:07:44.824+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:44.825542+0000) 2024-02-20T09:07:45.154 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:07:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:07:44.941+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:44.942480+0000) 2024-02-20T09:07:45.429 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:07:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:07:45.328+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 58441 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:07:45.429 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:07:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:07:45.321+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:45.322671+0000) 2024-02-20T09:07:45.430 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:07:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:07:45.219+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:07:45.430 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:07:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:07:45.219+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:07:45.430 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:07:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:07:45.219+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:07:45.430 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:07:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:07:45.321+0000 7f8a5f86e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:45.323310+0000) 2024-02-20T09:07:45.430 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:07:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:07:45.150+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:07:45.431 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:07:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:07:45.151+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:07:45.431 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:07:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:07:45.151+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:07:45.431 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:07:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:07:45.151+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:07:45.431 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:07:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:07:45.324+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:45.325895+0000) 2024-02-20T09:07:45.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:07:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:07:45.427+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:45.427961+0000) 2024-02-20T09:07:45.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:07:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:07:45.509+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:07:45.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:07:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:07:45.509+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:07:45.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:07:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:07:45.509+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:07:46.164 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:07:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:07:45.824+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:45.825678+0000) 2024-02-20T09:07:46.164 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:07:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:07:45.942+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:45.942682+0000) 2024-02-20T09:07:46.433 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:07:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:07:46.258+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:07:46.433 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:07:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:07:46.258+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:07:46.433 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:07:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:07:46.258+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:07:46.433 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:07:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:07:46.162+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:07:46.433 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:07:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:07:46.162+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:07:46.433 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:07:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:07:46.162+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:07:46.434 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:07:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:07:46.162+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:07:46.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:07:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:07:46.430+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:46.431489+0000) 2024-02-20T09:07:46.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:07:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:07:46.463+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:07:46.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:07:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:07:46.463+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:07:46.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:07:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:07:46.463+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:07:47.184 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:07:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:07:46.824+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:46.825787+0000) 2024-02-20T09:07:47.185 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:07:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:07:46.942+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:46.942839+0000) 2024-02-20T09:07:47.436 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:07:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:07:47.250+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:07:47.436 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:07:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:07:47.250+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:07:47.436 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:07:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:07:47.250+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:07:47.437 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:07:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:07:47.422+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:07:47.437 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:07:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:07:47.423+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:07:47.437 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:07:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:07:47.423+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:07:47.437 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:07:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:07:47.182+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:07:47.437 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:07:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:07:47.182+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:07:47.437 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:07:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:07:47.182+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:07:47.438 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:07:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:07:47.182+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:07:47.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:07:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:07:47.433+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:47.434552+0000) 2024-02-20T09:07:48.222 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:07:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:07:47.825+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:47.825945+0000) 2024-02-20T09:07:48.222 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:07:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:07:47.942+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:47.943017+0000) 2024-02-20T09:07:48.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:07:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:07:48.407+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:07:48.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:07:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:07:48.407+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:07:48.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:07:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:07:48.407+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:07:48.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:07:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:07:48.282+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:07:48.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:07:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:07:48.282+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:07:48.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:07:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:07:48.282+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:07:48.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:07:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:07:48.220+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:07:48.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:07:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:07:48.220+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:07:48.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:07:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:07:48.220+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:07:48.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:07:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:07:48.220+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:07:48.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:07:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:07:48.436+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:48.437494+0000) 2024-02-20T09:07:49.182 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:07:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:07:48.825+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:48.826067+0000) 2024-02-20T09:07:49.183 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:07:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:07:48.942+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:48.943166+0000) 2024-02-20T09:07:49.433 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:07:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:07:49.300+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:07:49.434 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:07:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:07:49.300+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:07:49.434 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:07:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:07:49.300+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:07:49.434 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:07:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:07:49.180+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:07:49.434 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:07:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:07:49.180+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:07:49.434 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:07:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:07:49.180+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:07:49.434 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:07:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:07:49.180+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:07:49.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:07:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:07:49.436+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:49.438131+0000) 2024-02-20T09:07:49.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:07:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:07:49.431+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:07:49.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:07:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:07:49.431+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:07:49.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:07:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:07:49.431+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:07:50.149 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:07:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:07:49.825+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:49.826225+0000) 2024-02-20T09:07:50.149 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:07:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:07:49.942+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:49.943318+0000) 2024-02-20T09:07:50.429 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:07:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:07:50.337+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 58514 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:07:50.429 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:07:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:07:50.330+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:50.332237+0000) 2024-02-20T09:07:50.430 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:07:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:07:50.427+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:07:50.430 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:07:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:07:50.427+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:07:50.430 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:07:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:07:50.274+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:07:50.430 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:07:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:07:50.274+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:07:50.430 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:07:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:07:50.274+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:07:50.430 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:07:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:07:50.330+0000 7f8a5f86e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:50.331883+0000) 2024-02-20T09:07:50.431 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:07:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:07:50.146+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:07:50.431 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:07:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:07:50.146+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:07:50.431 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:07:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:07:50.146+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:07:50.431 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:07:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:07:50.146+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:07:50.431 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:07:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:07:50.334+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:50.335855+0000) 2024-02-20T09:07:50.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:07:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:07:50.437+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:50.438449+0000) 2024-02-20T09:07:50.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:07:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:07:50.427+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:07:51.106 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:07:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:07:50.825+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:50.826456+0000) 2024-02-20T09:07:51.107 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:07:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:07:50.943+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:50.943483+0000) 2024-02-20T09:07:51.408 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:07:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:07:51.303+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:07:51.408 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:07:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:07:51.303+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:07:51.408 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:07:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:07:51.303+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:07:51.409 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:07:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:07:51.104+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:07:51.409 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:07:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:07:51.104+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:07:51.409 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:07:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:07:51.104+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:07:51.409 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:07:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:07:51.104+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:07:51.673 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:07:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:07:51.440+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:51.441065+0000) 2024-02-20T09:07:51.673 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:07:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:07:51.406+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:07:51.673 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:07:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:07:51.406+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:07:51.673 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:07:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:07:51.406+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:07:51.944 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:07:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:07:51.826+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:51.826568+0000) 2024-02-20T09:07:52.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:07:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:07:51.943+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:51.943659+0000) 2024-02-20T09:07:52.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:07:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:07:52.101+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:07:52.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:07:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:07:52.101+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:07:52.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:07:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:07:52.101+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:07:52.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:07:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:07:52.101+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:07:52.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:07:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:07:52.443+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:52.443873+0000) 2024-02-20T09:07:52.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:07:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:07:52.258+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:07:52.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:07:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:07:52.258+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:07:52.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:07:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:07:52.258+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:07:52.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:07:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:07:52.388+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:07:52.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:07:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:07:52.388+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:07:52.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:07:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:07:52.388+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:07:53.070 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:07:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:07:52.826+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:52.826729+0000) 2024-02-20T09:07:53.071 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:07:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:07:52.943+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:52.943850+0000) 2024-02-20T09:07:53.363 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:07:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:07:53.239+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:07:53.363 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:07:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:07:53.239+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:07:53.363 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:07:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:07:53.239+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:07:53.364 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:07:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:07:53.068+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:07:53.364 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:07:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:07:53.069+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:07:53.364 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:07:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:07:53.069+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:07:53.364 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:07:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:07:53.069+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:07:53.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:07:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:07:53.446+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:53.446728+0000) 2024-02-20T09:07:53.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:07:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:07:53.361+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:07:53.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:07:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:07:53.361+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:07:53.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:07:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:07:53.361+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:07:54.104 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:07:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:07:53.826+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:53.826887+0000) 2024-02-20T09:07:54.105 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:07:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:07:53.943+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:53.944066+0000) 2024-02-20T09:07:54.411 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:07:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:07:54.248+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:07:54.411 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:07:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:07:54.248+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:07:54.411 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:07:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:07:54.248+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:07:54.411 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:07:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:07:54.102+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:07:54.411 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:07:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:07:54.102+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:07:54.412 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:07:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:07:54.102+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:07:54.412 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:07:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:07:54.102+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:07:54.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:07:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:07:54.446+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:54.447320+0000) 2024-02-20T09:07:54.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:07:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:07:54.409+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:07:54.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:07:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:07:54.409+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:07:54.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:07:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:07:54.409+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:07:55.089 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:07:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:07:54.826+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:54.827061+0000) 2024-02-20T09:07:55.090 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:07:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:07:54.943+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:54.944280+0000) 2024-02-20T09:07:55.342 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:07:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:07:55.229+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:07:55.342 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:07:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:07:55.229+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:07:55.342 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:07:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:07:55.229+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:07:55.342 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:07:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:07:55.087+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:07:55.342 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:07:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:07:55.087+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:07:55.342 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:07:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:07:55.087+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:07:55.343 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:07:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:07:55.088+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:07:55.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:07:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:07:55.340+0000 7f8a5f86e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:55.341100+0000) 2024-02-20T09:07:55.743 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:07:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:07:55.347+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 58587 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:07:55.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:07:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:07:55.344+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:55.344514+0000) 2024-02-20T09:07:55.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:07:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:07:55.447+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:55.447626+0000) 2024-02-20T09:07:55.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:07:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:07:55.340+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:55.341267+0000) 2024-02-20T09:07:55.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:07:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:07:55.442+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:07:55.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:07:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:07:55.442+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:07:55.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:07:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:07:55.442+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:07:56.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:07:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:07:55.826+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:55.827273+0000) 2024-02-20T09:07:56.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:07:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:07:55.944+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:55.944474+0000) 2024-02-20T09:07:56.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:07:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:07:56.051+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:07:56.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:07:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:07:56.051+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:07:56.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:07:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:07:56.051+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:07:56.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:07:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:07:56.052+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:07:56.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:07:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:07:56.449+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:56.450130+0000) 2024-02-20T09:07:56.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:07:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:07:56.461+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:07:56.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:07:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:07:56.461+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:07:56.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:07:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:07:56.461+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:07:56.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:07:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:07:56.259+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:07:56.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:07:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:07:56.259+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:07:56.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:07:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:07:56.259+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:07:57.082 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:07:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:07:56.944+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:56.944603+0000) 2024-02-20T09:07:57.082 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:07:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:07:56.827+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:56.827420+0000) 2024-02-20T09:07:57.420 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:07:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:07:57.080+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:07:57.420 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:07:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:07:57.080+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:07:57.420 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:07:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:07:57.080+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:07:57.420 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:07:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:07:57.080+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:07:57.420 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:07:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:07:57.293+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:07:57.421 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:07:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:07:57.294+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:07:57.421 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:07:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:07:57.294+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:07:57.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:07:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:07:57.453+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:57.454049+0000) 2024-02-20T09:07:57.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:07:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:07:57.418+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:07:57.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:07:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:07:57.418+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:07:57.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:07:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:07:57.418+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:07:58.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:07:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:07:57.827+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:57.827549+0000) 2024-02-20T09:07:58.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:07:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:07:57.944+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:57.944740+0000) 2024-02-20T09:07:58.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:07:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:07:58.032+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:07:58.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:07:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:07:58.032+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:07:58.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:07:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:07:58.032+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:07:58.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:07:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:07:58.032+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:07:58.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:07:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:07:58.456+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:58.456928+0000) 2024-02-20T09:07:58.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:07:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:07:58.378+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:07:58.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:07:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:07:58.378+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:07:58.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:07:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:07:58.378+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:07:58.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:07:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:07:58.323+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:07:58.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:07:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:07:58.323+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:07:58.744 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:07:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:07:58.323+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:07:59.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:07:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:07:58.944+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:58.944892+0000) 2024-02-20T09:07:59.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:07:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:07:58.827+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:58.827668+0000) 2024-02-20T09:07:59.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:07:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:07:59.029+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:07:59.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:07:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:07:59.030+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:07:59.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:07:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:07:59.030+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:07:59.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:07:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:07:59.030+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:07:59.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:07:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:07:59.457+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:59.457475+0000) 2024-02-20T09:07:59.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:07:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:07:59.352+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:07:59.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:07:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:07:59.352+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:07:59.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:07:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:07:59.352+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:07:59.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:07:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:07:59.294+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:07:59.744 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:07:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:07:59.294+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:07:59.744 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:07:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:07:59.294+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:08:00.080 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:07:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:07:59.827+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:59.827850+0000) 2024-02-20T09:08:00.080 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:07:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:07:59.944+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:07:59.945076+0000) 2024-02-20T09:08:00.331 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:08:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:08:00.078+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:08:00.331 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:08:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:08:00.078+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:08:00.331 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:08:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:08:00.078+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:08:00.331 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:08:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:08:00.078+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:08:00.743 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:08:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:08:00.356+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 58660 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:08:00.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:08:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:08:00.352+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:00.353282+0000) 2024-02-20T09:08:00.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:08:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:08:00.457+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:00.457734+0000) 2024-02-20T09:08:00.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:08:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:08:00.349+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:00.349758+0000) 2024-02-20T09:08:00.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:08:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:08:00.358+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:08:00.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:08:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:08:00.358+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:08:00.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:08:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:08:00.358+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:08:00.744 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:08:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:08:00.329+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:08:00.744 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:08:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:08:00.329+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:08:00.744 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:08:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:08:00.329+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:08:00.744 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:08:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:08:00.349+0000 7f8a6006f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:00.349937+0000) 2024-02-20T09:08:01.075 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:08:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:08:00.827+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:00.828047+0000) 2024-02-20T09:08:01.076 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:08:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:08:00.944+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:00.945198+0000) 2024-02-20T09:08:01.445 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:08:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:08:01.301+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:08:01.446 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:08:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:08:01.301+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:08:01.446 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:08:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:08:01.301+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:08:01.446 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:08:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:08:01.321+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:08:01.446 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:08:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:08:01.321+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:08:01.446 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:08:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:08:01.321+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:08:01.447 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:08:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:08:01.074+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:08:01.447 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:08:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:08:01.074+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:08:01.447 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:08:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:08:01.074+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:08:01.447 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:08:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:08:01.074+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:08:01.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:08:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:08:01.459+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:01.460168+0000) 2024-02-20T09:08:02.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:08:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:08:01.827+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:01.828212+0000) 2024-02-20T09:08:02.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:08:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:08:01.944+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:01.945329+0000) 2024-02-20T09:08:02.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:08:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:08:02.035+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:08:02.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:08:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:08:02.035+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:08:02.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:08:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:08:02.035+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:08:02.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:08:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:08:02.035+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:08:02.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:08:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:08:02.462+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:02.462938+0000) 2024-02-20T09:08:02.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:08:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:08:02.274+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:08:02.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:08:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:08:02.274+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:08:02.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:08:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:08:02.274+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:08:02.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:08:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:08:02.346+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:08:02.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:08:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:08:02.346+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:08:02.744 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:08:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:08:02.346+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:08:03.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:08:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:08:02.828+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:02.828433+0000) 2024-02-20T09:08:03.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:08:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:08:02.945+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:02.945472+0000) 2024-02-20T09:08:03.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:08:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:08:03.023+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:08:03.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:08:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:08:03.023+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:08:03.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:08:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:08:03.023+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:08:03.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:08:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:08:03.023+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:08:03.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:08:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:08:03.465+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:03.465684+0000) 2024-02-20T09:08:03.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:08:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:08:03.312+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:08:03.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:08:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:08:03.312+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:08:03.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:08:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:08:03.312+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:08:03.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:08:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:08:03.280+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:08:03.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:08:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:08:03.280+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:08:03.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:08:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:08:03.280+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:08:04.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:08:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:08:03.828+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:03.828584+0000) 2024-02-20T09:08:04.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:08:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:08:03.945+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:03.945630+0000) 2024-02-20T09:08:04.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:08:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:08:03.992+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:08:04.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:08:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:08:03.992+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:08:04.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:08:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:08:03.992+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:08:04.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:08:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:08:03.992+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:08:04.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:08:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:08:04.465+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:04.466296+0000) 2024-02-20T09:08:04.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:08:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:08:04.341+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:08:04.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:08:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:08:04.341+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:08:04.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:08:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:08:04.341+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:08:04.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:08:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:08:04.316+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:08:04.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:08:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:08:04.316+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:08:04.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:08:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:08:04.316+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:08:05.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:08:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:08:04.828+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:04.828768+0000) 2024-02-20T09:08:05.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:08:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:08:04.945+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:04.945830+0000) 2024-02-20T09:08:05.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:08:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:08:04.987+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:08:05.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:08:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:08:04.987+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:08:05.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:08:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:08:04.987+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:08:05.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:08:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:08:04.987+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:08:05.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:08:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:08:05.361+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:05.361973+0000) 2024-02-20T09:08:05.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:08:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:08:05.466+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:05.466596+0000) 2024-02-20T09:08:05.743 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:08:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:08:05.365+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 58732 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:08:05.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:08:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:08:05.322+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:08:05.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:08:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:08:05.322+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:08:05.744 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:08:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:08:05.322+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:08:05.744 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:08:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:08:05.357+0000 7f8a5f86e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:05.358220+0000) 2024-02-20T09:08:05.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:08:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:08:05.358+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:05.358748+0000) 2024-02-20T09:08:05.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:08:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:08:05.362+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:08:05.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:08:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:08:05.362+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:08:05.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:08:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:08:05.362+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:08:06.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:08:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:08:05.828+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:05.828956+0000) 2024-02-20T09:08:06.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:08:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:08:05.945+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:05.946014+0000) 2024-02-20T09:08:06.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:08:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:08:05.967+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:08:06.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:08:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:08:05.967+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:08:06.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:08:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:08:05.967+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:08:06.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:08:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:08:05.967+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:08:06.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:08:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:08:06.470+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:06.470212+0000) 2024-02-20T09:08:06.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:08:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:08:06.313+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:08:06.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:08:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:08:06.313+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:08:06.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:08:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:08:06.313+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:08:06.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:08:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:08:06.291+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:08:06.744 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:08:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:08:06.291+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:08:06.744 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:08:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:08:06.291+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:08:07.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:08:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:08:06.828+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:06.829109+0000) 2024-02-20T09:08:07.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:08:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:08:06.945+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:06.946176+0000) 2024-02-20T09:08:07.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:08:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:08:06.921+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:08:07.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:08:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:08:06.921+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:08:07.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:08:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:08:06.921+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:08:07.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:08:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:08:06.921+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:08:07.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:08:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:08:07.472+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:07.473159+0000) 2024-02-20T09:08:07.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:08:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:08:07.348+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:08:07.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:08:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:08:07.348+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:08:07.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:08:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:08:07.348+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:08:07.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:08:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:08:07.269+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:08:07.744 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:08:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:08:07.269+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:08:07.744 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:08:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:08:07.269+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:08:08.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:08:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:08:07.946+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:07.946358+0000) 2024-02-20T09:08:08.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:08:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:08:07.828+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:07.829313+0000) 2024-02-20T09:08:08.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:08:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:08:07.950+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:08:08.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:08:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:08:07.950+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:08:08.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:08:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:08:07.950+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:08:08.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:08:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:08:07.950+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:08:08.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:08:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:08:08.475+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:08.476035+0000) 2024-02-20T09:08:08.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:08:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:08:08.318+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:08:08.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:08:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:08:08.319+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:08:08.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:08:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:08:08.319+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:08:08.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:08:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:08:08.306+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:08:08.744 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:08:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:08:08.306+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:08:08.744 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:08:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:08:08.306+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:08:09.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:08:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:08:08.829+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:08.829450+0000) 2024-02-20T09:08:09.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:08:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:08:08.946+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:08.946517+0000) 2024-02-20T09:08:09.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:08:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:08:08.953+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:08:09.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:08:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:08:08.953+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:08:09.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:08:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:08:08.953+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:08:09.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:08:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:08:08.954+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:08:09.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:08:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:08:09.476+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:09.476619+0000) 2024-02-20T09:08:09.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:08:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:08:09.359+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:08:09.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:08:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:08:09.359+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:08:09.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:08:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:08:09.359+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:08:09.744 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:08:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:08:09.268+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:08:09.744 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:08:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:08:09.268+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:08:09.744 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:08:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:08:09.268+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:08:10.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:08:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:08:09.829+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:09.829587+0000) 2024-02-20T09:08:10.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:08:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:08:09.946+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:09.946701+0000) 2024-02-20T09:08:10.245 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:08:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:08:09.917+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:08:10.245 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:08:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:08:09.917+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:08:10.245 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:08:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:08:09.917+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:08:10.245 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:08:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:08:09.917+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:08:10.742 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:08:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:08:10.374+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 58805 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:08:10.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:08:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:08:10.370+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:10.370967+0000) 2024-02-20T09:08:10.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:08:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:08:10.476+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:10.476944+0000) 2024-02-20T09:08:10.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:08:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:08:10.367+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:10.367559+0000) 2024-02-20T09:08:10.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:08:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:08:10.399+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:08:10.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:08:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:08:10.399+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:08:10.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:08:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:08:10.399+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:08:10.744 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:08:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:08:10.261+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:08:10.744 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:08:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:08:10.261+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:08:10.744 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:08:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:08:10.261+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:08:10.744 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:08:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:08:10.366+0000 7f8a5f86e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:10.367387+0000) 2024-02-20T09:08:11.231 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:08:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:08:10.946+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:10.946842+0000) 2024-02-20T09:08:11.231 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:08:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:08:10.829+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:10.829737+0000) 2024-02-20T09:08:11.231 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:08:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:08:11.229+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:08:11.231 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:08:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:08:11.229+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:08:11.231 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:08:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:08:10.947+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:08:11.232 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:08:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:08:10.947+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:08:11.232 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:08:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:08:10.947+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:08:11.232 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:08:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:08:10.947+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:08:11.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:08:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:08:11.229+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:08:11.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:08:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:08:11.479+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:11.479395+0000) 2024-02-20T09:08:11.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:08:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:08:11.427+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:08:11.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:08:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:08:11.427+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:08:11.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:08:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:08:11.427+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:08:12.190 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:08:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:08:11.946+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:11.947017+0000) 2024-02-20T09:08:12.190 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:08:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:08:11.829+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:11.829912+0000) 2024-02-20T09:08:12.191 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:08:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:08:12.188+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:08:12.191 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:08:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:08:11.918+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:08:12.191 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:08:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:08:11.918+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:08:12.191 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:08:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:08:11.918+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:08:12.191 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:08:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:08:11.918+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:08:12.454 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:08:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:08:12.188+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:08:12.454 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:08:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:08:12.188+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:08:12.455 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:08:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:08:12.411+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:08:12.455 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:08:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:08:12.411+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:08:12.455 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:08:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:08:12.411+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:08:12.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:08:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:08:12.481+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:12.482359+0000) 2024-02-20T09:08:13.177 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:08:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:08:12.829+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:12.830159+0000) 2024-02-20T09:08:13.177 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:08:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:08:12.946+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:12.947225+0000) 2024-02-20T09:08:13.178 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:08:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:08:12.883+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:08:13.178 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:08:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:08:12.883+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:08:13.178 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:08:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:08:12.883+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:08:13.178 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:08:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:08:12.883+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:08:13.454 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:08:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:08:13.184+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:08:13.455 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:08:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:08:13.184+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:08:13.455 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:08:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:08:13.184+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:08:13.455 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:08:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:08:13.413+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:08:13.455 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:08:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:08:13.413+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:08:13.455 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:08:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:08:13.413+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:08:13.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:08:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:08:13.484+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:13.485310+0000) 2024-02-20T09:08:14.235 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:08:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:08:13.946+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:13.947429+0000) 2024-02-20T09:08:14.236 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:08:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:08:13.829+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:13.830348+0000) 2024-02-20T09:08:14.236 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:08:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:08:14.232+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:08:14.236 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:08:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:08:14.232+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:08:14.236 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:08:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:08:14.232+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:08:14.236 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:08:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:08:13.892+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:08:14.237 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:08:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:08:13.892+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:08:14.237 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:08:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:08:13.892+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:08:14.237 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:08:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:08:13.892+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:08:14.487 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:08:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:08:14.410+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:08:14.487 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:08:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:08:14.410+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:08:14.487 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:08:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:08:14.410+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:08:14.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:08:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:08:14.484+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:14.485992+0000) 2024-02-20T09:08:15.206 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:08:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:08:14.829+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:14.830586+0000) 2024-02-20T09:08:15.206 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:08:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:08:14.946+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:14.947590+0000) 2024-02-20T09:08:15.207 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:08:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:08:14.910+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:08:15.207 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:08:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:08:14.910+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:08:15.207 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:08:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:08:14.910+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:08:15.207 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:08:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:08:14.910+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:08:15.474 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:08:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:08:15.382+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 58877 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:08:15.475 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:08:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:08:15.378+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:15.379477+0000) 2024-02-20T09:08:15.475 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:08:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:08:15.203+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:08:15.475 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:08:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:08:15.203+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:08:15.475 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:08:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:08:15.203+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:08:15.475 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:08:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:08:15.375+0000 7f8a5f86e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:15.376392+0000) 2024-02-20T09:08:15.476 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:08:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:08:15.375+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:15.376716+0000) 2024-02-20T09:08:15.476 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:08:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:08:15.380+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:08:15.476 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:08:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:08:15.380+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:08:15.476 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:08:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:08:15.380+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:08:15.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:08:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:08:15.484+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:15.486268+0000) 2024-02-20T09:08:16.218 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:08:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:08:15.946+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:15.947732+0000) 2024-02-20T09:08:16.218 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:08:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:08:15.913+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:08:16.218 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:08:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:08:15.913+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:08:16.218 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:08:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:08:15.913+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:08:16.219 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:08:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:08:15.913+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:08:16.219 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:08:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:08:15.829+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:15.830764+0000) 2024-02-20T09:08:16.219 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:08:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:08:16.215+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:08:16.219 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:08:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:08:16.215+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:08:16.474 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:08:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:08:16.215+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:08:16.475 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:08:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:08:16.398+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:08:16.475 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:08:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:08:16.398+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:08:16.475 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:08:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:08:16.398+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:08:16.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:08:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:08:16.487+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:16.488581+0000) 2024-02-20T09:08:17.193 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:08:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:08:16.946+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:16.947880+0000) 2024-02-20T09:08:17.194 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:08:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:08:16.829+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:16.830938+0000) 2024-02-20T09:08:17.194 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:08:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:08:17.190+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:08:17.194 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:08:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:08:17.190+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:08:17.194 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:08:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:08:17.190+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:08:17.194 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:08:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:08:16.938+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:08:17.194 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:08:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:08:16.938+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:08:17.195 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:08:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:08:16.938+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:08:17.195 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:08:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:08:16.938+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:08:17.458 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:08:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:08:17.434+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:08:17.458 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:08:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:08:17.434+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:08:17.459 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:08:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:08:17.434+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:08:17.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:08:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:08:17.489+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:17.491287+0000) 2024-02-20T09:08:18.230 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:08:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:08:17.947+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:17.948065+0000) 2024-02-20T09:08:18.230 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:08:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:08:17.830+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:17.831074+0000) 2024-02-20T09:08:18.230 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:08:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:08:18.227+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:08:18.231 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:08:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:08:17.969+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:08:18.231 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:08:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:08:17.969+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:08:18.231 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:08:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:08:17.969+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:08:18.231 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:08:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:08:17.969+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:08:18.492 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:08:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:08:18.227+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:08:18.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:08:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:08:18.227+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:08:18.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:08:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:08:18.414+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:08:18.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:08:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:08:18.414+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:08:18.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:08:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:08:18.414+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:08:18.821 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:08:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:08:18.492+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:18.494363+0000) 2024-02-20T09:08:19.199 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:08:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:08:18.946+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:18.948267+0000) 2024-02-20T09:08:19.199 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:08:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:08:18.829+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:18.831185+0000) 2024-02-20T09:08:19.199 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:08:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:08:19.196+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:08:19.200 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:08:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:08:18.944+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:08:19.200 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:08:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:08:18.944+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:08:19.200 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:08:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:08:18.944+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:08:19.200 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:08:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:08:18.944+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:08:19.459 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:08:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:08:19.196+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:08:19.459 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:08:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:08:19.196+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:08:19.459 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:08:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:08:19.422+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:08:19.459 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:08:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:08:19.422+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:08:19.459 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:08:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:08:19.422+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:08:19.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:08:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:08:19.493+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:19.495075+0000) 2024-02-20T09:08:20.184 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:08:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:08:19.947+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:19.948436+0000) 2024-02-20T09:08:20.184 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:08:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:08:19.830+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:19.831327+0000) 2024-02-20T09:08:20.185 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:08:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:08:20.181+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:08:20.185 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:08:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:08:19.964+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:08:20.185 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:08:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:08:19.964+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:08:20.185 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:08:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:08:19.964+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:08:20.185 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:08:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:08:19.964+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:08:20.459 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:08:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:08:20.391+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 58950 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:08:20.459 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:08:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:08:20.387+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:20.388499+0000) 2024-02-20T09:08:20.460 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:08:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:08:20.181+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:08:20.460 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:08:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:08:20.181+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:08:20.460 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:08:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:08:20.384+0000 7f8a5f86e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:20.385776+0000) 2024-02-20T09:08:20.460 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:08:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:08:20.384+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:20.386275+0000) 2024-02-20T09:08:20.460 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:08:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:08:20.402+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:08:20.460 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:08:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:08:20.402+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:08:20.461 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:08:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:08:20.402+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:08:20.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:08:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:08:20.493+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:20.495351+0000) 2024-02-20T09:08:21.190 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:08:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:08:20.947+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:20.948623+0000) 2024-02-20T09:08:21.190 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:08:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:08:20.830+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:20.831537+0000) 2024-02-20T09:08:21.190 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:08:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:08:21.187+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:08:21.191 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:08:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:08:21.187+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:08:21.191 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:08:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:08:21.187+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:08:21.191 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:08:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:08:20.916+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:08:21.191 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:08:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:08:20.916+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:08:21.191 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:08:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:08:20.916+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:08:21.191 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:08:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:08:20.916+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:08:21.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:08:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:08:21.496+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:21.497840+0000) 2024-02-20T09:08:21.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:08:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:08:21.449+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:08:21.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:08:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:08:21.449+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:08:21.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:08:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:08:21.449+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:08:22.199 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:08:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:08:21.947+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:21.948799+0000) 2024-02-20T09:08:22.200 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:08:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:08:21.830+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:21.831689+0000) 2024-02-20T09:08:22.200 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:08:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:08:22.197+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:08:22.200 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:08:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:08:22.197+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:08:22.200 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:08:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:08:21.935+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:08:22.200 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:08:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:08:21.935+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:08:22.201 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:08:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:08:21.935+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:08:22.201 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:08:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:08:21.935+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:08:22.463 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:08:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:08:22.197+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:08:22.464 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:08:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:08:22.428+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:08:22.464 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:08:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:08:22.428+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:08:22.464 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:08:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:08:22.428+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:08:22.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:08:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:08:22.500+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:22.501496+0000) 2024-02-20T09:08:23.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:08:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:08:22.831+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:22.831863+0000) 2024-02-20T09:08:23.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:08:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:08:22.948+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:22.948975+0000) 2024-02-20T09:08:23.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:08:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:08:22.918+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:08:23.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:08:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:08:22.918+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:08:23.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:08:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:08:22.918+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:08:23.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:08:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:08:22.918+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:08:23.505 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:08:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:08:23.438+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:08:23.506 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:08:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:08:23.438+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:08:23.506 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:08:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:08:23.438+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:08:23.506 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:08:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:08:23.246+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:08:23.506 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:08:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:08:23.246+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:08:23.506 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:08:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:08:23.246+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:08:23.822 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:08:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:08:23.502+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:23.504260+0000) 2024-02-20T09:08:24.202 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:08:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:08:23.948+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:23.949158+0000) 2024-02-20T09:08:24.203 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:08:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:08:23.830+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:23.832090+0000) 2024-02-20T09:08:24.203 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:08:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:08:24.200+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:08:24.203 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:08:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:08:23.910+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:08:24.203 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:08:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:08:23.910+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:08:24.204 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:08:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:08:23.910+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:08:24.204 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:08:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:08:23.910+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:08:24.464 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:08:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:08:24.428+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:08:24.464 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:08:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:08:24.428+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:08:24.464 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:08:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:08:24.428+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:08:24.465 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:08:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:08:24.200+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:08:24.465 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:08:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:08:24.200+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:08:24.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:08:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:08:24.504+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:24.504996+0000) 2024-02-20T09:08:25.160 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:08:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:08:24.948+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:24.949364+0000) 2024-02-20T09:08:25.161 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:08:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:08:24.934+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:08:25.161 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:08:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:08:24.935+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:08:25.161 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:08:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:08:24.935+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:08:25.161 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:08:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:08:24.935+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:08:25.161 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:08:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:08:24.830+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:24.832269+0000) 2024-02-20T09:08:25.162 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:08:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:08:25.158+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:08:25.162 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:08:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:08:25.158+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:08:25.162 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:08:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:08:25.158+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:08:25.410 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:08:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:08:25.400+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 59022 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:08:25.411 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:08:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:08:25.393+0000 7f8a5f86e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:25.394869+0000) 2024-02-20T09:08:25.411 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:08:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:08:25.396+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:25.397755+0000) 2024-02-20T09:08:25.412 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:08:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:08:25.393+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:25.394637+0000) 2024-02-20T09:08:25.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:08:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:08:25.503+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:25.505340+0000) 2024-02-20T09:08:25.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:08:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:08:25.422+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:08:25.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:08:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:08:25.422+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:08:25.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:08:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:08:25.422+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:08:26.127 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:08:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:08:25.948+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:25.949609+0000) 2024-02-20T09:08:26.127 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:08:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:08:25.831+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:25.832476+0000) 2024-02-20T09:08:26.127 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:08:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:08:26.124+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:08:26.127 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:08:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:08:25.886+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:08:26.128 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:08:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:08:25.886+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:08:26.128 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:08:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:08:25.886+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:08:26.128 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:08:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:08:25.886+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:08:26.467 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:08:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:08:26.124+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:08:26.467 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:08:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:08:26.124+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:08:26.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:08:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:08:26.506+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:26.507660+0000) 2024-02-20T09:08:26.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:08:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:08:26.470+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:08:26.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:08:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:08:26.470+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:08:26.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:08:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:08:26.470+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:08:27.138 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:08:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:08:26.948+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:26.949708+0000) 2024-02-20T09:08:27.139 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:08:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:08:26.831+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:26.832651+0000) 2024-02-20T09:08:27.139 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:08:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:08:27.135+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:08:27.139 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:08:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:08:26.907+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:08:27.139 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:08:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:08:26.907+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:08:27.139 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:08:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:08:26.907+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:08:27.140 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:08:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:08:26.907+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:08:27.447 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:08:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:08:27.136+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:08:27.447 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:08:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:08:27.136+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:08:27.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:08:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:08:27.509+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:27.511279+0000) 2024-02-20T09:08:27.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:08:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:08:27.444+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:08:27.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:08:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:08:27.444+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:08:27.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:08:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:08:27.444+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:08:28.164 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:08:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:08:27.832+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:27.832874+0000) 2024-02-20T09:08:28.164 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:08:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:08:28.161+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:08:28.164 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:08:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:08:27.949+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:27.949910+0000) 2024-02-20T09:08:28.165 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:08:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:08:27.941+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:08:28.165 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:08:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:08:27.941+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:08:28.165 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:08:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:08:27.941+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:08:28.165 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:08:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:08:27.941+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:08:28.467 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:08:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:08:28.161+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:08:28.468 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:08:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:08:28.161+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:08:28.468 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:08:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:08:28.399+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:08:28.468 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:08:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:08:28.399+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:08:28.468 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:08:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:08:28.399+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:08:28.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:08:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:08:28.512+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:28.514168+0000) 2024-02-20T09:08:29.150 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:08:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:08:28.831+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:28.833079+0000) 2024-02-20T09:08:29.150 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:08:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:08:29.148+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:08:29.151 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:08:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:08:28.949+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:28.950104+0000) 2024-02-20T09:08:29.151 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:08:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:08:28.905+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:08:29.151 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:08:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:08:28.905+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:08:29.151 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:08:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:08:28.905+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:08:29.151 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:08:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:08:28.905+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:08:29.468 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:08:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:08:29.392+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:08:29.468 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:08:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:08:29.392+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:08:29.468 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:08:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:08:29.392+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:08:29.468 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:08:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:08:29.148+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:08:29.468 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:08:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:08:29.148+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:08:29.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:08:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:08:29.514+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:29.514811+0000) 2024-02-20T09:08:30.121 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:08:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:08:29.949+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:29.950239+0000) 2024-02-20T09:08:30.121 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:08:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:08:29.938+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:08:30.122 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:08:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:08:29.938+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:08:30.122 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:08:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:08:29.938+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:08:30.122 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:08:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:08:29.938+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:08:30.122 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:08:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:08:29.832+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:29.833255+0000) 2024-02-20T09:08:30.122 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:08:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:08:30.118+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:08:30.122 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:08:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:08:30.118+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:08:30.122 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:08:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:08:30.118+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:08:30.399 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:08:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:08:30.355+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:08:30.399 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:08:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:08:30.355+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:08:30.399 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:08:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:08:30.355+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:08:30.742 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:08:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:08:30.409+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 59095 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:08:30.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:08:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:08:30.402+0000 7f8a5f86e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:30.403672+0000) 2024-02-20T09:08:30.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:08:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:08:30.402+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:30.404015+0000) 2024-02-20T09:08:30.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:08:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:08:30.405+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:30.407070+0000) 2024-02-20T09:08:30.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:08:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:08:30.514+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:30.515170+0000) 2024-02-20T09:08:31.106 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:08:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:08:30.949+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:30.950423+0000) 2024-02-20T09:08:31.106 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:08:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:08:30.832+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:30.833417+0000) 2024-02-20T09:08:31.106 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:08:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:08:31.103+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:08:31.107 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:08:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:08:30.957+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:08:31.107 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:08:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:08:30.957+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:08:31.107 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:08:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:08:30.957+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:08:31.107 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:08:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:08:30.957+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:08:31.368 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:08:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:08:31.103+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:08:31.368 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:08:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:08:31.103+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:08:31.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:08:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:08:31.518+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:31.518791+0000) 2024-02-20T09:08:31.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:08:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:08:31.366+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:08:31.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:08:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:08:31.366+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:08:31.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:08:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:08:31.366+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:08:32.096 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:08:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:08:31.949+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:31.950542+0000) 2024-02-20T09:08:32.097 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:08:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:08:31.832+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:31.833585+0000) 2024-02-20T09:08:32.097 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:08:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:08:32.094+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:08:32.097 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:08:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:08:32.094+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:08:32.097 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:08:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:08:32.094+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:08:32.097 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:08:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:08:31.943+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:08:32.098 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:08:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:08:31.943+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:08:32.098 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:08:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:08:31.943+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:08:32.098 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:08:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:08:31.943+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:08:32.471 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:08:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:08:32.342+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:08:32.471 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:08:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:08:32.342+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:08:32.472 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:08:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:08:32.342+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:08:32.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:08:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:08:32.521+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:32.522489+0000) 2024-02-20T09:08:33.076 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:08:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:08:32.832+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:32.833707+0000) 2024-02-20T09:08:33.077 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:08:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:08:33.074+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:08:33.077 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:08:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:08:32.950+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:32.950702+0000) 2024-02-20T09:08:33.077 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:08:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:08:32.969+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:08:33.077 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:08:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:08:32.969+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:08:33.077 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:08:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:08:32.969+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:08:33.078 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:08:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:08:32.969+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:08:33.335 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:08:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:08:33.074+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:08:33.336 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:08:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:08:33.074+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:08:33.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:08:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:08:33.525+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:33.526076+0000) 2024-02-20T09:08:33.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:08:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:08:33.333+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:08:33.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:08:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:08:33.333+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:08:33.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:08:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:08:33.333+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:08:34.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:08:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:08:33.950+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:33.950879+0000) 2024-02-20T09:08:34.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:08:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:08:33.832+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:33.833855+0000) 2024-02-20T09:08:34.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:08:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:08:34.043+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:08:34.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:08:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:08:34.043+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:08:34.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:08:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:08:34.043+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:08:34.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:08:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:08:33.994+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:08:34.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:08:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:08:33.994+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:08:34.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:08:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:08:33.994+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:08:34.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:08:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:08:33.994+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:08:34.681 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:08:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:08:34.525+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:34.526775+0000) 2024-02-20T09:08:34.681 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:08:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:08:34.326+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:08:34.681 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:08:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:08:34.326+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:08:34.681 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:08:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:08:34.326+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:08:34.952 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:08:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:08:34.833+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:34.834035+0000) 2024-02-20T09:08:35.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:08:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:08:34.949+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:34.951036+0000) 2024-02-20T09:08:35.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:08:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:08:35.040+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:08:35.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:08:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:08:35.040+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:08:35.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:08:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:08:35.040+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:08:35.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:08:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:08:35.040+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:08:35.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:08:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:08:35.065+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:08:35.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:08:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:08:35.065+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:08:35.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:08:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:08:35.065+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:08:35.742 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:08:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:08:35.417+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 59167 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:08:35.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:08:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:08:35.410+0000 7f8a5f86e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:35.412019+0000) 2024-02-20T09:08:35.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:08:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:08:35.414+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:35.416272+0000) 2024-02-20T09:08:35.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:08:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:08:35.526+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:35.527084+0000) 2024-02-20T09:08:35.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:08:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:08:35.355+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:08:35.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:08:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:08:35.355+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:08:35.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:08:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:08:35.355+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:08:35.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:08:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:08:35.410+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:35.411966+0000) 2024-02-20T09:08:36.104 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:08:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:08:35.950+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:35.951186+0000) 2024-02-20T09:08:36.117 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:08:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:08:36.058+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:08:36.117 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:08:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:08:36.058+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:08:36.117 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:08:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:08:36.058+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:08:36.117 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:08:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:08:36.058+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:08:36.118 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:08:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:08:35.832+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:35.834246+0000) 2024-02-20T09:08:36.118 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:08:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:08:36.101+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:08:36.118 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:08:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:08:36.101+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:08:36.118 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:08:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:08:36.101+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:08:36.474 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:08:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:08:36.342+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:08:36.475 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:08:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:08:36.342+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:08:36.475 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:08:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:08:36.342+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:08:36.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:08:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:08:36.528+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:36.529454+0000) 2024-02-20T09:08:37.133 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:08:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:08:36.950+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:36.951330+0000) 2024-02-20T09:08:37.134 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:08:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:08:36.833+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:36.834416+0000) 2024-02-20T09:08:37.134 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:08:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:08:37.130+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:08:37.134 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:08:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:08:37.130+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:08:37.134 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:08:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:08:37.048+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:08:37.134 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:08:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:08:37.048+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:08:37.134 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:08:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:08:37.048+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:08:37.134 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:08:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:08:37.048+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:08:37.475 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:08:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:08:37.130+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:08:37.475 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:08:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:08:37.305+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:08:37.476 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:08:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:08:37.305+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:08:37.476 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:08:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:08:37.305+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:08:37.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:08:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:08:37.531+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:37.532446+0000) 2024-02-20T09:08:38.084 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:08:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:08:37.950+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:37.951539+0000) 2024-02-20T09:08:38.084 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:08:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:08:37.834+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:37.834598+0000) 2024-02-20T09:08:38.085 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:08:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:08:38.082+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:08:38.085 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:08:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:08:38.025+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:08:38.085 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:08:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:08:38.025+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:08:38.085 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:08:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:08:38.025+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:08:38.085 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:08:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:08:38.025+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:08:38.335 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:08:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:08:38.082+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:08:38.335 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:08:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:08:38.082+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:08:38.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:08:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:08:38.534+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:38.535242+0000) 2024-02-20T09:08:38.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:08:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:08:38.333+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:08:38.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:08:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:08:38.333+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:08:38.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:08:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:08:38.333+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:08:39.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:08:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:08:38.950+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:38.951669+0000) 2024-02-20T09:08:39.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:08:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:08:39.010+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:08:39.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:08:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:08:39.010+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:08:39.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:08:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:08:39.010+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:08:39.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:08:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:08:39.010+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:08:39.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:08:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:08:38.834+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:38.834781+0000) 2024-02-20T09:08:39.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:08:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:08:39.055+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:08:39.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:08:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:08:39.055+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:08:39.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:08:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:08:39.055+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:08:39.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:08:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:08:39.534+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:39.535982+0000) 2024-02-20T09:08:39.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:08:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:08:39.306+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:08:39.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:08:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:08:39.306+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:08:39.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:08:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:08:39.306+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:08:40.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:08:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:08:39.951+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:39.951833+0000) 2024-02-20T09:08:40.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:08:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:08:40.008+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:08:40.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:08:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:08:40.008+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:08:40.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:08:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:08:40.008+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:08:40.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:08:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:08:40.008+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:08:40.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:08:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:08:39.834+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:39.834985+0000) 2024-02-20T09:08:40.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:08:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:08:40.048+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:08:40.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:08:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:08:40.048+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:08:40.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:08:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:08:40.048+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:08:40.537 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:08:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:08:40.427+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 59239 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:08:40.538 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:08:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:08:40.420+0000 7f8a5f86e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:40.421444+0000) 2024-02-20T09:08:40.538 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:08:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:08:40.423+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:40.424676+0000) 2024-02-20T09:08:40.538 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:08:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:08:40.276+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:08:40.538 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:08:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:08:40.276+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:08:40.538 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:08:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:08:40.276+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:08:40.539 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:08:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:08:40.419+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:40.421003+0000) 2024-02-20T09:08:40.824 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:08:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:08:40.534+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:40.536368+0000) 2024-02-20T09:08:41.079 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:08:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:08:40.951+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:40.952047+0000) 2024-02-20T09:08:41.079 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:08:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:08:40.833+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:40.835156+0000) 2024-02-20T09:08:41.079 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:08:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:08:41.077+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:08:41.079 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:08:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:08:40.973+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:08:41.080 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:08:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:08:40.973+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:08:41.080 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:08:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:08:40.973+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:08:41.080 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:08:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:08:40.973+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:08:41.479 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:08:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:08:41.077+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:08:41.479 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:08:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:08:41.077+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:08:41.479 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:08:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:08:41.326+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:08:41.480 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:08:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:08:41.326+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:08:41.480 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:08:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:08:41.326+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:08:41.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:08:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:08:41.538+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:41.539006+0000) 2024-02-20T09:08:42.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:08:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:08:41.950+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:41.952219+0000) 2024-02-20T09:08:42.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:08:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:08:41.833+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:41.835274+0000) 2024-02-20T09:08:42.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:08:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:08:42.055+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:08:42.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:08:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:08:42.055+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:08:42.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:08:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:08:42.055+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:08:42.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:08:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:08:41.977+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:08:42.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:08:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:08:41.977+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:08:42.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:08:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:08:41.977+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:08:42.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:08:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:08:41.977+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:08:42.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:08:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:08:42.541+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:42.541987+0000) 2024-02-20T09:08:42.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:08:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:08:42.313+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:08:42.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:08:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:08:42.313+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:08:42.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:08:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:08:42.314+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:08:43.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:08:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:08:42.951+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:42.952421+0000) 2024-02-20T09:08:43.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:08:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:08:42.834+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:42.835473+0000) 2024-02-20T09:08:43.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:08:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:08:43.027+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:08:43.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:08:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:08:43.027+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:08:43.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:08:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:08:43.027+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:08:43.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:08:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:08:42.942+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:08:43.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:08:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:08:42.942+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:08:43.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:08:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:08:42.942+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:08:43.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:08:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:08:42.942+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:08:43.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:08:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:08:43.544+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:43.544837+0000) 2024-02-20T09:08:43.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:08:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:08:43.351+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:08:43.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:08:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:08:43.351+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:08:43.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:08:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:08:43.351+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:08:44.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:08:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:08:43.952+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:43.952603+0000) 2024-02-20T09:08:44.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:08:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:08:43.834+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:43.835647+0000) 2024-02-20T09:08:44.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:08:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:08:44.058+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:08:44.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:08:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:08:44.058+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:08:44.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:08:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:08:44.058+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:08:44.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:08:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:08:43.903+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:08:44.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:08:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:08:43.903+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:08:44.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:08:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:08:43.903+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:08:44.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:08:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:08:43.903+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:08:44.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:08:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:08:44.545+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:44.545626+0000) 2024-02-20T09:08:44.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:08:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:08:44.365+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:08:44.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:08:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:08:44.365+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:08:44.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:08:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:08:44.365+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:08:45.102 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:08:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:08:44.952+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:44.952789+0000) 2024-02-20T09:08:45.102 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:08:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:08:44.908+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:08:45.102 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:08:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:08:44.908+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:08:45.103 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:08:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:08:44.908+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:08:45.103 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:08:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:08:44.908+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:08:45.103 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:08:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:08:44.835+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:44.835783+0000) 2024-02-20T09:08:45.103 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:08:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:08:45.100+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:08:45.103 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:08:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:08:45.100+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:08:45.431 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:08:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:08:45.100+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:08:45.431 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:08:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:08:45.325+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:08:45.431 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:08:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:08:45.327+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:08:45.432 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:08:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:08:45.327+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:08:45.432 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:08:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:08:45.429+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:45.429827+0000) 2024-02-20T09:08:45.742 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:08:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:08:45.437+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 59312 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:08:45.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:08:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:08:45.430+0000 7f8a5f86e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:45.430858+0000) 2024-02-20T09:08:45.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:08:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:08:45.434+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:45.434724+0000) 2024-02-20T09:08:45.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:08:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:08:45.545+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:45.545874+0000) 2024-02-20T09:08:46.078 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:08:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:08:45.952+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:45.952918+0000) 2024-02-20T09:08:46.078 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:08:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:08:45.835+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:45.835937+0000) 2024-02-20T09:08:46.079 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:08:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:08:46.076+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:08:46.079 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:08:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:08:45.894+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:08:46.079 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:08:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:08:45.894+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:08:46.079 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:08:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:08:45.894+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:08:46.079 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:08:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:08:45.894+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:08:46.479 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:08:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:08:46.076+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:08:46.479 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:08:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:08:46.076+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:08:46.480 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:08:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:08:46.322+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:08:46.480 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:08:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:08:46.322+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:08:46.480 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:08:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:08:46.322+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:08:46.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:08:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:08:46.548+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:46.548416+0000) 2024-02-20T09:08:47.083 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:08:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:08:46.952+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:46.953019+0000) 2024-02-20T09:08:47.084 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:08:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:08:46.835+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:46.836082+0000) 2024-02-20T09:08:47.084 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:08:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:08:47.082+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:08:47.084 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:08:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:08:46.906+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:08:47.084 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:08:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:08:46.906+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:08:47.084 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:08:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:08:46.906+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:08:47.085 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:08:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:08:46.906+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:08:47.479 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:08:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:08:47.082+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:08:47.480 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:08:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:08:47.082+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:08:47.480 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:08:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:08:47.314+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:08:47.480 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:08:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:08:47.314+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:08:47.480 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:08:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:08:47.314+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:08:47.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:08:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:08:47.550+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:47.551276+0000) 2024-02-20T09:08:48.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:08:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:08:47.952+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:47.953204+0000) 2024-02-20T09:08:48.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:08:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:08:47.835+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:47.836270+0000) 2024-02-20T09:08:48.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:08:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:08:48.032+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:08:48.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:08:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:08:48.032+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:08:48.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:08:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:08:48.032+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:08:48.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:08:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:08:47.930+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:08:48.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:08:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:08:47.930+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:08:48.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:08:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:08:47.930+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:08:48.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:08:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:08:47.930+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:08:48.555 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:08:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:08:48.297+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:08:48.555 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:08:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:08:48.297+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:08:48.556 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:08:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:08:48.297+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:08:48.826 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:08:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:08:48.553+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:48.554142+0000) 2024-02-20T09:08:49.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:08:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:08:48.952+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:48.953332+0000) 2024-02-20T09:08:49.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:08:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:08:48.836+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:48.836484+0000) 2024-02-20T09:08:49.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:08:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:08:49.014+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:08:49.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:08:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:08:49.014+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:08:49.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:08:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:08:49.014+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:08:49.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:08:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:08:48.926+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:08:49.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:08:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:08:48.926+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:08:49.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:08:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:08:48.926+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:08:49.245 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:08:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:08:48.926+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:08:49.556 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:08:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:08:49.264+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:08:49.556 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:08:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:08:49.264+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:08:49.556 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:08:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:08:49.264+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:08:49.827 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:08:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:08:49.554+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:49.555029+0000) 2024-02-20T09:08:50.233 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:08:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:08:49.953+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:49.953501+0000) 2024-02-20T09:08:50.233 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:08:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:08:49.908+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:08:50.234 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:08:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:08:49.908+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:08:50.234 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:08:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:08:49.908+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:08:50.234 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:08:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:08:49.908+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:08:50.234 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:08:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:08:49.836+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:49.836619+0000) 2024-02-20T09:08:50.234 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:08:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:08:50.027+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:08:50.234 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:08:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:08:50.027+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:08:50.234 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:08:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:08:50.027+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:08:50.485 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:08:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:08:50.446+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 59384 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:08:50.485 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:08:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:08:50.439+0000 7f8a6006f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:50.439472+0000) 2024-02-20T09:08:50.486 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:08:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:08:50.442+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:50.442651+0000) 2024-02-20T09:08:50.486 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:08:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:08:50.231+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:08:50.486 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:08:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:08:50.231+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:08:50.486 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:08:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:08:50.231+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:08:50.487 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:08:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:08:50.439+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:50.439566+0000) 2024-02-20T09:08:50.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:08:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:08:50.554+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:50.555287+0000) 2024-02-20T09:08:51.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:08:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:08:50.953+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:50.953662+0000) 2024-02-20T09:08:51.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:08:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:08:50.836+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:50.836770+0000) 2024-02-20T09:08:51.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:08:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:08:51.008+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:08:51.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:08:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:08:51.008+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:08:51.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:08:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:08:51.008+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:08:51.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:08:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:08:50.912+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:08:51.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:08:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:08:50.912+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:08:51.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:08:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:08:50.912+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:08:51.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:08:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:08:50.912+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:08:51.558 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:08:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:08:51.249+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:08:51.559 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:08:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:08:51.249+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:08:51.559 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:08:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:08:51.249+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:08:51.827 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:08:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:08:51.557+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:51.557513+0000) 2024-02-20T09:08:52.233 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:08:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:08:51.953+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:51.953828+0000) 2024-02-20T09:08:52.233 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:08:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:08:52.231+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:08:52.233 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:08:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:08:52.231+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:08:52.234 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:08:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:08:51.836+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:51.836936+0000) 2024-02-20T09:08:52.234 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:08:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:08:52.039+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:08:52.234 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:08:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:08:52.039+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:08:52.234 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:08:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:08:52.039+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:08:52.234 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:08:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:08:51.869+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:08:52.235 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:08:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:08:51.869+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:08:52.235 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:08:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:08:51.869+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:08:52.235 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:08:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:08:51.870+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:08:52.489 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:08:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:08:52.231+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:08:52.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:08:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:08:52.560+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:52.560439+0000) 2024-02-20T09:08:53.113 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:08:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:08:52.953+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:52.954002+0000) 2024-02-20T09:08:53.113 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:08:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:08:52.858+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:08:53.113 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:08:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:08:52.858+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:08:53.113 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:08:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:08:52.858+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:08:53.113 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:08:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:08:52.858+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:08:53.114 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:08:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:08:52.836+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:52.837148+0000) 2024-02-20T09:08:53.114 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:08:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:08:52.995+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:08:53.114 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:08:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:08:52.995+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:08:53.114 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:08:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:08:52.995+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:08:53.480 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:08:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:08:53.197+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:08:53.480 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:08:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:08:53.197+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:08:53.481 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:08:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:08:53.197+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:08:53.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:08:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:08:53.563+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:53.563327+0000) 2024-02-20T09:08:54.204 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:08:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:08:53.953+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:53.954155+0000) 2024-02-20T09:08:54.205 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:08:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:08:54.202+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:08:54.205 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:08:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:08:53.836+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:53.837260+0000) 2024-02-20T09:08:54.205 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:08:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:08:54.042+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:08:54.205 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:08:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:08:54.042+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:08:54.205 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:08:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:08:54.042+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:08:54.206 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:08:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:08:53.905+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:08:54.206 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:08:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:08:53.905+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:08:54.206 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:08:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:08:53.905+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:08:54.206 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:08:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:08:53.905+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:08:54.481 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:08:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:08:54.203+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:08:54.481 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:08:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:08:54.203+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:08:54.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:08:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:08:54.563+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:54.564029+0000) 2024-02-20T09:08:55.195 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:08:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:08:54.883+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:08:55.195 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:08:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:08:54.883+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:08:55.196 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:08:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:08:54.883+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:08:55.196 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:08:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:08:54.883+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:08:55.196 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:08:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:08:54.954+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:54.954313+0000) 2024-02-20T09:08:55.196 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:08:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:08:55.193+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:08:55.197 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:08:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:08:55.193+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:08:55.197 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:08:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:08:55.193+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:08:55.197 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:08:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:08:54.837+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:54.837453+0000) 2024-02-20T09:08:55.197 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:08:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:08:55.031+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:08:55.197 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:08:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:08:55.031+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:08:55.197 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:08:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:08:55.031+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:08:55.742 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:08:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:08:55.455+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 59457 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:08:55.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:08:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:08:55.447+0000 7f8a6006f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:55.448138+0000) 2024-02-20T09:08:55.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:08:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:08:55.448+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:55.448958+0000) 2024-02-20T09:08:55.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:08:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:08:55.451+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:55.452011+0000) 2024-02-20T09:08:55.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:08:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:08:55.563+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:55.564332+0000) 2024-02-20T09:08:56.161 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:08:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:08:55.954+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:55.954485+0000) 2024-02-20T09:08:56.161 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:08:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:08:56.160+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:08:56.162 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:08:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:08:55.837+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:55.837629+0000) 2024-02-20T09:08:56.162 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:08:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:08:56.067+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:08:56.162 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:08:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:08:56.067+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:08:56.162 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:08:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:08:56.067+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:08:56.162 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:08:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:08:55.911+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:08:56.162 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:08:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:08:55.911+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:08:56.163 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:08:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:08:55.911+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:08:56.163 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:08:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:08:55.911+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:08:56.481 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:08:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:08:56.160+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:08:56.481 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:08:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:08:56.160+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:08:56.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:08:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:08:56.566+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:56.567018+0000) 2024-02-20T09:08:57.110 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:08:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:08:56.954+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:56.954582+0000) 2024-02-20T09:08:57.110 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:08:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:08:56.837+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:56.837746+0000) 2024-02-20T09:08:57.111 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:08:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:08:57.108+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:08:57.111 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:08:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:08:57.108+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:08:57.111 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:08:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:08:56.932+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:08:57.111 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:08:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:08:56.932+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:08:57.111 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:08:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:08:56.933+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:08:57.111 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:08:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:08:56.933+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:08:57.481 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:08:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:08:57.108+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:08:57.481 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:08:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:08:57.175+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:08:57.482 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:08:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:08:57.175+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:08:57.482 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:08:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:08:57.175+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:08:57.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:08:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:08:57.569+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:57.569900+0000) 2024-02-20T09:08:58.092 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:08:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:08:57.954+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:57.954724+0000) 2024-02-20T09:08:58.092 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:08:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:08:57.837+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:57.837961+0000) 2024-02-20T09:08:58.092 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:08:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:08:58.090+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:08:58.093 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:08:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:08:57.900+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:08:58.093 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:08:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:08:57.900+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:08:58.093 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:08:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:08:57.900+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:08:58.093 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:08:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:08:57.900+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:08:58.481 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:08:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:08:58.090+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:08:58.482 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:08:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:08:58.090+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:08:58.482 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:08:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:08:58.216+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:08:58.482 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:08:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:08:58.216+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:08:58.482 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:08:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:08:58.216+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:08:58.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:08:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:08:58.572+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:58.572937+0000) 2024-02-20T09:08:59.229 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:08:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:08:58.954+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:58.954889+0000) 2024-02-20T09:08:59.229 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:08:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:08:59.227+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:08:59.229 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:08:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:08:58.837+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:58.838166+0000) 2024-02-20T09:08:59.229 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:08:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:08:59.057+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:08:59.230 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:08:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:08:59.057+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:08:59.230 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:08:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:08:59.057+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:08:59.230 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:08:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:08:58.868+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:08:59.230 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:08:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:08:58.868+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:08:59.230 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:08:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:08:58.868+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:08:59.230 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:08:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:08:58.869+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:08:59.481 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:08:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:08:59.227+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:08:59.482 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:08:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:08:59.227+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:08:59.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:08:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:08:59.573+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:59.573857+0000) 2024-02-20T09:09:00.230 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:08:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:08:59.954+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:59.955094+0000) 2024-02-20T09:09:00.231 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:08:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:08:59.888+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:09:00.231 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:08:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:08:59.888+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:09:00.231 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:08:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:08:59.888+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:09:00.231 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:08:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:08:59.888+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:09:00.232 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:08:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:08:59.838+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:08:59.838355+0000) 2024-02-20T09:09:00.232 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:09:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:09:00.049+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:09:00.232 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:09:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:09:00.049+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:09:00.232 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:09:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:09:00.049+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:09:00.482 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:09:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:09:00.457+0000 7f8a5f86e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:00.457554+0000) 2024-02-20T09:09:00.482 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:09:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:09:00.464+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 59529 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:09:00.483 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:09:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:09:00.461+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:00.461760+0000) 2024-02-20T09:09:00.483 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:09:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:09:00.228+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:09:00.483 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:09:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:09:00.228+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:09:00.483 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:09:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:09:00.228+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:09:00.483 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:09:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:09:00.456+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:00.457049+0000) 2024-02-20T09:09:00.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:09:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:09:00.573+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:00.574158+0000) 2024-02-20T09:09:01.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:09:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:09:00.955+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:00.955260+0000) 2024-02-20T09:09:01.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:09:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:09:00.935+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:09:01.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:09:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:09:00.935+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:09:01.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:09:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:09:00.935+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:09:01.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:09:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:09:00.935+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:09:01.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:09:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:09:00.838+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:00.838538+0000) 2024-02-20T09:09:01.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:09:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:09:01.040+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:09:01.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:09:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:09:01.040+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:09:01.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:09:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:09:01.040+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:09:01.496 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:09:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:09:01.242+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:09:01.496 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:09:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:09:01.242+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:09:01.497 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:09:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:09:01.242+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:09:01.829 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:09:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:09:01.576+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:01.576487+0000) 2024-02-20T09:09:02.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:09:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:09:01.955+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:01.955434+0000) 2024-02-20T09:09:02.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:09:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:09:01.838+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:01.838658+0000) 2024-02-20T09:09:02.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:09:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:09:02.004+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:09:02.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:09:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:09:02.004+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:09:02.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:09:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:09:02.004+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:09:02.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:09:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:09:01.930+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:09:02.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:09:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:09:01.930+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:09:02.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:09:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:09:01.930+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:09:02.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:09:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:09:01.930+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:09:02.580 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:09:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:09:02.257+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:09:02.580 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:09:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:09:02.257+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:09:02.581 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:09:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:09:02.257+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:09:02.840 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:09:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:09:02.579+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:02.579194+0000) 2024-02-20T09:09:03.228 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:09:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:09:02.902+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:09:03.228 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:09:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:09:02.902+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:09:03.228 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:09:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:09:02.902+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:09:03.229 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:09:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:09:02.902+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:09:03.229 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:09:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:09:02.838+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:02.838883+0000) 2024-02-20T09:09:03.229 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:09:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:09:03.017+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:09:03.229 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:09:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:09:03.017+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:09:03.229 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:09:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:09:03.017+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:09:03.230 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:09:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:09:02.955+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:02.955557+0000) 2024-02-20T09:09:03.230 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:09:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:09:03.226+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:09:03.230 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:09:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:09:03.226+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:09:03.230 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:09:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:09:03.226+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:09:03.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:09:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:09:03.582+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:03.582962+0000) 2024-02-20T09:09:04.234 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:09:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:09:03.871+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:09:04.234 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:09:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:09:03.871+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:09:04.234 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:09:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:09:03.871+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:09:04.234 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:09:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:09:03.871+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:09:04.235 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:09:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:09:03.838+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:03.839074+0000) 2024-02-20T09:09:04.235 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:09:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:09:03.980+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:09:04.235 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:09:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:09:03.980+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:09:04.235 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:09:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:09:03.980+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:09:04.235 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:09:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:09:03.955+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:03.955726+0000) 2024-02-20T09:09:04.235 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:09:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:09:04.232+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:09:04.235 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:09:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:09:04.232+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:09:04.236 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:09:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:09:04.232+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:09:04.829 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:09:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:09:04.583+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:04.583872+0000) 2024-02-20T09:09:05.211 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:09:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:09:04.838+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:04.839248+0000) 2024-02-20T09:09:05.212 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:09:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:09:04.956+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:09:05.212 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:09:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:09:04.956+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:09:05.212 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:09:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:09:04.956+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:09:05.212 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:09:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:09:04.846+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:09:05.212 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:09:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:09:04.846+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:09:05.212 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:09:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:09:04.846+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:09:05.213 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:09:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:09:04.846+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:09:05.213 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:09:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:09:04.955+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:04.955932+0000) 2024-02-20T09:09:05.213 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:09:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:09:05.210+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:09:05.213 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:09:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:09:05.210+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:09:05.213 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:09:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:09:05.210+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:09:05.742 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:09:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:09:05.473+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 59602 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:09:05.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:09:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:09:05.466+0000 7f8a6006f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:05.466477+0000) 2024-02-20T09:09:05.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:09:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:09:05.465+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:05.466066+0000) 2024-02-20T09:09:05.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:09:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:09:05.469+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:05.470374+0000) 2024-02-20T09:09:05.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:09:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:09:05.583+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:05.584191+0000) 2024-02-20T09:09:06.216 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:09:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:09:05.956+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:05.956062+0000) 2024-02-20T09:09:06.216 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:09:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:09:06.214+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:09:06.216 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:09:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:09:05.886+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:09:06.217 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:09:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:09:05.886+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:09:06.217 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:09:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:09:05.886+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:09:06.217 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:09:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:09:05.886+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:09:06.217 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:09:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:09:05.839+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:05.839436+0000) 2024-02-20T09:09:06.217 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:09:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:09:05.942+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:09:06.217 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:09:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:09:05.942+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:09:06.217 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:09:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:09:05.942+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:09:06.482 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:09:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:09:06.214+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:09:06.483 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:09:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:09:06.214+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:09:06.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:09:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:09:06.585+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:06.586704+0000) 2024-02-20T09:09:07.186 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:09:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:09:06.955+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:06.956223+0000) 2024-02-20T09:09:07.187 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:09:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:09:07.184+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:09:07.187 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:09:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:09:06.837+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:09:07.187 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:09:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:09:06.837+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:09:07.187 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:09:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:09:06.837+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:09:07.188 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:09:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:09:06.837+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:09:07.188 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:09:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:09:06.838+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:06.839583+0000) 2024-02-20T09:09:07.188 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:09:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:09:06.965+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:09:07.188 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:09:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:09:06.965+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:09:07.188 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:09:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:09:06.965+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:09:07.482 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:09:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:09:07.184+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:09:07.483 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:09:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:09:07.184+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:09:07.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:09:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:09:07.588+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:07.589570+0000) 2024-02-20T09:09:08.142 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:09:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:09:07.955+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:07.956451+0000) 2024-02-20T09:09:08.142 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:09:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:09:08.140+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:09:08.143 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:09:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:09:07.838+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:07.839790+0000) 2024-02-20T09:09:08.143 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:09:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:09:07.970+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:09:08.143 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:09:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:09:07.970+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:09:08.143 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:09:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:09:07.970+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:09:08.143 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:09:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:09:07.845+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:09:08.143 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:09:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:09:07.845+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:09:08.143 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:09:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:09:07.845+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:09:08.144 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:09:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:09:07.845+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:09:08.483 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:09:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:09:08.140+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:09:08.483 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:09:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:09:08.140+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:09:08.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:09:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:09:08.591+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:08.592369+0000) 2024-02-20T09:09:09.177 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:09:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:09:08.955+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:08.956622+0000) 2024-02-20T09:09:09.178 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:09:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:09:08.824+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:09:09.178 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:09:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:09:08.824+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:09:09.178 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:09:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:09:08.824+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:09:09.178 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:09:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:09:08.824+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:09:09.179 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:09:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:09:08.838+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:08.839981+0000) 2024-02-20T09:09:09.179 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:09:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:09:08.948+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:09:09.179 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:09:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:09:08.948+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:09:09.179 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:09:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:09:08.948+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:09:09.483 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:09:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:09:09.175+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:09:09.483 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:09:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:09:09.175+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:09:09.483 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:09:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:09:09.175+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:09:09.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:09:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:09:09.592+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:09.593336+0000) 2024-02-20T09:09:10.176 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:09:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:09:09.955+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:09.956780+0000) 2024-02-20T09:09:10.176 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:09:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:09:10.173+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:09:10.177 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:09:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:09:09.838+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:09.840128+0000) 2024-02-20T09:09:10.177 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:09:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:09:09.959+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:09:10.177 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:09:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:09:09.959+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:09:10.177 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:09:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:09:09.959+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:09:10.177 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:09:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:09:09.792+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:09:10.178 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:09:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:09:09.792+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:09:10.178 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:09:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:09:09.792+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:09:10.178 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:09:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:09:09.792+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:09:10.476 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:09:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:09:10.173+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:09:10.477 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:09:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:09:10.173+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:09:10.477 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:09:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:09:10.473+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:10.475243+0000) 2024-02-20T09:09:10.742 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:09:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:09:10.481+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 59674 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:09:10.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:09:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:09:10.474+0000 7f8a5f86e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:10.476245+0000) 2024-02-20T09:09:10.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:09:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:09:10.478+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:10.479678+0000) 2024-02-20T09:09:10.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:09:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:09:10.592+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:10.593695+0000) 2024-02-20T09:09:11.215 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:09:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:09:10.824+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:09:11.216 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:09:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:09:10.824+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:09:11.216 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:09:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:09:10.824+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:09:11.216 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:09:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:09:10.824+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:09:11.216 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:09:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:09:10.838+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:10.840333+0000) 2024-02-20T09:09:11.216 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:09:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:09:10.963+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:09:11.217 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:09:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:09:10.963+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:09:11.217 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:09:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:09:10.963+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:09:11.217 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:09:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:09:10.955+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:10.956954+0000) 2024-02-20T09:09:11.217 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:09:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:09:11.213+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:09:11.217 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:09:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:09:11.213+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:09:11.218 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:09:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:09:11.213+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:09:11.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:09:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:09:11.595+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:11.597363+0000) 2024-02-20T09:09:12.200 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:09:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:09:11.956+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:11.957150+0000) 2024-02-20T09:09:12.200 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:09:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:09:12.197+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:09:12.200 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:09:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:09:11.839+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:11.840478+0000) 2024-02-20T09:09:12.201 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:09:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:09:12.002+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:09:12.201 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:09:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:09:12.002+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:09:12.201 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:09:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:09:12.002+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:09:12.201 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:09:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:09:11.812+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:09:12.201 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:09:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:09:11.812+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:09:12.201 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:09:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:09:11.812+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:09:12.202 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:09:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:09:11.812+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:09:12.483 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:09:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:09:12.197+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:09:12.484 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:09:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:09:12.197+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:09:12.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:09:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:09:12.598+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:12.600171+0000) 2024-02-20T09:09:13.187 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:09:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:09:12.955+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:12.957314+0000) 2024-02-20T09:09:13.188 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:09:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:09:13.184+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:09:13.188 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:09:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:09:13.184+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:09:13.188 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:09:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:09:12.839+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:12.840648+0000) 2024-02-20T09:09:13.189 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:09:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:09:12.980+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:09:13.189 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:09:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:09:12.980+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:09:13.189 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:09:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:09:12.980+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:09:13.189 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:09:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:09:12.821+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:09:13.189 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:09:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:09:12.821+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:09:13.189 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:09:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:09:12.821+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:09:13.189 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:09:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:09:12.821+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:09:13.484 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:09:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:09:13.184+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:09:13.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:09:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:09:13.602+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:13.603178+0000) 2024-02-20T09:09:14.220 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:09:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:09:13.956+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:13.957443+0000) 2024-02-20T09:09:14.220 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:09:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:09:14.217+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:09:14.220 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:09:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:09:13.839+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:13.840830+0000) 2024-02-20T09:09:14.221 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:09:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:09:14.009+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:09:14.221 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:09:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:09:14.009+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:09:14.221 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:09:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:09:14.009+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:09:14.221 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:09:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:09:13.852+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:09:14.221 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:09:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:09:13.852+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:09:14.221 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:09:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:09:13.852+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:09:14.222 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:09:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:09:13.853+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:09:14.484 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:09:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:09:14.217+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:09:14.484 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:09:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:09:14.217+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:09:14.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:09:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:09:14.603+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:14.603905+0000) 2024-02-20T09:09:15.186 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:09:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:09:14.956+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:14.957604+0000) 2024-02-20T09:09:15.186 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:09:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:09:15.183+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:09:15.187 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:09:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:09:14.840+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:14.841013+0000) 2024-02-20T09:09:15.187 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:09:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:09:14.965+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:09:15.187 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:09:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:09:14.965+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:09:15.187 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:09:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:09:14.965+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:09:15.187 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:09:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:09:14.845+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:09:15.188 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:09:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:09:14.845+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:09:15.188 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:09:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:09:14.845+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:09:15.188 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:09:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:09:14.845+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:09:15.484 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:09:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:09:15.183+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:09:15.484 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:09:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:09:15.183+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:09:15.742 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:09:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:09:15.490+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 59747 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:09:15.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:09:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:09:15.483+0000 7f8a5f86e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:15.484526+0000) 2024-02-20T09:09:15.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:09:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:09:15.483+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:15.484647+0000) 2024-02-20T09:09:15.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:09:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:09:15.487+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:15.488419+0000) 2024-02-20T09:09:15.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:09:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:09:15.603+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:15.604174+0000) 2024-02-20T09:09:16.210 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:09:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:09:15.839+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:15.841221+0000) 2024-02-20T09:09:16.210 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:09:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:09:15.950+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:09:16.210 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:09:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:09:15.950+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:09:16.210 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:09:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:09:15.950+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:09:16.211 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:09:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:09:15.956+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:15.957716+0000) 2024-02-20T09:09:16.211 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:09:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:09:16.207+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:09:16.211 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:09:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:09:15.836+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:09:16.211 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:09:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:09:15.836+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:09:16.212 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:09:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:09:15.836+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:09:16.212 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:09:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:09:15.836+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:09:16.484 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:09:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:09:16.207+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:09:16.484 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:09:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:09:16.207+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:09:16.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:09:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:09:16.605+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:16.606462+0000) 2024-02-20T09:09:17.173 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:09:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:09:16.956+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:16.957845+0000) 2024-02-20T09:09:17.173 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:09:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:09:17.170+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:09:17.174 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:09:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:09:16.840+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:16.841360+0000) 2024-02-20T09:09:17.174 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:09:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:09:16.970+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:09:17.175 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:09:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:09:16.970+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:09:17.175 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:09:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:09:16.970+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:09:17.175 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:09:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:09:16.792+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:09:17.176 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:09:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:09:16.792+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:09:17.176 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:09:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:09:16.792+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:09:17.176 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:09:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:09:16.792+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:09:17.484 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:09:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:09:17.170+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:09:17.485 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:09:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:09:17.170+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:09:17.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:09:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:09:17.608+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:17.609240+0000) 2024-02-20T09:09:18.217 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:09:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:09:17.956+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:17.957988+0000) 2024-02-20T09:09:18.217 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:09:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:09:18.214+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:09:18.217 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:09:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:09:17.840+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:17.841587+0000) 2024-02-20T09:09:18.218 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:09:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:09:17.971+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:09:18.218 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:09:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:09:17.971+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:09:18.218 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:09:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:09:17.971+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:09:18.218 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:09:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:09:17.768+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:09:18.218 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:09:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:09:17.768+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:09:18.218 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:09:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:09:17.768+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:09:18.219 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:09:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:09:17.768+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:09:18.484 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:09:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:09:18.214+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:09:18.485 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:09:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:09:18.214+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:09:18.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:09:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:09:18.612+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:18.612971+0000) 2024-02-20T09:09:19.209 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:09:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:09:18.957+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:18.958186+0000) 2024-02-20T09:09:19.209 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:09:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:09:19.206+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:09:19.210 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:09:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:09:18.784+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:09:19.210 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:09:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:09:18.784+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:09:19.210 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:09:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:09:18.784+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:09:19.210 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:09:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:09:18.785+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:09:19.210 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:09:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:09:18.840+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:18.841716+0000) 2024-02-20T09:09:19.210 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:09:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:09:19.008+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:09:19.211 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:09:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:09:19.008+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:09:19.211 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:09:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:09:19.008+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:09:19.485 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:09:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:09:19.206+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:09:19.485 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:09:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:09:19.206+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:09:19.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:09:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:09:19.612+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:19.613785+0000) 2024-02-20T09:09:20.213 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:09:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:09:19.957+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:19.958372+0000) 2024-02-20T09:09:20.213 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:09:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:09:20.211+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:09:20.214 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:09:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:09:19.783+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:09:20.214 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:09:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:09:19.783+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:09:20.214 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:09:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:09:19.783+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:09:20.214 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:09:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:09:19.783+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:09:20.214 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:09:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:09:19.841+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:19.841899+0000) 2024-02-20T09:09:20.214 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:09:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:09:20.026+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:09:20.215 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:09:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:09:20.026+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:09:20.215 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:09:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:09:20.026+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:09:20.485 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:09:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:09:20.211+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:09:20.485 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:09:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:09:20.211+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:09:20.742 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:09:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:09:20.498+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 59819 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:09:20.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:09:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:09:20.491+0000 7f8a6006f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:20.492849+0000) 2024-02-20T09:09:20.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:09:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:09:20.491+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:20.493289+0000) 2024-02-20T09:09:20.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:09:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:09:20.495+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:20.496846+0000) 2024-02-20T09:09:20.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:09:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:09:20.613+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:20.614061+0000) 2024-02-20T09:09:21.224 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:09:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:09:20.957+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:20.958577+0000) 2024-02-20T09:09:21.224 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:09:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:09:21.222+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:09:21.225 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:09:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:09:20.840+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:20.842073+0000) 2024-02-20T09:09:21.225 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:09:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:09:20.976+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:09:21.225 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:09:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:09:20.976+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:09:21.225 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:09:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:09:20.976+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:09:21.225 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:09:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:09:20.814+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:09:21.226 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:09:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:09:20.814+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:09:21.226 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:09:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:09:20.814+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:09:21.226 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:09:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:09:20.814+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:09:21.485 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:09:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:09:21.222+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:09:21.485 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:09:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:09:21.222+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:09:21.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:09:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:09:21.615+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:21.616900+0000) 2024-02-20T09:09:22.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:09:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:09:21.958+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:21.958791+0000) 2024-02-20T09:09:22.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:09:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:09:21.840+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:21.842185+0000) 2024-02-20T09:09:22.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:09:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:09:21.939+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:09:22.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:09:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:09:21.939+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:09:22.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:09:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:09:21.939+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:09:22.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:09:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:09:21.820+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:09:22.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:09:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:09:21.820+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:09:22.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:09:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:09:21.820+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:09:22.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:09:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:09:21.821+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:09:22.514 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:09:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:09:22.259+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:09:22.515 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:09:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:09:22.259+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:09:22.515 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:09:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:09:22.259+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:09:22.832 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:09:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:09:22.619+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:22.619821+0000) 2024-02-20T09:09:23.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:09:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:09:22.958+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:22.958964+0000) 2024-02-20T09:09:23.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:09:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:09:22.841+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:22.842385+0000) 2024-02-20T09:09:23.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:09:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:09:22.974+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:09:23.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:09:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:09:22.974+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:09:23.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:09:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:09:22.974+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:09:23.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:09:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:09:22.865+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:09:23.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:09:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:09:22.865+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:09:23.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:09:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:09:22.865+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:09:23.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:09:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:09:22.865+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:09:23.515 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:09:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:09:23.247+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:09:23.515 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:09:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:09:23.247+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:09:23.515 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:09:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:09:23.247+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:09:23.832 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:09:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:09:23.621+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:23.622572+0000) 2024-02-20T09:09:24.237 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:09:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:09:23.958+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:23.959141+0000) 2024-02-20T09:09:24.237 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:09:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:09:23.841+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:23.842554+0000) 2024-02-20T09:09:24.238 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:09:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:09:23.996+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:09:24.238 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:09:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:09:23.996+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:09:24.238 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:09:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:09:23.996+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:09:24.238 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:09:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:09:23.845+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:09:24.238 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:09:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:09:23.845+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:09:24.238 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:09:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:09:23.845+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:09:24.238 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:09:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:09:23.845+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:09:24.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:09:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:09:24.234+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:09:24.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:09:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:09:24.234+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:09:24.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:09:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:09:24.234+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:09:24.833 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:09:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:09:24.622+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:24.623469+0000) 2024-02-20T09:09:25.193 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:09:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:09:24.958+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:24.959356+0000) 2024-02-20T09:09:25.193 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:09:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:09:25.191+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:09:25.193 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:09:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:09:24.850+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:09:25.194 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:09:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:09:24.850+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:09:25.194 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:09:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:09:24.850+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:09:25.194 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:09:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:09:24.850+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:09:25.194 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:09:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:09:24.841+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:24.842725+0000) 2024-02-20T09:09:25.194 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:09:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:09:25.006+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:09:25.194 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:09:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:09:25.006+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:09:25.195 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:09:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:09:25.006+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:09:25.486 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:09:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:09:25.191+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:09:25.486 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:09:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:09:25.191+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:09:25.742 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:09:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:09:25.507+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 59892 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:09:25.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:09:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:09:25.500+0000 7f8a5f86e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:25.501610+0000) 2024-02-20T09:09:25.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:09:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:09:25.500+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:25.502361+0000) 2024-02-20T09:09:25.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:09:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:09:25.504+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:25.505516+0000) 2024-02-20T09:09:25.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:09:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:09:25.623+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:25.623808+0000) 2024-02-20T09:09:26.163 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:09:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:09:25.958+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:25.959515+0000) 2024-02-20T09:09:26.163 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:09:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:09:26.161+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:09:26.163 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:09:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:09:25.841+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:25.842845+0000) 2024-02-20T09:09:26.164 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:09:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:09:26.044+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:09:26.164 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:09:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:09:26.044+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:09:26.164 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:09:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:09:26.044+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:09:26.164 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:09:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:09:25.837+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:09:26.164 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:09:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:09:25.837+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:09:26.164 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:09:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:09:25.837+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:09:26.165 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:09:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:09:25.837+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:09:26.486 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:09:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:09:26.161+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:09:26.486 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:09:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:09:26.161+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:09:26.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:09:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:09:26.625+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:26.626492+0000) 2024-02-20T09:09:27.186 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:09:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:09:26.958+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:26.959621+0000) 2024-02-20T09:09:27.186 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:09:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:09:27.184+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:09:27.187 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:09:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:09:26.862+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:09:27.187 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:09:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:09:26.862+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:09:27.187 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:09:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:09:26.862+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:09:27.187 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:09:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:09:26.862+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:09:27.188 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:09:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:09:26.842+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:26.842996+0000) 2024-02-20T09:09:27.188 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:09:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:09:27.032+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:09:27.188 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:09:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:09:27.032+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:09:27.188 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:09:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:09:27.032+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:09:27.486 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:09:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:09:27.184+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:09:27.486 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:09:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:09:27.184+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:09:27.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:09:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:09:27.628+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:27.629265+0000) 2024-02-20T09:09:28.138 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:09:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:09:27.830+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:09:28.138 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:09:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:09:27.830+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:09:28.138 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:09:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:09:27.830+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:09:28.139 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:09:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:09:27.830+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:09:28.139 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:09:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:09:27.959+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:27.959747+0000) 2024-02-20T09:09:28.139 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:09:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:09:28.136+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:09:28.139 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:09:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:09:28.136+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:09:28.139 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:09:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:09:27.841+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:27.843179+0000) 2024-02-20T09:09:28.140 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:09:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:09:28.024+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:09:28.140 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:09:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:09:28.024+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:09:28.140 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:09:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:09:28.024+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:09:28.486 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:09:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:09:28.136+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:09:28.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:09:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:09:28.631+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:28.632085+0000) 2024-02-20T09:09:29.105 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:09:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:09:28.958+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:28.959861+0000) 2024-02-20T09:09:29.106 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:09:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:09:29.102+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:09:29.106 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:09:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:09:28.842+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:28.843334+0000) 2024-02-20T09:09:29.106 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:09:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:09:29.043+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:09:29.106 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:09:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:09:29.043+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:09:29.106 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:09:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:09:29.043+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:09:29.107 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:09:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:09:28.803+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:09:29.107 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:09:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:09:28.803+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:09:29.107 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:09:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:09:28.803+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:09:29.107 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:09:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:09:28.803+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:09:29.486 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:09:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:09:29.102+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:09:29.486 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:09:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:09:29.102+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:09:29.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:09:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:09:29.632+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:29.632970+0000) 2024-02-20T09:09:30.101 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:09:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:09:29.959+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:29.960045+0000) 2024-02-20T09:09:30.101 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:09:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:09:29.843+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:09:30.101 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:09:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:09:29.843+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:09:30.101 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:09:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:09:29.843+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:09:30.101 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:09:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:09:29.843+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:09:30.102 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:09:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:09:29.842+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:29.843518+0000) 2024-02-20T09:09:30.102 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:09:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:09:30.049+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:09:30.102 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:09:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:09:30.049+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:09:30.102 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:09:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:09:30.049+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:09:30.403 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:09:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:09:30.098+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:09:30.404 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:09:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:09:30.098+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:09:30.404 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:09:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:09:30.098+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:09:30.742 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:09:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:09:30.516+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 59964 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:09:30.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:09:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:09:30.510+0000 7f8a5f86e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:30.511584+0000) 2024-02-20T09:09:30.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:09:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:09:30.509+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:30.510675+0000) 2024-02-20T09:09:30.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:09:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:09:30.512+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:30.513884+0000) 2024-02-20T09:09:30.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:09:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:09:30.632+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:30.633302+0000) 2024-02-20T09:09:31.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:09:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:09:30.821+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:09:31.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:09:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:09:30.821+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:09:31.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:09:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:09:30.821+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:09:31.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:09:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:09:30.821+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:09:31.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:09:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:09:30.843+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:30.843722+0000) 2024-02-20T09:09:31.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:09:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:09:31.021+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:09:31.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:09:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:09:31.021+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:09:31.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:09:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:09:31.021+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:09:31.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:09:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:09:30.958+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:30.960208+0000) 2024-02-20T09:09:31.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:09:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:09:31.070+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:09:31.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:09:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:09:31.070+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:09:31.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:09:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:09:31.070+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:09:31.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:09:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:09:31.635+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:31.635722+0000) 2024-02-20T09:09:32.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:09:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:09:31.842+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:31.843894+0000) 2024-02-20T09:09:32.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:09:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:09:31.976+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:09:32.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:09:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:09:31.976+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:09:32.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:09:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:09:31.976+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:09:32.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:09:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:09:31.815+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:09:32.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:09:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:09:31.815+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:09:32.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:09:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:09:31.815+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:09:32.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:09:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:09:31.815+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:09:32.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:09:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:09:31.959+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:31.960394+0000) 2024-02-20T09:09:32.245 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:09:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:09:32.040+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:09:32.245 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:09:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:09:32.040+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:09:32.245 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:09:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:09:32.040+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:09:32.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:09:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:09:32.638+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:32.638591+0000) 2024-02-20T09:09:33.072 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:09:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:09:32.814+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:09:33.073 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:09:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:09:32.814+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:09:33.073 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:09:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:09:32.814+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:09:33.073 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:09:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:09:32.814+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:09:33.073 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:09:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:09:32.960+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:32.960583+0000) 2024-02-20T09:09:33.074 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:09:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:09:33.070+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:09:33.074 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:09:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:09:33.070+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:09:33.074 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:09:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:09:32.842+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:32.844105+0000) 2024-02-20T09:09:33.074 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:09:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:09:33.017+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:09:33.074 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:09:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:09:33.017+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:09:33.074 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:09:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:09:33.017+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:09:33.487 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:09:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:09:33.070+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:09:33.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:09:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:09:33.640+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:33.641496+0000) 2024-02-20T09:09:34.112 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:09:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:09:33.812+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:09:34.112 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:09:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:09:33.812+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:09:34.112 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:09:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:09:33.812+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:09:34.112 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:09:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:09:33.812+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:09:34.113 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:09:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:09:33.960+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:33.960788+0000) 2024-02-20T09:09:34.113 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:09:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:09:34.109+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:09:34.113 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:09:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:09:34.109+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:09:34.113 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:09:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:09:33.842+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:33.844279+0000) 2024-02-20T09:09:34.113 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:09:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:09:33.966+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:09:34.113 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:09:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:09:33.966+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:09:34.114 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:09:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:09:33.966+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:09:34.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:09:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:09:34.642+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:34.642599+0000) 2024-02-20T09:09:34.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:09:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:09:34.109+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:09:35.154 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:09:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:09:34.843+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:34.844463+0000) 2024-02-20T09:09:35.155 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:09:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:09:35.005+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:09:35.155 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:09:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:09:35.005+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:09:35.155 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:09:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:09:35.005+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:09:35.155 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:09:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:09:34.854+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:09:35.155 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:09:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:09:34.854+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:09:35.156 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:09:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:09:34.854+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:09:35.156 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:09:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:09:34.854+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:09:35.156 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:09:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:09:34.960+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:34.960941+0000) 2024-02-20T09:09:35.156 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:09:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:09:35.152+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:09:35.156 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:09:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:09:35.152+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:09:35.156 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:09:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:09:35.152+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:09:35.742 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:09:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:09:35.518+0000 7f8a5f86e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:35.520039+0000) 2024-02-20T09:09:35.743 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:09:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:09:35.524+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 60037 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:09:35.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:09:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:09:35.517+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:35.519242+0000) 2024-02-20T09:09:35.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:09:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:09:35.520+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:35.521837+0000) 2024-02-20T09:09:35.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:09:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:09:35.642+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:35.642898+0000) 2024-02-20T09:09:36.180 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:09:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:09:35.844+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:35.844651+0000) 2024-02-20T09:09:36.181 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:09:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:09:36.010+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:09:36.181 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:09:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:09:36.010+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:09:36.181 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:09:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:09:36.010+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:09:36.181 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:09:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:09:35.868+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:09:36.181 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:09:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:09:35.868+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:09:36.181 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:09:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:09:35.868+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:09:36.182 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:09:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:09:35.868+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:09:36.182 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:09:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:09:35.960+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:35.961120+0000) 2024-02-20T09:09:36.182 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:09:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:09:36.178+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:09:36.182 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:09:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:09:36.178+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:09:36.182 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:09:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:09:36.178+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:09:36.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:09:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:09:36.645+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:36.645610+0000) 2024-02-20T09:09:37.192 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:09:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:09:36.844+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:36.844803+0000) 2024-02-20T09:09:37.193 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:09:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:09:36.975+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:09:37.193 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:09:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:09:36.975+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:09:37.193 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:09:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:09:36.975+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:09:37.193 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:09:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:09:36.878+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:09:37.193 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:09:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:09:36.878+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:09:37.193 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:09:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:09:36.878+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:09:37.194 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:09:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:09:36.878+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:09:37.194 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:09:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:09:36.960+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:36.961271+0000) 2024-02-20T09:09:37.194 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:09:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:09:37.190+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:09:37.194 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:09:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:09:37.190+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:09:37.194 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:09:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:09:37.190+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:09:37.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:09:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:09:37.647+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:37.648309+0000) 2024-02-20T09:09:38.227 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:09:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:09:37.844+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:37.845008+0000) 2024-02-20T09:09:38.227 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:09:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:09:37.933+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:09:38.228 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:09:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:09:37.933+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:09:38.228 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:09:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:09:37.933+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:09:38.229 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:09:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:09:37.843+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:09:38.229 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:09:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:09:37.843+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:09:38.229 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:09:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:09:37.843+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:09:38.230 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:09:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:09:37.843+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:09:38.230 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:09:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:09:37.961+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:37.961481+0000) 2024-02-20T09:09:38.230 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:09:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:09:38.225+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:09:38.231 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:09:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:09:38.225+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:09:38.231 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:09:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:09:38.225+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:09:38.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:09:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:09:38.650+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:38.651143+0000) 2024-02-20T09:09:39.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:09:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:09:38.844+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:38.845219+0000) 2024-02-20T09:09:39.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:09:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:09:38.965+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:09:39.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:09:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:09:38.965+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:09:39.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:09:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:09:38.965+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:09:39.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:09:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:09:38.961+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:38.961711+0000) 2024-02-20T09:09:39.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:09:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:09:39.239+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:09:39.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:09:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:09:39.239+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:09:39.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:09:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:09:38.859+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:09:39.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:09:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:09:38.859+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:09:39.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:09:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:09:38.859+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:09:39.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:09:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:09:38.859+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:09:39.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:09:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:09:39.239+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:09:39.835 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:09:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:09:39.651+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:39.652139+0000) 2024-02-20T09:09:40.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:09:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:09:39.845+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:39.845387+0000) 2024-02-20T09:09:40.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:09:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:09:39.999+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:09:40.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:09:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:09:39.999+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:09:40.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:09:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:09:39.999+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:09:40.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:09:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:09:39.853+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:09:40.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:09:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:09:39.853+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:09:40.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:09:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:09:39.853+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:09:40.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:09:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:09:39.854+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:09:40.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:09:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:09:39.961+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:39.961861+0000) 2024-02-20T09:09:40.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:09:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:09:40.240+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:09:40.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:09:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:09:40.240+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:09:40.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:09:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:09:40.240+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:09:40.830 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:09:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:09:40.535+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 60109 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:09:40.831 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:09:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:09:40.528+0000 7f8a5f86e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:40.528605+0000) 2024-02-20T09:09:40.831 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:09:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:09:40.527+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:40.528352+0000) 2024-02-20T09:09:40.831 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:09:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:09:40.530+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:40.531027+0000) 2024-02-20T09:09:40.831 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:09:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:09:40.652+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:40.652445+0000) 2024-02-20T09:09:40.831 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:09:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:09:40.828+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:09:40.832 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:09:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:09:40.828+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:09:40.832 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:09:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:09:40.828+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:09:41.224 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:09:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:09:40.828+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:09:41.225 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:09:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:09:40.845+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:40.845588+0000) 2024-02-20T09:09:41.225 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:09:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:09:40.960+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:09:41.225 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:09:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:09:40.960+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:09:41.225 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:09:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:09:40.960+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:09:41.225 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:09:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:09:40.961+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:40.961985+0000) 2024-02-20T09:09:41.225 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:09:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:09:41.222+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:09:41.225 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:09:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:09:41.222+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:09:41.226 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:09:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:09:41.222+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:09:41.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:09:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:09:41.654+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:41.654905+0000) 2024-02-20T09:09:42.190 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:09:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:09:41.841+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:09:42.190 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:09:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:09:41.841+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:09:42.191 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:09:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:09:41.841+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:09:42.191 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:09:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:09:41.841+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:09:42.191 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:09:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:09:41.845+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:41.845707+0000) 2024-02-20T09:09:42.191 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:09:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:09:41.941+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:09:42.191 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:09:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:09:41.941+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:09:42.191 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:09:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:09:41.941+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:09:42.192 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:09:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:09:41.961+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:41.962157+0000) 2024-02-20T09:09:42.192 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:09:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:09:42.188+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:09:42.192 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:09:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:09:42.188+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:09:42.192 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:09:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:09:42.188+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:09:42.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:09:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:09:42.657+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:42.657704+0000) 2024-02-20T09:09:43.232 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:09:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:09:42.805+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:09:43.232 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:09:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:09:42.805+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:09:43.232 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:09:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:09:42.805+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:09:43.233 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:09:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:09:42.805+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:09:43.233 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:09:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:09:42.845+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:42.845907+0000) 2024-02-20T09:09:43.233 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:09:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:09:42.932+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:09:43.234 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:09:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:09:42.932+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:09:43.234 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:09:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:09:42.932+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:09:43.234 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:09:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:09:42.961+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:42.962336+0000) 2024-02-20T09:09:43.234 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:09:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:09:43.229+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:09:43.235 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:09:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:09:43.230+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:09:43.235 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:09:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:09:43.230+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:09:43.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:09:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:09:43.660+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:43.660462+0000) 2024-02-20T09:09:44.195 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:09:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:09:43.845+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:43.846110+0000) 2024-02-20T09:09:44.196 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:09:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:09:43.948+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:09:44.196 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:09:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:09:43.948+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:09:44.196 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:09:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:09:43.948+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:09:44.196 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:09:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:09:43.962+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:43.962524+0000) 2024-02-20T09:09:44.196 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:09:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:09:44.193+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:09:44.197 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:09:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:09:44.193+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:09:44.197 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:09:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:09:43.817+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:09:44.197 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:09:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:09:43.817+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:09:44.197 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:09:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:09:43.817+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:09:44.197 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:09:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:09:43.817+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:09:44.489 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:09:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:09:44.193+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:09:44.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:09:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:09:44.660+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:44.661312+0000) 2024-02-20T09:09:45.177 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:09:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:09:44.845+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:44.846316+0000) 2024-02-20T09:09:45.177 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:09:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:09:44.908+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:09:45.177 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:09:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:09:44.908+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:09:45.177 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:09:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:09:44.908+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:09:45.178 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:09:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:09:44.768+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:09:45.178 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:09:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:09:44.768+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:09:45.178 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:09:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:09:44.768+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:09:45.178 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:09:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:09:44.768+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:09:45.179 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:09:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:09:44.962+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:44.962706+0000) 2024-02-20T09:09:45.179 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:09:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:09:45.175+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:09:45.179 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:09:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:09:45.175+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:09:45.179 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:09:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:09:45.175+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:09:45.742 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:09:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:09:45.543+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 60182 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:09:45.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:09:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:09:45.536+0000 7f8a5f86e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:45.536783+0000) 2024-02-20T09:09:45.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:09:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:09:45.537+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:45.537545+0000) 2024-02-20T09:09:45.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:09:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:09:45.539+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:45.540335+0000) 2024-02-20T09:09:45.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:09:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:09:45.661+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:45.661610+0000) 2024-02-20T09:09:45.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:09:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:09:45.733+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:09:45.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:09:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:09:45.733+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:09:45.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:09:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:09:45.733+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:09:45.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:09:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:09:45.733+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:09:46.197 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:09:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:09:45.962+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:45.962866+0000) 2024-02-20T09:09:46.198 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:09:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:09:46.195+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:09:46.198 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:09:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:09:45.846+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:45.846451+0000) 2024-02-20T09:09:46.198 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:09:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:09:45.927+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:09:46.198 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:09:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:09:45.927+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:09:46.198 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:09:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:09:45.927+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:09:46.489 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:09:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:09:46.195+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:09:46.490 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:09:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:09:46.195+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:09:46.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:09:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:09:46.664+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:46.664336+0000) 2024-02-20T09:09:47.176 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:09:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:09:46.846+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:46.846590+0000) 2024-02-20T09:09:47.177 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:09:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:09:46.922+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:09:47.177 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:09:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:09:46.922+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:09:47.177 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:09:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:09:46.922+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:09:47.177 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:09:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:09:46.753+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:09:47.177 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:09:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:09:46.753+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:09:47.177 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:09:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:09:46.753+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:09:47.178 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:09:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:09:46.754+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:09:47.178 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:09:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:09:46.962+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:46.963011+0000) 2024-02-20T09:09:47.178 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:09:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:09:47.175+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:09:47.178 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:09:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:09:47.175+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:09:47.178 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:09:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:09:47.175+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:09:47.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:09:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:09:47.666+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:47.667136+0000) 2024-02-20T09:09:47.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:09:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:09:47.729+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:09:47.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:09:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:09:47.729+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:09:47.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:09:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:09:47.729+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:09:47.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:09:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:09:47.730+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:09:48.205 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:09:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:09:47.962+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:47.963191+0000) 2024-02-20T09:09:48.206 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:09:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:09:48.204+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:09:48.206 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:09:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:09:47.846+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:47.846795+0000) 2024-02-20T09:09:48.206 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:09:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:09:47.942+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:09:48.206 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:09:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:09:47.942+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:09:48.206 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:09:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:09:47.942+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:09:48.490 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:09:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:09:48.204+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:09:48.490 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:09:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:09:48.204+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:09:48.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:09:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:09:48.669+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:48.669856+0000) 2024-02-20T09:09:49.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:09:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:09:48.963+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:48.963360+0000) 2024-02-20T09:09:49.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:09:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:09:48.773+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:09:49.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:09:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:09:48.773+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:09:49.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:09:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:09:48.773+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:09:49.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:09:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:09:48.773+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:09:49.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:09:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:09:48.846+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:48.846962+0000) 2024-02-20T09:09:49.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:09:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:09:48.924+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:09:49.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:09:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:09:48.924+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:09:49.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:09:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:09:48.924+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:09:49.542 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:09:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:09:49.250+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:09:49.542 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:09:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:09:49.250+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:09:49.542 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:09:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:09:49.250+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:09:49.837 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:09:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:09:49.670+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:49.670908+0000) 2024-02-20T09:09:49.838 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:09:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:09:49.727+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:09:49.838 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:09:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:09:49.727+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:09:49.838 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:09:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:09:49.727+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:09:49.838 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:09:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:09:49.727+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:09:50.208 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:09:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:09:49.963+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:49.963479+0000) 2024-02-20T09:09:50.209 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:09:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:09:50.206+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:09:50.209 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:09:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:09:49.846+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:49.847138+0000) 2024-02-20T09:09:50.209 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:09:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:09:49.958+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:09:50.209 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:09:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:09:49.958+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:09:50.209 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:09:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:09:49.958+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:09:50.490 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:09:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:09:50.206+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:09:50.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:09:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:09:50.206+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:09:50.742 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:09:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:09:50.545+0000 7f8a5f86e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:50.545487+0000) 2024-02-20T09:09:50.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:09:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:09:50.548+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:50.548686+0000) 2024-02-20T09:09:50.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:09:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:09:50.670+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:50.671147+0000) 2024-02-20T09:09:50.743 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:09:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:09:50.551+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 60254 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:09:50.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:09:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:09:50.545+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:50.545836+0000) 2024-02-20T09:09:51.182 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:09:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:09:50.963+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:50.963612+0000) 2024-02-20T09:09:51.182 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:09:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:09:50.846+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:50.847298+0000) 2024-02-20T09:09:51.183 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:09:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:09:50.953+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:09:51.183 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:09:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:09:50.953+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:09:51.183 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:09:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:09:50.953+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:09:51.183 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:09:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:09:50.749+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:09:51.183 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:09:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:09:50.749+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:09:51.183 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:09:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:09:50.749+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:09:51.183 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:09:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:09:50.749+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:09:51.481 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:09:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:09:51.180+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:09:51.481 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:09:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:09:51.180+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:09:51.481 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:09:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:09:51.180+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:09:51.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:09:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:09:51.673+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:51.673830+0000) 2024-02-20T09:09:51.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:09:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:09:51.723+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:09:51.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:09:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:09:51.723+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:09:51.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:09:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:09:51.723+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:09:51.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:09:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:09:51.723+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:09:52.202 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:09:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:09:51.963+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:51.963746+0000) 2024-02-20T09:09:52.202 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:09:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:09:52.200+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:09:52.202 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:09:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:09:52.200+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:09:52.202 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:09:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:09:52.200+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:09:52.202 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:09:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:09:51.847+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:51.847455+0000) 2024-02-20T09:09:52.203 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:09:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:09:51.922+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:09:52.203 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:09:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:09:51.922+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:09:52.203 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:09:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:09:51.922+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:09:52.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:09:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:09:52.676+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:52.676679+0000) 2024-02-20T09:09:53.171 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:09:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:09:52.847+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:52.847586+0000) 2024-02-20T09:09:53.172 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:09:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:09:52.877+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:09:53.172 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:09:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:09:52.877+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:09:53.172 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:09:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:09:52.877+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:09:53.172 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:09:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:09:52.762+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:09:53.172 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:09:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:09:52.762+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:09:53.172 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:09:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:09:52.762+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:09:53.172 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:09:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:09:52.762+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:09:53.173 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:09:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:09:52.963+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:52.963918+0000) 2024-02-20T09:09:53.173 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:09:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:09:53.169+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:09:53.173 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:09:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:09:53.169+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:09:53.173 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:09:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:09:53.169+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:09:53.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:09:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:09:53.679+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:53.679471+0000) 2024-02-20T09:09:54.193 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:09:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:09:53.963+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:53.964019+0000) 2024-02-20T09:09:54.194 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:09:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:09:54.192+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:09:54.194 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:09:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:09:53.847+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:53.847769+0000) 2024-02-20T09:09:54.194 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:09:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:09:53.897+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:09:54.194 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:09:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:09:53.897+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:09:54.194 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:09:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:09:53.897+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:09:54.195 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:09:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:09:53.809+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:09:54.195 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:09:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:09:53.809+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:09:54.195 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:09:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:09:53.809+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:09:54.195 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:09:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:09:53.809+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:09:54.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:09:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:09:54.192+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:09:54.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:09:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:09:54.192+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:09:54.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:09:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:09:54.680+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:54.680290+0000) 2024-02-20T09:09:55.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:09:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:09:54.963+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:54.964172+0000) 2024-02-20T09:09:55.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:09:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:09:55.239+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:09:55.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:09:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:09:55.239+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:09:55.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:09:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:09:54.847+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:54.847948+0000) 2024-02-20T09:09:55.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:09:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:09:54.858+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:09:55.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:09:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:09:54.858+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:09:55.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:09:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:09:54.858+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:09:55.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:09:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:09:54.788+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:09:55.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:09:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:09:54.788+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:09:55.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:09:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:09:54.788+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:09:55.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:09:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:09:54.788+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:09:55.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:09:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:09:55.239+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:09:55.742 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:09:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:09:55.560+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 60327 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:09:55.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:09:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:09:55.554+0000 7f8a6006f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:55.554782+0000) 2024-02-20T09:09:55.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:09:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:09:55.553+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:55.554217+0000) 2024-02-20T09:09:55.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:09:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:09:55.556+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:55.557219+0000) 2024-02-20T09:09:55.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:09:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:09:55.680+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:55.680583+0000) 2024-02-20T09:09:56.221 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:09:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:09:55.964+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:55.964324+0000) 2024-02-20T09:09:56.222 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:09:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:09:56.220+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:09:56.222 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:09:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:09:55.788+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:09:56.222 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:09:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:09:55.788+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:09:56.222 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:09:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:09:55.788+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:09:56.222 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:09:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:09:55.788+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:09:56.223 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:09:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:09:55.846+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:09:56.223 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:09:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:09:55.846+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:09:56.223 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:09:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:09:55.846+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:09:56.223 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:09:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:09:55.847+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:55.848126+0000) 2024-02-20T09:09:56.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:09:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:09:56.220+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:09:56.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:09:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:09:56.220+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:09:56.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:09:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:09:56.683+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:56.683375+0000) 2024-02-20T09:09:57.218 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:09:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:09:56.964+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:56.964475+0000) 2024-02-20T09:09:57.219 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:09:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:09:57.217+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:09:57.219 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:09:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:09:56.834+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:09:57.219 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:09:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:09:56.834+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:09:57.219 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:09:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:09:56.834+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:09:57.220 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:09:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:09:56.834+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:09:57.220 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:09:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:09:56.817+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:09:57.220 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:09:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:09:56.817+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:09:57.220 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:09:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:09:56.817+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:09:57.220 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:09:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:09:56.847+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:56.848234+0000) 2024-02-20T09:09:57.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:09:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:09:57.217+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:09:57.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:09:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:09:57.217+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:09:57.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:09:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:09:57.686+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:57.686215+0000) 2024-02-20T09:09:58.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:09:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:09:57.964+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:57.964596+0000) 2024-02-20T09:09:58.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:09:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:09:57.867+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:09:58.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:09:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:09:57.867+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:09:58.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:09:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:09:57.867+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:09:58.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:09:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:09:57.868+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:09:58.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:09:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:09:57.848+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:57.848436+0000) 2024-02-20T09:09:58.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:09:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:09:57.848+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:09:58.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:09:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:09:57.848+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:09:58.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:09:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:09:57.848+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:09:58.551 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:09:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:09:58.245+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:09:58.552 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:09:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:09:58.245+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:09:58.552 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:09:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:09:58.245+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:09:58.838 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:09:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:09:58.688+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:58.689128+0000) 2024-02-20T09:09:59.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:09:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:09:58.964+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:58.964741+0000) 2024-02-20T09:09:59.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:09:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:09:58.879+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:09:59.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:09:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:09:58.879+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:09:59.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:09:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:09:58.879+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:09:59.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:09:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:09:58.880+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:09:59.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:09:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:09:58.848+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:58.848581+0000) 2024-02-20T09:09:59.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:09:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:09:58.848+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:09:59.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:09:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:09:58.848+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:09:59.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:09:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:09:58.848+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:09:59.551 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:09:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:09:59.282+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:09:59.551 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:09:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:09:59.282+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:09:59.552 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:09:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:09:59.282+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:09:59.833 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:09:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:09:59.689+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:59.689918+0000) 2024-02-20T09:09:59.833 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:09:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:09:59.830+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:09:59.833 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:09:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:09:59.830+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:09:59.834 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:09:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:09:59.830+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:09:59.834 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:09:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:09:59.830+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:10:00.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:09:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:09:59.963+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:59.964875+0000) 2024-02-20T09:10:00.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:09:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:09:59.847+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:09:59.848745+0000) 2024-02-20T09:10:00.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:09:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:09:59.855+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:10:00.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:09:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:09:59.855+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:10:00.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:09:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:09:59.855+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:10:00.551 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:10:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:10:00.284+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:10:00.552 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:10:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:10:00.284+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:10:00.552 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:10:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:10:00.284+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:10:00.839 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:10:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:10:00.565+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 60399 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:10:00.839 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:10:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:10:00.560+0000 7f8a5f86e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:00.562313+0000) 2024-02-20T09:10:00.839 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:10:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:10:00.563+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:00.565125+0000) 2024-02-20T09:10:00.839 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:10:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:10:00.688+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:00.690127+0000) 2024-02-20T09:10:00.840 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:10:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:10:00.560+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:00.562375+0000) 2024-02-20T09:10:01.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:10:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:10:00.963+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:00.964993+0000) 2024-02-20T09:10:01.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:10:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:10:00.845+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:10:01.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:10:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:10:00.845+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:10:01.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:10:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:10:00.845+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:10:01.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:10:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:10:00.845+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:10:01.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:10:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:10:00.847+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:00.848878+0000) 2024-02-20T09:10:01.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:10:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:10:00.856+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:10:01.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:10:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:10:00.856+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:10:01.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:10:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:10:00.856+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:10:01.693 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:10:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:10:01.302+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:10:01.693 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:10:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:10:01.302+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:10:01.694 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:10:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:10:01.302+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:10:01.966 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:10:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:10:01.847+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:01.848981+0000) 2024-02-20T09:10:01.966 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:10:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:10:01.857+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:10:01.966 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:10:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:10:01.858+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:10:01.967 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:10:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:10:01.858+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:10:01.967 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:10:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:10:01.690+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:01.692350+0000) 2024-02-20T09:10:01.967 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:10:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:10:01.855+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:10:01.967 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:10:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:10:01.855+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:10:01.967 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:10:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:10:01.855+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:10:01.968 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:10:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:10:01.855+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:10:02.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:10:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:10:01.963+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:01.965119+0000) 2024-02-20T09:10:02.696 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:10:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:10:02.335+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:10:02.696 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:10:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:10:02.335+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:10:02.696 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:10:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:10:02.335+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:10:02.966 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:10:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:10:02.693+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:02.694657+0000) 2024-02-20T09:10:02.966 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:10:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:10:02.809+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:10:02.967 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:10:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:10:02.809+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:10:02.967 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:10:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:10:02.809+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:10:02.967 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:10:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:10:02.809+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:10:02.967 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:10:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:10:02.829+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:10:02.967 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:10:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:10:02.829+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:10:02.967 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:10:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:10:02.829+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:10:02.967 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:10:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:10:02.847+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:02.849122+0000) 2024-02-20T09:10:03.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:10:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:10:02.963+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:02.965227+0000) 2024-02-20T09:10:03.698 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:10:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:10:03.325+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:10:03.698 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:10:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:10:03.325+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:10:03.698 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:10:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:10:03.325+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:10:03.966 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:10:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:10:03.695+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:03.696752+0000) 2024-02-20T09:10:03.967 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:10:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:10:03.820+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:10:03.967 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:10:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:10:03.820+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:10:03.967 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:10:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:10:03.820+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:10:03.967 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:10:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:10:03.820+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:10:03.967 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:10:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:10:03.847+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:03.849236+0000) 2024-02-20T09:10:03.967 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:10:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:10:03.867+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:10:03.968 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:10:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:10:03.867+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:10:03.968 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:10:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:10:03.867+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:10:04.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:10:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:10:03.963+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:03.965333+0000) 2024-02-20T09:10:04.555 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:10:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:10:04.293+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:10:04.555 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:10:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:10:04.293+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:10:04.555 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:10:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:10:04.293+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:10:04.839 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:10:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:10:04.696+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:04.697747+0000) 2024-02-20T09:10:05.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:10:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:10:04.964+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:04.965459+0000) 2024-02-20T09:10:05.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:10:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:10:04.847+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:04.849372+0000) 2024-02-20T09:10:05.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:10:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:10:04.853+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:10:05.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:10:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:10:04.853+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:10:05.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:10:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:10:04.853+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:10:05.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:10:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:10:04.854+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:10:05.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:10:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:10:04.854+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:10:05.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:10:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:10:04.854+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:10:05.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:10:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:10:04.854+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:10:05.555 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:10:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:10:05.269+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:10:05.555 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:10:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:10:05.269+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:10:05.555 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:10:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:10:05.269+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:10:05.812 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:10:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:10:05.572+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 60472 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:10:05.813 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:10:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:10:05.567+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:05.568778+0000) 2024-02-20T09:10:05.813 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:10:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:10:05.570+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:05.571875+0000) 2024-02-20T09:10:05.814 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:10:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:10:05.696+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:05.698032+0000) 2024-02-20T09:10:05.814 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:10:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:10:05.567+0000 7f8a6006f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:05.569168+0000) 2024-02-20T09:10:05.814 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:10:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:10:05.810+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:10:05.814 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:10:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:10:05.810+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:10:05.814 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:10:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:10:05.810+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:10:06.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:10:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:10:05.848+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:05.849514+0000) 2024-02-20T09:10:06.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:10:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:10:05.964+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:05.965575+0000) 2024-02-20T09:10:06.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:10:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:10:05.857+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:10:06.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:10:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:10:05.857+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:10:06.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:10:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:10:05.857+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:10:06.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:10:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:10:05.857+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:10:06.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:10:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:10:06.291+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:10:06.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:10:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:10:06.291+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:10:06.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:10:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:10:06.291+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:10:06.791 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:10:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:10:06.698+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:06.700080+0000) 2024-02-20T09:10:07.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:10:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:10:06.964+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:06.965674+0000) 2024-02-20T09:10:07.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:10:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:10:06.813+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:10:07.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:10:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:10:06.813+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:10:07.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:10:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:10:06.813+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:10:07.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:10:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:10:06.813+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:10:07.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:10:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:10:06.788+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:10:07.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:10:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:10:06.788+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:10:07.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:10:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:10:06.788+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:10:07.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:10:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:10:06.848+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:06.849613+0000) 2024-02-20T09:10:07.557 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:10:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:10:07.289+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:10:07.557 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:10:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:10:07.289+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:10:07.558 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:10:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:10:07.289+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:10:07.839 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:10:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:10:07.801+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:10:07.840 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:10:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:10:07.801+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:10:07.840 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:10:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:10:07.801+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:10:07.840 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:10:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:10:07.700+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:07.702113+0000) 2024-02-20T09:10:07.840 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:10:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:10:07.801+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:10:07.840 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:10:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:10:07.801+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:10:07.840 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:10:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:10:07.801+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:10:07.840 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:10:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:10:07.801+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:10:08.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:10:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:10:07.848+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:07.849731+0000) 2024-02-20T09:10:08.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:10:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:10:07.964+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:07.965821+0000) 2024-02-20T09:10:08.706 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:10:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:10:08.317+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:10:08.706 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:10:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:10:08.317+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:10:08.706 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:10:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:10:08.317+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:10:08.967 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:10:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:10:08.761+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:10:09.021 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:10:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:10:08.761+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:10:09.022 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:10:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:10:08.761+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:10:09.022 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:10:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:10:08.848+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:08.849885+0000) 2024-02-20T09:10:09.022 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:10:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:10:08.703+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:08.704665+0000) 2024-02-20T09:10:09.022 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:10:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:10:08.810+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:10:09.022 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:10:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:10:08.810+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:10:09.022 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:10:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:10:08.810+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:10:09.022 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:10:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:10:08.810+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:10:09.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:10:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:10:08.964+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:08.965932+0000) 2024-02-20T09:10:09.558 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:10:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:10:09.293+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:10:09.559 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:10:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:10:09.293+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:10:09.559 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:10:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:10:09.293+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:10:09.840 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:10:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:10:09.798+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:10:09.840 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:10:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:10:09.798+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:10:09.840 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:10:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:10:09.798+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:10:09.841 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:10:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:10:09.704+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:09.705474+0000) 2024-02-20T09:10:09.841 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:10:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:10:09.763+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:10:09.841 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:10:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:10:09.763+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:10:09.841 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:10:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:10:09.763+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:10:09.841 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:10:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:10:09.763+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:10:10.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:10:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:10:09.848+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:09.850011+0000) 2024-02-20T09:10:10.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:10:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:10:09.964+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:09.966048+0000) 2024-02-20T09:10:10.558 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:10:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:10:10.268+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:10:10.559 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:10:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:10:10.268+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:10:10.559 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:10:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:10:10.268+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:10:10.820 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:10:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:10:10.579+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 60544 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:10:10.821 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:10:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:10:10.574+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:10.575502+0000) 2024-02-20T09:10:10.821 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:10:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:10:10.573+0000 7f8a5f86e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:10.575130+0000) 2024-02-20T09:10:10.821 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:10:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:10:10.818+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:10:10.821 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:10:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:10:10.818+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:10:10.821 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:10:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:10:10.818+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:10:10.822 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:10:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:10:10.576+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:10.577738+0000) 2024-02-20T09:10:10.822 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:10:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:10:10.704+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:10.705677+0000) 2024-02-20T09:10:10.822 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:10:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:10:10.799+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:10:10.822 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:10:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:10:10.799+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:10:10.822 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:10:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:10:10.799+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:10:10.822 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:10:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:10:10.799+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:10:11.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:10:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:10:10.848+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:10.850129+0000) 2024-02-20T09:10:11.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:10:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:10:10.964+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:10.966165+0000) 2024-02-20T09:10:11.559 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:10:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:10:11.271+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:10:11.559 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:10:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:10:11.271+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:10:11.559 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:10:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:10:11.271+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:10:11.840 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:10:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:10:11.706+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:11.707668+0000) 2024-02-20T09:10:11.840 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:10:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:10:11.780+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:10:11.841 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:10:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:10:11.780+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:10:11.841 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:10:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:10:11.780+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:10:12.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:10:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:10:11.848+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:11.850243+0000) 2024-02-20T09:10:12.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:10:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:10:11.964+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:11.966303+0000) 2024-02-20T09:10:12.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:10:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:10:11.839+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:10:12.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:10:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:10:11.839+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:10:12.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:10:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:10:11.839+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:10:12.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:10:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:10:11.839+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:10:12.561 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:10:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:10:12.283+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:10:12.562 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:10:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:10:12.283+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:10:12.562 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:10:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:10:12.283+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:10:12.827 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:10:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:10:12.708+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:12.709736+0000) 2024-02-20T09:10:13.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:10:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:10:12.965+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:12.966391+0000) 2024-02-20T09:10:13.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:10:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:10:12.837+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:10:13.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:10:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:10:12.837+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:10:13.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:10:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:10:12.837+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:10:13.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:10:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:10:12.837+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:10:13.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:10:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:10:12.824+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:10:13.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:10:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:10:12.824+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:10:13.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:10:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:10:12.824+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:10:13.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:10:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:10:12.849+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:12.850379+0000) 2024-02-20T09:10:13.713 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:10:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:10:13.312+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:10:13.713 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:10:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:10:13.312+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:10:13.713 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:10:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:10:13.312+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:10:13.967 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:10:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:10:13.847+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:10:13.968 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:10:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:10:13.847+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:10:13.968 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:10:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:10:13.847+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:10:13.968 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:10:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:10:13.849+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:13.850518+0000) 2024-02-20T09:10:13.968 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:10:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:10:13.710+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:13.711804+0000) 2024-02-20T09:10:13.968 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:10:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:10:13.809+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:10:13.968 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:10:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:10:13.809+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:10:13.969 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:10:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:10:13.809+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:10:13.969 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:10:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:10:13.810+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:10:14.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:10:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:10:13.965+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:13.966520+0000) 2024-02-20T09:10:14.562 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:10:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:10:14.273+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:10:14.562 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:10:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:10:14.273+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:10:14.563 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:10:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:10:14.273+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:10:14.824 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:10:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:10:14.711+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:14.712778+0000) 2024-02-20T09:10:14.824 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:10:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:10:14.803+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:10:14.824 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:10:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:10:14.803+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:10:14.824 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:10:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:10:14.803+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:10:14.825 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:10:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:10:14.803+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:10:15.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:10:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:10:14.965+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:14.966607+0000) 2024-02-20T09:10:15.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:10:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:10:14.821+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:10:15.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:10:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:10:14.821+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:10:15.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:10:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:10:14.821+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:10:15.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:10:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:10:14.849+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:14.850648+0000) 2024-02-20T09:10:15.562 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:10:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:10:15.273+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:10:15.563 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:10:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:10:15.273+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:10:15.563 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:10:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:10:15.273+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:10:15.841 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:10:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:10:15.580+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:15.582122+0000) 2024-02-20T09:10:15.841 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:10:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:10:15.586+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 60617 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:10:15.841 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:10:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:10:15.580+0000 7f8a6006f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:15.581757+0000) 2024-02-20T09:10:15.842 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:10:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:10:15.584+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:15.585678+0000) 2024-02-20T09:10:15.842 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:10:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:10:15.711+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:15.713036+0000) 2024-02-20T09:10:15.842 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:10:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:10:15.772+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:10:15.842 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:10:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:10:15.772+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:10:15.842 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:10:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:10:15.772+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:10:15.842 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:10:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:10:15.772+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:10:16.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:10:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:10:15.965+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:15.966715+0000) 2024-02-20T09:10:16.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:10:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:10:15.849+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:15.850760+0000) 2024-02-20T09:10:16.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:10:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:10:15.855+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:10:16.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:10:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:10:15.855+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:10:16.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:10:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:10:15.855+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:10:16.563 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:10:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:10:16.248+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:10:16.563 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:10:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:10:16.248+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:10:16.563 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:10:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:10:16.248+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:10:16.828 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:10:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:10:16.713+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:16.715005+0000) 2024-02-20T09:10:16.829 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:10:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:10:16.795+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:10:16.829 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:10:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:10:16.795+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:10:16.829 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:10:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:10:16.795+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:10:16.830 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:10:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:10:16.795+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:10:17.229 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:10:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:10:16.965+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:16.966788+0000) 2024-02-20T09:10:17.229 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:10:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:10:17.226+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:10:17.229 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:10:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:10:17.226+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:10:17.229 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:10:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:10:16.825+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:10:17.230 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:10:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:10:16.825+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:10:17.230 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:10:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:10:16.825+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:10:17.230 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:10:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:10:16.849+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:16.850901+0000) 2024-02-20T09:10:17.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:10:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:10:17.226+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:10:17.798 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:10:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:10:17.715+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:17.717148+0000) 2024-02-20T09:10:18.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:10:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:10:17.965+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:17.966994+0000) 2024-02-20T09:10:18.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:10:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:10:17.795+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:10:18.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:10:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:10:17.795+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:10:18.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:10:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:10:17.795+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:10:18.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:10:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:10:17.849+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:17.851049+0000) 2024-02-20T09:10:18.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:10:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:10:17.808+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:10:18.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:10:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:10:17.808+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:10:18.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:10:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:10:17.808+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:10:18.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:10:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:10:17.809+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:10:18.567 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:10:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:10:18.246+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:10:18.567 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:10:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:10:18.246+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:10:18.567 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:10:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:10:18.246+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:10:18.827 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:10:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:10:18.768+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:10:18.828 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:10:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:10:18.768+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:10:18.828 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:10:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:10:18.768+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:10:18.828 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:10:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:10:18.717+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:18.719361+0000) 2024-02-20T09:10:18.828 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:10:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:10:18.825+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:10:18.828 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:10:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:10:18.825+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:10:18.828 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:10:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:10:18.825+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:10:18.828 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:10:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:10:18.825+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:10:19.204 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:10:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:10:18.849+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:18.851202+0000) 2024-02-20T09:10:19.205 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:10:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:10:18.965+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:18.967182+0000) 2024-02-20T09:10:19.205 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:10:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:10:19.202+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:10:19.205 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:10:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:10:19.202+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:10:19.205 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:10:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:10:19.202+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:10:19.762 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:10:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:10:19.718+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:19.720016+0000) 2024-02-20T09:10:20.166 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:10:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:10:19.965+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:19.967352+0000) 2024-02-20T09:10:20.167 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:10:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:10:19.760+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:10:20.167 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:10:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:10:19.760+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:10:20.167 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:10:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:10:19.760+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:10:20.167 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:10:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:10:19.849+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:19.851345+0000) 2024-02-20T09:10:20.167 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:10:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:10:19.826+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:10:20.167 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:10:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:10:19.826+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:10:20.167 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:10:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:10:19.826+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:10:20.168 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:10:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:10:19.826+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:10:20.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:10:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:10:20.163+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:10:20.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:10:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:10:20.163+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:10:20.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:10:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:10:20.163+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:10:20.842 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:10:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:10:20.592+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 60689 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:10:20.842 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:10:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:10:20.587+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:20.588899+0000) 2024-02-20T09:10:20.843 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:10:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:10:20.590+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:20.591702+0000) 2024-02-20T09:10:20.843 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:10:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:10:20.718+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:20.720203+0000) 2024-02-20T09:10:20.843 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:10:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:10:20.587+0000 7f8a6006f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:20.589022+0000) 2024-02-20T09:10:20.843 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:10:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:10:20.720+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:10:20.843 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:10:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:10:20.720+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:10:20.843 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:10:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:10:20.720+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:10:21.136 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:10:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:10:20.850+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:20.851478+0000) 2024-02-20T09:10:21.137 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:10:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:10:20.966+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:20.967534+0000) 2024-02-20T09:10:21.137 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:10:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:10:20.860+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:10:21.137 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:10:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:10:20.860+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:10:21.138 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:10:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:10:20.860+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:10:21.138 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:10:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:10:20.861+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:10:21.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:10:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:10:21.134+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:10:21.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:10:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:10:21.134+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:10:21.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:10:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:10:21.134+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:10:21.841 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:10:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:10:21.721+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:21.722671+0000) 2024-02-20T09:10:21.842 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:10:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:10:21.687+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:10:21.842 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:10:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:10:21.687+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:10:21.842 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:10:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:10:21.687+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:10:22.093 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:10:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:10:21.850+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:21.851580+0000) 2024-02-20T09:10:22.093 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:10:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:10:21.966+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:21.967648+0000) 2024-02-20T09:10:22.093 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:10:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:10:22.090+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:10:22.094 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:10:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:10:22.090+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:10:22.094 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:10:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:10:21.859+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:10:22.094 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:10:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:10:21.859+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:10:22.094 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:10:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:10:21.859+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:10:22.094 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:10:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:10:21.859+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:10:22.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:10:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:10:22.090+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:10:22.841 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:10:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:10:22.724+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:22.725460+0000) 2024-02-20T09:10:22.842 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:10:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:10:22.724+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:10:22.842 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:10:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:10:22.724+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:10:22.842 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:10:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:10:22.724+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:10:23.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:10:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:10:22.850+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:22.851727+0000) 2024-02-20T09:10:23.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:10:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:10:22.851+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:10:23.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:10:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:10:22.851+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:10:23.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:10:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:10:22.851+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:10:23.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:10:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:10:22.851+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:10:23.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:10:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:10:22.966+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:22.967765+0000) 2024-02-20T09:10:23.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:10:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:10:23.044+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:10:23.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:10:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:10:23.044+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:10:23.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:10:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:10:23.044+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:10:23.757 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:10:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:10:23.727+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:23.729108+0000) 2024-02-20T09:10:24.010 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:10:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:10:23.755+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:10:24.010 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:10:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:10:23.755+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:10:24.010 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:10:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:10:23.755+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:10:24.010 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:10:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:10:23.850+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:23.851866+0000) 2024-02-20T09:10:24.011 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:10:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:10:23.841+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:10:24.011 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:10:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:10:23.841+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:10:24.011 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:10:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:10:23.841+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:10:24.011 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:10:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:10:23.841+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:10:24.012 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:10:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:10:23.966+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:23.967874+0000) 2024-02-20T09:10:24.012 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:10:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:10:24.007+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:10:24.012 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:10:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:10:24.007+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:10:24.012 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:10:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:10:24.007+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:10:24.745 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:10:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:10:24.728+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:24.730174+0000) 2024-02-20T09:10:25.059 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:10:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:10:24.829+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:10:25.059 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:10:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:10:24.829+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:10:25.059 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:10:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:10:24.829+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:10:25.059 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:10:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:10:24.829+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:10:25.060 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:10:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:10:24.742+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:10:25.060 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:10:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:10:24.742+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:10:25.060 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:10:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:10:24.743+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:10:25.060 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:10:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:10:24.850+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:24.851974+0000) 2024-02-20T09:10:25.060 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:10:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:10:24.966+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:24.968009+0000) 2024-02-20T09:10:25.061 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:10:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:10:25.056+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:10:25.061 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:10:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:10:25.056+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:10:25.061 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:10:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:10:25.056+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:10:25.745 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:10:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:10:25.594+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:25.595618+0000) 2024-02-20T09:10:25.746 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:10:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:10:25.600+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 60762 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:10:25.746 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:10:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:10:25.596+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:25.597657+0000) 2024-02-20T09:10:25.746 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:10:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:10:25.729+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:25.730419+0000) 2024-02-20T09:10:25.747 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:10:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:10:25.594+0000 7f8a6006f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:25.596008+0000) 2024-02-20T09:10:25.747 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:10:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:10:25.742+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:10:25.747 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:10:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:10:25.742+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:10:25.747 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:10:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:10:25.742+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:10:26.050 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:10:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:10:25.850+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:25.852175+0000) 2024-02-20T09:10:26.051 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:10:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:10:25.834+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:10:26.051 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:10:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:10:25.834+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:10:26.051 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:10:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:10:25.834+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:10:26.051 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:10:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:10:25.834+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:10:26.051 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:10:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:10:25.966+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:25.968206+0000) 2024-02-20T09:10:26.051 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:10:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:10:26.047+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:10:26.052 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:10:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:10:26.047+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:10:26.052 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:10:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:10:26.047+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:10:26.756 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:10:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:10:26.732+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:26.732564+0000) 2024-02-20T09:10:27.081 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:10:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:10:26.801+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:10:27.081 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:10:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:10:26.801+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:10:27.081 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:10:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:10:26.801+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:10:27.082 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:10:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:10:26.801+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:10:27.082 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:10:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:10:26.754+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:10:27.082 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:10:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:10:26.754+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:10:27.082 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:10:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:10:26.754+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:10:27.082 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:10:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:10:26.850+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:26.852325+0000) 2024-02-20T09:10:27.083 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:10:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:10:26.966+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:26.968378+0000) 2024-02-20T09:10:27.083 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:10:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:10:27.078+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:10:27.083 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:10:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:10:27.078+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:10:27.751 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:10:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:10:27.734+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:27.734681+0000) 2024-02-20T09:10:27.751 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:10:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:10:27.078+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:10:28.069 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:10:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:10:27.967+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:27.968536+0000) 2024-02-20T09:10:28.069 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:10:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:10:28.067+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:10:28.070 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:10:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:10:27.748+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:10:28.070 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:10:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:10:27.748+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:10:28.070 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:10:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:10:27.748+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:10:28.070 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:10:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:10:27.851+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:27.852501+0000) 2024-02-20T09:10:28.071 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:10:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:10:27.753+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:10:28.071 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:10:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:10:27.753+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:10:28.071 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:10:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:10:27.753+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:10:28.071 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:10:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:10:27.753+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:10:28.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:10:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:10:28.067+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:10:28.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:10:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:10:28.067+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:10:28.751 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:10:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:10:28.743+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:10:28.751 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:10:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:10:28.743+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:10:28.751 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:10:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:10:28.743+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:10:28.752 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:10:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:10:28.736+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:28.736894+0000) 2024-02-20T09:10:28.752 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:10:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:10:28.748+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:10:28.752 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:10:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:10:28.748+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:10:28.752 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:10:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:10:28.748+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:10:28.752 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:10:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:10:28.748+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:10:29.095 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:10:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:10:28.852+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:28.852626+0000) 2024-02-20T09:10:29.096 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:10:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:10:28.968+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:28.968692+0000) 2024-02-20T09:10:29.096 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:10:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:10:29.092+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:10:29.096 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:10:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:10:29.092+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:10:29.096 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:10:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:10:29.092+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:10:29.779 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:10:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:10:29.736+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:29.737534+0000) 2024-02-20T09:10:29.780 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:10:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:10:29.777+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:10:29.780 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:10:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:10:29.713+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:10:29.780 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:10:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:10:29.713+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:10:29.780 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:10:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:10:29.713+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:10:30.072 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:10:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:10:29.852+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:29.852733+0000) 2024-02-20T09:10:30.073 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:10:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:10:29.777+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:10:30.073 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:10:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:10:29.777+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:10:30.073 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:10:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:10:29.777+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:10:30.074 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:10:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:10:29.967+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:29.968807+0000) 2024-02-20T09:10:30.074 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:10:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:10:30.070+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:10:30.074 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:10:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:10:30.070+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:10:30.074 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:10:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:10:30.070+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:10:30.682 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:10:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:10:30.607+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 60834 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:10:30.682 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:10:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:10:30.601+0000 7f8a5f86e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:30.603238+0000) 2024-02-20T09:10:30.683 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:10:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:10:30.604+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:30.606135+0000) 2024-02-20T09:10:30.683 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:10:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:10:30.601+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:30.602774+0000) 2024-02-20T09:10:30.970 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:10:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:10:30.680+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:10:30.970 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:10:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:10:30.680+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:10:30.971 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:10:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:10:30.680+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:10:30.971 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:10:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:10:30.851+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:30.852881+0000) 2024-02-20T09:10:30.971 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:10:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:10:30.737+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:30.737707+0000) 2024-02-20T09:10:30.971 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:10:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:10:30.760+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:10:30.971 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:10:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:10:30.760+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:10:30.971 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:10:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:10:30.760+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:10:30.971 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:10:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:10:30.760+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:10:31.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:10:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:10:30.967+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:30.968911+0000) 2024-02-20T09:10:31.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:10:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:10:31.078+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:10:31.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:10:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:10:31.078+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:10:31.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:10:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:10:31.078+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:10:31.725 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:10:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:10:31.659+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:10:31.725 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:10:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:10:31.659+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:10:31.725 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:10:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:10:31.659+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:10:31.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:10:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:10:31.852+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:31.853000+0000) 2024-02-20T09:10:31.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:10:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:10:31.968+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:31.969094+0000) 2024-02-20T09:10:31.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:10:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:10:31.723+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:10:31.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:10:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:10:31.723+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:10:31.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:10:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:10:31.723+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:10:31.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:10:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:10:31.723+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:10:31.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:10:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:10:31.739+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:31.739744+0000) 2024-02-20T09:10:32.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:10:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:10:32.090+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:10:32.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:10:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:10:32.090+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:10:32.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:10:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:10:32.090+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:10:32.843 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:10:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:10:32.698+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:10:32.843 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:10:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:10:32.698+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:10:32.843 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:10:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:10:32.698+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:10:32.843 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:10:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:10:32.741+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:32.741789+0000) 2024-02-20T09:10:32.844 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:10:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:10:32.743+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:10:32.844 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:10:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:10:32.743+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:10:32.844 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:10:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:10:32.743+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:10:32.844 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:10:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:10:32.744+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:10:33.131 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:10:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:10:32.852+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:32.853199+0000) 2024-02-20T09:10:33.131 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:10:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:10:32.968+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:32.969306+0000) 2024-02-20T09:10:33.131 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:10:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:10:33.129+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:10:33.131 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:10:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:10:33.129+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:10:33.131 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:10:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:10:33.129+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:10:33.765 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:10:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:10:33.743+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:33.743894+0000) 2024-02-20T09:10:33.765 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:10:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:10:33.764+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:10:33.766 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:10:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:10:33.649+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:10:33.766 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:10:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:10:33.649+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:10:33.766 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:10:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:10:33.649+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:03.284314+0000 (oldest deadline 2024-02-20T09:00:23.284314+0000) 2024-02-20T09:10:34.121 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:10:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:10:33.852+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:33.853369+0000) 2024-02-20T09:10:34.121 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:10:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:10:33.764+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:10:34.121 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:10:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:10:33.764+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:10:34.121 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:10:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:10:33.764+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:10:34.122 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:10:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:10:33.969+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:33.969451+0000) 2024-02-20T09:10:34.122 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:10:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:10:34.119+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:10:34.122 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:10:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:10:34.119+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:10:34.122 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:10:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:10:34.119+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:10:35.166 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:10:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:10:34.853+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:34.853550+0000) 2024-02-20T09:10:35.167 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:10:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:10:34.969+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:34.969578+0000) 2024-02-20T09:10:35.167 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:10:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:10:35.165+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:10:35.167 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:10:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:10:35.165+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:10:35.167 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:10:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:10:35.165+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:10:35.168 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:10:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:10:34.744+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:34.744623+0000) 2024-02-20T09:10:35.168 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:10:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:10:34.792+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:10:35.168 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:10:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:10:34.792+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:10:35.168 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:10:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:10:34.792+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:10:35.168 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:10:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:10:34.793+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:10:35.792 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:10:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:10:35.615+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 60907 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:10:35.792 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:10:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:10:35.609+0000 7f8a5f86e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:35.609862+0000) 2024-02-20T09:10:35.793 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:10:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:10:35.609+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:35.610002+0000) 2024-02-20T09:10:35.793 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:10:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:10:35.612+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:35.612427+0000) 2024-02-20T09:10:35.793 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:10:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:10:35.744+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:35.744906+0000) 2024-02-20T09:10:35.793 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:10:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:10:35.790+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:10:35.793 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:10:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:10:35.790+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:10:35.794 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:10:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:10:35.790+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:10:35.794 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:10:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:10:35.790+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:10:36.169 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:10:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:10:35.853+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:35.853705+0000) 2024-02-20T09:10:36.170 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:10:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:10:35.969+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:35.969691+0000) 2024-02-20T09:10:36.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:10:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:10:36.168+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:10:36.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:10:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:10:36.168+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:10:36.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:10:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:10:36.168+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:10:37.135 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:10:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:10:36.853+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:36.853831+0000) 2024-02-20T09:10:37.135 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:10:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:10:36.746+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:36.747063+0000) 2024-02-20T09:10:37.135 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:10:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:10:36.826+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:10:37.135 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:10:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:10:36.826+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:10:37.136 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:10:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:10:36.826+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:10:37.136 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:10:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:10:36.826+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:10:37.136 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:10:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:10:36.969+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:36.969770+0000) 2024-02-20T09:10:37.136 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:10:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:10:37.133+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:10:37.136 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:10:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:10:37.133+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:10:37.136 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:10:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:10:37.133+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:10:38.172 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:10:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:10:37.853+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:37.853953+0000) 2024-02-20T09:10:38.172 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:10:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:10:37.748+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:37.749137+0000) 2024-02-20T09:10:38.172 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:10:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:10:37.793+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:10:38.173 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:10:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:10:37.793+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:10:38.173 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:10:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:10:37.793+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:10:38.173 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:10:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:10:37.793+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:10:38.173 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:10:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:10:37.969+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:37.969908+0000) 2024-02-20T09:10:38.173 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:10:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:10:38.170+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:10:38.173 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:10:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:10:38.170+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:10:38.173 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:10:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:10:38.170+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:10:39.162 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:10:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:10:38.853+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:38.854057+0000) 2024-02-20T09:10:39.163 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:10:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:10:38.969+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:38.970042+0000) 2024-02-20T09:10:39.163 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:10:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:10:39.160+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:10:39.163 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:10:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:10:39.160+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:10:39.163 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:10:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:10:39.160+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:10:39.163 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:10:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:10:38.750+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:38.751190+0000) 2024-02-20T09:10:39.163 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:10:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:10:38.811+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:10:39.164 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:10:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:10:38.811+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:10:39.164 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:10:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:10:38.811+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:10:39.164 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:10:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:10:38.811+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:10:40.167 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:10:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:10:39.853+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:39.854206+0000) 2024-02-20T09:10:40.168 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:10:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:10:39.969+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:39.970201+0000) 2024-02-20T09:10:40.168 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:10:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:10:40.166+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:10:40.169 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:10:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:10:39.751+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:39.752075+0000) 2024-02-20T09:10:40.169 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:10:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:10:39.844+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:10:40.169 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:10:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:10:39.844+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:10:40.169 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:10:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:10:39.844+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:10:40.169 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:10:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:10:39.844+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:10:40.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:10:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:10:40.166+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:10:40.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:10:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:10:40.166+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:10:40.753 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:10:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:10:40.622+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 60979 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:10:40.754 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:10:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:10:40.616+0000 7f8a5f86e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:40.616889+0000) 2024-02-20T09:10:40.754 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:10:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:10:40.620+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:40.620574+0000) 2024-02-20T09:10:40.754 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:10:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:10:40.616+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:40.616969+0000) 2024-02-20T09:10:41.192 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:10:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:10:40.853+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:40.854318+0000) 2024-02-20T09:10:41.192 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:10:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:10:40.969+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:40.970316+0000) 2024-02-20T09:10:41.192 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:10:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:10:41.190+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:10:41.193 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:10:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:10:40.751+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:40.752370+0000) 2024-02-20T09:10:41.193 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:10:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:10:40.804+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:10:41.193 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:10:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:10:40.804+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:10:41.193 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:10:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:10:40.804+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:10:41.193 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:10:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:10:40.804+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:10:41.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:10:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:10:41.190+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:10:41.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:10:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:10:41.190+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:10:42.231 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:10:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:10:41.854+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:41.854414+0000) 2024-02-20T09:10:42.232 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:10:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:10:41.970+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:41.970462+0000) 2024-02-20T09:10:42.232 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:10:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:10:42.229+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:10:42.232 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:10:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:10:41.754+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:41.754464+0000) 2024-02-20T09:10:42.232 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:10:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:10:41.816+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:10:42.232 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:10:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:10:41.816+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:10:42.232 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:10:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:10:41.816+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:10:42.233 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:10:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:10:41.816+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:10:42.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:10:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:10:42.229+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:10:42.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:10:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:10:42.229+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:10:43.201 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:10:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:10:42.854+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:42.854566+0000) 2024-02-20T09:10:43.202 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:10:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:10:42.970+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:42.970575+0000) 2024-02-20T09:10:43.202 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:10:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:10:43.199+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:10:43.202 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:10:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:10:43.199+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:10:43.202 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:10:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:10:43.200+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:10:43.202 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:10:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:10:42.757+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:42.758003+0000) 2024-02-20T09:10:43.202 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:10:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:10:42.785+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:10:43.202 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:10:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:10:42.785+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:10:43.203 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:10:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:10:42.785+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:10:43.203 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:10:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:10:42.785+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:10:44.197 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:10:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:10:43.970+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:43.970716+0000) 2024-02-20T09:10:44.197 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:10:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:10:43.854+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:43.854677+0000) 2024-02-20T09:10:44.197 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:10:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:10:43.759+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:43.760157+0000) 2024-02-20T09:10:44.198 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:10:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:10:43.792+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:10:44.198 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:10:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:10:43.792+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:10:44.198 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:10:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:10:43.792+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:10:44.198 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:10:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:10:43.792+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:10:44.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:10:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:10:44.195+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:10:44.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:10:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:10:44.195+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:10:44.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:10:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:10:44.195+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:10:45.197 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:10:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:10:44.854+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:44.854827+0000) 2024-02-20T09:10:45.198 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:10:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:10:44.970+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:44.970848+0000) 2024-02-20T09:10:45.198 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:10:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:10:45.195+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:10:45.198 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:10:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:10:44.760+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:44.761258+0000) 2024-02-20T09:10:45.198 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:10:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:10:44.779+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:10:45.198 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:10:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:10:44.779+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:10:45.198 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:10:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:10:44.779+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:10:45.198 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:10:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:10:44.779+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:10:45.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:10:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:10:45.195+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:10:45.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:10:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:10:45.195+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:10:45.750 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:10:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:10:45.629+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 61052 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:10:45.750 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:10:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:10:45.623+0000 7f8a5f86e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:45.624344+0000) 2024-02-20T09:10:45.751 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:10:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:10:45.624+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:45.624406+0000) 2024-02-20T09:10:45.751 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:10:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:10:45.626+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:45.626742+0000) 2024-02-20T09:10:45.751 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:10:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:10:45.744+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:10:45.751 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:10:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:10:45.744+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:10:45.751 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:10:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:10:45.744+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:10:45.751 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:10:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:10:45.744+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:10:46.160 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:10:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:10:45.854+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:45.854938+0000) 2024-02-20T09:10:46.160 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:10:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:10:45.761+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:45.761512+0000) 2024-02-20T09:10:46.161 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:10:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:10:45.970+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:45.970980+0000) 2024-02-20T09:10:46.161 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:10:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:10:46.158+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:10:46.161 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:10:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:10:46.158+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:10:46.161 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:10:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:10:46.158+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:11.199905+0000 (oldest deadline 2024-02-20T09:00:31.199905+0000) 2024-02-20T09:10:47.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:10:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:10:46.854+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:46.855043+0000) 2024-02-20T09:10:47.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:10:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:10:46.970+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:46.971081+0000) 2024-02-20T09:10:47.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:10:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:10:46.763+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:46.763480+0000) 2024-02-20T09:10:47.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:10:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:10:46.789+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:10:47.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:10:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:10:46.789+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:10:47.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:10:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:10:46.789+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:10:47.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:10:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:10:46.789+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:10:48.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:10:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:10:47.854+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:47.855191+0000) 2024-02-20T09:10:48.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:10:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:10:47.970+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:47.971205+0000) 2024-02-20T09:10:48.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:10:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:10:47.765+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:47.765619+0000) 2024-02-20T09:10:48.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:10:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:10:47.839+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:10:48.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:10:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:10:47.839+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:10:48.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:10:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:10:47.839+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:10:48.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:10:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:10:47.839+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:10:49.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:10:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:10:48.854+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:48.855302+0000) 2024-02-20T09:10:49.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:10:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:10:48.970+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:48.971316+0000) 2024-02-20T09:10:49.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:10:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:10:48.767+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:48.767965+0000) 2024-02-20T09:10:49.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:10:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:10:48.842+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:10:49.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:10:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:10:48.842+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:10:49.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:10:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:10:48.842+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:10:49.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:10:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:10:48.842+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:10:50.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:10:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:10:49.855+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:49.855453+0000) 2024-02-20T09:10:50.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:10:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:10:49.971+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:49.971465+0000) 2024-02-20T09:10:50.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:10:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:10:49.768+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:49.768745+0000) 2024-02-20T09:10:50.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:10:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:10:49.890+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:10:50.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:10:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:10:49.890+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:10:50.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:10:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:10:49.890+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:10:50.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:10:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:10:49.890+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:10:50.658 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:10:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:10:50.636+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 61124 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:10:50.658 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:10:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:10:50.631+0000 7f8a5f86e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:50.631534+0000) 2024-02-20T09:10:50.659 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:10:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:10:50.634+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:50.634761+0000) 2024-02-20T09:10:50.659 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:10:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:10:50.631+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:50.631596+0000) 2024-02-20T09:10:50.973 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:10:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:10:50.855+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:50.855573+0000) 2024-02-20T09:10:50.973 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:10:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:10:50.768+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:50.768942+0000) 2024-02-20T09:10:50.974 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:10:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:10:50.867+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:10:50.974 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:10:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:10:50.867+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:10:50.974 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:10:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:10:50.867+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:10:50.974 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:10:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:10:50.867+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:10:51.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:10:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:10:50.971+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:50.971573+0000) 2024-02-20T09:10:52.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:10:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:10:51.855+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:51.855649+0000) 2024-02-20T09:10:52.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:10:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:10:51.971+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:51.971685+0000) 2024-02-20T09:10:52.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:10:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:10:51.771+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:51.771586+0000) 2024-02-20T09:10:52.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:10:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:10:51.886+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:10:52.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:10:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:10:51.886+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:10:52.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:10:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:10:51.886+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:10:52.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:10:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:10:51.886+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:10:53.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:10:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:10:52.855+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:52.855769+0000) 2024-02-20T09:10:53.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:10:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:10:52.970+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:52.971775+0000) 2024-02-20T09:10:53.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:10:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:10:52.773+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:52.773578+0000) 2024-02-20T09:10:53.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:10:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:10:52.850+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:10:53.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:10:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:10:52.850+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:10:53.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:10:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:10:52.850+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:10:53.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:10:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:10:52.850+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:10:54.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:10:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:10:53.854+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:53.855901+0000) 2024-02-20T09:10:54.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:10:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:10:53.970+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:53.971874+0000) 2024-02-20T09:10:54.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:10:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:10:53.774+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:53.775561+0000) 2024-02-20T09:10:54.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:10:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:10:53.870+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:10:54.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:10:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:10:53.870+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:10:54.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:10:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:10:53.870+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:10:54.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:10:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:10:53.870+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:10:55.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:10:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:10:54.854+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:54.856001+0000) 2024-02-20T09:10:55.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:10:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:10:54.970+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:54.972080+0000) 2024-02-20T09:10:55.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:10:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:10:54.774+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:54.776287+0000) 2024-02-20T09:10:55.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:10:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:10:54.913+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:10:55.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:10:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:10:54.913+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:10:55.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:10:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:10:54.913+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:10:55.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:10:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:10:54.913+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:10:55.763 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:10:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:10:55.637+0000 7f8a5f86e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:55.638517+0000) 2024-02-20T09:10:55.764 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:10:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:10:55.639+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:55.641115+0000) 2024-02-20T09:10:55.764 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:10:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:10:55.636+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:55.638178+0000) 2024-02-20T09:10:55.764 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:10:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:10:55.642+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 61197 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:10:56.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:10:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:10:55.854+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:55.856224+0000) 2024-02-20T09:10:56.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:10:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:10:55.970+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:55.972259+0000) 2024-02-20T09:10:56.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:10:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:10:55.775+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:55.776545+0000) 2024-02-20T09:10:56.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:10:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:10:55.943+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:10:56.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:10:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:10:55.943+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:10:56.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:10:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:10:55.943+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:10:56.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:10:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:10:55.943+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:10:57.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:10:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:10:56.854+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:56.856389+0000) 2024-02-20T09:10:57.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:10:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:10:56.970+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:56.972365+0000) 2024-02-20T09:10:57.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:10:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:10:56.778+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:56.780147+0000) 2024-02-20T09:10:57.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:10:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:10:56.905+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:10:57.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:10:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:10:56.905+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:10:57.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:10:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:10:56.905+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:10:57.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:10:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:10:56.905+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:10:58.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:10:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:10:57.971+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:57.972474+0000) 2024-02-20T09:10:58.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:10:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:10:57.855+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:57.856523+0000) 2024-02-20T09:10:58.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:10:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:10:57.782+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:57.783719+0000) 2024-02-20T09:10:58.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:10:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:10:57.954+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:10:58.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:10:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:10:57.954+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:10:58.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:10:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:10:57.954+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:00:27.896469+0000 (oldest deadline 2024-02-20T09:00:47.896469+0000) 2024-02-20T09:10:58.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:10:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:10:57.954+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:10:58.788 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:10:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:10:58.605+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:10:58.788 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:10:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:10:58.605+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:10:58.789 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:10:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:10:58.605+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:10:59.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:10:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:10:58.855+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:58.856674+0000) 2024-02-20T09:10:59.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:10:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:10:58.971+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:58.972631+0000) 2024-02-20T09:10:59.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:10:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:10:58.785+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:58.787283+0000) 2024-02-20T09:10:59.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:10:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:10:58.975+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:10:59.789 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:10:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:10:59.641+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:10:59.790 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:10:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:10:59.641+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:10:59.790 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:10:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:10:59.641+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:11:00.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:10:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:10:59.855+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:59.856814+0000) 2024-02-20T09:11:00.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:10:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:10:59.971+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:59.972803+0000) 2024-02-20T09:11:00.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:10:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:10:59.786+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:10:59.788350+0000) 2024-02-20T09:11:00.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:10:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:10:59.985+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:11:00.670 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:11:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:11:00.649+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 61269 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:11:00.671 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:11:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:11:00.647+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:00.648615+0000) 2024-02-20T09:11:00.671 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:11:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:11:00.644+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:00.645624+0000) 2024-02-20T09:11:00.671 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:11:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:11:00.644+0000 7f8a5f86e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:00.645561+0000) 2024-02-20T09:11:00.672 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:11:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:11:00.647+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:11:00.672 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:11:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:11:00.647+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:11:00.672 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:11:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:11:00.647+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:11:00.974 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:11:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:11:00.787+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:00.788580+0000) 2024-02-20T09:11:00.976 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:11:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:11:00.855+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:00.856988+0000) 2024-02-20T09:11:01.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:11:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:11:01.032+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:11:01.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:11:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:11:00.971+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:00.972916+0000) 2024-02-20T09:11:01.793 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:11:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:11:01.599+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:11:01.793 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:11:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:11:01.599+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:11:01.793 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:11:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:11:01.599+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:11:02.085 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:11:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:11:01.855+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:01.857108+0000) 2024-02-20T09:11:02.085 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:11:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:11:01.971+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:01.973043+0000) 2024-02-20T09:11:02.086 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:11:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:11:01.790+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:01.791874+0000) 2024-02-20T09:11:02.086 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:11:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:11:02.082+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:11:02.795 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:11:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:11:02.636+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:11:02.795 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:11:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:11:02.636+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:11:02.795 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:11:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:11:02.636+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:11:03.064 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:11:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:11:02.971+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:02.973171+0000) 2024-02-20T09:11:03.064 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:11:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:11:02.855+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:02.857224+0000) 2024-02-20T09:11:03.064 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:11:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:11:02.792+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:02.794155+0000) 2024-02-20T09:11:03.065 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:11:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:11:03.061+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:11:03.797 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:11:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:11:03.676+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:11:03.797 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:11:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:11:03.676+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:11:03.798 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:11:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:11:03.676+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:11:04.056 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:11:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:11:03.855+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:03.857336+0000) 2024-02-20T09:11:04.057 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:11:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:11:03.971+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:03.973317+0000) 2024-02-20T09:11:04.057 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:11:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:11:03.794+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:03.796334+0000) 2024-02-20T09:11:04.057 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:11:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:11:04.054+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:11:04.798 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:11:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:11:04.676+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:11:04.798 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:11:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:11:04.676+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:11:04.799 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:11:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:11:04.676+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:11:05.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:11:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:11:04.856+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:04.857441+0000) 2024-02-20T09:11:05.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:11:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:11:04.972+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:04.973432+0000) 2024-02-20T09:11:05.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:11:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:11:04.795+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:04.797186+0000) 2024-02-20T09:11:05.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:11:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:11:05.042+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:11:05.778 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:11:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:11:05.654+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:05.655981+0000) 2024-02-20T09:11:05.779 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:11:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:11:05.651+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:05.652756+0000) 2024-02-20T09:11:05.779 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:11:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:11:05.656+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 61342 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:11:05.779 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:11:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:11:05.651+0000 7f8a5f86e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:05.653030+0000) 2024-02-20T09:11:05.779 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:11:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:11:05.687+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:11:05.780 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:11:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:11:05.687+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:11:05.780 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:11:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:11:05.687+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:11:06.038 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:11:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:11:05.856+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:05.857547+0000) 2024-02-20T09:11:06.039 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:11:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:11:05.796+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:05.797432+0000) 2024-02-20T09:11:06.039 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:11:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:11:06.035+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:11:06.039 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:11:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:11:05.972+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:05.973535+0000) 2024-02-20T09:11:06.800 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:11:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:11:06.666+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:11:06.801 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:11:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:11:06.666+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:11:06.801 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:11:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:11:06.666+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:11:07.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:11:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:11:06.856+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:06.857642+0000) 2024-02-20T09:11:07.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:11:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:11:06.972+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:06.973648+0000) 2024-02-20T09:11:07.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:11:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:11:06.798+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:06.799543+0000) 2024-02-20T09:11:07.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:11:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:11:07.012+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:11:07.803 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:11:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:11:07.675+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:11:07.803 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:11:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:11:07.675+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:11:07.803 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:11:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:11:07.675+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:11:08.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:11:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:11:07.856+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:07.857784+0000) 2024-02-20T09:11:08.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:11:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:11:07.972+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:07.973743+0000) 2024-02-20T09:11:08.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:11:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:11:07.800+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:07.801958+0000) 2024-02-20T09:11:08.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:11:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:11:08.027+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:11:08.805 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:11:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:11:08.632+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:11:08.805 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:11:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:11:08.632+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:11:08.806 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:11:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:11:08.632+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:11:09.058 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:11:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:11:08.856+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:08.857876+0000) 2024-02-20T09:11:09.058 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:11:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:11:08.972+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:08.973854+0000) 2024-02-20T09:11:09.058 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:11:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:11:08.802+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:08.804289+0000) 2024-02-20T09:11:09.058 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:11:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:11:09.055+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:11:09.806 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:11:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:11:09.619+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:11:09.806 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:11:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:11:09.619+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:11:09.806 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:11:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:11:09.619+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:11:10.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:11:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:11:09.856+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:09.857997+0000) 2024-02-20T09:11:10.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:11:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:11:09.972+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:09.973968+0000) 2024-02-20T09:11:10.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:11:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:11:09.803+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:09.805116+0000) 2024-02-20T09:11:10.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:11:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:11:10.048+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:11:10.660 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:11:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:11:10.622+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:11:10.661 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:11:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:11:10.622+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:11:10.661 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:11:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:11:10.622+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:11:10.976 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:11:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:11:10.663+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 61414 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:11:11.029 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:11:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:11:10.657+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:10.659259+0000) 2024-02-20T09:11:11.030 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:11:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:11:10.657+0000 7f8a5f86e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:10.659147+0000) 2024-02-20T09:11:11.030 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:11:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:11:10.856+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:10.858110+0000) 2024-02-20T09:11:11.030 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:11:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:11:10.660+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:10.662165+0000) 2024-02-20T09:11:11.030 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:11:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:11:10.804+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:10.805471+0000) 2024-02-20T09:11:11.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:11:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:11:11.021+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:11:11.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:11:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:11:10.972+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:10.974104+0000) 2024-02-20T09:11:11.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:11:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:11:11.163+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:11:11.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:11:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:11:11.163+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:11:11.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:11:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:11:11.163+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:11:11.937 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:11:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:11:11.806+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:11.807611+0000) 2024-02-20T09:11:11.938 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:11:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:11:11.616+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:11:11.938 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:11:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:11:11.616+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:11:11.938 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:11:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:11:11.616+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:11:11.938 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:11:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:11:11.856+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:11.858228+0000) 2024-02-20T09:11:12.190 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:11:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:11:11.994+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:11:12.191 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:11:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:11:11.972+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:11.974233+0000) 2024-02-20T09:11:12.191 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:11:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:11:12.188+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:11:12.191 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:11:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:11:12.188+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:11:12.191 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:11:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:11:12.188+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:11:12.810 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:11:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:11:12.655+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:11:12.811 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:11:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:11:12.655+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:11:12.811 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:11:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:11:12.655+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:11:13.183 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:11:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:11:12.856+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:12.858372+0000) 2024-02-20T09:11:13.183 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:11:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:11:12.808+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:12.809722+0000) 2024-02-20T09:11:13.184 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:11:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:11:13.018+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:11:13.184 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:11:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:11:12.972+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:12.974364+0000) 2024-02-20T09:11:13.184 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:11:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:11:13.180+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:11:13.184 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:11:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:11:13.180+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:11:13.184 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:11:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:11:13.180+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:11:13.813 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:11:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:11:13.653+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:11:13.813 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:11:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:11:13.653+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:11:13.813 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:11:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:11:13.653+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:11:14.069 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:11:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:11:13.857+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:13.858508+0000) 2024-02-20T09:11:14.070 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:11:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:11:13.973+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:13.974523+0000) 2024-02-20T09:11:14.070 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:11:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:11:13.810+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:13.812058+0000) 2024-02-20T09:11:14.070 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:11:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:11:14.067+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:11:14.425 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:11:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:11:14.179+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:11:14.425 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:11:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:11:14.179+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:11:14.425 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:11:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:11:14.179+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:11:14.742 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:11:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:11:14.654+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:11:14.742 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:11:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:11:14.654+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:11:14.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:11:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:11:14.654+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:11:15.081 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:11:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:11:14.857+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:14.858624+0000) 2024-02-20T09:11:15.082 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:11:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:11:14.973+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:14.974637+0000) 2024-02-20T09:11:15.082 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:11:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:11:14.811+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:14.813172+0000) 2024-02-20T09:11:15.082 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:11:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:11:15.079+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:11:15.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:11:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:11:15.225+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:11:15.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:11:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:11:15.225+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:11:15.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:11:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:11:15.225+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:11:15.791 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:11:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:11:15.667+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:15.669331+0000) 2024-02-20T09:11:15.791 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:11:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:11:15.664+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:15.665806+0000) 2024-02-20T09:11:15.791 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:11:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:11:15.670+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 61487 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:11:15.792 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:11:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:11:15.664+0000 7f8a5f86e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:15.666116+0000) 2024-02-20T09:11:15.792 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:11:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:11:15.676+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:11:15.792 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:11:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:11:15.676+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:11:15.792 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:11:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:11:15.676+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:11:16.124 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:11:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:11:15.857+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:15.858763+0000) 2024-02-20T09:11:16.124 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:11:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:11:15.973+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:15.974745+0000) 2024-02-20T09:11:16.125 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:11:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:11:15.811+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:15.813376+0000) 2024-02-20T09:11:16.125 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:11:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:11:16.121+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:11:16.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:11:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:11:16.255+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:11:16.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:11:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:11:16.255+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:11:16.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:11:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:11:16.255+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:11:16.818 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:11:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:11:16.672+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:11:16.818 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:11:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:11:16.672+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:11:16.818 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:11:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:11:16.672+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:11:17.111 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:11:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:11:16.857+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:16.858905+0000) 2024-02-20T09:11:17.111 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:11:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:11:16.815+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:16.816978+0000) 2024-02-20T09:11:17.111 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:11:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:11:17.108+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:11:17.112 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:11:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:11:16.973+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:16.974859+0000) 2024-02-20T09:11:17.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:11:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:11:17.265+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:11:17.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:11:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:11:17.265+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:11:17.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:11:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:11:17.265+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:11:17.821 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:11:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:11:17.628+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:11:17.822 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:11:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:11:17.628+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:11:17.822 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:11:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:11:17.628+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:11:18.139 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:11:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:11:17.857+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:17.859015+0000) 2024-02-20T09:11:18.139 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:11:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:11:17.974+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:17.975016+0000) 2024-02-20T09:11:18.139 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:11:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:11:17.819+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:17.820476+0000) 2024-02-20T09:11:18.140 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:11:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:11:18.136+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:11:18.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:11:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:11:18.254+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:11:18.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:11:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:11:18.254+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:11:18.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:11:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:11:18.254+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:11:18.824 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:11:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:11:18.610+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:11:18.824 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:11:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:11:18.610+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:11:18.824 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:11:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:11:18.610+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:11:19.150 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:11:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:11:18.857+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:18.859109+0000) 2024-02-20T09:11:19.150 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:11:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:11:18.974+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:18.975168+0000) 2024-02-20T09:11:19.151 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:11:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:11:18.821+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:18.822629+0000) 2024-02-20T09:11:19.151 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:11:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:11:19.147+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:11:19.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:11:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:11:19.225+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:11:19.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:11:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:11:19.225+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:11:19.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:11:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:11:19.225+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:11:19.824 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:11:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:11:19.567+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:11:19.825 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:11:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:11:19.567+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:11:19.825 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:11:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:11:19.567+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:11:20.162 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:11:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:11:19.857+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:19.859224+0000) 2024-02-20T09:11:20.163 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:11:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:11:19.973+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:19.975252+0000) 2024-02-20T09:11:20.163 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:11:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:11:19.822+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:19.823717+0000) 2024-02-20T09:11:20.163 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:11:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:11:20.160+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:11:20.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:11:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:11:20.253+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:11:20.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:11:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:11:20.253+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:11:20.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:11:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:11:20.253+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:11:20.799 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:11:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:11:20.677+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 61559 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:11:20.799 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:11:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:11:20.674+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:20.675798+0000) 2024-02-20T09:11:20.800 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:11:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:11:20.672+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:20.673611+0000) 2024-02-20T09:11:20.800 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:11:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:11:20.564+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:11:20.800 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:11:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:11:20.564+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:11:20.800 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:11:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:11:20.564+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:11:20.801 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:11:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:11:20.672+0000 7f8a5f86e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:20.674013+0000) 2024-02-20T09:11:21.182 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:11:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:11:20.858+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:20.859389+0000) 2024-02-20T09:11:21.183 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:11:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:11:20.822+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:20.823938+0000) 2024-02-20T09:11:21.183 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:11:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:11:20.974+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:20.975463+0000) 2024-02-20T09:11:21.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:11:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:11:21.180+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:11:21.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:11:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:11:21.278+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:11:21.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:11:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:11:21.278+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:11:21.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:11:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:11:21.278+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:11:21.827 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:11:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:11:21.523+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:11:21.827 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:11:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:11:21.523+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:11:21.827 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:11:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:11:21.523+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:11:22.138 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:11:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:11:21.859+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:21.859549+0000) 2024-02-20T09:11:22.138 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:11:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:11:21.975+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:21.975602+0000) 2024-02-20T09:11:22.139 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:11:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:11:21.824+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:21.826003+0000) 2024-02-20T09:11:22.139 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:11:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:11:22.136+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:11:22.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:11:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:11:22.324+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:11:22.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:11:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:11:22.324+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:11:22.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:11:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:11:22.324+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:11:22.830 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:11:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:11:22.505+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:11:22.830 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:11:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:11:22.505+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:11:22.830 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:11:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:11:22.505+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:11:23.188 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:11:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:11:22.858+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:22.859681+0000) 2024-02-20T09:11:23.188 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:11:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:11:22.974+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:22.975748+0000) 2024-02-20T09:11:23.189 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:11:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:11:22.828+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:22.828518+0000) 2024-02-20T09:11:23.189 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:11:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:11:23.186+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:11:23.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:11:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:11:23.339+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:11:23.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:11:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:11:23.339+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:11:23.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:11:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:11:23.339+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:11:23.831 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:11:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:11:23.543+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:11:23.832 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:11:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:11:23.543+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:11:23.832 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:11:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:11:23.543+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:11:24.194 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:11:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:11:23.858+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:23.859775+0000) 2024-02-20T09:11:24.195 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:11:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:11:23.974+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:23.975873+0000) 2024-02-20T09:11:24.195 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:11:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:11:23.830+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:23.830759+0000) 2024-02-20T09:11:24.195 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:11:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:11:24.192+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:11:24.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:11:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:11:24.192+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:11:24.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:11:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:11:24.192+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:11:24.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:11:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:11:24.192+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:11:24.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:11:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:11:24.350+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:11:24.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:11:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:11:24.350+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:11:24.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:11:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:11:24.350+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:11:24.833 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:11:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:11:24.541+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:11:24.833 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:11:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:11:24.541+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:11:24.833 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:11:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:11:24.541+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:11:25.211 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:11:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:11:24.858+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:24.859939+0000) 2024-02-20T09:11:25.211 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:11:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:11:24.974+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:24.976043+0000) 2024-02-20T09:11:25.211 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:11:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:11:24.831+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:24.831861+0000) 2024-02-20T09:11:25.212 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:11:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:11:25.209+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:11:25.212 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:11:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:11:25.209+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:11:25.212 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:11:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:11:25.209+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:11:25.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:11:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:11:25.209+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:11:25.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:11:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:11:25.333+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:11:25.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:11:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:11:25.333+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:11:25.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:11:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:11:25.333+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:11:25.804 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:11:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:11:25.684+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 61632 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:11:25.804 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:11:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:11:25.683+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:25.684201+0000) 2024-02-20T09:11:25.805 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:11:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:11:25.679+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:25.680118+0000) 2024-02-20T09:11:25.805 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:11:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:11:25.579+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:11:25.805 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:11:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:11:25.579+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:11:25.805 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:11:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:11:25.579+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:11:25.805 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:11:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:11:25.679+0000 7f8a5f86e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:25.680341+0000) 2024-02-20T09:11:26.224 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:11:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:11:25.859+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:25.860108+0000) 2024-02-20T09:11:26.225 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:11:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:11:25.831+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:25.832093+0000) 2024-02-20T09:11:26.225 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:11:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:11:26.223+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:11:26.225 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:11:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:11:25.975+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:25.976160+0000) 2024-02-20T09:11:26.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:11:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:11:26.223+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:11:26.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:11:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:11:26.223+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:11:26.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:11:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:11:26.223+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:11:26.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:11:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:11:26.340+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:11:26.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:11:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:11:26.340+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:11:26.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:11:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:11:26.340+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:11:26.835 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:11:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:11:26.573+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:11:26.835 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:11:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:11:26.573+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:11:26.835 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:11:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:11:26.573+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:11:27.200 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:11:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:11:26.859+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:26.860277+0000) 2024-02-20T09:11:27.201 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:11:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:11:26.975+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:26.976318+0000) 2024-02-20T09:11:27.201 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:11:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:11:26.833+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:26.834097+0000) 2024-02-20T09:11:27.201 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:11:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:11:27.199+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:11:27.201 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:11:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:11:27.199+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:11:27.201 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:11:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:11:27.199+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:11:27.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:11:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:11:27.199+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:11:27.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:11:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:11:27.326+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:11:27.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:11:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:11:27.326+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:11:27.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:11:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:11:27.326+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:11:27.838 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:11:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:11:27.542+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:11:27.838 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:11:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:11:27.543+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:11:27.838 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:11:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:11:27.543+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:11:28.211 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:11:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:11:27.976+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:27.976503+0000) 2024-02-20T09:11:28.211 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:11:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:11:27.859+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:27.860366+0000) 2024-02-20T09:11:28.211 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:11:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:11:27.836+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:27.836677+0000) 2024-02-20T09:11:28.212 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:11:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:11:28.209+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:11:28.212 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:11:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:11:28.209+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:11:28.212 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:11:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:11:28.209+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:11:28.212 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:11:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:11:28.209+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:11:28.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:11:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:11:28.340+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:11:28.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:11:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:11:28.340+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:11:28.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:11:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:11:28.340+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:11:28.840 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:11:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:11:28.557+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:11:28.840 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:11:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:11:28.557+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:11:28.840 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:11:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:11:28.557+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:11:29.196 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:11:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:11:28.860+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:28.860526+0000) 2024-02-20T09:11:29.197 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:11:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:11:28.976+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:28.976631+0000) 2024-02-20T09:11:29.197 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:11:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:11:28.838+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:28.838970+0000) 2024-02-20T09:11:29.197 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:11:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:11:29.195+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:11:29.197 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:11:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:11:29.195+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:11:29.197 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:11:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:11:29.195+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:11:29.197 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:11:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:11:29.195+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:11:29.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:11:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:11:29.343+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:11:29.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:11:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:11:29.343+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:11:29.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:11:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:11:29.343+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:11:29.841 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:11:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:11:29.575+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:11:29.841 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:11:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:11:29.575+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:11:29.841 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:11:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:11:29.575+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:11:30.198 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:11:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:11:29.860+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:29.860647+0000) 2024-02-20T09:11:30.198 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:11:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:11:29.976+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:29.976760+0000) 2024-02-20T09:11:30.198 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:11:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:11:29.839+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:29.839969+0000) 2024-02-20T09:11:30.199 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:11:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:11:30.196+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:11:30.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:11:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:11:30.196+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:11:30.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:11:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:11:30.196+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:11:30.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:11:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:11:30.196+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:11:30.495 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:11:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:11:30.371+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:11:30.495 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:11:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:11:30.371+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:11:30.496 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:11:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:11:30.371+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:11:30.811 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:11:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:11:30.691+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 61704 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:11:30.812 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:11:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:11:30.688+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:30.689211+0000) 2024-02-20T09:11:30.812 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:11:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:11:30.686+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:30.686462+0000) 2024-02-20T09:11:30.812 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:11:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:11:30.579+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:11:30.813 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:11:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:11:30.579+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:11:30.813 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:11:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:11:30.579+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:11:30.813 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:11:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:11:30.686+0000 7f8a5f86e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:30.686630+0000) 2024-02-20T09:11:31.175 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:11:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:11:30.860+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:30.860777+0000) 2024-02-20T09:11:31.175 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:11:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:11:30.976+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:30.976884+0000) 2024-02-20T09:11:31.175 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:11:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:11:30.839+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:30.840216+0000) 2024-02-20T09:11:31.175 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:11:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:11:31.173+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:11:31.176 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:11:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:11:31.173+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:11:31.176 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:11:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:11:31.173+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:11:31.176 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:11:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:11:31.173+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:11:31.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:11:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:11:31.361+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:11:31.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:11:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:11:31.361+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:11:31.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:11:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:11:31.361+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:11:31.843 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:11:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:11:31.619+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:11:31.843 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:11:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:11:31.619+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:11:31.843 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:11:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:11:31.619+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:11:32.145 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:11:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:11:31.860+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:31.860894+0000) 2024-02-20T09:11:32.145 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:11:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:11:31.976+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:31.977013+0000) 2024-02-20T09:11:32.145 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:11:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:11:31.841+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:31.842147+0000) 2024-02-20T09:11:32.145 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:11:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:11:32.143+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:11:32.146 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:11:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:11:32.143+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:11:32.146 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:11:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:11:32.143+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:11:32.146 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:11:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:11:32.143+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:11:32.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:11:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:11:32.345+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:11:32.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:11:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:11:32.345+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:11:32.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:11:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:11:32.345+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:11:32.846 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:11:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:11:32.597+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:11:32.846 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:11:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:11:32.597+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:11:32.846 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:11:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:11:32.597+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:11:33.132 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:11:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:11:32.860+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:32.861025+0000) 2024-02-20T09:11:33.132 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:11:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:11:32.976+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:32.977140+0000) 2024-02-20T09:11:33.133 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:11:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:11:32.844+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:32.844791+0000) 2024-02-20T09:11:33.133 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:11:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:11:33.130+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:11:33.133 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:11:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:11:33.130+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:11:33.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:11:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:11:33.130+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:11:33.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:11:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:11:33.131+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:11:33.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:11:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:11:33.350+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:11:33.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:11:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:11:33.350+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:11:33.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:11:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:11:33.350+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:11:33.848 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:11:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:11:33.632+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:11:33.848 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:11:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:11:33.632+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:11:33.848 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:11:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:11:33.632+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:11:34.151 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:11:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:11:33.860+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:33.861191+0000) 2024-02-20T09:11:34.152 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:11:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:11:33.976+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:33.977287+0000) 2024-02-20T09:11:34.152 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:11:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:11:33.846+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:33.847060+0000) 2024-02-20T09:11:34.152 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:11:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:11:34.150+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:11:34.152 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:11:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:11:34.150+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:11:34.152 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:11:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:11:34.150+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:11:34.153 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:11:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:11:34.150+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:11:34.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:11:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:11:34.348+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:11:34.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:11:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:11:34.348+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:11:34.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:11:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:11:34.348+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:11:34.849 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:11:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:11:34.591+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:11:34.849 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:11:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:11:34.591+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:11:34.849 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:11:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:11:34.591+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:11:35.151 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:11:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:11:34.860+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:34.861346+0000) 2024-02-20T09:11:35.151 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:11:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:11:34.977+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:34.977460+0000) 2024-02-20T09:11:35.151 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:11:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:11:34.847+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:34.848098+0000) 2024-02-20T09:11:35.152 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:11:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:11:35.149+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:11:35.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:11:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:11:35.300+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:11:35.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:11:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:11:35.300+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:11:35.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:11:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:11:35.300+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:11:35.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:11:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:11:35.149+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:11:35.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:11:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:11:35.149+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:11:35.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:11:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:11:35.149+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:11:35.821 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:11:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:11:35.698+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 61777 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:11:35.822 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:11:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:11:35.696+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:35.696480+0000) 2024-02-20T09:11:35.822 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:11:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:11:35.693+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:35.693583+0000) 2024-02-20T09:11:35.822 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:11:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:11:35.550+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:11:35.823 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:11:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:11:35.550+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:11:35.823 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:11:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:11:35.550+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:11:35.823 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:11:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:11:35.693+0000 7f8a5f86e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:35.693513+0000) 2024-02-20T09:11:36.106 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:11:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:11:35.861+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:35.861456+0000) 2024-02-20T09:11:36.106 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:11:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:11:35.977+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:35.977575+0000) 2024-02-20T09:11:36.106 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:11:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:11:35.847+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:35.848314+0000) 2024-02-20T09:11:36.107 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:11:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:11:36.104+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:11:36.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:11:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:11:36.104+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:11:36.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:11:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:11:36.104+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:11:36.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:11:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:11:36.104+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:11:36.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:11:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:11:36.320+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:11:36.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:11:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:11:36.320+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:11:36.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:11:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:11:36.320+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:11:36.851 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:11:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:11:36.563+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:11:36.851 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:11:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:11:36.563+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:11:36.851 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:11:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:11:36.563+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:11:37.101 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:11:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:11:36.861+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:36.861535+0000) 2024-02-20T09:11:37.101 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:11:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:11:36.977+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:36.977651+0000) 2024-02-20T09:11:37.102 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:11:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:11:36.850+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:36.850613+0000) 2024-02-20T09:11:37.102 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:11:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:11:37.099+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:11:37.102 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:11:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:11:37.099+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:11:37.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:11:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:11:37.099+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:11:37.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:11:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:11:37.099+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:11:37.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:11:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:11:37.349+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:11:37.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:11:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:11:37.349+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:11:37.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:11:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:11:37.349+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:11:37.851 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:11:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:11:37.539+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:11:37.851 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:11:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:11:37.539+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:11:37.851 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:11:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:11:37.539+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:11:38.127 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:11:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:11:37.861+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:37.861678+0000) 2024-02-20T09:11:38.128 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:11:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:11:37.977+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:37.977772+0000) 2024-02-20T09:11:38.128 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:11:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:11:37.852+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:37.852655+0000) 2024-02-20T09:11:38.128 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:11:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:11:38.126+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:11:38.128 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:11:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:11:38.126+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:11:38.129 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:11:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:11:38.126+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:11:38.129 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:11:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:11:38.126+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:11:38.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:11:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:11:38.363+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:11:38.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:11:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:11:38.363+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:11:38.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:11:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:11:38.363+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:11:38.851 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:11:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:11:38.502+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:11:38.851 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:11:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:11:38.502+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:11:38.851 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:11:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:11:38.502+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:11:39.173 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:11:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:11:38.861+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:38.861833+0000) 2024-02-20T09:11:39.174 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:11:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:11:38.977+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:38.977918+0000) 2024-02-20T09:11:39.174 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:11:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:11:38.854+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:38.854741+0000) 2024-02-20T09:11:39.174 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:11:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:11:39.172+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:11:39.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:11:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:11:39.172+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:11:39.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:11:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:11:39.172+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:11:39.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:11:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:11:39.172+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:11:39.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:11:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:11:39.317+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:11:39.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:11:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:11:39.317+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:11:39.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:11:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:11:39.317+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:11:39.852 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:11:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:11:39.492+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:11:39.852 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:11:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:11:39.492+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:11:39.852 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:11:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:11:39.492+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:11:40.141 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:11:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:11:39.861+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:39.862015+0000) 2024-02-20T09:11:40.141 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:11:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:11:39.977+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:39.978066+0000) 2024-02-20T09:11:40.142 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:11:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:11:39.855+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:39.855539+0000) 2024-02-20T09:11:40.142 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:11:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:11:40.139+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:11:40.142 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:11:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:11:40.139+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:11:40.411 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:11:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:11:40.139+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:11:40.411 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:11:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:11:40.139+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:11:40.411 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:11:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:11:40.320+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:11:40.411 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:11:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:11:40.320+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:11:40.411 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:11:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:11:40.320+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:11:40.701 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:11:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:11:40.538+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:11:40.702 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:11:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:11:40.538+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:11:40.702 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:11:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:11:40.538+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:11:40.979 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:11:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:11:40.700+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:40.700716+0000) 2024-02-20T09:11:40.980 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:11:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:11:40.705+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 61849 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:11:40.980 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:11:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:11:40.702+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:40.702725+0000) 2024-02-20T09:11:40.980 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:11:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:11:40.855+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:40.855713+0000) 2024-02-20T09:11:40.981 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:11:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:11:40.700+0000 7f8a5f86e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:40.700519+0000) 2024-02-20T09:11:40.981 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:11:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:11:40.861+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:40.862220+0000) 2024-02-20T09:11:41.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:11:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:11:40.977+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:40.978248+0000) 2024-02-20T09:11:41.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:11:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:11:41.108+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:11:41.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:11:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:11:41.108+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:11:41.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:11:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:11:41.108+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:11:41.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:11:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:11:41.109+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:11:41.625 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:11:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:11:41.511+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:11:41.626 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:11:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:11:41.511+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:11:41.626 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:11:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:11:41.511+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:11:41.626 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:11:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:11:41.316+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:11:41.626 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:11:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:11:41.316+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:11:41.626 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:11:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:11:41.316+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:11:41.939 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:11:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:11:41.861+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:41.862368+0000) 2024-02-20T09:11:41.940 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:11:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:11:41.857+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:41.857661+0000) 2024-02-20T09:11:42.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:11:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:11:41.978+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:41.978418+0000) 2024-02-20T09:11:42.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:11:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:11:42.132+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:11:42.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:11:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:11:42.132+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:11:42.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:11:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:11:42.132+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:11:42.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:11:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:11:42.132+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:11:42.628 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:11:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:11:42.560+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:11:42.628 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:11:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:11:42.560+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:11:42.629 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:11:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:11:42.560+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:11:42.629 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:11:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:11:42.325+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:11:42.629 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:11:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:11:42.325+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:11:42.629 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:11:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:11:42.325+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:11:42.979 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:11:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:11:42.862+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:42.862525+0000) 2024-02-20T09:11:42.980 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:11:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:11:42.859+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:42.859645+0000) 2024-02-20T09:11:43.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:11:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:11:42.978+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:42.978553+0000) 2024-02-20T09:11:43.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:11:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:11:43.177+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:11:43.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:11:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:11:43.177+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:11:43.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:11:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:11:43.177+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:11:43.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:11:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:11:43.177+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:11:43.629 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:11:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:11:43.347+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:11:43.629 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:11:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:11:43.347+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:11:43.629 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:11:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:11:43.348+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:11:43.630 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:11:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:11:43.596+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:11:43.630 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:11:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:11:43.596+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:11:43.630 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:11:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:11:43.596+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:11:43.979 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:11:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:11:43.862+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:43.862620+0000) 2024-02-20T09:11:43.980 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:11:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:11:43.861+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:43.861658+0000) 2024-02-20T09:11:44.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:11:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:11:43.978+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:43.978640+0000) 2024-02-20T09:11:44.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:11:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:11:44.205+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:11:44.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:11:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:11:44.205+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:11:44.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:11:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:11:44.205+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:11:44.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:11:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:11:44.206+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:11:44.629 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:11:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:11:44.580+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:11:44.629 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:11:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:11:44.580+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:11:44.629 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:11:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:11:44.580+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:11:44.630 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:11:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:11:44.376+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:11:44.630 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:11:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:11:44.376+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:11:44.630 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:11:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:11:44.376+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:11:44.979 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:11:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:11:44.862+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:44.862713+0000) 2024-02-20T09:11:44.980 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:11:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:11:44.862+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:44.862475+0000) 2024-02-20T09:11:45.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:11:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:11:44.978+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:44.978757+0000) 2024-02-20T09:11:45.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:11:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:11:45.173+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:11:45.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:11:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:11:45.173+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:11:45.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:11:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:11:45.173+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:11:45.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:11:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:11:45.173+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:11:45.708 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:11:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:11:45.573+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:11:45.708 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:11:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:11:45.573+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:11:45.708 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:11:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:11:45.573+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:11:45.709 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:11:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:11:45.420+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:11:45.709 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:11:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:11:45.420+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:11:45.709 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:11:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:11:45.420+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:11:45.709 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:11:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:11:45.706+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:45.707369+0000) 2024-02-20T09:11:45.980 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:11:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:11:45.713+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 61921 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:11:45.980 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:11:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:11:45.707+0000 7f8a5f86e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:45.707421+0000) 2024-02-20T09:11:45.980 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:11:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:11:45.862+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:45.862810+0000) 2024-02-20T09:11:45.980 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:11:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:11:45.710+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:45.711314+0000) 2024-02-20T09:11:45.981 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:11:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:11:45.862+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:45.862704+0000) 2024-02-20T09:11:46.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:11:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:11:45.978+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:45.978887+0000) 2024-02-20T09:11:46.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:11:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:11:46.178+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:11:46.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:11:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:11:46.178+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:11:46.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:11:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:11:46.178+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:11:46.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:11:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:11:46.178+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:11:46.742 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:11:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:11:46.572+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:11:46.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:11:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:11:46.572+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:11:46.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:11:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:11:46.572+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:11:46.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:11:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:11:46.397+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:11:46.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:11:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:11:46.397+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:11:46.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:11:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:11:46.397+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:11:47.171 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:11:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:11:46.861+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:46.862920+0000) 2024-02-20T09:11:47.172 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:11:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:11:46.977+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:46.979005+0000) 2024-02-20T09:11:47.172 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:11:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:11:46.863+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:46.864770+0000) 2024-02-20T09:11:47.172 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:11:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:11:47.169+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:11:47.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:11:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:11:47.169+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:11:47.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:11:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:11:47.169+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:11:47.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:11:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:11:47.169+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:11:47.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:11:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:11:47.398+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:11:47.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:11:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:11:47.398+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:11:47.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:11:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:11:47.398+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:11:47.852 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:11:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:11:47.558+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:11:47.852 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:11:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:11:47.558+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:11:47.853 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:11:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:11:47.558+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:11:48.163 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:11:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:11:47.861+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:47.863039+0000) 2024-02-20T09:11:48.163 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:11:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:11:47.977+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:47.979116+0000) 2024-02-20T09:11:48.164 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:11:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:11:47.865+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:47.866806+0000) 2024-02-20T09:11:48.164 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:11:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:11:48.160+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:11:48.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:11:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:11:48.160+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:11:48.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:11:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:11:48.160+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:11:48.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:11:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:11:48.160+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:11:48.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:11:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:11:48.349+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:11:48.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:11:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:11:48.349+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:11:48.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:11:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:11:48.349+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:11:48.853 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:11:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:11:48.522+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:11:48.853 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:11:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:11:48.522+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:11:48.853 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:11:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:11:48.522+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:11:49.186 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:11:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:11:48.861+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:48.863147+0000) 2024-02-20T09:11:49.186 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:11:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:11:48.867+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:48.868902+0000) 2024-02-20T09:11:49.186 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:11:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:11:48.977+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:48.979242+0000) 2024-02-20T09:11:49.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:11:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:11:49.328+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:11:49.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:11:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:11:49.328+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:11:49.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:11:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:11:49.328+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:11:49.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:11:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:11:49.183+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:11:49.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:11:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:11:49.183+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:11:49.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:11:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:11:49.183+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:11:49.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:11:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:11:49.183+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:11:49.852 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:11:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:11:49.494+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:11:49.853 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:11:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:11:49.494+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:11:49.853 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:11:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:11:49.494+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:11:50.183 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:11:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:11:49.861+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:49.863275+0000) 2024-02-20T09:11:50.183 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:11:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:11:49.977+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:49.979364+0000) 2024-02-20T09:11:50.184 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:11:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:11:49.868+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:49.870190+0000) 2024-02-20T09:11:50.184 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:11:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:11:50.180+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:11:50.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:11:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:11:50.180+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:11:50.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:11:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:11:50.180+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:11:50.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:11:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:11:50.180+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:11:50.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:11:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:11:50.351+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:11:50.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:11:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:11:50.351+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:11:50.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:11:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:11:50.351+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:11:50.841 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:11:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:11:50.713+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:50.714912+0000) 2024-02-20T09:11:50.841 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:11:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:11:50.719+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 61993 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:11:50.842 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:11:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:11:50.716+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:50.717522+0000) 2024-02-20T09:11:50.842 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:11:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:11:50.519+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:11:50.842 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:11:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:11:50.519+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:11:50.842 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:11:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:11:50.519+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:11:50.842 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:11:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:11:50.714+0000 7f8a5f86e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:50.715409+0000) 2024-02-20T09:11:51.192 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:11:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:11:50.862+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:50.863432+0000) 2024-02-20T09:11:51.192 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:11:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:11:50.869+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:50.870456+0000) 2024-02-20T09:11:51.192 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:11:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:11:51.189+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:11:51.193 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:11:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:11:50.978+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:50.979504+0000) 2024-02-20T09:11:51.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:11:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:11:51.333+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:11:51.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:11:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:11:51.333+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:11:51.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:11:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:11:51.333+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:11:51.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:11:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:11:51.189+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:11:51.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:11:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:11:51.189+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:11:51.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:11:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:11:51.189+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:11:51.853 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:11:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:11:51.533+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:11:51.853 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:11:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:11:51.533+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:11:51.853 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:11:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:11:51.533+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:11:52.142 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:11:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:11:51.978+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:51.979610+0000) 2024-02-20T09:11:52.143 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:11:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:11:51.862+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:51.863514+0000) 2024-02-20T09:11:52.143 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:11:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:11:51.872+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:51.874079+0000) 2024-02-20T09:11:52.143 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:11:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:11:52.139+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:11:52.143 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:11:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:11:52.139+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:11:52.143 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:11:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:11:52.139+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:11:52.143 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:11:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:11:52.140+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:11:52.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:11:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:11:52.371+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:11:52.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:11:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:11:52.371+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:11:52.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:11:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:11:52.371+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:11:52.853 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:11:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:11:52.509+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:11:52.853 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:11:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:11:52.509+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:11:52.853 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:11:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:11:52.509+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:11:53.130 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:11:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:11:52.978+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:52.979748+0000) 2024-02-20T09:11:53.130 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:11:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:11:52.862+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:52.863645+0000) 2024-02-20T09:11:53.130 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:11:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:11:52.876+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:52.877629+0000) 2024-02-20T09:11:53.131 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:11:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:11:53.127+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:11:53.131 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:11:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:11:53.127+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:11:53.131 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:11:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:11:53.127+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:11:53.131 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:11:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:11:53.127+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:11:53.480 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:11:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:11:53.350+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:11:53.480 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:11:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:11:53.351+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:11:53.480 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:11:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:11:53.351+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:11:53.742 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:11:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:11:53.477+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:11:53.742 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:11:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:11:53.477+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:11:53.742 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:11:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:11:53.477+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:11:54.130 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:11:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:11:53.862+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:53.863765+0000) 2024-02-20T09:11:54.131 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:11:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:11:53.978+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:53.979945+0000) 2024-02-20T09:11:54.131 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:11:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:11:53.878+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:53.879631+0000) 2024-02-20T09:11:54.131 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:11:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:11:54.127+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:11:54.132 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:11:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:11:54.127+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:11:54.132 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:11:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:11:54.127+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:11:54.132 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:11:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:11:54.128+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:11:54.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:11:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:11:54.362+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:11:54.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:11:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:11:54.362+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:11:54.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:11:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:11:54.362+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:11:54.853 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:11:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:11:54.498+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:11:54.853 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:11:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:11:54.498+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:11:54.854 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:11:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:11:54.498+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:11:55.114 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:11:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:11:54.862+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:54.863967+0000) 2024-02-20T09:11:55.114 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:11:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:11:54.978+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:54.980116+0000) 2024-02-20T09:11:55.115 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:11:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:11:54.879+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:54.880670+0000) 2024-02-20T09:11:55.115 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:11:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:11:55.112+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:11:55.371 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:11:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:11:55.112+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:11:55.372 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:11:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:11:55.112+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:11:55.372 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:11:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:11:55.112+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:11:55.636 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:11:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:11:55.519+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:11:55.636 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:11:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:11:55.519+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:11:55.636 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:11:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:11:55.519+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:11:55.637 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:11:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:11:55.369+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:11:55.637 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:11:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:11:55.369+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:11:55.637 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:11:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:11:55.369+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:11:55.981 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:11:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:11:55.726+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 62066 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:11:55.981 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:11:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:11:55.721+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:55.722703+0000) 2024-02-20T09:11:55.982 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:11:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:11:55.723+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:55.724484+0000) 2024-02-20T09:11:55.982 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:11:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:11:55.879+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:55.880850+0000) 2024-02-20T09:11:55.982 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:11:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:11:55.720+0000 7f8a5f86e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:55.722375+0000) 2024-02-20T09:11:55.982 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:11:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:11:55.862+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:55.864111+0000) 2024-02-20T09:11:56.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:11:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:11:55.978+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:55.980228+0000) 2024-02-20T09:11:56.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:11:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:11:56.129+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:11:56.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:11:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:11:56.129+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:11:56.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:11:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:11:56.129+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:11:56.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:11:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:11:56.130+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:11:56.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:11:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:11:56.417+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:11:56.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:11:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:11:56.417+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:11:56.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:11:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:11:56.417+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:11:56.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:11:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:11:56.490+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:11:56.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:11:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:11:56.490+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:11:56.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:11:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:11:56.490+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:11:57.127 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:11:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:11:56.978+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:56.980345+0000) 2024-02-20T09:11:57.128 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:11:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:11:56.862+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:56.864237+0000) 2024-02-20T09:11:57.128 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:11:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:11:56.881+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:56.882841+0000) 2024-02-20T09:11:57.128 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:11:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:11:57.125+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:11:57.128 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:11:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:11:57.125+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:11:57.128 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:11:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:11:57.125+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:11:57.129 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:11:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:11:57.125+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:11:57.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:11:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:11:57.399+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:11:57.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:11:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:11:57.399+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:11:57.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:11:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:11:57.399+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:11:57.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:11:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:11:57.448+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:11:57.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:11:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:11:57.448+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:11:57.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:11:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:11:57.448+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:11:58.137 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:11:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:11:57.979+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:57.980476+0000) 2024-02-20T09:11:58.138 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:11:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:11:57.862+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:57.864355+0000) 2024-02-20T09:11:58.138 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:11:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:11:57.884+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:57.885592+0000) 2024-02-20T09:11:58.138 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:11:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:11:58.135+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:11:58.138 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:11:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:11:58.135+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:11:58.397 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:11:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:11:58.135+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:11:58.398 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:11:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:11:58.135+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:11:58.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:11:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:11:58.395+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:11:58.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:11:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:11:58.395+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:11:58.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:11:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:11:58.395+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:11:58.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:11:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:11:58.438+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:11:58.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:11:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:11:58.438+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:11:58.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:11:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:11:58.438+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:11:59.106 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:11:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:11:58.979+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:58.980567+0000) 2024-02-20T09:11:59.106 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:11:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:11:58.863+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:58.864474+0000) 2024-02-20T09:11:59.106 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:11:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:11:58.886+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:58.887769+0000) 2024-02-20T09:11:59.106 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:11:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:11:59.103+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:11:59.107 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:11:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:11:59.103+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:11:59.107 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:11:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:11:59.103+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:11:59.107 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:11:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:11:59.103+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:11:59.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:11:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:11:59.438+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:11:59.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:11:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:11:59.438+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:11:59.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:11:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:11:59.438+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:11:59.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:11:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:11:59.415+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:11:59.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:11:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:11:59.415+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:11:59.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:11:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:11:59.415+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:12:00.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:11:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:11:59.863+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:59.864567+0000) 2024-02-20T09:12:00.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:11:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:11:59.979+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:59.980726+0000) 2024-02-20T09:12:00.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:11:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:11:59.887+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:11:59.888742+0000) 2024-02-20T09:12:00.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:12:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:12:00.091+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:12:00.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:12:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:12:00.091+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:12:00.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:12:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:12:00.091+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:12:00.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:12:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:12:00.091+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:12:00.731 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:12:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:12:00.463+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:12:00.732 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:12:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:12:00.463+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:12:00.732 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:12:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:12:00.463+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:12:00.732 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:12:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:12:00.450+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:12:00.732 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:12:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:12:00.450+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:12:00.733 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:12:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:12:00.450+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:12:00.982 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:12:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:12:00.734+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 62126 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:12:00.982 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:12:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:12:00.728+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:00.730285+0000) 2024-02-20T09:12:00.982 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:12:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:12:00.732+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:00.734135+0000) 2024-02-20T09:12:00.983 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:12:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:12:00.887+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:00.889019+0000) 2024-02-20T09:12:00.983 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:12:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:12:00.728+0000 7f8a5f86e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:00.729590+0000) 2024-02-20T09:12:00.983 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:12:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:12:00.863+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:00.864678+0000) 2024-02-20T09:12:01.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:12:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:12:00.979+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:00.980937+0000) 2024-02-20T09:12:01.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:12:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:12:01.047+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:12:01.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:12:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:12:01.047+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:12:01.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:12:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:12:01.047+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:12:01.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:12:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:12:01.047+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:12:01.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:12:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:12:01.462+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:12:01.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:12:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:12:01.462+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:12:01.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:12:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:12:01.462+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:12:01.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:12:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:12:01.414+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:12:01.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:12:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:12:01.414+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:12:01.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:12:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:12:01.414+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:12:02.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:12:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:12:01.863+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:01.864784+0000) 2024-02-20T09:12:02.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:12:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:12:01.979+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:01.981108+0000) 2024-02-20T09:12:02.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:12:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:12:01.889+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:01.891311+0000) 2024-02-20T09:12:02.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:12:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:12:02.069+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:12:02.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:12:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:12:02.069+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:12:02.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:12:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:12:02.069+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:12:02.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:12:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:12:02.069+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:12:02.642 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:12:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:12:02.448+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:12:02.643 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:12:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:12:02.448+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:12:02.643 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:12:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:12:02.448+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:12:02.643 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:12:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:12:02.383+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:12:02.643 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:12:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:12:02.383+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:12:02.643 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:12:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:12:02.383+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:12:02.894 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:12:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:12:02.863+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:02.864968+0000) 2024-02-20T09:12:03.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:12:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:12:02.979+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:02.981282+0000) 2024-02-20T09:12:03.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:12:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:12:02.892+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:02.893526+0000) 2024-02-20T09:12:03.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:12:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:12:03.019+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:12:03.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:12:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:12:03.019+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:12:03.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:12:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:12:03.019+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:12:03.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:12:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:12:03.019+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:12:03.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:12:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:12:03.435+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:12:03.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:12:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:12:03.435+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:12:03.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:12:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:12:03.435+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:12:03.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:12:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:12:03.391+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:12:03.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:12:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:12:03.391+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:12:03.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:12:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:12:03.391+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:12:04.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:12:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:12:03.980+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:03.981428+0000) 2024-02-20T09:12:04.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:12:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:12:03.863+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:03.865133+0000) 2024-02-20T09:12:04.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:12:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:12:03.894+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:03.895601+0000) 2024-02-20T09:12:04.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:12:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:12:04.057+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:12:04.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:12:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:12:04.057+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:12:04.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:12:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:12:04.057+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:12:04.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:12:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:12:04.057+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:12:04.643 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:12:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:12:04.375+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:12:04.644 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:12:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:12:04.375+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:12:04.644 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:12:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:12:04.375+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:12:04.644 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:12:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:12:04.485+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:12:04.644 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:12:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:12:04.485+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:12:04.644 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:12:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:12:04.485+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:12:04.898 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:12:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:12:04.863+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:04.865325+0000) 2024-02-20T09:12:05.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:12:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:12:04.980+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:04.981566+0000) 2024-02-20T09:12:05.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:12:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:12:04.895+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:04.897033+0000) 2024-02-20T09:12:05.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:12:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:12:05.011+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:12:05.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:12:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:12:05.011+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:12:05.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:12:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:12:05.011+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:12:05.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:12:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:12:05.011+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:12:05.644 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:12:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:12:05.472+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:12:05.644 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:12:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:12:05.472+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:12:05.644 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:12:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:12:05.472+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:12:05.644 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:12:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:12:05.367+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:12:05.644 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:12:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:12:05.367+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:12:05.645 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:12:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:12:05.367+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:12:05.898 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:12:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:12:05.740+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 62184 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:12:05.898 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:12:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:12:05.738+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:05.740124+0000) 2024-02-20T09:12:05.899 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:12:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:12:05.735+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:05.736903+0000) 2024-02-20T09:12:05.899 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:12:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:12:05.735+0000 7f8a5f86e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:05.737019+0000) 2024-02-20T09:12:05.899 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:12:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:12:05.864+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:05.865515+0000) 2024-02-20T09:12:06.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:12:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:12:05.980+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:05.981719+0000) 2024-02-20T09:12:06.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:12:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:12:05.895+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:05.897243+0000) 2024-02-20T09:12:06.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:12:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:12:06.053+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:12:06.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:12:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:12:06.053+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:12:06.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:12:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:12:06.053+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:12:06.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:12:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:12:06.053+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:12:06.508 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:12:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:12:06.350+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:12:06.509 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:12:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:12:06.350+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:12:06.509 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:12:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:12:06.350+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:12:06.509 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:12:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:12:06.482+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:12:06.509 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:12:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:12:06.482+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:12:06.509 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:12:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:12:06.482+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:12:07.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:12:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:12:06.864+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:06.865597+0000) 2024-02-20T09:12:07.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:12:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:12:06.980+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:06.981812+0000) 2024-02-20T09:12:07.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:12:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:12:06.898+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:06.899573+0000) 2024-02-20T09:12:07.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:12:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:12:07.034+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:12:07.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:12:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:12:07.034+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:12:07.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:12:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:12:07.034+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:12:07.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:12:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:12:07.034+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:12:07.742 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:12:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:12:07.397+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:12:07.742 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:12:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:12:07.397+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:12:07.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:12:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:12:07.397+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:12:07.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:12:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:12:07.487+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:12:07.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:12:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:12:07.487+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:12:07.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:12:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:12:07.487+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:12:08.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:12:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:12:07.864+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:07.865739+0000) 2024-02-20T09:12:08.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:12:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:12:07.980+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:07.981924+0000) 2024-02-20T09:12:08.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:12:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:12:07.900+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:07.901682+0000) 2024-02-20T09:12:08.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:12:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:12:08.058+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:12:08.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:12:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:12:08.058+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:12:08.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:12:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:12:08.058+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:12:08.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:12:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:12:08.058+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:12:08.742 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:12:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:12:08.410+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:12:08.742 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:12:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:12:08.410+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:12:08.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:12:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:12:08.410+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:12:08.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:12:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:12:08.461+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:12:08.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:12:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:12:08.461+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:12:08.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:12:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:12:08.461+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:12:09.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:12:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:12:08.864+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:08.865901+0000) 2024-02-20T09:12:09.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:12:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:12:08.980+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:08.982061+0000) 2024-02-20T09:12:09.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:12:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:12:08.902+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:08.904378+0000) 2024-02-20T09:12:09.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:12:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:12:09.095+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:12:09.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:12:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:12:09.095+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:12:09.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:12:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:12:09.095+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:12:09.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:12:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:12:09.095+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:12:09.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:12:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:12:09.485+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:12:09.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:12:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:12:09.485+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:12:09.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:12:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:12:09.485+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:12:09.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:12:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:12:09.430+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:12:09.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:12:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:12:09.430+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:12:09.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:12:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:12:09.430+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:12:10.145 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:12:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:12:09.864+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:09.866059+0000) 2024-02-20T09:12:10.146 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:12:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:12:09.980+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:09.982193+0000) 2024-02-20T09:12:10.146 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:12:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:12:09.903+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:09.905273+0000) 2024-02-20T09:12:10.146 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:12:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:12:10.143+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:12:10.146 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:12:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:12:10.143+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:12:10.412 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:12:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:12:10.143+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:12:10.413 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:12:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:12:10.143+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:12:10.742 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:12:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:12:10.447+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:12:10.742 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:12:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:12:10.447+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:12:10.742 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:12:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:12:10.447+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:12:10.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:12:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:12:10.440+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:12:10.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:12:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:12:10.440+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:12:10.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:12:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:12:10.440+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:12:11.184 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:12:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:12:10.747+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 62241 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:12:11.185 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:12:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:12:10.742+0000 7f8a5f86e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:10.744028+0000) 2024-02-20T09:12:11.185 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:12:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:12:10.864+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:10.866168+0000) 2024-02-20T09:12:11.185 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:12:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:12:10.742+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:10.743508+0000) 2024-02-20T09:12:11.185 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:12:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:12:10.980+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:10.982307+0000) 2024-02-20T09:12:11.186 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:12:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:12:10.746+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:10.747846+0000) 2024-02-20T09:12:11.186 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:12:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:12:10.904+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:10.905503+0000) 2024-02-20T09:12:11.186 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:12:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:12:11.181+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:12:11.186 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:12:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:12:11.181+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:12:11.186 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:12:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:12:11.181+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:12:11.475 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:12:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:12:11.182+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:12:11.742 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:12:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:12:11.476+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:12:11.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:12:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:12:11.477+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:12:11.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:12:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:12:11.477+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:12:11.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:12:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:12:11.473+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:12:11.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:12:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:12:11.473+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:12:11.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:12:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:12:11.473+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:12:12.225 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:12:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:12:11.864+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:11.866272+0000) 2024-02-20T09:12:12.225 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:12:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:12:11.906+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:11.907566+0000) 2024-02-20T09:12:12.226 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:12:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:12:11.981+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:11.982480+0000) 2024-02-20T09:12:12.481 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:12:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:12:12.222+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:12:12.481 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:12:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:12:12.222+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:12:12.481 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:12:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:12:12.222+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:12:12.481 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:12:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:12:12.222+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:12:12.742 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:12:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:12:12.496+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:12:12.742 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:12:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:12:12.496+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:12:12.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:12:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:12:12.496+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:12:12.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:12:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:12:12.479+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:12:12.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:12:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:12:12.479+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:12:12.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:12:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:12:12.479+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:12:13.190 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:12:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:12:12.864+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:12.866388+0000) 2024-02-20T09:12:13.191 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:12:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:12:12.981+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:12.982583+0000) 2024-02-20T09:12:13.191 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:12:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:12:12.908+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:12.909574+0000) 2024-02-20T09:12:13.191 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:12:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:12:13.188+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:12:13.191 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:12:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:12:13.188+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:12:13.191 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:12:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:12:13.188+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:12:13.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:12:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:12:13.188+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:12:13.856 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:12:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:12:13.508+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:12:13.856 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:12:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:12:13.508+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:12:13.856 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:12:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:12:13.508+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:12:13.856 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:12:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:12:13.507+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:12:13.857 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:12:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:12:13.507+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:12:13.857 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:12:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:12:13.507+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:12:14.182 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:12:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:12:13.865+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:13.866489+0000) 2024-02-20T09:12:14.182 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:12:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:12:13.981+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:13.982696+0000) 2024-02-20T09:12:14.183 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:12:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:12:13.911+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:13.913214+0000) 2024-02-20T09:12:14.183 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:12:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:12:14.179+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:12:14.183 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:12:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:12:14.179+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:12:14.183 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:12:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:12:14.179+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:12:14.183 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:12:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:12:14.179+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:12:14.856 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:12:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:12:14.521+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:12:14.856 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:12:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:12:14.521+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:12:14.856 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:12:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:12:14.521+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:12:14.856 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:12:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:12:14.553+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:12:14.856 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:12:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:12:14.553+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:12:14.857 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:12:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:12:14.553+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:12:15.207 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:12:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:12:14.866+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:14.866617+0000) 2024-02-20T09:12:15.208 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:12:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:12:14.982+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:14.982809+0000) 2024-02-20T09:12:15.208 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:12:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:12:14.913+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:14.914455+0000) 2024-02-20T09:12:15.208 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:12:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:12:15.205+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:12:15.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:12:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:12:15.205+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:12:15.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:12:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:12:15.205+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:12:15.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:12:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:12:15.205+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:12:15.763 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:12:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:12:15.754+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 62299 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:12:15.763 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:12:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:12:15.752+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:15.753771+0000) 2024-02-20T09:12:15.764 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:12:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:12:15.541+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:12:15.764 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:12:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:12:15.541+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:12:15.764 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:12:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:12:15.541+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:12:15.764 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:12:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:12:15.748+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:15.750052+0000) 2024-02-20T09:12:15.764 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:12:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:12:15.579+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:12:15.764 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:12:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:12:15.579+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:12:15.765 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:12:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:12:15.579+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:12:15.765 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:12:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:12:15.748+0000 7f8a5f86e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:15.750362+0000) 2024-02-20T09:12:16.214 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:12:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:12:15.981+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:15.982918+0000) 2024-02-20T09:12:16.214 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:12:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:12:15.865+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:15.866711+0000) 2024-02-20T09:12:16.214 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:12:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:12:15.913+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:15.914692+0000) 2024-02-20T09:12:16.215 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:12:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:12:16.211+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:12:16.215 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:12:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:12:16.211+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:12:16.215 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:12:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:12:16.211+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:12:16.215 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:12:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:12:16.211+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:12:16.856 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:12:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:12:16.626+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:12:16.857 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:12:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:12:16.626+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:12:16.857 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:12:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:12:16.626+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:12:16.857 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:12:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:12:16.507+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:12:16.857 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:12:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:12:16.507+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:12:16.857 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:12:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:12:16.507+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:12:17.202 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:12:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:12:16.981+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:16.982999+0000) 2024-02-20T09:12:17.202 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:12:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:12:16.866+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:16.866818+0000) 2024-02-20T09:12:17.202 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:12:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:12:16.916+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:16.916765+0000) 2024-02-20T09:12:17.202 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:12:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:12:17.200+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:12:17.203 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:12:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:12:17.200+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:12:17.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:12:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:12:17.200+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:12:17.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:12:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:12:17.200+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:12:17.856 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:12:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:12:17.586+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:12:17.856 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:12:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:12:17.586+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:12:17.856 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:12:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:12:17.586+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:12:17.857 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:12:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:12:17.529+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:12:17.857 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:12:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:12:17.529+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:12:17.857 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:12:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:12:17.529+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:12:18.204 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:12:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:12:17.866+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:17.866968+0000) 2024-02-20T09:12:18.204 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:12:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:12:17.919+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:17.919544+0000) 2024-02-20T09:12:18.204 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:12:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:12:17.981+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:17.983114+0000) 2024-02-20T09:12:18.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:12:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:12:18.202+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:12:18.495 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:12:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:12:18.202+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:12:18.495 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:12:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:12:18.202+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:12:18.495 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:12:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:12:18.202+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:12:18.856 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:12:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:12:18.523+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:12:18.856 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:12:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:12:18.523+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:12:18.857 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:12:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:12:18.523+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:12:18.857 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:12:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:12:18.635+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:12:18.857 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:12:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:12:18.635+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:12:18.857 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:12:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:12:18.635+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:12:19.170 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:12:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:12:18.982+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:18.983240+0000) 2024-02-20T09:12:19.170 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:12:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:12:18.866+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:18.867121+0000) 2024-02-20T09:12:19.171 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:12:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:12:18.921+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:18.921606+0000) 2024-02-20T09:12:19.171 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:12:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:12:19.168+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:12:19.479 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:12:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:12:19.168+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:12:19.479 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:12:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:12:19.168+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:12:19.479 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:12:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:12:19.168+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:12:19.742 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:12:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:12:19.609+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:12:19.742 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:12:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:12:19.609+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:12:19.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:12:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:12:19.609+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:12:19.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:12:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:12:19.477+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:12:19.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:12:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:12:19.477+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:12:19.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:12:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:12:19.477+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:12:20.147 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:12:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:12:19.866+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:19.867220+0000) 2024-02-20T09:12:20.147 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:12:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:12:19.982+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:19.983367+0000) 2024-02-20T09:12:20.147 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:12:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:12:19.922+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:19.922473+0000) 2024-02-20T09:12:20.148 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:12:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:12:20.145+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:12:20.148 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:12:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:12:20.145+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:12:20.412 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:12:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:12:20.145+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:12:20.413 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:12:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:12:20.145+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:12:20.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:12:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:12:20.440+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:12:20.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:12:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:12:20.440+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:12:20.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:12:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:12:20.440+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:12:20.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:12:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:12:20.566+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:12:20.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:12:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:12:20.566+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:12:20.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:12:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:12:20.566+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:12:21.189 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:12:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:12:20.762+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 62358 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:12:21.190 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:12:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:12:20.756+0000 7f8a5f86e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:20.757045+0000) 2024-02-20T09:12:21.190 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:12:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:12:20.866+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:20.867338+0000) 2024-02-20T09:12:21.190 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:12:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:12:20.756+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:20.757278+0000) 2024-02-20T09:12:21.190 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:12:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:12:20.983+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:20.983485+0000) 2024-02-20T09:12:21.191 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:12:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:12:20.759+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:20.759601+0000) 2024-02-20T09:12:21.191 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:12:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:12:20.922+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:20.922712+0000) 2024-02-20T09:12:21.191 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:12:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:12:21.187+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:12:21.191 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:12:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:12:21.187+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:12:21.191 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:12:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:12:21.188+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:12:21.191 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:12:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:12:21.188+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:12:21.742 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:12:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:12:21.597+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:12:21.742 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:12:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:12:21.597+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:12:21.742 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:12:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:12:21.597+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:12:21.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:12:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:12:21.450+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:12:21.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:12:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:12:21.450+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:12:21.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:12:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:12:21.450+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:12:22.186 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:12:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:12:21.983+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:21.983604+0000) 2024-02-20T09:12:22.186 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:12:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:12:21.867+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:21.867441+0000) 2024-02-20T09:12:22.187 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:12:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:12:21.925+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:21.926363+0000) 2024-02-20T09:12:22.187 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:12:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:12:22.184+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:12:22.187 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:12:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:12:22.184+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:12:22.187 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:12:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:12:22.184+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:12:22.187 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:12:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:12:22.184+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:12:22.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:12:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:12:22.417+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:12:22.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:12:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:12:22.417+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:12:22.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:12:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:12:22.417+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:12:22.857 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:12:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:12:22.611+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:12:22.857 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:12:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:12:22.611+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:12:22.857 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:12:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:12:22.611+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:12:23.221 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:12:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:12:22.867+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:22.867572+0000) 2024-02-20T09:12:23.221 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:12:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:12:22.983+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:22.983691+0000) 2024-02-20T09:12:23.221 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:12:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:12:22.929+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:22.930004+0000) 2024-02-20T09:12:23.221 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:12:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:12:23.219+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:12:23.222 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:12:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:12:23.219+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:12:23.222 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:12:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:12:23.219+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:12:23.222 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:12:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:12:23.219+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:12:23.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:12:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:12:23.390+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:12:23.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:12:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:12:23.390+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:12:23.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:12:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:12:23.390+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:12:23.857 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:12:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:12:23.645+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:12:23.857 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:12:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:12:23.645+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:12:23.857 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:12:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:12:23.645+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:12:24.196 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:12:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:12:23.867+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:23.867665+0000) 2024-02-20T09:12:24.196 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:12:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:12:23.983+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:23.983781+0000) 2024-02-20T09:12:24.196 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:12:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:12:23.933+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:23.933700+0000) 2024-02-20T09:12:24.196 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:12:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:12:24.194+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:12:24.197 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:12:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:12:24.194+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:12:24.197 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:12:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:12:24.194+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:12:24.197 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:12:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:12:24.194+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:12:24.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:12:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:12:24.389+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:12:24.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:12:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:12:24.389+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:12:24.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:12:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:12:24.389+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:12:24.857 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:12:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:12:24.654+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:12:24.857 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:12:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:12:24.654+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:12:24.858 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:12:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:12:24.654+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:12:25.192 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:12:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:12:24.867+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:24.867757+0000) 2024-02-20T09:12:25.193 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:12:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:12:24.983+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:24.983956+0000) 2024-02-20T09:12:25.193 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:12:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:12:24.934+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:24.935123+0000) 2024-02-20T09:12:25.193 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:12:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:12:25.191+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:12:25.193 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:12:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:12:25.191+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:12:25.194 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:12:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:12:25.191+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:12:25.194 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:12:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:12:25.191+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:12:25.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:12:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:12:25.345+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:12:25.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:12:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:12:25.345+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:12:25.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:12:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:12:25.345+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:12:25.765 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:12:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:12:25.694+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:12:25.765 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:12:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:12:25.694+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:12:25.765 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:12:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:12:25.694+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:12:26.210 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:12:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:12:25.769+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 62416 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:12:26.210 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:12:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:12:25.763+0000 7f8a5f86e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:25.764030+0000) 2024-02-20T09:12:26.210 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:12:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:12:25.867+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:25.867852+0000) 2024-02-20T09:12:26.211 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:12:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:12:25.763+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:25.764069+0000) 2024-02-20T09:12:26.211 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:12:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:12:25.983+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:25.984138+0000) 2024-02-20T09:12:26.211 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:12:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:12:25.767+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:25.767548+0000) 2024-02-20T09:12:26.211 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:12:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:12:25.934+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:25.935338+0000) 2024-02-20T09:12:26.211 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:12:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:12:26.208+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:12:26.211 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:12:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:12:26.208+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:12:26.211 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:12:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:12:26.208+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:12:26.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:12:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:12:26.208+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:12:26.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:12:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:12:26.335+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:12:26.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:12:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:12:26.335+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:12:26.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:12:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:12:26.335+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:12:26.857 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:12:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:12:26.675+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:12:26.857 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:12:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:12:26.675+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:12:26.858 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:12:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:12:26.675+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:12:27.222 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:12:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:12:26.867+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:26.867935+0000) 2024-02-20T09:12:27.222 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:12:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:12:26.983+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:26.984274+0000) 2024-02-20T09:12:27.223 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:12:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:12:26.936+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:26.937313+0000) 2024-02-20T09:12:27.223 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:12:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:12:27.220+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:12:27.223 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:12:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:12:27.220+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:12:27.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:12:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:12:27.220+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:12:27.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:12:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:12:27.221+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:12:27.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:12:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:12:27.326+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:12:27.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:12:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:12:27.326+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:12:27.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:12:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:12:27.326+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:12:27.857 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:12:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:12:27.694+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:12:27.858 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:12:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:12:27.694+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:12:27.858 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:12:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:12:27.694+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:12:28.184 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:12:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:12:27.867+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:27.868033+0000) 2024-02-20T09:12:28.185 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:12:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:12:27.984+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:27.984483+0000) 2024-02-20T09:12:28.185 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:12:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:12:27.940+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:27.940865+0000) 2024-02-20T09:12:28.185 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:12:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:12:28.183+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:12:28.185 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:12:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:12:28.183+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:12:28.185 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:12:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:12:28.183+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:12:28.186 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:12:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:12:28.183+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:12:28.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:12:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:12:28.336+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:12:28.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:12:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:12:28.336+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:12:28.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:12:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:12:28.336+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:12:28.858 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:12:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:12:28.665+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:12:28.858 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:12:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:12:28.665+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:12:28.858 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:12:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:12:28.665+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:12:29.227 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:12:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:12:28.867+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:28.868130+0000) 2024-02-20T09:12:29.228 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:12:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:12:28.984+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:28.984622+0000) 2024-02-20T09:12:29.228 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:12:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:12:28.943+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:28.944176+0000) 2024-02-20T09:12:29.228 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:12:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:12:29.225+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:12:29.228 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:12:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:12:29.225+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:12:29.228 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:12:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:12:29.225+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:12:29.228 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:12:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:12:29.226+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:12:29.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:12:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:12:29.333+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:12:29.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:12:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:12:29.333+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:12:29.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:12:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:12:29.333+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:12:29.858 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:12:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:12:29.643+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:12:29.858 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:12:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:12:29.643+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:12:29.858 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:12:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:12:29.643+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:12:30.229 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:12:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:12:29.867+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:29.868256+0000) 2024-02-20T09:12:30.230 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:12:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:12:29.945+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:29.945806+0000) 2024-02-20T09:12:30.230 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:12:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:12:29.984+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:29.984781+0000) 2024-02-20T09:12:30.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:12:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:12:30.228+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:12:30.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:12:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:12:30.228+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:12:30.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:12:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:12:30.228+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:12:30.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:12:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:12:30.228+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:12:30.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:12:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:12:30.334+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:12:30.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:12:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:12:30.334+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:12:30.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:12:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:12:30.334+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:12:30.772 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:12:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:12:30.649+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:12:30.773 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:12:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:12:30.649+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:12:30.773 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:12:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:12:30.649+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:12:31.188 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:12:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:12:30.776+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 62473 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:12:31.189 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:12:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:12:30.771+0000 7f8a5f86e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:30.771520+0000) 2024-02-20T09:12:31.189 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:12:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:12:30.867+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:30.868385+0000) 2024-02-20T09:12:31.189 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:12:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:12:30.771+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:30.771861+0000) 2024-02-20T09:12:31.190 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:12:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:12:30.984+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:30.984928+0000) 2024-02-20T09:12:31.190 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:12:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:12:30.775+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:30.775593+0000) 2024-02-20T09:12:31.190 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:12:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:12:30.945+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:30.946022+0000) 2024-02-20T09:12:31.190 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:12:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:12:31.187+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:12:31.190 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:12:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:12:31.187+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:12:31.190 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:12:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:12:31.187+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:12:31.191 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:12:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:12:31.187+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:12:31.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:12:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:12:31.362+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:12:31.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:12:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:12:31.362+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:12:31.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:12:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:12:31.362+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:12:31.858 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:12:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:12:31.630+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:12:31.859 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:12:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:12:31.630+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:12:31.859 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:12:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:12:31.630+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:12:32.231 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:12:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:12:31.984+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:31.985084+0000) 2024-02-20T09:12:32.232 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:12:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:12:31.868+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:31.868474+0000) 2024-02-20T09:12:32.232 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:12:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:12:31.947+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:31.948121+0000) 2024-02-20T09:12:32.232 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:12:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:12:32.229+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:12:32.232 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:12:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:12:32.229+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:12:32.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:12:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:12:32.229+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:12:32.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:12:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:12:32.230+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:12:32.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:12:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:12:32.366+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:12:32.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:12:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:12:32.366+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:12:32.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:12:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:12:32.366+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:12:32.858 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:12:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:12:32.624+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:12:32.858 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:12:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:12:32.624+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:12:32.859 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:12:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:12:32.624+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:12:33.240 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:12:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:12:32.868+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:32.868578+0000) 2024-02-20T09:12:33.240 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:12:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:12:32.984+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:32.985202+0000) 2024-02-20T09:12:33.240 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:12:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:12:32.949+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:32.950111+0000) 2024-02-20T09:12:33.240 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:12:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:12:33.238+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:12:33.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:12:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:12:33.238+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:12:33.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:12:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:12:33.238+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:12:33.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:12:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:12:33.238+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:12:33.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:12:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:12:33.373+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:12:33.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:12:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:12:33.373+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:12:33.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:12:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:12:33.373+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:12:33.858 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:12:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:12:33.673+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:12:33.859 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:12:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:12:33.673+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:12:33.859 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:12:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:12:33.673+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:12:34.216 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:12:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:12:33.868+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:33.868672+0000) 2024-02-20T09:12:34.217 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:12:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:12:33.984+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:33.985380+0000) 2024-02-20T09:12:34.217 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:12:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:12:33.951+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:33.952119+0000) 2024-02-20T09:12:34.217 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:12:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:12:34.215+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:12:34.217 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:12:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:12:34.215+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:12:34.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:12:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:12:34.215+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:12:34.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:12:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:12:34.215+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:12:34.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:12:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:12:34.348+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:12:34.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:12:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:12:34.348+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:12:34.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:12:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:12:34.348+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:12:34.858 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:12:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:12:34.695+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:12:34.859 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:12:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:12:34.695+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:12:34.859 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:12:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:12:34.695+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:12:35.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:12:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:12:34.985+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:34.985536+0000) 2024-02-20T09:12:35.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:12:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:12:34.868+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:34.868769+0000) 2024-02-20T09:12:35.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:12:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:12:34.953+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:34.953582+0000) 2024-02-20T09:12:35.512 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:12:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:12:35.242+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:12:35.512 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:12:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:12:35.242+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:12:35.512 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:12:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:12:35.242+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:12:35.513 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:12:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:12:35.242+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:12:35.513 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:12:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:12:35.384+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:12:35.513 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:12:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:12:35.384+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:12:35.513 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:12:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:12:35.384+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:12:35.779 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:12:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:12:35.683+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:12:35.779 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:12:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:12:35.683+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:12:35.779 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:12:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:12:35.683+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:12:36.238 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:12:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:12:35.784+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 62527 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:12:36.238 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:12:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:12:35.778+0000 7f8a5f86e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:35.778445+0000) 2024-02-20T09:12:36.239 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:12:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:12:35.868+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:35.868863+0000) 2024-02-20T09:12:36.239 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:12:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:12:35.778+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:35.779327+0000) 2024-02-20T09:12:36.239 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:12:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:12:35.985+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:35.985661+0000) 2024-02-20T09:12:36.239 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:12:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:12:35.782+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:35.782834+0000) 2024-02-20T09:12:36.239 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:12:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:12:35.953+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:35.953915+0000) 2024-02-20T09:12:36.240 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:12:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:12:36.236+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:12:36.240 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:12:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:12:36.236+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:12:36.240 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:12:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:12:36.236+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:12:36.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:12:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:12:36.236+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:12:36.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:12:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:12:36.380+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:12:36.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:12:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:12:36.380+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:12:36.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:12:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:12:36.380+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:12:36.859 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:12:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:12:36.672+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:12:36.859 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:12:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:12:36.672+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:12:36.859 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:12:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:12:36.672+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:12:37.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:12:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:12:36.955+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:36.956143+0000) 2024-02-20T09:12:37.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:12:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:12:36.985+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:36.985736+0000) 2024-02-20T09:12:37.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:12:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:12:36.868+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:36.868969+0000) 2024-02-20T09:12:37.669 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:12:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:12:37.375+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:12:37.669 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:12:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:12:37.375+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:12:37.669 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:12:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:12:37.375+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:12:37.670 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:12:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:12:37.265+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:12:37.670 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:12:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:12:37.265+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:12:37.670 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:12:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:12:37.265+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:12:37.670 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:12:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:12:37.265+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:12:37.959 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:12:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:12:37.673+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:12:38.028 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:12:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:12:37.673+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:12:38.028 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:12:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:12:37.673+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:12:38.029 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:12:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:12:37.868+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:37.869097+0000) 2024-02-20T09:12:38.235 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:12:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:12:37.985+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:37.985833+0000) 2024-02-20T09:12:38.235 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:12:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:12:37.957+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:37.958269+0000) 2024-02-20T09:12:38.235 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:12:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:12:38.233+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:12:38.235 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:12:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:12:38.233+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:12:38.236 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:12:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:12:38.233+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:12:38.236 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:12:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:12:38.233+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:12:38.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:12:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:12:38.369+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:12:38.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:12:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:12:38.369+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:12:38.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:12:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:12:38.369+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:12:38.859 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:12:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:12:38.708+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:12:38.859 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:12:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:12:38.708+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:12:38.860 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:12:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:12:38.708+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:12:39.203 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:12:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:12:38.868+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:38.869196+0000) 2024-02-20T09:12:39.204 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:12:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:12:38.960+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:38.960470+0000) 2024-02-20T09:12:39.204 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:12:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:12:38.985+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:38.986008+0000) 2024-02-20T09:12:39.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:12:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:12:39.399+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:12:39.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:12:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:12:39.399+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:12:39.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:12:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:12:39.399+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:12:39.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:12:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:12:39.202+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:12:39.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:12:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:12:39.202+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:12:39.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:12:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:12:39.202+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:12:39.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:12:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:12:39.202+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:12:39.859 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:12:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:12:39.688+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:12:39.860 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:12:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:12:39.688+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:12:39.860 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:12:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:12:39.688+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:12:40.210 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:12:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:12:39.867+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:39.869298+0000) 2024-02-20T09:12:40.210 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:12:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:12:39.960+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:39.961453+0000) 2024-02-20T09:12:40.211 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:12:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:12:39.984+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:39.986152+0000) 2024-02-20T09:12:40.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:12:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:12:40.388+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:12:40.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:12:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:12:40.388+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:12:40.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:12:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:12:40.388+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:12:40.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:12:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:12:40.207+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:12:40.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:12:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:12:40.207+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:12:40.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:12:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:12:40.207+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:12:40.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:12:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:12:40.207+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:12:40.787 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:12:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:12:40.708+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:12:40.787 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:12:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:12:40.708+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:12:40.787 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:12:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:12:40.708+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:12:40.787 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:12:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:12:40.784+0000 7f8a5f86e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:40.785790+0000) 2024-02-20T09:12:41.182 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:12:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:12:40.789+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 62582 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:12:41.182 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:12:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:12:40.868+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:40.869422+0000) 2024-02-20T09:12:41.182 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:12:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:12:40.788+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:40.789451+0000) 2024-02-20T09:12:41.183 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:12:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:12:40.960+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:40.961634+0000) 2024-02-20T09:12:41.183 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:12:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:12:40.785+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:40.786746+0000) 2024-02-20T09:12:41.183 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:12:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:12:40.984+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:40.986301+0000) 2024-02-20T09:12:41.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:12:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:12:41.356+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:12:41.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:12:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:12:41.356+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:12:41.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:12:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:12:41.356+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:12:41.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:12:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:12:41.179+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:12:41.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:12:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:12:41.179+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:12:41.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:12:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:12:41.179+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:12:41.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:12:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:12:41.179+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:12:41.860 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:12:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:12:41.681+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:12:41.860 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:12:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:12:41.681+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:12:41.860 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:12:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:12:41.681+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:12:42.173 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:12:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:12:41.868+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:41.869507+0000) 2024-02-20T09:12:42.174 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:12:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:12:41.962+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:41.963770+0000) 2024-02-20T09:12:42.174 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:12:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:12:41.985+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:41.986415+0000) 2024-02-20T09:12:42.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:12:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:12:42.371+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:12:42.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:12:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:12:42.371+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:12:42.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:12:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:12:42.371+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:12:42.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:12:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:12:42.170+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:12:42.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:12:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:12:42.170+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:12:42.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:12:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:12:42.170+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:12:42.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:12:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:12:42.171+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:12:42.860 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:12:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:12:42.721+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:12:42.860 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:12:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:12:42.721+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:12:42.860 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:12:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:12:42.721+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:12:43.166 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:12:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:12:42.985+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:42.986579+0000) 2024-02-20T09:12:43.166 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:12:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:12:42.868+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:42.869616+0000) 2024-02-20T09:12:43.166 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:12:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:12:42.966+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:42.967460+0000) 2024-02-20T09:12:43.167 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:12:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:12:43.163+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:12:43.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:12:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:12:43.163+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:12:43.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:12:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:12:43.163+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:12:43.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:12:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:12:43.163+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:12:43.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:12:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:12:43.405+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:12:43.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:12:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:12:43.405+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:12:43.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:12:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:12:43.405+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:12:43.860 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:12:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:12:43.731+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:12:43.860 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:12:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:12:43.731+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:12:43.860 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:12:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:12:43.731+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:12:44.181 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:12:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:12:43.868+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:43.869726+0000) 2024-02-20T09:12:44.182 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:12:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:12:43.985+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:43.986721+0000) 2024-02-20T09:12:44.182 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:12:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:12:43.968+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:43.969522+0000) 2024-02-20T09:12:44.182 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:12:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:12:44.179+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:12:44.182 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:12:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:12:44.179+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:12:44.182 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:12:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:12:44.179+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:12:44.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:12:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:12:44.179+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:12:44.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:12:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:12:44.421+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:12:44.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:12:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:12:44.421+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:12:44.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:12:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:12:44.421+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:12:44.860 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:12:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:12:44.694+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:12:44.860 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:12:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:12:44.694+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:12:44.860 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:12:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:12:44.694+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:12:45.139 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:12:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:12:44.868+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:44.869860+0000) 2024-02-20T09:12:45.139 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:12:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:12:44.985+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:44.986852+0000) 2024-02-20T09:12:45.140 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:12:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:12:44.969+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:44.970415+0000) 2024-02-20T09:12:45.140 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:12:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:12:45.136+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:12:45.140 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:12:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:12:45.136+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:12:45.428 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:12:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:12:45.136+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:12:45.428 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:12:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:12:45.136+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:12:45.721 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:12:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:12:45.426+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:12:45.722 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:12:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:12:45.426+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:12:45.722 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:12:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:12:45.426+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:12:45.972 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:12:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:12:45.797+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 62636 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:12:45.972 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:12:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:12:45.794+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:45.795918+0000) 2024-02-20T09:12:45.973 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:12:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:12:45.790+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:45.792212+0000) 2024-02-20T09:12:45.973 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:12:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:12:45.719+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:12:45.973 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:12:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:12:45.719+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:12:45.973 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:12:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:12:45.719+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:12:45.973 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:12:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:12:45.791+0000 7f8a5f86e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:45.793177+0000) 2024-02-20T09:12:45.973 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:12:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:12:45.868+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:45.869981+0000) 2024-02-20T09:12:46.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:12:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:12:45.985+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:45.987039+0000) 2024-02-20T09:12:46.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:12:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:12:45.969+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:45.970685+0000) 2024-02-20T09:12:46.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:12:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:12:46.132+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:12:46.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:12:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:12:46.132+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:12:46.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:12:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:12:46.132+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:12:46.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:12:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:12:46.133+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:12:46.674 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:12:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:12:46.416+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:12:46.675 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:12:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:12:46.416+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:12:46.675 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:12:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:12:46.416+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:12:46.974 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:12:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:12:46.675+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:12:46.975 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:12:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:12:46.675+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:12:46.975 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:12:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:12:46.675+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:12:46.975 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:12:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:12:46.868+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:46.870077+0000) 2024-02-20T09:12:47.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:12:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:12:46.985+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:46.987207+0000) 2024-02-20T09:12:47.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:12:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:12:46.971+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:46.973326+0000) 2024-02-20T09:12:47.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:12:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:12:47.121+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:12:47.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:12:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:12:47.121+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:12:47.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:12:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:12:47.121+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:12:47.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:12:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:12:47.121+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:12:47.742 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:12:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:12:47.712+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:12:47.742 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:12:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:12:47.712+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:12:47.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:12:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:12:47.712+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:12:47.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:12:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:12:47.465+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:12:47.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:12:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:12:47.465+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:12:47.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:12:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:12:47.465+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:12:48.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:12:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:12:47.868+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:47.870183+0000) 2024-02-20T09:12:48.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:12:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:12:47.985+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:47.987341+0000) 2024-02-20T09:12:48.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:12:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:12:47.974+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:47.976294+0000) 2024-02-20T09:12:48.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:12:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:12:48.094+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:12:48.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:12:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:12:48.094+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:12:48.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:12:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:12:48.094+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:12:48.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:12:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:12:48.094+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:12:48.861 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:12:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:12:48.748+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:12:48.862 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:12:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:12:48.748+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:12:48.862 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:12:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:12:48.748+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:12:48.862 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:12:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:12:48.515+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:12:48.862 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:12:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:12:48.515+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:12:48.862 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:12:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:12:48.515+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:12:49.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:12:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:12:48.868+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:48.870284+0000) 2024-02-20T09:12:49.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:12:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:12:48.986+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:48.987506+0000) 2024-02-20T09:12:49.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:12:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:12:48.977+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:48.978465+0000) 2024-02-20T09:12:49.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:12:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:12:49.075+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:12:49.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:12:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:12:49.075+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:12:49.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:12:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:12:49.075+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:12:49.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:12:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:12:49.075+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:12:49.861 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:12:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:12:49.747+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:12:49.861 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:12:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:12:49.747+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:12:49.861 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:12:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:12:49.747+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:12:49.861 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:12:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:12:49.550+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:12:49.862 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:12:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:12:49.550+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:12:49.862 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:12:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:12:49.550+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:12:50.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:12:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:12:49.868+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:49.870386+0000) 2024-02-20T09:12:50.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:12:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:12:49.986+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:49.987648+0000) 2024-02-20T09:12:50.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:12:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:12:49.978+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:49.980011+0000) 2024-02-20T09:12:50.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:12:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:12:50.104+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:12:50.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:12:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:12:50.104+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:12:50.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:12:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:12:50.104+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:12:50.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:12:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:12:50.104+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:12:50.677 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:12:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:12:50.553+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:12:50.678 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:12:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:12:50.553+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:12:50.678 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:12:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:12:50.553+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:12:50.981 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:12:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:12:50.801+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:50.802542+0000) 2024-02-20T09:12:50.981 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:12:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:12:50.798+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:50.799530+0000) 2024-02-20T09:12:50.982 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:12:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:12:50.804+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 62688 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:12:50.982 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:12:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:12:50.738+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:12:50.982 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:12:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:12:50.738+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:12:50.982 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:12:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:12:50.738+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:12:50.982 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:12:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:12:50.798+0000 7f8a5f86e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:50.799460+0000) 2024-02-20T09:12:50.982 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:12:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:12:50.869+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:50.870491+0000) 2024-02-20T09:12:51.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:12:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:12:50.986+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:50.987791+0000) 2024-02-20T09:12:51.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:12:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:12:50.978+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:50.980203+0000) 2024-02-20T09:12:51.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:12:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:12:51.140+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:12:51.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:12:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:12:51.140+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:12:51.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:12:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:12:51.140+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:12:51.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:12:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:12:51.140+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:12:51.861 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:12:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:12:51.704+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:12:51.861 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:12:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:12:51.704+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:12:51.861 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:12:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:12:51.704+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:12:51.862 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:12:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:12:51.534+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:12:51.862 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:12:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:12:51.534+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:12:51.862 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:12:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:12:51.534+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:12:52.179 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:12:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:12:51.869+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:51.870573+0000) 2024-02-20T09:12:52.180 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:12:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:12:51.986+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:51.987895+0000) 2024-02-20T09:12:52.180 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:12:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:12:51.982+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:51.983754+0000) 2024-02-20T09:12:52.180 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:12:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:12:52.177+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:12:52.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:12:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:12:52.177+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:12:52.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:12:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:12:52.177+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:12:52.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:12:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:12:52.177+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:12:52.861 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:12:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:12:52.567+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:12:52.861 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:12:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:12:52.567+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:12:52.862 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:12:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:12:52.567+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:12:52.862 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:12:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:12:52.662+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:12:52.862 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:12:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:12:52.662+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:12:52.862 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:12:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:12:52.662+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:12:53.218 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:12:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:12:52.869+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:52.870678+0000) 2024-02-20T09:12:53.219 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:12:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:12:52.986+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:52.988020+0000) 2024-02-20T09:12:53.219 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:12:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:12:52.984+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:52.985903+0000) 2024-02-20T09:12:53.219 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:12:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:12:53.216+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:12:53.219 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:12:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:12:53.216+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:12:53.219 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:12:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:12:53.216+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:12:53.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:12:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:12:53.216+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:12:53.861 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:12:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:12:53.659+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:12:53.861 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:12:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:12:53.659+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:12:53.862 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:12:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:12:53.659+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:12:53.862 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:12:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:12:53.571+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:12:53.862 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:12:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:12:53.571+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:12:53.862 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:12:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:12:53.571+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:12:54.206 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:12:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:12:53.869+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:53.870772+0000) 2024-02-20T09:12:54.207 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:12:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:12:53.986+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:53.988105+0000) 2024-02-20T09:12:54.207 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:12:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:12:53.987+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:53.988582+0000) 2024-02-20T09:12:54.207 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:12:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:12:54.204+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:12:54.207 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:12:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:12:54.204+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:12:54.207 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:12:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:12:54.204+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:12:54.208 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:12:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:12:54.204+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:12:54.861 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:12:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:12:54.699+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:12:54.862 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:12:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:12:54.699+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:12:54.862 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:12:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:12:54.699+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:12:54.862 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:12:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:12:54.606+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:12:54.862 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:12:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:12:54.606+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:12:54.862 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:12:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:12:54.606+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:12:55.236 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:12:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:12:54.986+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:54.988215+0000) 2024-02-20T09:12:55.237 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:12:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:12:54.869+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:54.870886+0000) 2024-02-20T09:12:55.237 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:12:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:12:54.988+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:54.989512+0000) 2024-02-20T09:12:55.237 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:12:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:12:55.233+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:12:55.237 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:12:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:12:55.233+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:12:55.237 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:12:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:12:55.233+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:12:55.237 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:12:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:12:55.234+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:12:55.808 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:12:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:12:55.641+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:12:55.808 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:12:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:12:55.641+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:12:55.808 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:12:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:12:55.641+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:12:55.808 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:12:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:12:55.670+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:12:55.809 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:12:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:12:55.670+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:12:55.809 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:12:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:12:55.670+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:12:55.809 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:12:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:12:55.805+0000 7f8a5f86e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:55.806874+0000) 2024-02-20T09:12:56.242 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:12:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:12:55.811+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 62739 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:12:56.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:12:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:12:55.869+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:55.871028+0000) 2024-02-20T09:12:56.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:12:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:12:55.809+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:55.810403+0000) 2024-02-20T09:12:56.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:12:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:12:55.988+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:55.989698+0000) 2024-02-20T09:12:56.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:12:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:12:55.806+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:55.807434+0000) 2024-02-20T09:12:56.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:12:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:12:55.986+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:55.988328+0000) 2024-02-20T09:12:56.681 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:12:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:12:56.269+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:12:56.681 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:12:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:12:56.269+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:12:56.682 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:12:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:12:56.269+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:12:56.682 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:12:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:12:56.269+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:12:56.989 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:12:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:12:56.680+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:12:56.989 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:12:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:12:56.680+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:12:56.990 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:12:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:12:56.680+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:12:56.990 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:12:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:12:56.697+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:12:56.990 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:12:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:12:56.697+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:12:56.990 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:12:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:12:56.697+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:12:56.990 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:12:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:12:56.869+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:56.871147+0000) 2024-02-20T09:12:57.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:12:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:12:56.990+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:56.991700+0000) 2024-02-20T09:12:57.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:12:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:12:56.987+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:56.988449+0000) 2024-02-20T09:12:57.515 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:12:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:12:57.241+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:12:57.515 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:12:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:12:57.241+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:12:57.515 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:12:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:12:57.241+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:12:57.515 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:12:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:12:57.241+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:12:57.862 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:12:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:12:57.678+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:12:57.862 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:12:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:12:57.678+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:12:57.862 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:12:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:12:57.678+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:12:57.862 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:12:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:12:57.728+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:12:57.863 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:12:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:12:57.728+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:12:57.863 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:12:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:12:57.728+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:12:58.203 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:12:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:12:57.987+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:57.988588+0000) 2024-02-20T09:12:58.203 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:12:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:12:57.869+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:57.871283+0000) 2024-02-20T09:12:58.203 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:12:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:12:57.992+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:57.993837+0000) 2024-02-20T09:12:58.204 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:12:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:12:58.200+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:12:58.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:12:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:12:58.200+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:12:58.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:12:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:12:58.200+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:12:58.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:12:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:12:58.200+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:12:58.779 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:12:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:12:58.652+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:12:58.779 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:12:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:12:58.652+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:12:58.779 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:12:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:12:58.652+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:12:59.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:12:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:12:58.870+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:58.871431+0000) 2024-02-20T09:12:59.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:12:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:12:58.994+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:58.996065+0000) 2024-02-20T09:12:59.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:12:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:12:58.776+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:12:59.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:12:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:12:58.776+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:12:59.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:12:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:12:58.776+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:12:59.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:12:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:12:58.987+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:58.988729+0000) 2024-02-20T09:12:59.515 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:12:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:12:59.247+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:12:59.515 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:12:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:12:59.247+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:12:59.515 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:12:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:12:59.247+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:12:59.515 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:12:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:12:59.247+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:12:59.785 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:12:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:12:59.634+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:12:59.785 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:12:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:12:59.634+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:12:59.785 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:12:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:12:59.634+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:13:00.220 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:12:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:12:59.870+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:59.871556+0000) 2024-02-20T09:13:00.220 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:12:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:12:59.995+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:59.997149+0000) 2024-02-20T09:13:00.220 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:13:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:13:00.217+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:13:00.221 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:12:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:12:59.782+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:13:00.221 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:12:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:12:59.782+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:13:00.221 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:12:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:12:59.782+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:13:00.221 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:12:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:12:59.987+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:12:59.988860+0000) 2024-02-20T09:13:00.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:13:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:13:00.217+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:13:00.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:13:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:13:00.217+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:13:00.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:13:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:13:00.217+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:13:00.816 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:13:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:13:00.650+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:13:00.816 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:13:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:13:00.650+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:13:00.816 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:13:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:13:00.650+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:13:00.816 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:13:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:13:00.758+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:13:00.816 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:13:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:13:00.758+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:13:00.817 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:13:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:13:00.758+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:13:01.173 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:13:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:13:00.818+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 62794 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:13:01.173 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:13:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:13:00.813+0000 7f8a5f86e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:00.814897+0000) 2024-02-20T09:13:01.173 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:13:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:13:00.870+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:00.871699+0000) 2024-02-20T09:13:01.173 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:13:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:13:00.813+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:00.815252+0000) 2024-02-20T09:13:01.174 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:13:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:13:00.987+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:00.988951+0000) 2024-02-20T09:13:01.174 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:13:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:13:00.816+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:00.818245+0000) 2024-02-20T09:13:01.174 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:13:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:13:00.996+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:00.997495+0000) 2024-02-20T09:13:01.174 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:13:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:13:01.170+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:13:01.174 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:13:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:13:01.170+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:13:01.174 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:13:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:13:01.170+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:13:01.174 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:13:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:13:01.170+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:13:01.862 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:13:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:13:01.671+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:13:01.863 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:13:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:13:01.671+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:13:01.863 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:13:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:13:01.671+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:13:01.863 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:13:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:13:01.745+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:13:01.863 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:13:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:13:01.745+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:13:01.863 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:13:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:13:01.745+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:13:02.189 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:13:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:13:01.870+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:01.871800+0000) 2024-02-20T09:13:02.189 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:13:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:13:01.987+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:01.989072+0000) 2024-02-20T09:13:02.190 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:13:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:13:01.998+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:01.999765+0000) 2024-02-20T09:13:02.190 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:13:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:13:02.186+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:13:02.190 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:13:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:13:02.186+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:13:02.190 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:13:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:13:02.186+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:13:02.190 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:13:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:13:02.186+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:13:02.862 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:13:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:13:02.714+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:13:02.863 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:13:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:13:02.714+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:13:02.863 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:13:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:13:02.714+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:13:02.863 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:13:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:13:02.636+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:13:02.863 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:13:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:13:02.636+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:13:02.863 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:13:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:13:02.636+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:13:03.234 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:13:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:13:02.870+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:02.871941+0000) 2024-02-20T09:13:03.234 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:13:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:13:02.987+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:02.989189+0000) 2024-02-20T09:13:03.235 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:13:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:13:03.000+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:03.001811+0000) 2024-02-20T09:13:03.235 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:13:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:13:03.231+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:13:03.235 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:13:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:13:03.231+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:13:03.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:13:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:13:03.231+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:13:03.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:13:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:13:03.231+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:13:03.862 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:13:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:13:03.679+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:13:03.863 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:13:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:13:03.679+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:13:03.863 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:13:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:13:03.679+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:13:03.863 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:13:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:13:03.668+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:13:03.863 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:13:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:13:03.668+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:13:03.863 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:13:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:13:03.668+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:13:04.239 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:13:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:13:03.987+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:03.989339+0000) 2024-02-20T09:13:04.239 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:13:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:13:03.870+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:03.872041+0000) 2024-02-20T09:13:04.239 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:13:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:13:04.002+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:04.003834+0000) 2024-02-20T09:13:04.240 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:13:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:13:04.236+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:13:04.240 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:13:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:13:04.236+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:13:04.240 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:13:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:13:04.236+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:13:04.240 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:13:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:13:04.236+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:13:04.863 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:13:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:13:04.638+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:13:04.863 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:13:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:13:04.638+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:13:04.863 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:13:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:13:04.638+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:13:04.863 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:13:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:13:04.703+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:13:04.864 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:13:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:13:04.703+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:13:04.864 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:13:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:13:04.703+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:13:05.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:13:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:13:04.988+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:04.989513+0000) 2024-02-20T09:13:05.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:13:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:13:04.870+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:04.872186+0000) 2024-02-20T09:13:05.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:13:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:13:05.003+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:05.004756+0000) 2024-02-20T09:13:05.597 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:13:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:13:05.286+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:13:05.597 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:13:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:13:05.286+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:13:05.597 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:13:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:13:05.286+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:13:05.597 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:13:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:13:05.286+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:13:05.863 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:13:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:13:05.822+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:05.824030+0000) 2024-02-20T09:13:05.863 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:13:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:13:05.825+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 62848 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:13:05.863 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:13:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:13:05.594+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:13:05.864 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:13:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:13:05.594+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:13:05.864 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:13:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:13:05.594+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:13:05.864 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:13:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:13:05.820+0000 7f8a5f86e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:05.821906+0000) 2024-02-20T09:13:05.864 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:13:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:13:05.673+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:13:05.864 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:13:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:13:05.673+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:13:05.864 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:13:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:13:05.673+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:13:05.865 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:13:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:13:05.819+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:05.821318+0000) 2024-02-20T09:13:06.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:13:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:13:05.870+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:05.872307+0000) 2024-02-20T09:13:06.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:13:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:13:06.003+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:06.005027+0000) 2024-02-20T09:13:06.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:13:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:13:05.988+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:05.989631+0000) 2024-02-20T09:13:06.516 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:13:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:13:06.288+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:13:06.516 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:13:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:13:06.288+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:13:06.516 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:13:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:13:06.288+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:13:06.517 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:13:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:13:06.288+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:13:06.863 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:13:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:13:06.700+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:13:06.863 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:13:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:13:06.700+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:13:06.863 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:13:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:13:06.700+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:13:06.864 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:13:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:13:06.566+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:13:06.864 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:13:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:13:06.566+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:13:06.864 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:13:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:13:06.566+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:13:07.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:13:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:13:06.988+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:06.989734+0000) 2024-02-20T09:13:07.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:13:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:13:06.871+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:06.872428+0000) 2024-02-20T09:13:07.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:13:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:13:07.005+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:07.007049+0000) 2024-02-20T09:13:07.694 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:13:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:13:07.550+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:13:07.694 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:13:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:13:07.550+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:13:07.694 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:13:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:13:07.550+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:13:07.694 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:13:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:13:07.328+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:13:07.694 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:13:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:13:07.328+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:13:07.694 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:13:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:13:07.328+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:13:07.695 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:13:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:13:07.328+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:13:07.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:13:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:13:07.871+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:07.872521+0000) 2024-02-20T09:13:07.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:13:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:13:07.733+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:13:07.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:13:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:13:07.733+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:13:07.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:13:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:13:07.733+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:13:08.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:13:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:13:07.989+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:07.989886+0000) 2024-02-20T09:13:08.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:13:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:13:08.008+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:08.009299+0000) 2024-02-20T09:13:08.695 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:13:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:13:08.503+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:13:08.695 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:13:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:13:08.503+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:13:08.695 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:13:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:13:08.503+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:13:08.695 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:13:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:13:08.362+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:13:08.696 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:13:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:13:08.362+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:13:08.696 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:13:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:13:08.362+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:13:08.696 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:13:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:13:08.362+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:13:08.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:13:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:13:08.871+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:08.872655+0000) 2024-02-20T09:13:08.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:13:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:13:08.717+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:13:08.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:13:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:13:08.717+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:13:08.992 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:13:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:13:08.717+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:13:09.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:13:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:13:09.010+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:09.011325+0000) 2024-02-20T09:13:09.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:13:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:13:08.988+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:08.990000+0000) 2024-02-20T09:13:09.680 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:13:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:13:09.461+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:13:09.680 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:13:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:13:09.461+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:13:09.680 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:13:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:13:09.461+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:13:09.680 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:13:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:13:09.385+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:13:09.681 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:13:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:13:09.385+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:13:09.681 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:13:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:13:09.385+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:13:09.681 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:13:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:13:09.385+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:13:09.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:13:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:13:09.872+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:09.872771+0000) 2024-02-20T09:13:09.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:13:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:13:09.677+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:13:09.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:13:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:13:09.677+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:13:09.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:13:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:13:09.677+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:13:10.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:13:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:13:10.011+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:10.012238+0000) 2024-02-20T09:13:10.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:13:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:13:09.989+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:09.990137+0000) 2024-02-20T09:13:10.695 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:13:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:13:10.415+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:13:10.695 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:13:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:13:10.415+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:13:10.695 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:13:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:13:10.415+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:13:10.696 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:13:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:13:10.629+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:13:10.696 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:13:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:13:10.629+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:13:10.696 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:13:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:13:10.629+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:13:10.696 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:13:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:13:10.402+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:13:10.697 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:13:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:13:10.402+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:13:10.697 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:13:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:13:10.402+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:13:10.697 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:13:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:13:10.402+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:13:10.953 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:13:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:13:10.832+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 62903 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:13:10.953 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:13:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:13:10.830+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:10.831614+0000) 2024-02-20T09:13:10.953 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:13:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:13:10.826+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:10.828217+0000) 2024-02-20T09:13:10.954 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:13:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:13:10.827+0000 7f8a5f86e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:10.829005+0000) 2024-02-20T09:13:10.954 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:13:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:13:10.872+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:10.872887+0000) 2024-02-20T09:13:11.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:13:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:13:11.012+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:11.012501+0000) 2024-02-20T09:13:11.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:13:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:13:10.989+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:10.990310+0000) 2024-02-20T09:13:11.671 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:13:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:13:11.411+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:13:11.671 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:13:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:13:11.411+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:13:11.671 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:13:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:13:11.411+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:13:11.671 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:13:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:13:11.412+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:13:11.672 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:13:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:13:11.415+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:13:11.672 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:13:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:13:11.415+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:13:11.672 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:13:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:13:11.415+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:13:11.946 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:13:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:13:11.872+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:11.872990+0000) 2024-02-20T09:13:11.947 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:13:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:13:11.669+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:13:11.947 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:13:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:13:11.669+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:13:11.947 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:13:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:13:11.669+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:13:12.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:13:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:13:12.014+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:12.014687+0000) 2024-02-20T09:13:12.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:13:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:13:11.990+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:11.990485+0000) 2024-02-20T09:13:12.698 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:13:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:13:12.446+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:13:12.698 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:13:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:13:12.446+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:13:12.698 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:13:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:13:12.446+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:13:12.698 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:13:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:13:12.646+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:13:12.698 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:13:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:13:12.646+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:13:12.699 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:13:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:13:12.646+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:13:12.699 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:13:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:13:12.443+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:13:12.699 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:13:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:13:12.443+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:13:12.699 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:13:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:13:12.443+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:13:12.699 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:13:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:13:12.444+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:13:12.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:13:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:13:12.872+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:12.873097+0000) 2024-02-20T09:13:13.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:13:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:13:13.017+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:13.018276+0000) 2024-02-20T09:13:13.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:13:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:13:12.990+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:12.990623+0000) 2024-02-20T09:13:13.698 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:13:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:13:13.437+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:13:13.699 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:13:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:13:13.437+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:13:13.699 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:13:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:13:13.437+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:13:13.699 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:13:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:13:13.649+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:13:13.699 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:13:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:13:13.649+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:13:13.699 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:13:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:13:13.649+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:13:13.700 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:13:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:13:13.416+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:13:13.700 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:13:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:13:13.416+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:13:13.700 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:13:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:13:13.416+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:13:13.700 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:13:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:13:13.416+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:13:13.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:13:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:13:13.872+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:13.873259+0000) 2024-02-20T09:13:14.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:13:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:13:14.020+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:14.020911+0000) 2024-02-20T09:13:14.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:13:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:13:13.990+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:13.990755+0000) 2024-02-20T09:13:14.699 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:13:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:13:14.687+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:13:14.699 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:13:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:13:14.687+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:13:14.699 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:13:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:13:14.687+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:13:14.699 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:13:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:13:14.477+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:13:14.699 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:13:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:13:14.477+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:13:14.700 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:13:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:13:14.477+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:13:14.700 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:13:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:13:14.446+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:13:14.700 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:13:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:13:14.446+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:13:14.700 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:13:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:13:14.446+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:13:14.700 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:13:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:13:14.447+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:13:14.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:13:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:13:14.872+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:14.873384+0000) 2024-02-20T09:13:15.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:13:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:13:15.022+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:15.022511+0000) 2024-02-20T09:13:15.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:13:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:13:14.990+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:14.990867+0000) 2024-02-20T09:13:15.699 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:13:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:13:15.439+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:13:15.699 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:13:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:13:15.439+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:13:15.699 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:13:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:13:15.439+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:13:15.699 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:13:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:13:15.687+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:13:15.700 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:13:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:13:15.687+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:13:15.700 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:13:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:13:15.687+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:13:15.700 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:13:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:13:15.464+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:13:15.700 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:13:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:13:15.464+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:13:15.700 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:13:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:13:15.464+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:13:15.700 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:13:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:13:15.464+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:13:15.961 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:13:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:13:15.840+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 62960 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:13:15.961 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:13:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:13:15.838+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:15.838616+0000) 2024-02-20T09:13:15.961 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:13:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:13:15.834+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:15.835050+0000) 2024-02-20T09:13:15.962 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:13:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:13:15.834+0000 7f8a5f86e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:15.834951+0000) 2024-02-20T09:13:15.962 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:13:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:13:15.873+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:15.873492+0000) 2024-02-20T09:13:16.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:13:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:13:16.022+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:16.022761+0000) 2024-02-20T09:13:16.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:13:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:13:15.990+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:15.990984+0000) 2024-02-20T09:13:16.699 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:13:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:13:16.437+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:13:16.699 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:13:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:13:16.437+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:13:16.699 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:13:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:13:16.437+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:13:16.699 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:13:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:13:16.437+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:13:16.700 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:13:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:13:16.463+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:13:16.700 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:13:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:13:16.463+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:13:16.700 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:13:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:13:16.463+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:13:16.700 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:13:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:13:16.682+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:13:16.700 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:13:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:13:16.682+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:13:16.701 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:13:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:13:16.682+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:13:16.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:13:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:13:16.873+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:16.873570+0000) 2024-02-20T09:13:17.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:13:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:13:17.024+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:17.024935+0000) 2024-02-20T09:13:17.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:13:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:13:16.990+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:16.991098+0000) 2024-02-20T09:13:17.683 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:13:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:13:17.487+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:13:17.683 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:13:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:13:17.487+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:13:17.683 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:13:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:13:17.487+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:13:17.683 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:13:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:13:17.404+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:13:17.684 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:13:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:13:17.404+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:13:17.684 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:13:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:13:17.404+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:13:17.684 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:13:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:13:17.404+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:13:17.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:13:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:13:17.873+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:17.873667+0000) 2024-02-20T09:13:17.992 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:13:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:13:17.681+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:13:17.992 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:13:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:13:17.681+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:13:17.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:13:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:13:17.681+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:13:18.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:13:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:13:18.026+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:18.027072+0000) 2024-02-20T09:13:18.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:13:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:13:17.990+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:17.991205+0000) 2024-02-20T09:13:18.654 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:13:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:13:18.518+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:13:18.655 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:13:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:13:18.518+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:13:18.655 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:13:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:13:18.518+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:13:18.655 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:13:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:13:18.380+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:13:18.655 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:13:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:13:18.380+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:13:18.655 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:13:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:13:18.380+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:13:18.655 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:13:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:13:18.381+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:13:18.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:13:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:13:18.873+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:18.873769+0000) 2024-02-20T09:13:18.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:13:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:13:18.653+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:13:18.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:13:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:13:18.653+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:13:18.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:13:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:13:18.653+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:13:19.341 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:13:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:13:18.990+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:18.991334+0000) 2024-02-20T09:13:19.342 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:13:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:13:19.029+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:19.029712+0000) 2024-02-20T09:13:19.342 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:13:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:13:19.339+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:13:19.342 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:13:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:13:19.339+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:13:19.343 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:13:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:13:19.339+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:13:19.611 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:13:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:13:19.339+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:13:19.611 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:13:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:13:19.484+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:13:19.611 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:13:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:13:19.484+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:13:19.611 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:13:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:13:19.484+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:13:19.864 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:13:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:13:19.609+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:13:19.865 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:13:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:13:19.609+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:13:19.865 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:13:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:13:19.609+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:13:20.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:13:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:13:19.873+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:19.873869+0000) 2024-02-20T09:13:20.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:13:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:13:20.030+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:20.030969+0000) 2024-02-20T09:13:20.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:13:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:13:19.991+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:19.991479+0000) 2024-02-20T09:13:20.703 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:13:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:13:20.518+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:13:20.703 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:13:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:13:20.518+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:13:20.703 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:13:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:13:20.518+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:13:20.704 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:13:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:13:20.359+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:13:20.704 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:13:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:13:20.359+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:13:20.704 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:13:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:13:20.359+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:13:20.704 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:13:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:13:20.359+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:13:20.704 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:13:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:13:20.596+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:13:20.705 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:13:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:13:20.596+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:13:20.705 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:13:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:13:20.596+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:13:20.964 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:13:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:13:20.844+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:20.844562+0000) 2024-02-20T09:13:20.965 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:13:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:13:20.841+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:20.842055+0000) 2024-02-20T09:13:20.965 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:13:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:13:20.841+0000 7f8a5f86e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:20.841648+0000) 2024-02-20T09:13:20.965 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:13:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:13:20.873+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:20.873969+0000) 2024-02-20T09:13:20.966 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:13:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:13:20.847+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 63018 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:13:21.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:13:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:13:21.030+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:21.031229+0000) 2024-02-20T09:13:21.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:13:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:13:20.991+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:20.991577+0000) 2024-02-20T09:13:21.643 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:13:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:13:21.489+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:13:21.643 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:13:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:13:21.489+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:13:21.643 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:13:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:13:21.489+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:13:21.643 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:13:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:13:21.364+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:13:21.644 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:13:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:13:21.364+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:13:21.644 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:13:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:13:21.364+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:13:21.644 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:13:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:13:21.364+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:13:21.947 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:13:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:13:21.873+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:21.874060+0000) 2024-02-20T09:13:21.948 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:13:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:13:21.641+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:13:21.948 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:13:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:13:21.641+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:13:21.948 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:13:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:13:21.641+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:13:22.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:13:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:13:22.033+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:22.033871+0000) 2024-02-20T09:13:22.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:13:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:13:21.991+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:21.991668+0000) 2024-02-20T09:13:22.706 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:13:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:13:22.470+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:13:22.706 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:13:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:13:22.470+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:13:22.706 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:13:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:13:22.470+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:13:22.706 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:13:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:13:22.396+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:13:22.707 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:13:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:13:22.396+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:13:22.707 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:13:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:13:22.396+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:13:22.707 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:13:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:13:22.396+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:13:22.707 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:13:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:13:22.645+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:13:22.707 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:13:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:13:22.645+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:13:22.707 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:13:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:13:22.645+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:13:22.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:13:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:13:22.873+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:22.874193+0000) 2024-02-20T09:13:23.392 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:13:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:13:22.991+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:22.991756+0000) 2024-02-20T09:13:23.393 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:13:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:13:23.036+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:23.037311+0000) 2024-02-20T09:13:23.707 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:13:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:13:23.484+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:13:23.707 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:13:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:13:23.484+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:13:23.707 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:13:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:13:23.484+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:13:23.707 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:13:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:13:23.627+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:13:23.708 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:13:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:13:23.627+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:13:23.708 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:13:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:13:23.627+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:13:23.711 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:13:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:13:23.391+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:13:23.711 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:13:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:13:23.391+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:13:23.711 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:13:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:13:23.391+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:13:23.711 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:13:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:13:23.391+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:13:23.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:13:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:13:23.873+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:23.874307+0000) 2024-02-20T09:13:24.410 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:13:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:13:23.991+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:23.991844+0000) 2024-02-20T09:13:24.410 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:13:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:13:24.040+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:24.040747+0000) 2024-02-20T09:13:24.410 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:13:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:13:24.408+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:13:24.411 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:13:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:13:24.408+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:13:24.671 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:13:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:13:24.408+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:13:24.671 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:13:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:13:24.408+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:13:24.672 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:13:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:13:24.439+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:13:24.672 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:13:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:13:24.439+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:13:24.672 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:13:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:13:24.439+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:13:24.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:13:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:13:24.874+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:24.874420+0000) 2024-02-20T09:13:24.992 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:13:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:13:24.669+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:13:24.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:13:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:13:24.669+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:13:24.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:13:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:13:24.669+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:13:25.405 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:13:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:13:24.991+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:24.991937+0000) 2024-02-20T09:13:25.405 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:13:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:13:25.041+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:25.042323+0000) 2024-02-20T09:13:25.405 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:13:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:13:25.403+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:13:25.405 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:13:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:13:25.403+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:13:25.406 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:13:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:13:25.403+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:13:25.406 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:13:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:13:25.403+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:13:25.707 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:13:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:13:25.621+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:13:25.707 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:13:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:13:25.621+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:13:25.707 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:13:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:13:25.621+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:13:25.708 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:13:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:13:25.446+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:13:25.708 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:13:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:13:25.446+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:13:25.708 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:13:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:13:25.446+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:13:25.973 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:13:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:13:25.850+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:25.851262+0000) 2024-02-20T09:13:25.974 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:13:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:13:25.848+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:25.849048+0000) 2024-02-20T09:13:25.974 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:13:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:13:25.848+0000 7f8a5f86e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:25.849335+0000) 2024-02-20T09:13:25.974 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:13:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:13:25.874+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:25.874527+0000) 2024-02-20T09:13:25.975 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:13:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:13:25.853+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 63075 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:13:26.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:13:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:13:26.042+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:26.042564+0000) 2024-02-20T09:13:26.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:13:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:13:25.991+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:25.992123+0000) 2024-02-20T09:13:26.707 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:13:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:13:26.405+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:13:26.708 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:13:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:13:26.405+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:13:26.708 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:13:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:13:26.405+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:13:26.708 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:13:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:13:26.586+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:13:26.708 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:13:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:13:26.586+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:13:26.708 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:13:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:13:26.586+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:13:26.709 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:13:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:13:26.439+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:13:26.709 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:13:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:13:26.439+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:13:26.709 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:13:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:13:26.439+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:13:26.709 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:13:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:13:26.439+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:13:26.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:13:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:13:26.874+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:26.874610+0000) 2024-02-20T09:13:27.370 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:13:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:13:27.044+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:27.044763+0000) 2024-02-20T09:13:27.371 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:13:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:13:26.991+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:26.992292+0000) 2024-02-20T09:13:27.710 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:13:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:13:27.369+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:13:27.710 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:13:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:13:27.369+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:13:27.710 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:13:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:13:27.369+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:13:27.711 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:13:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:13:27.576+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:13:27.711 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:13:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:13:27.576+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:13:27.711 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:13:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:13:27.576+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:13:27.711 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:13:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:13:27.429+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:13:27.711 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:13:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:13:27.429+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:13:27.711 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:13:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:13:27.429+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:13:27.711 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:13:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:13:27.429+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:13:27.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:13:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:13:27.874+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:27.874703+0000) 2024-02-20T09:13:28.378 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:13:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:13:28.046+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:28.046792+0000) 2024-02-20T09:13:28.378 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:13:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:13:27.992+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:27.992414+0000) 2024-02-20T09:13:28.711 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:13:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:13:28.606+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:13:28.711 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:13:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:13:28.606+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:13:28.711 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:13:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:13:28.606+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:13:28.712 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:13:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:13:28.377+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:13:28.712 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:13:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:13:28.377+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:13:28.712 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:13:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:13:28.377+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:13:28.712 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:13:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:13:28.381+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:13:28.712 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:13:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:13:28.381+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:13:28.713 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:13:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:13:28.381+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:13:28.713 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:13:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:13:28.381+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:13:28.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:13:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:13:28.874+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:28.874804+0000) 2024-02-20T09:13:29.403 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:13:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:13:28.992+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:28.992508+0000) 2024-02-20T09:13:29.403 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:13:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:13:29.048+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:29.048971+0000) 2024-02-20T09:13:29.403 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:13:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:13:29.400+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:13:29.403 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:13:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:13:29.400+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:13:29.404 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:13:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:13:29.400+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:13:29.404 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:13:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:13:29.401+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:13:29.711 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:13:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:13:29.412+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:13:29.711 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:13:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:13:29.412+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:13:29.712 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:13:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:13:29.412+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:13:29.712 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:13:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:13:29.566+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:13:29.712 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:13:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:13:29.566+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:13:29.712 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:13:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:13:29.566+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:13:29.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:13:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:13:29.874+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:29.874906+0000) 2024-02-20T09:13:30.384 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:13:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:13:30.050+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:30.050665+0000) 2024-02-20T09:13:30.384 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:13:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:13:29.992+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:29.992606+0000) 2024-02-20T09:13:30.711 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:13:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:13:30.526+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:13:30.712 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:13:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:13:30.526+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:13:30.712 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:13:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:13:30.526+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:13:30.712 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:13:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:13:30.382+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:13:30.712 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:13:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:13:30.382+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:13:30.712 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:13:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:13:30.382+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:13:30.713 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:13:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:13:30.411+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:13:30.713 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:13:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:13:30.411+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:13:30.713 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:13:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:13:30.411+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:13:30.713 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:13:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:13:30.411+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:13:30.980 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:13:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:13:30.861+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 63133 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:13:30.981 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:13:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:13:30.857+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:30.857976+0000) 2024-02-20T09:13:30.981 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:13:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:13:30.855+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:30.855546+0000) 2024-02-20T09:13:30.981 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:13:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:13:30.854+0000 7f8a5f86e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:30.855352+0000) 2024-02-20T09:13:30.982 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:13:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:13:30.874+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:30.875005+0000) 2024-02-20T09:13:31.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:13:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:13:30.992+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:30.992709+0000) 2024-02-20T09:13:31.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:13:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:13:31.050+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:31.050890+0000) 2024-02-20T09:13:31.711 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:13:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:13:31.368+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:13:31.712 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:13:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:13:31.368+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:13:31.712 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:13:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:13:31.368+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:13:31.712 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:13:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:13:31.567+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:13:31.712 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:13:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:13:31.567+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:13:31.712 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:13:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:13:31.567+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:13:31.713 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:13:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:13:31.372+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:13:31.713 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:13:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:13:31.372+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:13:31.713 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:13:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:13:31.372+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:13:31.713 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:13:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:13:31.372+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:13:31.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:13:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:13:31.874+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:31.875091+0000) 2024-02-20T09:13:32.328 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:13:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:13:32.052+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:32.052972+0000) 2024-02-20T09:13:32.328 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:13:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:13:31.992+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:31.992800+0000) 2024-02-20T09:13:32.664 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:13:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:13:32.326+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:13:32.665 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:13:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:13:32.326+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:13:32.665 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:13:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:13:32.326+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:13:32.665 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:13:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:13:32.567+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:13:32.665 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:13:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:13:32.567+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:13:32.665 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:13:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:13:32.567+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:13:32.666 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:13:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:13:32.406+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:13:32.666 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:13:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:13:32.406+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:13:32.666 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:13:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:13:32.406+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:13:32.666 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:13:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:13:32.406+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:13:32.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:13:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:13:32.873+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:32.875189+0000) 2024-02-20T09:13:33.323 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:13:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:13:33.055+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:33.056527+0000) 2024-02-20T09:13:33.323 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:13:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:13:32.991+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:32.992920+0000) 2024-02-20T09:13:33.715 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:13:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:13:33.453+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:13:33.715 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:13:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:13:33.453+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:13:33.715 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:13:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:13:33.453+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:13:33.716 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:13:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:13:33.453+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:13:33.716 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:13:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:13:33.320+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:13:33.716 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:13:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:13:33.320+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:13:33.716 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:13:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:13:33.320+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:13:33.717 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:13:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:13:33.540+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:13:33.717 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:13:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:13:33.540+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:13:33.717 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:13:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:13:33.540+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:13:33.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:13:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:13:33.873+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:33.875283+0000) 2024-02-20T09:13:34.356 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:13:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:13:33.991+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:33.993007+0000) 2024-02-20T09:13:34.357 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:13:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:13:34.057+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:34.058668+0000) 2024-02-20T09:13:34.715 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:13:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:13:34.354+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:13:34.716 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:13:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:13:34.354+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:13:34.716 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:13:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:13:34.354+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:13:34.716 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:13:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:13:34.470+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:13:34.716 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:13:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:13:34.470+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:13:34.716 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:13:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:13:34.470+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:13:34.717 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:13:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:13:34.471+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:13:34.717 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:13:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:13:34.503+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:13:34.717 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:13:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:13:34.503+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:13:34.717 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:13:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:13:34.503+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:13:34.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:13:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:13:34.873+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:34.875389+0000) 2024-02-20T09:13:35.371 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:13:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:13:34.991+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:34.993095+0000) 2024-02-20T09:13:35.371 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:13:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:13:35.058+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:35.059630+0000) 2024-02-20T09:13:35.715 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:13:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:13:35.368+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:13:35.716 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:13:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:13:35.368+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:13:35.716 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:13:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:13:35.368+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:13:35.716 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:13:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:13:35.485+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:13:35.716 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:13:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:13:35.485+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:13:35.716 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:13:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:13:35.485+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:13:35.717 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:13:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:13:35.466+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:13:35.717 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:13:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:13:35.466+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:13:35.717 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:13:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:13:35.466+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:13:35.717 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:13:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:13:35.466+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:13:35.985 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:13:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:13:35.866+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 63190 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:13:35.986 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:13:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:13:35.864+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:35.865657+0000) 2024-02-20T09:13:35.986 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:13:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:13:35.861+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:35.863186+0000) 2024-02-20T09:13:35.987 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:13:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:13:35.861+0000 7f8a5f86e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:35.862604+0000) 2024-02-20T09:13:35.987 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:13:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:13:35.874+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:35.875496+0000) 2024-02-20T09:13:36.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:13:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:13:36.058+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:36.059805+0000) 2024-02-20T09:13:36.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:13:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:13:35.991+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:35.993186+0000) 2024-02-20T09:13:36.503 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:13:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:13:36.376+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:13:36.503 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:13:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:13:36.376+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:13:36.503 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:13:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:13:36.376+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:13:36.503 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:13:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:13:36.442+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:13:36.503 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:13:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:13:36.442+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:13:36.504 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:13:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:13:36.442+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:13:36.504 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:13:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:13:36.442+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:13:36.866 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:13:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:13:36.500+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:13:36.866 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:13:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:13:36.500+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:13:36.867 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:13:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:13:36.500+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:13:37.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:13:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:13:36.991+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:36.993297+0000) 2024-02-20T09:13:37.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:13:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:13:36.874+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:36.875584+0000) 2024-02-20T09:13:37.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:13:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:13:37.061+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:37.062990+0000) 2024-02-20T09:13:37.719 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:13:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:13:37.371+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:13:37.719 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:13:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:13:37.371+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:13:37.720 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:13:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:13:37.371+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:13:37.720 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:13:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:13:37.549+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:13:37.720 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:13:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:13:37.549+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:13:37.720 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:13:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:13:37.549+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:13:37.721 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:13:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:13:37.436+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:13:37.721 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:13:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:13:37.436+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:13:37.721 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:13:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:13:37.436+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:13:37.721 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:13:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:13:37.436+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:13:37.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:13:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:13:37.874+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:37.875690+0000) 2024-02-20T09:13:38.404 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:13:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:13:38.063+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:38.064969+0000) 2024-02-20T09:13:38.405 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:13:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:13:37.992+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:37.993437+0000) 2024-02-20T09:13:38.720 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:13:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:13:38.517+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:13:38.721 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:13:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:13:38.517+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:13:38.721 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:13:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:13:38.517+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:13:38.721 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:13:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:13:38.416+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:13:38.721 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:13:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:13:38.416+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:13:38.721 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:13:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:13:38.416+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:13:38.722 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:13:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:13:38.416+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:13:38.722 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:13:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:13:38.402+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:13:38.722 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:13:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:13:38.402+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:13:38.722 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:13:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:13:38.402+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:13:38.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:13:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:13:38.874+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:38.875788+0000) 2024-02-20T09:13:39.371 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:13:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:13:39.067+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:39.068538+0000) 2024-02-20T09:13:39.371 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:13:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:13:38.992+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:38.993530+0000) 2024-02-20T09:13:39.720 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:13:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:13:39.368+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:13:39.721 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:13:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:13:39.368+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:13:39.721 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:13:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:13:39.368+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:13:39.721 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:13:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:13:39.547+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:13:39.721 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:13:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:13:39.547+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:13:39.722 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:13:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:13:39.547+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:13:39.722 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:13:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:13:39.461+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:13:39.722 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:13:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:13:39.461+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:13:39.722 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:13:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:13:39.461+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:13:39.722 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:13:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:13:39.461+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:13:39.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:13:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:13:39.874+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:39.875888+0000) 2024-02-20T09:13:40.342 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:13:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:13:40.068+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:40.069500+0000) 2024-02-20T09:13:40.342 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:13:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:13:39.992+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:39.993627+0000) 2024-02-20T09:13:40.721 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:13:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:13:40.521+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:13:40.721 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:13:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:13:40.521+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:13:40.721 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:13:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:13:40.521+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:13:40.722 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:13:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:13:40.339+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:13:40.722 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:13:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:13:40.339+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:13:40.722 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:13:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:13:40.339+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:13:40.722 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:13:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:13:40.429+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:13:40.722 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:13:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:13:40.429+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:13:40.722 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:13:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:13:40.429+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:13:40.722 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:13:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:13:40.430+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:13:40.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:13:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:13:40.870+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:40.872117+0000) 2024-02-20T09:13:40.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:13:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:13:40.868+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:40.869904+0000) 2024-02-20T09:13:40.993 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:13:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:13:40.873+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 63248 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:13:40.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:13:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:13:40.867+0000 7f8a5f86e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:40.869308+0000) 2024-02-20T09:13:40.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:13:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:13:40.874+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:40.875990+0000) 2024-02-20T09:13:41.306 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:13:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:13:41.068+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:41.069678+0000) 2024-02-20T09:13:41.306 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:13:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:13:40.992+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:40.993731+0000) 2024-02-20T09:13:41.721 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:13:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:13:41.398+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:13:41.721 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:13:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:13:41.398+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:13:41.721 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:13:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:13:41.398+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:13:41.721 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:13:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:13:41.398+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:13:41.722 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:13:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:13:41.485+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:13:41.722 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:13:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:13:41.485+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:13:41.722 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:13:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:13:41.485+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:13:41.722 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:13:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:13:41.303+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:13:41.722 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:13:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:13:41.303+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:13:41.722 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:13:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:13:41.303+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:13:41.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:13:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:13:41.874+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:41.876096+0000) 2024-02-20T09:13:42.318 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:13:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:13:42.071+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:42.073231+0000) 2024-02-20T09:13:42.318 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:13:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:13:41.992+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:41.993821+0000) 2024-02-20T09:13:42.724 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:13:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:13:42.458+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:13:42.724 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:13:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:13:42.458+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:13:42.724 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:13:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:13:42.458+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:13:42.724 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:13:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:13:42.315+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:13:42.724 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:13:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:13:42.315+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:13:42.725 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:13:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:13:42.315+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:13:42.725 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:13:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:13:42.354+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:13:42.725 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:13:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:13:42.354+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:13:42.725 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:13:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:13:42.354+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:13:42.725 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:13:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:13:42.354+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:13:42.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:13:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:13:42.874+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:42.876211+0000) 2024-02-20T09:13:43.296 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:13:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:13:43.074+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:43.075750+0000) 2024-02-20T09:13:43.296 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:13:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:13:42.992+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:42.993934+0000) 2024-02-20T09:13:43.725 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:13:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:13:43.483+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:13:43.725 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:13:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:13:43.483+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:13:43.725 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:13:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:13:43.483+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:13:43.725 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:13:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:13:43.293+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:13:43.726 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:13:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:13:43.293+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:13:43.726 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:13:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:13:43.293+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:13:43.726 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:13:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:13:43.326+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:13:43.726 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:13:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:13:43.326+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:13:43.726 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:13:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:13:43.326+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:13:43.726 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:13:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:13:43.326+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:13:43.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:13:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:13:43.874+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:43.876320+0000) 2024-02-20T09:13:44.277 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:13:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:13:44.077+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:44.078435+0000) 2024-02-20T09:13:44.277 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:13:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:13:43.992+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:43.994055+0000) 2024-02-20T09:13:44.725 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:13:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:13:44.360+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:13:44.725 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:13:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:13:44.360+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:13:44.725 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:13:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:13:44.360+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:13:44.725 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:13:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:13:44.360+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:13:44.726 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:13:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:13:44.274+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:13:44.726 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:13:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:13:44.274+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:13:44.726 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:13:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:13:44.274+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:13:44.726 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:13:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:13:44.518+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:13:44.727 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:13:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:13:44.518+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:13:44.727 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:13:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:13:44.518+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:13:44.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:13:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:13:44.875+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:44.876441+0000) 2024-02-20T09:13:45.273 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:13:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:13:44.992+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:44.994146+0000) 2024-02-20T09:13:45.274 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:13:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:13:45.078+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:45.079440+0000) 2024-02-20T09:13:45.725 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:13:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:13:45.492+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:13:45.725 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:13:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:13:45.492+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:13:45.725 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:13:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:13:45.492+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:13:45.726 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:13:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:13:45.358+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:13:45.726 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:13:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:13:45.358+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:13:45.726 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:13:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:13:45.358+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:13:45.726 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:13:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:13:45.358+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:13:45.726 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:13:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:13:45.271+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:13:45.727 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:13:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:13:45.271+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:13:45.727 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:13:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:13:45.271+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:13:45.992 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:13:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:13:45.880+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 63305 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:13:45.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:13:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:13:45.878+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:45.879576+0000) 2024-02-20T09:13:45.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:13:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:13:45.874+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:45.876208+0000) 2024-02-20T09:13:45.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:13:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:13:45.874+0000 7f8a5f86e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:45.876269+0000) 2024-02-20T09:13:45.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:13:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:13:45.875+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:45.876533+0000) 2024-02-20T09:13:46.258 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:13:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:13:45.992+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:45.994265+0000) 2024-02-20T09:13:46.258 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:13:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:13:46.078+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:46.079624+0000) 2024-02-20T09:13:46.521 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:13:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:13:46.476+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:13:46.521 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:13:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:13:46.476+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:13:46.521 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:13:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:13:46.476+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:13:46.521 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:13:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:13:46.255+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:13:46.522 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:13:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:13:46.255+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:13:46.522 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:13:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:13:46.255+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:13:46.522 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:13:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:13:46.382+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:13:46.522 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:13:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:13:46.382+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:13:46.522 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:13:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:13:46.382+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:13:46.522 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:13:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:13:46.382+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:13:47.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:13:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:13:46.875+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:46.876611+0000) 2024-02-20T09:13:47.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:13:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:13:46.992+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:46.994377+0000) 2024-02-20T09:13:47.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:13:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:13:47.080+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:47.081687+0000) 2024-02-20T09:13:47.521 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:13:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:13:47.410+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:13:47.521 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:13:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:13:47.410+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:13:47.521 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:13:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:13:47.410+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:13:47.521 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:13:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:13:47.410+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:13:47.522 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:13:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:13:47.247+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:13:47.522 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:13:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:13:47.247+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:13:47.522 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:13:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:13:47.247+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:13:47.522 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:13:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:13:47.435+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:13:47.522 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:13:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:13:47.435+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:13:47.523 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:13:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:13:47.435+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:13:48.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:13:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:13:47.875+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:47.876705+0000) 2024-02-20T09:13:48.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:13:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:13:47.993+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:47.994471+0000) 2024-02-20T09:13:48.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:13:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:13:48.083+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:48.085343+0000) 2024-02-20T09:13:48.730 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:13:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:13:48.470+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:13:48.730 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:13:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:13:48.470+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:13:48.730 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:13:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:13:48.470+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:13:48.731 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:13:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:13:48.282+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:13:48.731 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:13:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:13:48.283+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:13:48.731 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:13:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:13:48.283+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:13:48.731 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:13:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:13:48.366+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:13:48.731 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:13:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:13:48.366+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:13:48.731 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:13:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:13:48.366+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:13:48.732 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:13:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:13:48.366+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:13:48.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:13:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:13:48.875+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:48.876812+0000) 2024-02-20T09:13:49.308 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:13:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:13:49.086+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:49.087447+0000) 2024-02-20T09:13:49.309 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:13:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:13:48.993+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:48.994558+0000) 2024-02-20T09:13:49.730 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:13:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:13:49.306+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:13:49.730 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:13:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:13:49.306+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:13:49.730 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:13:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:13:49.306+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:13:49.731 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:13:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:13:49.426+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:13:49.731 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:13:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:13:49.426+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:13:49.731 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:13:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:13:49.426+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:13:49.731 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:13:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:13:49.415+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:13:49.731 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:13:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:13:49.415+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:13:49.732 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:13:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:13:49.415+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:13:49.732 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:13:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:13:49.415+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:13:49.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:13:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:13:49.875+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:49.876994+0000) 2024-02-20T09:13:50.295 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:13:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:13:50.087+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:50.088446+0000) 2024-02-20T09:13:50.295 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:13:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:13:49.993+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:49.994662+0000) 2024-02-20T09:13:50.730 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:13:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:13:50.293+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:13:50.730 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:13:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:13:50.293+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:13:50.731 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:13:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:13:50.293+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:13:50.731 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:13:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:13:50.432+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:13:50.731 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:13:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:13:50.432+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:13:50.731 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:13:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:13:50.432+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:13:50.731 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:13:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:13:50.453+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:13:50.732 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:13:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:13:50.453+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:13:50.732 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:13:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:13:50.453+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:13:50.732 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:13:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:13:50.453+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:13:50.992 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:13:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:13:50.881+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:50.883364+0000) 2024-02-20T09:13:50.993 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:13:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:13:50.887+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 63363 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:13:50.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:13:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:13:50.885+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:50.886777+0000) 2024-02-20T09:13:50.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:13:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:13:50.875+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:50.877165+0000) 2024-02-20T09:13:50.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:13:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:13:50.882+0000 7f8a5f86e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:50.883482+0000) 2024-02-20T09:13:51.321 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:13:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:13:51.087+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:51.088606+0000) 2024-02-20T09:13:51.321 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:13:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:13:50.993+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:50.994784+0000) 2024-02-20T09:13:51.730 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:13:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:13:51.318+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:13:51.731 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:13:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:13:51.318+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:13:51.731 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:13:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:13:51.318+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:13:51.731 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:13:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:13:51.446+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:13:51.731 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:13:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:13:51.446+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:13:51.731 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:13:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:13:51.446+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:13:51.732 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:13:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:13:51.484+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:13:51.732 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:13:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:13:51.484+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:13:51.732 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:13:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:13:51.484+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:13:51.732 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:13:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:13:51.484+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:13:51.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:13:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:13:51.875+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:51.877345+0000) 2024-02-20T09:13:52.335 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:13:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:13:52.089+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:52.091327+0000) 2024-02-20T09:13:52.335 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:13:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:13:51.993+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:51.994874+0000) 2024-02-20T09:13:52.628 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:13:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:13:52.475+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:13:52.628 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:13:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:13:52.475+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:13:52.628 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:13:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:13:52.475+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:13:52.629 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:13:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:13:52.332+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:13:52.629 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:13:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:13:52.332+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:13:52.629 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:13:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:13:52.332+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:13:52.629 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:13:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:13:52.530+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:13:52.630 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:13:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:13:52.530+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:13:52.630 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:13:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:13:52.530+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:13:52.630 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:13:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:13:52.530+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:13:53.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:13:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:13:52.876+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:52.877474+0000) 2024-02-20T09:13:53.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:13:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:13:53.092+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:53.094139+0000) 2024-02-20T09:13:53.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:13:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:13:52.993+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:52.995016+0000) 2024-02-20T09:13:53.628 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:13:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:13:53.482+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:13:53.628 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:13:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:13:53.482+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:13:53.629 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:13:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:13:53.482+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:13:53.629 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:13:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:13:53.483+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:13:53.629 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:13:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:13:53.357+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:13:53.629 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:13:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:13:53.357+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:13:53.629 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:13:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:13:53.357+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:13:53.630 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:13:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:13:53.493+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:13:53.630 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:13:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:13:53.493+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:13:53.630 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:13:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:13:53.493+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:13:54.168 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:13:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:13:54.094+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:54.096373+0000) 2024-02-20T09:13:54.169 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:13:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:13:53.993+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:53.995217+0000) 2024-02-20T09:13:54.169 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:13:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:13:53.876+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:53.877610+0000) 2024-02-20T09:13:54.492 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:13:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:13:54.372+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:13:54.492 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:13:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:13:54.372+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:13:54.492 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:13:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:13:54.372+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:13:54.869 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:13:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:13:54.497+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:13:54.869 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:13:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:13:54.497+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:13:54.869 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:13:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:13:54.497+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:13:54.869 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:13:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:13:54.523+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:13:54.870 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:13:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:13:54.523+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:13:54.870 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:13:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:13:54.523+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:13:54.870 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:13:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:13:54.523+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:13:55.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:13:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:13:54.876+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:54.877741+0000) 2024-02-20T09:13:55.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:13:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:13:55.096+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:55.097449+0000) 2024-02-20T09:13:55.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:13:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:13:54.993+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:54.995370+0000) 2024-02-20T09:13:55.629 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:13:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:13:55.324+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:13:55.629 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:13:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:13:55.324+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:13:55.629 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:13:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:13:55.324+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:13:55.629 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:13:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:13:55.473+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:13:55.629 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:13:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:13:55.473+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:13:55.630 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:13:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:13:55.473+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:13:55.630 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:13:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:13:55.572+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:13:55.630 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:13:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:13:55.572+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:13:55.630 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:13:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:13:55.572+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:13:55.630 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:13:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:13:55.573+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:13:56.242 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:13:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:13:55.895+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 63420 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:13:56.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:13:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:13:55.876+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:55.877861+0000) 2024-02-20T09:13:56.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:13:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:13:55.889+0000 7f8a5f86e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:55.890536+0000) 2024-02-20T09:13:56.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:13:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:13:55.891+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:55.892880+0000) 2024-02-20T09:13:56.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:13:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:13:56.096+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:56.097620+0000) 2024-02-20T09:13:56.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:13:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:13:55.888+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:55.890319+0000) 2024-02-20T09:13:56.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:13:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:13:55.994+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:55.995441+0000) 2024-02-20T09:13:56.628 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:13:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:13:56.342+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:13:56.629 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:13:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:13:56.342+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:13:56.629 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:13:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:13:56.342+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:13:56.629 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:13:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:13:56.455+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:13:56.629 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:13:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:13:56.455+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:13:56.629 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:13:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:13:56.455+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:13:56.630 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:13:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:13:56.577+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:13:56.630 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:13:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:13:56.577+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:13:56.630 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:13:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:13:56.577+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:13:56.630 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:13:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:13:56.578+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:13:57.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:13:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:13:56.876+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:56.877948+0000) 2024-02-20T09:13:57.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:13:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:13:57.098+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:57.100327+0000) 2024-02-20T09:13:57.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:13:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:13:56.994+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:56.995522+0000) 2024-02-20T09:13:57.629 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:13:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:13:57.490+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:13:57.629 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:13:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:13:57.490+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:13:57.629 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:13:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:13:57.490+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:13:57.629 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:13:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:13:57.305+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:13:57.630 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:13:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:13:57.305+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:13:57.630 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:13:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:13:57.305+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:13:57.630 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:13:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:13:57.536+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:13:57.630 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:13:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:13:57.536+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:13:57.630 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:13:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:13:57.536+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:13:57.630 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:13:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:13:57.536+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:13:58.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:13:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:13:57.876+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:57.878043+0000) 2024-02-20T09:13:58.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:13:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:13:58.100+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:58.102299+0000) 2024-02-20T09:13:58.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:13:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:13:57.994+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:57.995614+0000) 2024-02-20T09:13:58.567 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:13:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:13:58.306+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:13:58.568 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:13:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:13:58.306+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:13:58.568 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:13:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:13:58.306+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:13:58.568 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:13:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:13:58.536+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:13:58.568 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:13:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:13:58.536+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:13:58.568 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:13:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:13:58.536+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:13:58.869 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:13:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:13:58.564+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:13:58.869 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:13:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:13:58.564+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:13:58.869 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:13:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:13:58.565+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:13:58.869 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:13:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:13:58.565+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:13:59.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:13:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:13:58.876+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:58.878157+0000) 2024-02-20T09:13:59.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:13:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:13:59.103+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:59.104771+0000) 2024-02-20T09:13:59.245 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:13:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:13:58.994+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:58.995717+0000) 2024-02-20T09:13:59.629 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:13:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:13:59.331+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:13:59.629 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:13:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:13:59.331+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:13:59.629 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:13:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:13:59.331+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:13:59.630 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:13:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:13:59.564+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:13:59.630 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:13:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:13:59.564+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:13:59.630 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:13:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:13:59.564+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:13:59.630 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:13:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:13:59.538+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:13:59.630 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:13:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:13:59.538+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:13:59.630 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:13:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:13:59.538+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:13:59.631 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:13:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:13:59.538+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:14:00.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:13:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:13:59.994+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:59.995830+0000) 2024-02-20T09:14:00.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:13:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:13:59.876+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:13:59.878273+0000) 2024-02-20T09:14:00.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:14:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:14:00.104+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:00.105992+0000) 2024-02-20T09:14:00.548 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:14:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:14:00.286+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:14:00.549 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:14:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:14:00.286+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:14:00.549 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:14:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:14:00.286+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:14:00.870 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:14:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:14:00.546+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:14:00.870 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:14:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:14:00.546+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:14:00.870 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:14:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:14:00.546+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:14:00.870 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:14:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:14:00.546+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:14:00.870 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:14:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:14:00.605+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:14:00.871 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:14:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:14:00.605+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:14:00.871 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:14:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:14:00.605+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:14:01.242 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:14:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:14:00.902+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 63477 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:14:01.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:14:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:14:00.877+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:00.878419+0000) 2024-02-20T09:14:01.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:14:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:14:00.896+0000 7f8a5f86e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:00.898340+0000) 2024-02-20T09:14:01.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:14:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:14:00.899+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:00.900727+0000) 2024-02-20T09:14:01.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:14:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:14:01.104+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:01.106238+0000) 2024-02-20T09:14:01.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:14:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:14:00.896+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:00.897651+0000) 2024-02-20T09:14:01.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:14:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:14:00.994+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:00.995952+0000) 2024-02-20T09:14:01.523 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:14:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:14:01.254+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:14:01.523 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:14:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:14:01.254+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:14:01.523 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:14:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:14:01.254+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:14:01.869 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:14:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:14:01.653+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:14:01.870 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:14:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:14:01.653+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:14:01.870 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:14:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:14:01.653+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:14:01.870 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:14:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:14:01.565+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:14:01.870 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:14:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:14:01.565+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:14:01.870 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:14:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:14:01.565+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:14:01.871 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:14:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:14:01.566+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:14:02.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:14:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:14:01.994+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:01.996069+0000) 2024-02-20T09:14:02.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:14:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:14:01.877+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:01.878511+0000) 2024-02-20T09:14:02.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:14:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:14:02.106+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:02.108266+0000) 2024-02-20T09:14:02.567 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:14:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:14:02.299+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:14:02.568 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:14:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:14:02.299+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:14:02.568 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:14:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:14:02.299+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:14:02.870 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:14:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:14:02.565+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:14:02.870 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:14:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:14:02.565+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:14:02.870 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:14:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:14:02.565+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:14:02.871 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:14:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:14:02.565+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:14:02.871 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:14:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:14:02.677+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:14:02.871 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:14:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:14:02.677+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:14:02.872 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:14:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:14:02.677+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:14:03.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:14:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:14:02.877+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:02.878600+0000) 2024-02-20T09:14:03.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:14:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:14:03.109+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:03.110737+0000) 2024-02-20T09:14:03.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:14:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:14:02.994+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:02.996156+0000) 2024-02-20T09:14:03.523 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:14:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:14:03.257+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:14:03.524 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:14:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:14:03.257+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:14:03.524 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:14:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:14:03.257+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:14:03.870 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:14:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:14:03.591+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:14:03.870 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:14:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:14:03.592+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:14:03.870 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:14:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:14:03.592+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:14:03.870 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:14:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:14:03.592+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:14:03.871 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:14:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:14:03.677+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:14:03.871 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:14:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:14:03.677+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:14:03.871 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:14:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:14:03.677+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:14:04.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:14:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:14:03.877+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:03.878721+0000) 2024-02-20T09:14:04.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:14:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:14:04.113+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:04.114416+0000) 2024-02-20T09:14:04.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:14:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:14:03.994+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:03.996248+0000) 2024-02-20T09:14:04.613 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:14:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:14:04.279+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:14:04.613 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:14:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:14:04.279+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:14:04.613 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:14:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:14:04.279+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:14:04.870 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:14:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:14:04.722+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:14:04.870 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:14:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:14:04.722+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:14:04.870 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:14:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:14:04.722+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:14:04.870 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:14:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:14:04.611+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:14:04.871 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:14:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:14:04.611+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:14:04.871 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:14:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:14:04.611+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:14:04.871 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:14:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:14:04.611+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:14:05.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:14:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:14:04.995+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:04.996346+0000) 2024-02-20T09:14:05.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:14:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:14:04.878+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:04.878814+0000) 2024-02-20T09:14:05.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:14:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:14:05.115+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:05.115818+0000) 2024-02-20T09:14:05.523 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:14:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:14:05.248+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:14:05.524 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:14:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:14:05.248+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:14:05.524 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:14:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:14:05.248+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:14:05.870 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:14:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:14:05.680+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:14:05.870 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:14:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:14:05.680+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:14:05.870 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:14:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:14:05.680+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:14:05.871 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:14:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:14:05.581+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:14:05.871 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:14:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:14:05.581+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:14:05.871 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:14:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:14:05.581+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:14:05.871 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:14:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:14:05.582+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:14:06.242 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:14:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:14:05.909+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 63534 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:14:06.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:14:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:14:05.878+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:05.878988+0000) 2024-02-20T09:14:06.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:14:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:14:05.904+0000 7f8a5f86e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:05.904713+0000) 2024-02-20T09:14:06.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:14:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:14:05.906+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:05.907340+0000) 2024-02-20T09:14:06.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:14:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:14:06.115+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:06.116073+0000) 2024-02-20T09:14:06.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:14:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:14:05.904+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:05.904904+0000) 2024-02-20T09:14:06.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:14:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:14:05.996+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:05.996503+0000) 2024-02-20T09:14:06.523 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:14:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:14:06.282+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:14:06.524 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:14:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:14:06.282+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:14:06.524 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:14:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:14:06.282+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:14:06.870 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:14:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:14:06.664+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:14:06.870 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:14:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:14:06.664+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:14:06.870 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:14:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:14:06.664+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:14:06.871 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:14:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:14:06.611+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:14:06.871 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:14:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:14:06.611+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:14:06.871 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:14:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:14:06.611+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:14:06.871 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:14:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:14:06.611+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:14:07.120 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:14:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:14:06.996+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:06.996597+0000) 2024-02-20T09:14:07.121 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:14:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:14:06.878+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:06.879145+0000) 2024-02-20T09:14:07.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:14:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:14:07.119+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:07.119579+0000) 2024-02-20T09:14:07.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:14:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:14:07.238+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:14:07.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:14:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:14:07.238+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:14:07.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:14:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:14:07.238+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:14:07.870 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:14:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:14:07.622+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:14:07.870 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:14:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:14:07.622+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:14:07.870 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:14:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:14:07.622+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:14:07.871 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:14:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:14:07.615+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:14:07.871 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:14:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:14:07.615+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:14:07.871 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:14:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:14:07.615+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:14:07.871 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:14:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:14:07.615+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:14:08.124 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:14:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:14:07.878+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:07.879255+0000) 2024-02-20T09:14:08.124 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:14:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:14:07.996+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:07.996715+0000) 2024-02-20T09:14:08.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:14:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:14:08.122+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:08.123106+0000) 2024-02-20T09:14:08.492 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:14:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:14:08.189+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:14:08.492 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:14:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:14:08.189+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:14:08.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:14:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:14:08.189+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:14:08.870 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:14:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:14:08.636+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:14:08.870 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:14:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:14:08.636+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:14:08.870 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:14:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:14:08.636+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:14:08.871 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:14:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:14:08.599+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:14:08.871 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:14:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:14:08.599+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:14:08.871 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:14:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:14:08.599+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:14:08.871 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:14:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:14:08.599+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:14:09.126 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:14:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:14:08.878+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:08.879376+0000) 2024-02-20T09:14:09.126 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:14:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:14:08.996+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:08.996864+0000) 2024-02-20T09:14:09.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:14:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:14:09.125+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:09.125443+0000) 2024-02-20T09:14:09.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:14:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:14:09.187+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:14:09.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:14:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:14:09.187+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:14:09.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:14:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:14:09.187+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:14:09.871 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:14:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:14:09.606+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:14:09.871 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:14:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:14:09.606+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:14:09.871 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:14:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:14:09.606+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:14:09.871 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:14:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:14:09.583+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:14:09.871 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:14:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:14:09.583+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:14:09.871 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:14:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:14:09.583+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:14:09.871 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:14:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:14:09.583+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:14:10.127 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:14:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:14:09.879+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:09.879476+0000) 2024-02-20T09:14:10.128 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:14:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:14:09.996+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:09.996959+0000) 2024-02-20T09:14:10.419 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:14:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:14:10.126+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:10.126514+0000) 2024-02-20T09:14:10.499 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:14:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:14:10.159+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:14:10.499 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:14:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:14:10.159+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:14:10.500 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:14:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:14:10.159+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:14:10.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:14:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:14:10.611+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:14:10.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:14:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:14:10.611+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:14:10.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:14:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:14:10.611+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:14:10.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:14:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:14:10.610+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:14:10.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:14:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:14:10.610+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:14:10.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:14:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:14:10.610+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:14:10.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:14:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:14:10.611+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:14:10.998 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:14:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:14:10.914+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:10.915334+0000) 2024-02-20T09:14:10.998 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:14:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:14:10.910+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:10.911312+0000) 2024-02-20T09:14:10.999 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:14:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:14:10.916+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 63592 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:14:10.999 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:14:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:14:10.879+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:10.879564+0000) 2024-02-20T09:14:10.999 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:14:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:14:10.911+0000 7f8a5f86e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:10.911458+0000) 2024-02-20T09:14:11.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:14:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:14:10.996+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:10.997080+0000) 2024-02-20T09:14:11.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:14:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:14:11.126+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:11.126807+0000) 2024-02-20T09:14:11.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:14:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:14:11.155+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:14:11.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:14:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:14:11.155+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:14:11.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:14:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:14:11.155+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:14:11.871 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:14:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:14:11.637+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:14:11.871 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:14:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:14:11.637+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:14:11.871 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:14:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:14:11.637+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:14:11.871 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:14:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:14:11.583+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:14:11.871 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:14:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:14:11.583+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:14:11.871 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:14:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:14:11.583+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:14:11.872 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:14:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:14:11.584+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:14:12.130 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:14:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:14:11.996+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:11.997198+0000) 2024-02-20T09:14:12.130 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:14:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:14:11.879+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:11.879673+0000) 2024-02-20T09:14:12.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:14:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:14:12.128+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:12.128931+0000) 2024-02-20T09:14:12.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:14:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:14:12.165+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:14:12.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:14:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:14:12.165+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:14:12.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:14:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:14:12.165+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:14:12.871 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:14:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:14:12.616+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:14:12.871 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:14:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:14:12.616+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:14:12.871 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:14:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:14:12.616+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:14:12.871 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:14:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:14:12.616+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:14:12.871 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:14:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:14:12.629+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:14:12.872 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:14:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:14:12.629+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:14:12.872 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:14:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:14:12.629+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:14:13.132 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:14:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:14:12.879+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:12.879867+0000) 2024-02-20T09:14:13.132 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:14:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:14:12.996+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:12.997335+0000) 2024-02-20T09:14:13.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:14:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:14:13.130+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:13.131096+0000) 2024-02-20T09:14:13.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:14:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:14:13.132+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:14:13.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:14:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:14:13.132+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:14:13.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:14:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:14:13.132+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:14:13.871 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:14:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:14:13.584+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:14:13.871 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:14:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:14:13.584+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:14:13.871 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:14:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:14:13.584+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:14:13.871 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:14:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:14:13.659+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:14:13.871 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:14:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:14:13.659+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:14:13.872 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:14:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:14:13.659+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:14:13.872 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:14:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:14:13.659+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:14:14.134 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:14:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:14:13.997+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:13.997444+0000) 2024-02-20T09:14:14.134 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:14:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:14:13.879+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:13.880057+0000) 2024-02-20T09:14:14.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:14:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:14:14.132+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:14.133176+0000) 2024-02-20T09:14:14.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:14:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:14:14.163+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:14:14.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:14:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:14:14.163+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:14:14.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:14:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:14:14.163+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:14:14.871 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:14:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:14:14.571+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:14:14.871 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:14:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:14:14.571+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:14:14.871 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:14:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:14:14.571+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:14:14.872 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:14:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:14:14.619+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:14:14.872 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:14:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:14:14.619+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:14:14.872 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:14:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:14:14.619+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:14:14.872 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:14:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:14:14.619+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:14:15.136 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:14:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:14:14.997+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:14.997531+0000) 2024-02-20T09:14:15.136 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:14:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:14:14.879+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:14.880236+0000) 2024-02-20T09:14:15.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:14:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:14:15.134+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:15.134967+0000) 2024-02-20T09:14:15.492 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:14:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:14:15.157+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:14:15.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:14:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:14:15.157+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:14:15.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:14:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:14:15.157+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:14:15.871 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:14:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:14:15.553+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:14:15.871 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:14:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:14:15.553+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:14:15.871 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:14:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:14:15.553+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:14:15.872 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:14:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:14:15.644+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:14:15.872 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:14:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:14:15.644+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:14:15.872 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:14:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:14:15.644+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:14:15.872 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:14:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:14:15.644+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:14:16.136 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:14:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:14:15.921+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:15.922022+0000) 2024-02-20T09:14:16.136 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:14:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:14:15.923+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 63649 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:14:16.137 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:14:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:14:15.880+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:15.880465+0000) 2024-02-20T09:14:16.137 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:14:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:14:15.917+0000 7f8a5f86e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:15.917967+0000) 2024-02-20T09:14:16.137 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:14:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:14:15.917+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:15.918060+0000) 2024-02-20T09:14:16.137 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:14:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:14:15.997+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:15.997649+0000) 2024-02-20T09:14:16.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:14:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:14:16.134+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:16.135216+0000) 2024-02-20T09:14:16.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:14:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:14:16.187+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:14:16.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:14:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:14:16.187+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:14:16.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:14:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:14:16.187+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:14:16.872 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:14:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:14:16.617+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:14:16.872 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:14:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:14:16.617+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:14:16.872 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:14:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:14:16.617+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:14:16.873 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:14:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:14:16.617+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:14:16.873 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:14:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:14:16.542+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:14:16.873 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:14:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:14:16.542+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:14:16.873 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:14:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:14:16.542+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:14:17.139 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:14:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:14:16.997+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:16.997726+0000) 2024-02-20T09:14:17.139 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:14:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:14:16.880+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:16.880605+0000) 2024-02-20T09:14:17.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:14:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:14:17.137+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:17.138078+0000) 2024-02-20T09:14:17.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:14:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:14:17.224+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:14:17.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:14:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:14:17.224+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:14:17.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:14:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:14:17.224+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:14:17.871 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:14:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:14:17.529+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:14:17.872 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:14:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:14:17.529+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:14:17.872 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:14:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:14:17.529+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:14:17.872 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:14:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:14:17.611+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:14:17.872 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:14:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:14:17.611+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:14:17.872 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:14:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:14:17.611+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:14:17.873 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:14:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:14:17.612+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:14:18.141 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:14:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:14:17.997+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:17.997820+0000) 2024-02-20T09:14:18.141 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:14:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:14:17.880+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:17.880709+0000) 2024-02-20T09:14:18.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:14:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:14:18.139+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:18.140249+0000) 2024-02-20T09:14:18.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:14:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:14:18.227+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:14:18.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:14:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:14:18.227+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:14:18.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:14:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:14:18.227+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:14:18.872 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:14:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:14:18.576+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:14:18.872 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:14:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:14:18.576+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:14:18.872 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:14:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:14:18.576+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:14:18.872 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:14:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:14:18.627+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:14:18.872 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:14:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:14:18.627+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:14:18.873 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:14:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:14:18.627+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:14:18.873 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:14:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:14:18.627+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:14:19.144 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:14:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:14:18.880+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:18.880831+0000) 2024-02-20T09:14:19.144 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:14:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:14:18.997+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:18.997938+0000) 2024-02-20T09:14:19.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:14:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:14:19.142+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:19.142853+0000) 2024-02-20T09:14:19.494 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:14:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:14:19.240+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:14:19.494 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:14:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:14:19.240+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:14:19.494 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:14:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:14:19.240+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:14:19.872 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:14:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:14:19.564+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:14:19.872 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:14:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:14:19.565+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:14:19.872 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:14:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:14:19.565+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:14:19.872 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:14:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:14:19.626+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:14:19.872 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:14:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:14:19.626+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:14:19.873 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:14:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:14:19.626+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:14:19.873 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:14:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:14:19.626+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:14:20.145 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:14:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:14:19.880+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:19.880978+0000) 2024-02-20T09:14:20.146 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:14:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:14:19.997+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:19.998061+0000) 2024-02-20T09:14:20.420 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:14:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:14:20.144+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:20.144549+0000) 2024-02-20T09:14:20.420 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:14:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:14:20.228+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:14:20.421 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:14:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:14:20.228+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:14:20.421 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:14:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:14:20.228+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:14:20.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:14:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:14:20.541+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:14:20.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:14:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:14:20.541+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:14:20.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:14:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:14:20.541+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:14:20.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:14:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:14:20.642+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:14:20.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:14:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:14:20.642+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:14:20.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:14:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:14:20.642+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:14:20.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:14:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:14:20.642+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:14:21.050 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:14:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:14:20.930+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 63707 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:14:21.050 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:14:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:14:20.928+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:20.929029+0000) 2024-02-20T09:14:21.051 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:14:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:14:20.880+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:20.881078+0000) 2024-02-20T09:14:21.051 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:14:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:14:20.925+0000 7f8a5f86e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:20.925502+0000) 2024-02-20T09:14:21.051 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:14:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:14:20.924+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:20.924996+0000) 2024-02-20T09:14:21.051 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:14:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:14:20.997+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:20.998179+0000) 2024-02-20T09:14:21.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:14:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:14:21.144+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:21.144773+0000) 2024-02-20T09:14:21.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:14:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:14:21.188+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:14:21.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:14:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:14:21.188+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:14:21.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:14:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:14:21.188+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:14:21.816 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:14:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:14:21.571+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:14:21.817 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:14:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:14:21.571+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:14:21.817 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:14:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:14:21.571+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:14:21.817 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:14:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:14:21.625+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:14:21.817 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:14:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:14:21.625+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:14:21.817 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:14:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:14:21.625+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:14:21.817 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:14:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:14:21.625+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:14:22.147 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:14:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:14:21.880+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:21.881225+0000) 2024-02-20T09:14:22.148 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:14:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:14:21.997+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:21.998320+0000) 2024-02-20T09:14:22.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:14:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:14:22.146+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:22.146737+0000) 2024-02-20T09:14:22.492 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:14:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:14:22.147+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:14:22.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:14:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:14:22.147+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:14:22.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:14:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:14:22.147+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:14:22.872 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:14:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:14:22.553+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:14:22.872 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:14:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:14:22.553+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:14:22.872 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:14:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:14:22.553+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:14:22.873 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:14:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:14:22.625+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:14:22.873 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:14:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:14:22.625+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:14:22.873 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:14:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:14:22.625+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:14:22.873 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:14:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:14:22.625+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:14:23.150 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:14:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:14:22.880+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:22.881354+0000) 2024-02-20T09:14:23.150 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:14:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:14:22.998+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:22.998449+0000) 2024-02-20T09:14:23.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:14:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:14:23.148+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:23.149080+0000) 2024-02-20T09:14:23.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:14:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:14:23.189+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:14:23.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:14:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:14:23.189+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:14:23.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:14:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:14:23.189+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:14:23.872 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:14:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:14:23.536+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:14:23.872 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:14:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:14:23.536+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:14:23.872 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:14:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:14:23.536+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:14:23.873 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:14:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:14:23.601+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:14:23.873 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:14:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:14:23.601+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:14:23.873 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:14:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:14:23.601+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:14:23.873 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:14:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:14:23.601+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:14:24.152 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:14:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:14:23.998+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:23.998537+0000) 2024-02-20T09:14:24.152 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:14:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:14:23.881+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:23.881444+0000) 2024-02-20T09:14:24.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:14:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:14:24.150+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:24.151281+0000) 2024-02-20T09:14:24.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:14:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:14:24.163+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:14:24.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:14:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:14:24.163+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:14:24.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:14:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:14:24.163+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:14:24.872 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:14:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:14:24.579+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:14:24.872 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:14:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:14:24.579+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:14:24.873 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:14:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:14:24.579+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:14:24.873 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:14:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:14:24.628+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:14:24.873 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:14:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:14:24.628+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:14:24.873 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:14:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:14:24.628+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:14:24.873 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:14:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:14:24.628+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:14:25.153 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:14:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:14:24.881+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:24.881540+0000) 2024-02-20T09:14:25.153 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:14:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:14:24.998+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:24.998628+0000) 2024-02-20T09:14:25.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:14:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:14:25.152+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:25.152391+0000) 2024-02-20T09:14:25.492 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:14:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:14:25.166+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:14:25.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:14:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:14:25.166+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:14:25.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:14:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:14:25.166+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:14:25.872 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:14:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:14:25.588+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:14:25.873 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:14:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:14:25.588+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:14:25.873 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:14:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:14:25.588+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:14:25.873 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:14:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:14:25.641+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:14:25.873 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:14:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:14:25.641+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:14:25.873 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:14:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:14:25.641+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:14:25.873 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:14:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:14:25.641+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:14:26.153 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:14:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:14:25.936+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 63764 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:14:26.154 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:14:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:14:25.934+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:25.935631+0000) 2024-02-20T09:14:26.154 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:14:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:14:25.880+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:25.881640+0000) 2024-02-20T09:14:26.154 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:14:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:14:25.930+0000 7f8a5f86e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:25.932244+0000) 2024-02-20T09:14:26.155 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:14:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:14:25.931+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:25.932548+0000) 2024-02-20T09:14:26.155 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:14:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:14:25.997+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:25.998717+0000) 2024-02-20T09:14:26.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:14:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:14:26.151+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:26.152630+0000) 2024-02-20T09:14:26.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:14:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:14:26.152+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:14:26.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:14:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:14:26.152+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:14:26.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:14:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:14:26.152+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:14:26.872 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:14:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:14:26.636+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:14:26.873 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:14:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:14:26.636+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:14:26.873 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:14:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:14:26.636+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:14:26.873 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:14:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:14:26.690+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:14:26.873 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:14:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:14:26.690+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:14:26.873 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:14:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:14:26.690+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:14:26.873 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:14:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:14:26.690+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:14:27.147 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:14:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:14:26.997+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:26.998793+0000) 2024-02-20T09:14:27.147 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:14:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:14:26.880+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:26.881785+0000) 2024-02-20T09:14:27.147 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:14:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:14:27.144+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:14:27.147 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:14:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:14:27.144+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:14:27.148 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:14:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:14:27.144+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:14:27.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:14:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:14:27.154+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:27.155555+0000) 2024-02-20T09:14:27.872 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:14:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:14:27.612+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:14:27.873 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:14:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:14:27.612+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:14:27.873 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:14:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:14:27.612+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:14:27.873 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:14:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:14:27.703+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:14:27.873 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:14:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:14:27.703+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:14:27.873 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:14:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:14:27.703+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:14:27.873 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:14:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:14:27.703+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:14:28.150 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:14:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:14:27.997+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:27.998948+0000) 2024-02-20T09:14:28.150 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:14:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:14:27.880+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:27.881866+0000) 2024-02-20T09:14:28.150 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:14:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:14:28.147+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:14:28.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:14:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:14:28.156+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:28.158143+0000) 2024-02-20T09:14:28.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:14:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:14:28.147+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:14:28.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:14:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:14:28.147+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:14:28.873 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:14:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:14:28.595+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:14:28.873 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:14:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:14:28.595+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:14:28.873 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:14:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:14:28.595+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:14:28.873 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:14:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:14:28.682+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:14:28.873 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:14:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:14:28.682+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:14:28.874 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:14:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:14:28.682+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:14:28.874 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:14:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:14:28.682+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:14:29.135 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:14:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:14:28.997+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:28.999035+0000) 2024-02-20T09:14:29.135 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:14:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:14:28.880+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:28.881963+0000) 2024-02-20T09:14:29.136 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:14:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:14:29.133+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:14:29.136 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:14:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:14:29.133+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:14:29.136 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:14:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:14:29.133+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:14:29.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:14:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:14:29.159+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:29.160682+0000) 2024-02-20T09:14:29.873 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:14:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:14:29.608+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:14:29.873 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:14:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:14:29.608+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:14:29.873 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:14:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:14:29.608+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:14:29.873 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:14:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:14:29.638+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:14:29.874 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:14:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:14:29.638+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:14:29.874 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:14:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:14:29.638+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:14:29.874 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:14:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:14:29.639+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:14:30.129 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:14:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:14:29.997+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:29.999145+0000) 2024-02-20T09:14:30.130 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:14:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:14:29.880+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:29.882087+0000) 2024-02-20T09:14:30.130 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:14:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:14:30.127+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:14:30.130 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:14:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:14:30.127+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:14:30.130 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:14:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:14:30.127+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:14:30.421 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:14:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:14:30.160+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:30.161929+0000) 2024-02-20T09:14:30.688 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:14:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:14:30.563+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:14:30.688 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:14:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:14:30.563+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:14:30.689 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:14:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:14:30.563+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:14:30.940 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:14:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:14:30.685+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:14:30.940 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:14:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:14:30.685+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:14:30.941 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:14:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:14:30.685+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:14:30.941 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:14:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:14:30.686+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:14:30.941 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:14:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:14:30.880+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:30.882210+0000) 2024-02-20T09:14:31.242 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:14:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:14:30.943+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 63822 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:14:31.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:14:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:14:30.938+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:30.939812+0000) 2024-02-20T09:14:31.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:14:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:14:30.997+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:30.999265+0000) 2024-02-20T09:14:31.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:14:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:14:30.941+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:30.942616+0000) 2024-02-20T09:14:31.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:14:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:14:31.160+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:31.162193+0000) 2024-02-20T09:14:31.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:14:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:14:30.937+0000 7f8a5f86e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:30.939333+0000) 2024-02-20T09:14:31.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:14:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:14:31.141+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:14:31.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:14:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:14:31.141+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:14:31.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:14:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:14:31.141+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:14:31.873 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:14:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:14:31.583+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:14:31.873 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:14:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:14:31.583+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:14:31.873 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:14:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:14:31.583+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:14:31.874 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:14:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:14:31.732+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:14:31.874 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:14:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:14:31.732+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:14:31.874 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:14:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:14:31.732+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:14:31.874 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:14:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:14:31.732+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:14:32.164 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:14:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:14:31.997+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:31.999376+0000) 2024-02-20T09:14:32.164 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:14:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:14:31.880+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:31.882385+0000) 2024-02-20T09:14:32.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:14:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:14:32.164+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:32.165552+0000) 2024-02-20T09:14:32.492 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:14:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:14:32.161+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:14:32.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:14:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:14:32.161+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:14:32.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:14:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:14:32.161+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:14:32.873 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:14:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:14:32.561+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:14:32.873 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:14:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:14:32.561+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:14:32.874 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:14:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:14:32.561+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:14:32.874 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:14:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:14:32.692+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:14:32.874 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:14:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:14:32.692+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:14:32.874 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:14:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:14:32.692+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:14:32.874 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:14:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:14:32.692+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:14:33.130 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:14:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:14:32.998+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:32.999475+0000) 2024-02-20T09:14:33.130 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:14:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:14:32.881+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:32.882517+0000) 2024-02-20T09:14:33.130 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:14:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:14:33.127+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:14:33.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:14:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:14:33.167+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:33.169054+0000) 2024-02-20T09:14:33.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:14:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:14:33.127+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:14:33.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:14:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:14:33.127+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:14:33.873 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:14:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:14:33.538+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:14:33.873 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:14:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:14:33.538+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:14:33.873 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:14:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:14:33.538+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:14:33.874 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:14:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:14:33.690+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:14:33.874 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:14:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:14:33.690+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:14:33.874 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:14:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:14:33.690+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:14:33.874 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:14:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:14:33.690+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:14:34.172 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:14:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:14:33.998+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:33.999564+0000) 2024-02-20T09:14:34.172 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:14:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:14:33.881+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:33.882627+0000) 2024-02-20T09:14:34.173 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:14:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:14:34.088+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:14:34.173 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:14:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:14:34.088+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:14:34.173 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:14:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:14:34.088+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:14:34.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:14:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:14:34.170+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:34.171387+0000) 2024-02-20T09:14:34.873 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:14:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:14:34.535+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:14:34.874 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:14:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:14:34.535+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:14:34.874 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:14:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:14:34.535+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:14:34.874 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:14:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:14:34.700+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:14:34.874 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:14:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:14:34.700+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:14:34.874 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:14:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:14:34.700+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:14:34.874 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:14:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:14:34.700+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:14:35.174 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:14:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:14:34.998+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:34.999676+0000) 2024-02-20T09:14:35.174 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:14:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:14:34.881+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:34.882761+0000) 2024-02-20T09:14:35.175 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:14:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:14:35.098+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:14:35.175 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:14:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:14:35.098+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:14:35.175 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:14:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:14:35.098+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:14:35.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:14:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:14:35.171+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:35.173311+0000) 2024-02-20T09:14:35.873 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:14:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:14:35.578+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:14:35.874 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:14:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:14:35.578+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:14:35.874 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:14:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:14:35.578+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:14:35.874 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:14:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:14:35.697+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:14:35.874 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:14:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:14:35.697+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:14:35.874 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:14:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:14:35.697+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:14:35.874 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:14:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:14:35.697+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:14:36.174 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:14:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:14:35.950+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 63879 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:14:36.175 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:14:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:14:35.948+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:35.950291+0000) 2024-02-20T09:14:36.176 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:14:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:14:35.945+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:35.946550+0000) 2024-02-20T09:14:36.176 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:14:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:14:35.998+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:35.999843+0000) 2024-02-20T09:14:36.176 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:14:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:14:35.881+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:35.882861+0000) 2024-02-20T09:14:36.176 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:14:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:14:35.945+0000 7f8a5f86e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:35.946506+0000) 2024-02-20T09:14:36.177 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:14:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:14:36.050+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:14:36.177 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:14:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:14:36.050+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:14:36.177 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:14:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:14:36.050+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:14:36.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:14:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:14:36.172+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:36.173533+0000) 2024-02-20T09:14:36.873 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:14:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:14:36.557+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:14:36.874 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:14:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:14:36.557+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:14:36.874 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:14:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:14:36.557+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:14:36.874 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:14:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:14:36.714+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:14:36.874 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:14:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:14:36.714+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:14:36.874 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:14:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:14:36.714+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:14:36.874 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:14:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:14:36.714+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:14:37.177 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:14:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:14:36.998+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:36.999945+0000) 2024-02-20T09:14:37.177 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:14:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:14:36.881+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:36.883002+0000) 2024-02-20T09:14:37.177 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:14:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:14:37.020+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:14:37.177 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:14:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:14:37.020+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:14:37.177 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:14:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:14:37.020+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:14:37.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:14:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:14:37.174+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:37.175969+0000) 2024-02-20T09:14:37.874 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:14:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:14:37.517+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:14:37.874 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:14:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:14:37.517+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:14:37.874 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:14:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:14:37.517+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:14:37.874 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:14:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:14:37.696+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:14:37.874 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:14:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:14:37.696+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:14:37.875 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:14:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:14:37.696+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:14:37.875 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:14:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:14:37.696+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:14:38.180 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:14:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:14:37.998+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:38.000046+0000) 2024-02-20T09:14:38.180 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:14:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:14:37.881+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:37.883095+0000) 2024-02-20T09:14:38.181 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:14:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:14:37.977+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:14:38.181 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:14:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:14:37.977+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:14:38.181 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:14:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:14:37.977+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:14:38.488 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:14:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:14:38.178+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:38.179434+0000) 2024-02-20T09:14:38.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:14:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:14:38.485+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:14:38.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:14:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:14:38.485+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:14:38.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:14:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:14:38.485+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:14:38.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:14:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:14:38.668+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:14:38.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:14:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:14:38.668+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:14:38.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:14:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:14:38.668+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:14:38.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:14:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:14:38.668+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:14:39.183 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:14:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:14:38.998+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:39.000167+0000) 2024-02-20T09:14:39.183 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:14:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:14:38.881+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:38.883227+0000) 2024-02-20T09:14:39.183 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:14:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:14:38.939+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:14:39.183 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:14:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:14:38.939+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:14:39.183 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:14:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:14:38.939+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:14:39.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:14:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:14:39.180+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:39.181841+0000) 2024-02-20T09:14:39.770 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:14:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:14:39.634+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:14:39.770 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:14:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:14:39.634+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:14:39.771 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:14:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:14:39.634+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:14:39.771 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:14:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:14:39.634+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:14:39.771 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:14:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:14:39.512+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:14:39.771 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:14:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:14:39.512+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:14:39.771 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:14:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:14:39.512+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:14:40.184 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:14:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:14:39.998+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:40.000286+0000) 2024-02-20T09:14:40.185 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:14:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:14:39.881+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:39.883360+0000) 2024-02-20T09:14:40.185 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:14:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:14:39.910+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:14:40.185 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:14:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:14:39.910+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:14:40.185 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:14:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:14:39.910+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:14:40.475 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:14:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:14:40.182+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:40.183437+0000) 2024-02-20T09:14:40.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:14:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:14:40.472+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:14:40.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:14:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:14:40.472+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:14:40.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:14:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:14:40.472+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:14:40.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:14:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:14:40.636+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:14:40.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:14:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:14:40.636+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:14:40.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:14:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:14:40.636+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:14:40.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:14:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:14:40.636+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:14:41.077 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:14:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:14:40.957+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 63937 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:14:41.077 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:14:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:14:40.956+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:40.957748+0000) 2024-02-20T09:14:41.077 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:14:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:14:40.952+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:40.953522+0000) 2024-02-20T09:14:41.078 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:14:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:14:40.999+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:41.000426+0000) 2024-02-20T09:14:41.078 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:14:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:14:40.882+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:40.883470+0000) 2024-02-20T09:14:41.078 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:14:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:14:40.892+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:14:41.078 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:14:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:14:40.892+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:14:41.078 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:14:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:14:40.892+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:14:41.078 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:14:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:14:40.952+0000 7f8a5f86e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:40.953680+0000) 2024-02-20T09:14:41.466 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:14:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:14:41.182+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:41.183634+0000) 2024-02-20T09:14:41.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:14:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:14:41.464+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:14:41.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:14:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:14:41.464+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:14:41.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:14:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:14:41.464+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:14:41.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:14:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:14:41.627+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:14:41.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:14:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:14:41.627+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:14:41.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:14:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:14:41.627+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:14:41.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:14:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:14:41.627+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:14:42.186 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:14:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:14:41.999+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:42.000540+0000) 2024-02-20T09:14:42.187 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:14:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:14:41.878+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:14:42.187 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:14:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:14:41.878+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:14:42.187 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:14:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:14:41.878+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:14:42.187 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:14:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:14:41.882+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:41.883569+0000) 2024-02-20T09:14:42.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:14:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:14:42.184+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:42.185585+0000) 2024-02-20T09:14:42.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:14:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:14:42.432+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:14:42.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:14:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:14:42.432+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:14:42.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:14:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:14:42.432+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:14:42.841 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:14:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:14:42.590+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:14:42.842 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:14:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:14:42.590+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:14:42.842 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:14:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:14:42.590+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:14:42.842 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:14:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:14:42.590+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:14:43.189 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:14:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:14:42.999+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:43.000626+0000) 2024-02-20T09:14:43.189 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:14:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:14:42.839+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:14:43.189 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:14:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:14:42.839+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:14:43.189 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:14:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:14:42.839+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:14:43.189 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:14:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:14:42.882+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:42.883654+0000) 2024-02-20T09:14:43.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:14:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:14:43.186+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:43.188146+0000) 2024-02-20T09:14:43.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:14:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:14:43.415+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:14:43.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:14:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:14:43.415+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:14:43.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:14:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:14:43.415+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:14:43.849 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:14:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:14:43.627+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:14:43.849 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:14:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:14:43.627+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:14:43.850 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:14:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:14:43.627+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:14:43.850 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:14:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:14:43.628+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:14:44.192 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:14:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:14:43.999+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:44.000730+0000) 2024-02-20T09:14:44.193 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:14:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:14:43.847+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:14:44.193 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:14:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:14:43.847+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:14:44.193 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:14:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:14:43.847+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:14:44.193 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:14:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:14:43.882+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:43.883742+0000) 2024-02-20T09:14:44.443 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:14:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:14:44.190+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:44.191716+0000) 2024-02-20T09:14:44.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:14:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:14:44.441+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:14:44.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:14:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:14:44.441+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:14:44.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:14:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:14:44.441+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:14:44.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:14:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:14:44.597+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:14:44.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:14:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:14:44.597+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:14:44.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:14:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:14:44.597+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:14:44.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:14:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:14:44.597+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:14:45.194 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:14:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:14:44.999+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:45.000860+0000) 2024-02-20T09:14:45.195 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:14:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:14:44.841+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:14:45.195 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:14:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:14:44.841+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:14:45.195 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:14:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:14:44.841+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:14:45.195 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:14:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:14:44.882+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:44.883876+0000) 2024-02-20T09:14:45.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:14:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:14:45.192+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:45.193607+0000) 2024-02-20T09:14:45.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:14:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:14:45.438+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:14:45.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:14:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:14:45.438+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:14:45.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:14:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:14:45.438+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:14:45.822 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:14:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:14:45.600+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:14:45.823 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:14:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:14:45.600+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:14:45.823 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:14:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:14:45.600+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:14:45.823 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:14:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:14:45.600+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:14:46.085 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:14:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:14:45.965+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 63994 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:14:46.086 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:14:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:14:45.962+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:45.963617+0000) 2024-02-20T09:14:46.086 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:14:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:14:45.959+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:45.960558+0000) 2024-02-20T09:14:46.086 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:14:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:14:45.999+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:46.001035+0000) 2024-02-20T09:14:46.087 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:14:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:14:45.820+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:14:46.087 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:14:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:14:45.820+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:14:46.087 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:14:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:14:45.820+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:14:46.087 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:14:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:14:45.882+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:45.884010+0000) 2024-02-20T09:14:46.087 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:14:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:14:45.958+0000 7f8a5f86e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:45.960249+0000) 2024-02-20T09:14:46.445 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:14:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:14:46.192+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:46.193835+0000) 2024-02-20T09:14:46.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:14:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:14:46.557+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:14:46.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:14:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:14:46.557+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:14:46.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:14:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:14:46.557+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:14:46.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:14:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:14:46.557+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:14:46.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:14:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:14:46.442+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:14:46.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:14:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:14:46.442+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:14:46.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:14:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:14:46.442+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:14:47.197 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:14:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:14:46.999+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:47.001205+0000) 2024-02-20T09:14:47.197 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:14:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:14:46.806+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:14:47.197 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:14:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:14:46.806+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:14:47.197 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:14:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:14:46.806+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:14:47.197 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:14:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:14:46.882+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:46.884147+0000) 2024-02-20T09:14:47.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:14:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:14:47.194+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:47.195958+0000) 2024-02-20T09:14:47.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:14:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:14:47.411+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:14:47.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:14:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:14:47.411+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:14:47.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:14:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:14:47.411+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:14:47.791 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:14:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:14:47.526+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:14:47.791 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:14:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:14:47.526+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:14:47.791 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:14:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:14:47.526+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:14:47.791 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:14:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:14:47.526+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:14:48.199 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:14:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:14:47.999+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:48.001362+0000) 2024-02-20T09:14:48.199 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:14:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:14:47.788+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:14:48.200 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:14:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:14:47.788+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:14:48.200 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:14:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:14:47.788+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:14:48.200 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:14:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:14:47.882+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:47.884256+0000) 2024-02-20T09:14:48.481 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:14:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:14:48.441+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:14:48.481 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:14:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:14:48.441+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:14:48.481 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:14:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:14:48.441+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:14:48.481 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:14:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:14:48.196+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:48.198169+0000) 2024-02-20T09:14:48.482 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:14:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:14:48.478+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:14:48.482 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:14:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:14:48.478+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:14:48.482 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:14:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:14:48.478+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:14:48.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:14:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:14:48.478+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:14:49.201 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:14:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:14:49.000+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:49.001431+0000) 2024-02-20T09:14:49.202 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:14:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:14:48.835+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:14:49.202 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:14:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:14:48.835+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:14:49.202 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:14:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:14:48.835+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:14:49.202 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:14:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:14:48.883+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:48.884436+0000) 2024-02-20T09:14:49.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:14:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:14:49.199+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:49.200572+0000) 2024-02-20T09:14:49.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:14:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:14:49.427+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:14:49.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:14:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:14:49.427+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:14:49.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:14:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:14:49.427+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:14:49.778 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:14:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:14:49.496+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:14:49.778 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:14:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:14:49.496+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:14:49.778 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:14:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:14:49.496+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:14:49.778 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:14:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:14:49.496+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:14:50.203 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:14:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:14:50.000+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:50.001526+0000) 2024-02-20T09:14:50.203 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:14:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:14:49.791+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:14:50.203 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:14:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:14:49.791+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:14:50.204 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:14:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:14:49.791+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:14:50.204 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:14:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:14:49.883+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:49.884538+0000) 2024-02-20T09:14:50.454 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:14:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:14:50.200+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:50.202028+0000) 2024-02-20T09:14:50.454 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:14:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:14:50.407+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:14:50.454 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:14:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:14:50.407+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:14:50.454 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:14:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:14:50.407+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:14:50.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:14:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:14:50.451+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:14:50.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:14:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:14:50.451+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:14:50.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:14:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:14:50.451+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:14:50.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:14:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:14:50.451+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:14:51.091 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:14:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:14:50.972+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 64052 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:14:51.091 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:14:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:14:50.969+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:50.970501+0000) 2024-02-20T09:14:51.092 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:14:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:14:50.966+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:50.967669+0000) 2024-02-20T09:14:51.092 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:14:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:14:51.000+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:51.001620+0000) 2024-02-20T09:14:51.092 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:14:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:14:50.766+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:14:51.092 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:14:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:14:50.766+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:14:51.092 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:14:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:14:50.766+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:14:51.092 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:14:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:14:50.883+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:50.884655+0000) 2024-02-20T09:14:51.093 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:14:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:14:50.966+0000 7f8a5f86e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:50.967748+0000) 2024-02-20T09:14:51.371 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:14:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:14:51.200+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:51.202229+0000) 2024-02-20T09:14:51.635 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:14:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:14:51.369+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:14:51.635 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:14:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:14:51.369+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:14:51.635 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:14:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:14:51.369+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:14:51.636 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:14:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:14:51.439+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:14:51.636 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:14:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:14:51.439+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:14:51.636 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:14:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:14:51.439+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:14:51.636 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:14:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:14:51.439+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:14:51.886 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:14:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:14:51.753+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:14:51.886 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:14:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:14:51.753+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:14:51.886 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:14:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:14:51.753+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:14:52.206 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:14:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:14:51.883+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:51.884773+0000) 2024-02-20T09:14:52.207 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:14:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:14:52.000+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:52.001712+0000) 2024-02-20T09:14:52.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:14:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:14:52.204+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:52.205545+0000) 2024-02-20T09:14:52.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:14:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:14:52.439+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:14:52.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:14:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:14:52.439+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:14:52.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:14:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:14:52.440+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:14:52.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:14:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:14:52.440+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:14:52.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:14:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:14:52.391+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:14:52.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:14:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:14:52.391+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:14:52.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:14:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:14:52.391+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:14:52.782 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:14:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:14:52.759+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:14:52.782 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:14:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:14:52.759+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:14:52.782 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:14:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:14:52.759+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:14:53.209 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:14:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:14:52.883+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:52.884902+0000) 2024-02-20T09:14:53.209 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:14:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:14:53.000+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:53.001836+0000) 2024-02-20T09:14:53.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:14:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:14:53.436+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:14:53.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:14:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:14:53.436+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:14:53.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:14:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:14:53.436+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:14:53.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:14:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:14:53.206+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:53.208253+0000) 2024-02-20T09:14:53.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:14:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:14:53.407+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:14:53.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:14:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:14:53.407+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:14:53.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:14:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:14:53.407+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:14:53.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:14:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:14:53.407+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:14:53.783 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:14:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:14:53.768+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:14:53.783 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:14:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:14:53.768+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:14:53.783 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:14:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:14:53.768+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:14:54.211 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:14:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:14:53.883+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:53.885001+0000) 2024-02-20T09:14:54.211 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:14:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:14:54.000+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:54.001963+0000) 2024-02-20T09:14:54.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:14:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:14:54.438+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:14:54.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:14:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:14:54.439+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:14:54.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:14:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:14:54.439+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:14:54.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:14:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:14:54.209+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:54.210444+0000) 2024-02-20T09:14:54.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:14:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:14:54.407+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:14:54.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:14:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:14:54.407+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:14:54.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:14:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:14:54.407+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:14:54.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:14:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:14:54.407+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:14:54.783 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:14:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:14:54.732+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:14:54.783 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:14:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:14:54.732+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:14:54.783 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:14:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:14:54.732+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:14:55.212 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:14:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:14:54.883+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:54.885120+0000) 2024-02-20T09:14:55.213 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:14:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:14:55.000+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:55.002090+0000) 2024-02-20T09:14:55.478 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:14:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:14:55.210+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:55.211789+0000) 2024-02-20T09:14:55.479 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:14:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:14:55.440+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:14:55.479 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:14:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:14:55.440+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:14:55.479 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:14:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:14:55.440+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:14:55.479 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:14:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:14:55.440+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:14:55.742 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:14:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:14:55.683+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:14:55.742 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:14:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:14:55.683+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:14:55.742 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:14:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:14:55.683+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:14:55.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:14:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:14:55.476+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:14:55.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:14:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:14:55.476+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:14:55.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:14:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:14:55.476+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:14:56.099 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:14:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:14:55.976+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:55.978150+0000) 2024-02-20T09:14:56.100 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:14:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:14:55.978+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 64109 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:14:56.100 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:14:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:14:55.883+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:55.885236+0000) 2024-02-20T09:14:56.100 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:14:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:14:55.973+0000 7f8a5f86e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:55.974736+0000) 2024-02-20T09:14:56.100 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:14:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:14:55.973+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:55.974966+0000) 2024-02-20T09:14:56.101 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:14:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:14:56.000+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:56.002195+0000) 2024-02-20T09:14:56.438 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:14:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:14:56.210+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:56.211952+0000) 2024-02-20T09:14:56.691 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:14:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:14:56.504+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:14:56.692 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:14:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:14:56.504+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:14:56.692 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:14:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:14:56.504+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:14:56.692 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:14:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:14:56.435+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:14:56.692 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:14:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:14:56.435+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:14:56.693 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:14:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:14:56.435+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:14:56.693 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:14:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:14:56.436+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:14:56.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:14:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:14:56.690+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:14:56.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:14:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:14:56.690+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:14:56.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:14:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:14:56.690+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:14:56.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:14:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:14:56.883+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:56.885325+0000) 2024-02-20T09:14:57.396 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:14:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:14:57.000+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:57.002275+0000) 2024-02-20T09:14:57.397 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:14:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:14:57.214+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:57.215102+0000) 2024-02-20T09:14:57.397 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:14:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:14:57.394+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:14:57.673 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:14:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:14:57.394+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:14:57.673 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:14:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:14:57.394+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:14:57.673 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:14:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:14:57.394+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:14:57.673 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:14:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:14:57.475+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:14:57.673 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:14:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:14:57.475+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:14:57.674 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:14:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:14:57.475+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:14:57.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:14:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:14:57.671+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:14:57.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:14:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:14:57.671+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:14:57.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:14:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:14:57.671+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:14:57.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:14:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:14:57.885+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:57.885473+0000) 2024-02-20T09:14:58.392 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:14:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:14:58.216+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:58.217368+0000) 2024-02-20T09:14:58.393 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:14:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:14:58.002+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:58.002419+0000) 2024-02-20T09:14:58.660 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:14:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:14:58.481+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:14:58.661 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:14:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:14:58.481+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:14:58.661 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:14:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:14:58.481+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:14:58.661 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:14:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:14:58.391+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:14:58.661 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:14:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:14:58.391+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:14:58.661 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:14:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:14:58.391+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:14:58.661 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:14:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:14:58.391+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:14:58.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:14:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:14:58.659+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:14:58.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:14:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:14:58.659+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:14:58.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:14:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:14:58.659+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:14:58.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:14:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:14:58.885+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:58.885609+0000) 2024-02-20T09:14:59.356 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:14:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:14:59.002+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:59.002506+0000) 2024-02-20T09:14:59.356 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:14:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:14:59.219+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:59.219448+0000) 2024-02-20T09:14:59.356 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:14:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:14:59.354+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:14:59.357 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:14:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:14:59.354+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:14:59.636 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:14:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:14:59.354+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:14:59.636 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:14:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:14:59.354+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:14:59.636 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:14:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:14:59.448+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:14:59.637 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:14:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:14:59.448+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:14:59.637 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:14:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:14:59.448+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:14:59.887 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:14:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:14:59.639+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:14:59.887 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:14:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:14:59.639+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:14:59.887 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:14:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:14:59.639+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:15:00.221 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:14:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:14:59.885+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:14:59.885728+0000) 2024-02-20T09:15:00.222 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:15:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:15:00.002+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:00.002644+0000) 2024-02-20T09:15:00.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:15:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:15:00.220+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:00.220568+0000) 2024-02-20T09:15:00.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:15:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:15:00.328+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:15:00.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:15:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:15:00.328+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:15:00.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:15:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:15:00.328+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:15:00.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:15:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:15:00.328+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:15:00.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:15:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:15:00.424+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:15:00.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:15:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:15:00.424+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:15:00.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:15:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:15:00.424+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:15:00.787 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:15:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:15:00.590+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:15:00.788 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:15:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:15:00.590+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:15:00.788 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:15:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:15:00.590+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:15:01.107 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:15:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:15:00.987+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 64167 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:15:01.108 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:15:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:15:00.984+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:00.984842+0000) 2024-02-20T09:15:01.108 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:15:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:15:00.885+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:00.885893+0000) 2024-02-20T09:15:01.108 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:15:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:15:00.980+0000 7f8a5f86e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:00.981345+0000) 2024-02-20T09:15:01.108 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:15:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:15:00.981+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:00.981483+0000) 2024-02-20T09:15:01.108 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:15:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:15:01.002+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:01.002774+0000) 2024-02-20T09:15:01.392 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:15:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:15:01.220+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:01.220842+0000) 2024-02-20T09:15:01.392 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:15:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:15:01.288+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:15:01.392 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:15:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:15:01.288+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:15:01.392 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:15:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:15:01.288+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:15:01.392 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:15:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:15:01.288+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:15:01.742 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:15:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:15:01.600+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:15:01.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:15:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:15:01.600+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:15:01.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:15:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:15:01.600+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:15:01.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:15:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:15:01.390+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:15:01.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:15:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:15:01.390+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:15:01.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:15:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:15:01.390+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:15:02.003 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:15:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:15:01.885+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:01.886038+0000) 2024-02-20T09:15:02.279 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:15:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:15:02.223+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:02.223488+0000) 2024-02-20T09:15:02.280 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:15:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:15:02.002+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:02.002862+0000) 2024-02-20T09:15:02.531 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:15:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:15:02.365+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:15:02.531 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:15:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:15:02.365+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:15:02.531 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:15:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:15:02.365+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:15:02.531 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:15:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:15:02.278+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:15:02.531 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:15:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:15:02.278+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:15:02.531 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:15:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:15:02.278+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:15:02.532 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:15:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:15:02.278+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:15:02.790 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:15:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:15:02.582+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:15:02.790 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:15:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:15:02.582+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:15:02.790 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:15:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:15:02.582+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:15:03.227 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:15:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:15:03.002+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:03.002978+0000) 2024-02-20T09:15:03.227 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:15:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:15:02.885+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:02.886229+0000) 2024-02-20T09:15:03.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:15:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:15:03.350+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:15:03.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:15:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:15:03.350+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:15:03.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:15:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:15:03.350+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:15:03.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:15:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:15:03.225+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:03.225900+0000) 2024-02-20T09:15:03.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:15:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:15:03.297+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:15:03.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:15:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:15:03.297+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:15:03.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:15:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:15:03.297+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:15:03.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:15:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:15:03.297+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:15:03.791 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:15:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:15:03.622+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:15:03.791 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:15:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:15:03.622+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:15:03.791 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:15:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:15:03.622+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:15:04.229 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:15:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:15:03.885+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:03.886346+0000) 2024-02-20T09:15:04.229 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:15:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:15:04.002+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:04.003068+0000) 2024-02-20T09:15:04.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:15:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:15:04.325+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:15:04.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:15:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:15:04.325+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:15:04.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:15:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:15:04.325+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:15:04.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:15:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:15:04.227+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:04.228293+0000) 2024-02-20T09:15:04.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:15:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:15:04.322+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:15:04.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:15:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:15:04.322+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:15:04.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:15:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:15:04.322+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:15:04.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:15:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:15:04.322+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:15:04.791 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:15:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:15:04.604+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:15:04.791 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:15:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:15:04.604+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:15:04.791 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:15:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:15:04.604+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:15:05.230 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:15:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:15:04.886+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:04.886432+0000) 2024-02-20T09:15:05.231 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:15:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:15:05.002+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:05.003160+0000) 2024-02-20T09:15:05.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:15:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:15:05.330+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:15:05.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:15:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:15:05.330+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:15:05.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:15:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:15:05.330+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:15:05.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:15:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:15:05.229+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:05.229728+0000) 2024-02-20T09:15:05.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:15:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:15:05.277+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:15:05.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:15:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:15:05.277+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:15:05.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:15:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:15:05.277+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:15:05.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:15:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:15:05.277+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:15:05.791 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:15:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:15:05.571+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:15:05.792 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:15:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:15:05.571+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:15:05.792 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:15:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:15:05.571+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:15:06.114 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:15:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:15:05.994+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 64224 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:15:06.114 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:15:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:15:05.992+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:05.992514+0000) 2024-02-20T09:15:06.115 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:15:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:15:05.886+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:05.886542+0000) 2024-02-20T09:15:06.115 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:15:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:15:05.988+0000 7f8a5f86e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:05.989306+0000) 2024-02-20T09:15:06.115 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:15:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:15:05.988+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:05.989067+0000) 2024-02-20T09:15:06.115 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:15:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:15:06.002+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:06.003272+0000) 2024-02-20T09:15:06.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:15:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:15:06.355+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:15:06.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:15:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:15:06.355+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:15:06.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:15:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:15:06.355+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:15:06.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:15:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:15:06.229+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:06.229954+0000) 2024-02-20T09:15:06.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:15:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:15:06.242+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:15:06.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:15:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:15:06.242+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:15:06.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:15:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:15:06.242+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:15:06.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:15:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:15:06.242+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:15:06.791 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:15:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:15:06.618+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:15:06.792 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:15:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:15:06.618+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:15:06.792 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:15:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:15:06.618+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:15:07.232 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:15:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:15:06.886+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:06.886643+0000) 2024-02-20T09:15:07.233 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:15:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:15:07.002+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:07.003381+0000) 2024-02-20T09:15:07.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:15:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:15:07.305+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:15:07.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:15:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:15:07.305+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:15:07.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:15:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:15:07.305+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:15:07.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:15:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:15:07.231+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:15:07.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:15:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:15:07.231+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:15:07.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:15:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:15:07.231+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:15:07.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:15:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:15:07.231+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:15:07.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:15:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:15:07.231+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:07.231943+0000) 2024-02-20T09:15:07.795 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:15:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:15:07.627+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:15:07.795 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:15:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:15:07.627+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:15:07.795 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:15:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:15:07.627+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:15:08.199 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:15:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:15:07.886+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:07.886723+0000) 2024-02-20T09:15:08.199 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:15:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:15:08.003+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:08.003507+0000) 2024-02-20T09:15:08.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:15:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:15:08.197+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:15:08.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:15:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:15:08.197+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:15:08.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:15:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:15:08.197+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:15:08.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:15:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:15:08.197+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:15:08.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:15:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:15:08.235+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:08.235477+0000) 2024-02-20T09:15:08.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:15:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:15:08.264+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:15:08.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:15:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:15:08.264+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:15:08.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:15:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:15:08.264+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:15:08.796 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:15:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:15:08.598+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:15:08.797 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:15:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:15:08.598+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:15:08.797 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:15:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:15:08.598+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:15:09.208 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:15:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:15:09.003+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:09.003596+0000) 2024-02-20T09:15:09.209 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:15:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:15:08.886+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:08.886855+0000) 2024-02-20T09:15:09.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:15:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:15:09.275+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:15:09.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:15:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:15:09.275+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:15:09.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:15:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:15:09.275+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:15:09.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:15:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:15:09.207+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:15:09.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:15:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:15:09.207+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:15:09.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:15:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:15:09.207+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:15:09.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:15:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:15:09.207+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:15:09.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:15:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:15:09.237+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:09.237584+0000) 2024-02-20T09:15:09.796 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:15:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:15:09.625+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:15:09.797 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:15:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:15:09.625+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:15:09.797 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:15:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:15:09.625+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:15:10.185 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:15:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:15:09.886+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:09.886969+0000) 2024-02-20T09:15:10.185 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:15:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:15:10.003+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:10.003742+0000) 2024-02-20T09:15:10.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:15:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:15:10.262+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:15:10.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:15:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:15:10.262+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:15:10.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:15:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:15:10.262+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:15:10.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:15:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:15:10.182+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:15:10.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:15:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:15:10.182+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:15:10.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:15:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:15:10.182+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:15:10.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:15:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:15:10.182+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:15:10.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:15:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:15:10.239+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:10.239514+0000) 2024-02-20T09:15:10.796 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:15:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:15:10.589+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:15:10.797 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:15:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:15:10.589+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:15:10.797 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:15:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:15:10.589+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:15:11.121 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:15:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:15:11.001+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 64282 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:15:11.121 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:15:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:15:10.997+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:10.998237+0000) 2024-02-20T09:15:11.121 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:15:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:15:10.886+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:10.887095+0000) 2024-02-20T09:15:11.122 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:15:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:15:10.996+0000 7f8a5f86e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:10.996689+0000) 2024-02-20T09:15:11.122 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:15:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:15:10.995+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:10.995586+0000) 2024-02-20T09:15:11.122 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:15:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:15:11.003+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:11.003865+0000) 2024-02-20T09:15:11.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:15:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:15:11.265+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:15:11.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:15:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:15:11.265+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:15:11.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:15:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:15:11.265+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:15:11.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:15:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:15:11.217+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:15:11.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:15:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:15:11.217+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:15:11.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:15:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:15:11.217+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:15:11.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:15:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:15:11.217+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:15:11.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:15:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:15:11.239+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:11.239720+0000) 2024-02-20T09:15:11.797 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:15:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:15:11.605+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:15:11.797 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:15:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:15:11.605+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:15:11.797 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:15:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:15:11.605+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:15:12.203 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:15:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:15:11.886+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:11.887239+0000) 2024-02-20T09:15:12.204 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:15:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:15:12.003+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:12.004007+0000) 2024-02-20T09:15:12.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:15:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:15:12.202+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:15:12.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:15:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:15:12.202+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:15:12.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:15:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:15:12.202+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:15:12.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:15:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:15:12.202+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:15:12.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:15:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:15:12.241+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:12.242368+0000) 2024-02-20T09:15:12.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:15:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:15:12.251+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:15:12.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:15:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:15:12.251+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:15:12.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:15:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:15:12.251+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:15:12.800 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:15:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:15:12.640+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:15:12.800 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:15:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:15:12.640+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:15:12.800 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:15:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:15:12.640+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:15:13.219 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:15:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:15:12.886+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:12.887375+0000) 2024-02-20T09:15:13.219 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:15:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:15:13.003+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:13.004209+0000) 2024-02-20T09:15:13.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:15:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:15:13.217+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:15:13.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:15:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:15:13.217+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:15:13.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:15:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:15:13.217+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:15:13.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:15:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:15:13.242+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:15:13.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:15:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:15:13.242+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:15:13.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:15:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:15:13.242+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:15:13.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:15:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:15:13.242+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:15:13.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:15:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:15:13.244+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:13.244461+0000) 2024-02-20T09:15:13.801 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:15:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:15:13.675+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:15:13.801 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:15:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:15:13.675+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:15:13.801 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:15:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:15:13.675+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:15:14.231 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:15:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:15:13.887+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:13.887494+0000) 2024-02-20T09:15:14.232 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:15:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:15:14.004+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:14.004449+0000) 2024-02-20T09:15:14.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:15:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:15:14.236+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:15:14.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:15:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:15:14.236+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:15:14.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:15:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:15:14.236+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:15:14.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:15:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:15:14.236+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:15:14.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:15:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:15:14.246+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:14.246483+0000) 2024-02-20T09:15:14.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:15:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:15:14.230+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:15:14.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:15:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:15:14.230+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:15:14.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:15:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:15:14.230+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:15:14.801 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:15:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:15:14.660+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:15:14.801 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:15:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:15:14.660+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:15:14.801 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:15:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:15:14.660+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:15:15.195 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:15:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:15:14.887+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:14.887607+0000) 2024-02-20T09:15:15.195 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:15:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:15:15.004+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:15.004568+0000) 2024-02-20T09:15:15.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:15:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:15:15.193+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:15:15.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:15:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:15:15.193+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:15:15.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:15:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:15:15.193+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:15:15.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:15:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:15:15.193+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:15:15.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:15:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:15:15.247+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:15.247694+0000) 2024-02-20T09:15:15.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:15:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:15:15.220+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:15:15.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:15:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:15:15.220+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:15:15.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:15:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:15:15.220+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:15:15.801 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:15:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:15:15.662+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:15:15.801 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:15:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:15:15.662+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:15:15.802 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:15:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:15:15.662+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:15:16.127 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:15:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:15:16.008+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 64339 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:15:16.127 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:15:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:15:16.005+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:16.005715+0000) 2024-02-20T09:15:16.128 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:15:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:15:16.001+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:16.002385+0000) 2024-02-20T09:15:16.128 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:15:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:15:16.004+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:16.004701+0000) 2024-02-20T09:15:16.128 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:15:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:15:15.887+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:15.887755+0000) 2024-02-20T09:15:16.128 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:15:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:15:16.002+0000 7f8a5f86e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:16.002894+0000) 2024-02-20T09:15:16.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:15:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:15:16.235+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:15:16.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:15:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:15:16.235+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:15:16.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:15:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:15:16.235+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:15:16.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:15:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:15:16.175+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:15:16.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:15:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:15:16.175+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:15:16.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:15:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:15:16.175+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:15:16.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:15:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:15:16.175+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:15:16.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:15:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:15:16.247+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:16.247976+0000) 2024-02-20T09:15:16.801 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:15:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:15:16.696+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:15:16.802 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:15:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:15:16.696+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:15:16.802 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:15:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:15:16.696+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:15:17.154 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:15:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:15:16.887+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:16.887849+0000) 2024-02-20T09:15:17.154 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:15:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:15:17.004+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:17.004862+0000) 2024-02-20T09:15:17.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:15:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:15:17.209+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:15:17.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:15:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:15:17.209+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:15:17.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:15:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:15:17.209+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:15:17.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:15:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:15:17.152+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:15:17.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:15:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:15:17.152+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:15:17.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:15:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:15:17.152+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:15:17.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:15:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:15:17.153+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:15:17.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:15:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:15:17.250+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:17.251221+0000) 2024-02-20T09:15:17.803 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:15:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:15:17.720+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:15:17.804 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:15:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:15:17.720+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:15:17.804 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:15:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:15:17.720+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:15:18.125 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:15:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:15:17.887+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:17.887975+0000) 2024-02-20T09:15:18.125 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:15:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:15:18.004+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:18.005020+0000) 2024-02-20T09:15:18.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:15:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:15:18.259+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:15:18.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:15:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:15:18.259+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:15:18.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:15:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:15:18.259+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:15:18.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:15:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:15:18.123+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:15:18.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:15:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:15:18.123+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:15:18.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:15:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:15:18.123+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:15:18.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:15:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:15:18.123+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:15:18.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:15:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:15:18.254+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:18.254853+0000) 2024-02-20T09:15:18.805 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:15:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:15:18.750+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:15:18.805 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:15:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:15:18.750+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:15:18.805 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:15:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:15:18.750+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:15:19.107 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:15:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:15:18.886+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:18.888093+0000) 2024-02-20T09:15:19.107 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:15:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:15:19.003+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:19.005197+0000) 2024-02-20T09:15:19.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:15:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:15:19.278+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:15:19.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:15:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:15:19.278+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:15:19.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:15:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:15:19.278+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:15:19.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:15:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:15:19.105+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:15:19.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:15:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:15:19.105+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:15:19.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:15:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:15:19.105+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:15:19.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:15:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:15:19.105+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:15:19.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:15:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:15:19.257+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:19.258441+0000) 2024-02-20T09:15:20.060 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:15:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:15:20.003+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:20.005349+0000) 2024-02-20T09:15:20.061 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:15:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:15:19.781+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:15:20.061 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:15:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:15:19.781+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:15:20.061 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:15:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:15:19.781+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:15:20.061 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:15:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:15:19.886+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:19.888196+0000) 2024-02-20T09:15:20.311 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:15:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:15:20.058+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:15:20.311 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:15:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:15:20.058+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:15:20.311 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:15:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:15:20.058+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:15:20.312 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:15:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:15:20.058+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:15:20.312 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:15:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:15:20.258+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:20.259639+0000) 2024-02-20T09:15:20.638 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:15:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:15:20.308+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:15:20.639 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:15:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:15:20.308+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:15:20.639 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:15:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:15:20.308+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:15:20.889 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:15:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:15:20.802+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:15:20.889 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:15:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:15:20.802+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:15:20.889 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:15:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:15:20.802+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:15:21.242 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:15:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:15:21.014+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 64395 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:15:21.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:15:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:15:21.004+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:21.005500+0000) 2024-02-20T09:15:21.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:15:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:15:21.008+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:21.009573+0000) 2024-02-20T09:15:21.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:15:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:15:20.886+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:20.888298+0000) 2024-02-20T09:15:21.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:15:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:15:21.008+0000 7f8a5f86e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:21.010311+0000) 2024-02-20T09:15:21.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:15:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:15:21.011+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:21.012691+0000) 2024-02-20T09:15:21.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:15:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:15:21.070+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:15:21.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:15:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:15:21.070+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:15:21.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:15:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:15:21.070+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:15:21.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:15:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:15:21.070+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:15:21.533 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:15:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:15:21.258+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:21.259829+0000) 2024-02-20T09:15:21.533 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:15:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:15:21.346+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:15:21.533 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:15:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:15:21.346+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:15:21.533 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:15:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:15:21.346+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:15:21.805 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:15:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:15:21.754+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:15:21.805 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:15:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:15:21.754+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:15:21.805 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:15:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:15:21.754+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:15:22.098 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:15:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:15:21.887+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:21.888436+0000) 2024-02-20T09:15:22.098 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:15:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:15:22.004+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:22.005658+0000) 2024-02-20T09:15:22.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:15:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:15:22.335+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:15:22.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:15:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:15:22.335+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:15:22.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:15:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:15:22.335+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:15:22.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:15:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:15:22.095+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:15:22.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:15:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:15:22.095+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:15:22.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:15:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:15:22.095+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:15:22.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:15:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:15:22.095+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:15:22.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:15:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:15:22.260+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:22.261860+0000) 2024-02-20T09:15:23.092 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:15:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:15:23.004+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:23.005733+0000) 2024-02-20T09:15:23.093 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:15:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:15:22.805+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:15:23.093 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:15:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:15:22.805+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:15:23.093 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:15:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:15:22.805+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:15:23.093 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:15:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:15:22.887+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:22.888524+0000) 2024-02-20T09:15:23.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:15:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:15:23.334+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:15:23.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:15:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:15:23.334+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:15:23.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:15:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:15:23.334+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:15:23.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:15:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:15:23.090+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:15:23.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:15:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:15:23.090+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:15:23.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:15:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:15:23.090+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:15:23.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:15:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:15:23.090+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:15:23.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:15:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:15:23.262+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:23.263850+0000) 2024-02-20T09:15:24.089 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:15:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:15:24.004+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:24.005829+0000) 2024-02-20T09:15:24.089 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:15:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:15:23.786+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:15:24.090 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:15:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:15:23.786+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:15:24.090 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:15:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:15:23.786+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:15:24.090 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:15:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:15:23.887+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:23.888636+0000) 2024-02-20T09:15:24.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:15:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:15:24.299+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:15:24.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:15:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:15:24.299+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:15:24.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:15:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:15:24.299+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:15:24.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:15:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:15:24.086+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:15:24.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:15:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:15:24.086+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:15:24.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:15:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:15:24.086+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:15:24.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:15:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:15:24.086+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:15:24.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:15:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:15:24.266+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:24.267383+0000) 2024-02-20T09:15:24.810 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:15:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:15:24.773+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:15:24.810 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:15:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:15:24.773+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:15:24.810 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:15:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:15:24.773+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:15:25.063 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:15:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:15:24.887+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:24.888730+0000) 2024-02-20T09:15:25.063 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:15:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:15:25.004+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:25.005927+0000) 2024-02-20T09:15:25.314 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:15:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:15:25.060+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:15:25.315 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:15:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:15:25.060+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:15:25.315 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:15:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:15:25.060+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:15:25.315 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:15:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:15:25.061+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:15:25.315 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:15:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:15:25.267+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:25.268561+0000) 2024-02-20T09:15:25.638 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:15:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:15:25.312+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:15:25.639 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:15:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:15:25.312+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:15:25.639 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:15:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:15:25.312+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:15:25.890 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:15:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:15:25.739+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:15:25.890 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:15:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:15:25.739+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:15:25.890 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:15:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:15:25.739+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:15:26.142 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:15:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:15:26.021+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 64434 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:15:26.142 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:15:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:15:25.887+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:25.888850+0000) 2024-02-20T09:15:26.142 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:15:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:15:26.015+0000 7f8a5f86e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:26.016792+0000) 2024-02-20T09:15:26.142 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:15:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:15:26.004+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:26.006035+0000) 2024-02-20T09:15:26.143 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:15:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:15:26.015+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:26.016711+0000) 2024-02-20T09:15:26.143 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:15:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:15:26.019+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:26.020487+0000) 2024-02-20T09:15:26.143 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:15:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:15:26.041+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:15:26.143 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:15:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:15:26.041+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:15:26.143 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:15:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:15:26.041+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:15:26.143 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:15:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:15:26.041+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:15:26.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:15:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:15:26.267+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:26.268790+0000) 2024-02-20T09:15:26.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:15:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:15:26.271+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:15:26.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:15:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:15:26.271+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:15:26.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:15:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:15:26.271+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:15:26.810 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:15:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:15:26.770+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:15:26.810 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:15:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:15:26.770+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:15:26.810 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:15:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:15:26.770+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:15:27.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:15:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:15:26.887+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:26.889009+0000) 2024-02-20T09:15:27.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:15:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:15:27.004+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:27.006164+0000) 2024-02-20T09:15:27.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:15:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:15:27.044+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:15:27.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:15:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:15:27.044+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:15:27.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:15:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:15:27.044+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:15:27.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:15:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:15:27.044+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:15:27.534 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:15:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:15:27.270+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:27.271428+0000) 2024-02-20T09:15:27.534 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:15:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:15:27.258+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:15:27.534 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:15:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:15:27.258+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:15:27.534 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:15:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:15:27.258+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:15:27.813 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:15:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:15:27.764+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:15:27.813 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:15:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:15:27.765+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:15:27.813 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:15:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:15:27.765+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:15:28.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:15:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:15:28.004+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:28.006268+0000) 2024-02-20T09:15:28.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:15:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:15:27.887+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:27.889147+0000) 2024-02-20T09:15:28.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:15:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:15:28.014+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:15:28.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:15:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:15:28.014+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:15:28.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:15:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:15:28.014+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:15:28.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:15:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:15:28.014+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:15:28.533 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:15:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:15:28.273+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:28.275272+0000) 2024-02-20T09:15:28.534 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:15:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:15:28.256+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:15:28.534 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:15:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:15:28.256+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:15:28.534 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:15:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:15:28.256+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:15:28.814 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:15:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:15:28.761+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:15:28.814 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:15:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:15:28.761+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:15:28.814 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:15:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:15:28.761+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:15:29.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:15:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:15:29.004+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:29.006384+0000) 2024-02-20T09:15:29.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:15:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:15:28.887+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:28.889287+0000) 2024-02-20T09:15:29.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:15:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:15:28.966+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:15:29.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:15:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:15:28.966+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:15:29.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:15:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:15:28.966+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:15:29.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:15:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:15:28.966+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:15:29.534 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:15:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:15:29.275+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:29.277363+0000) 2024-02-20T09:15:29.534 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:15:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:15:29.293+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:15:29.534 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:15:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:15:29.293+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:15:29.535 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:15:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:15:29.293+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:15:30.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:15:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:15:30.005+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:30.006483+0000) 2024-02-20T09:15:30.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:15:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:15:29.803+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:15:30.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:15:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:15:29.803+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:15:30.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:15:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:15:29.803+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:15:30.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:15:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:15:29.888+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:29.889452+0000) 2024-02-20T09:15:30.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:15:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:15:29.924+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:15:30.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:15:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:15:29.924+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:15:30.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:15:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:15:29.924+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:15:30.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:15:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:15:29.924+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:15:30.534 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:15:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:15:30.277+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:30.278727+0000) 2024-02-20T09:15:30.534 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:15:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:15:30.276+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:15:30.534 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:15:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:15:30.276+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:15:30.534 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:15:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:15:30.276+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:15:31.147 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:15:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:15:31.028+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 64474 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:15:31.148 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:15:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:15:31.005+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:31.006578+0000) 2024-02-20T09:15:31.148 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:15:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:15:31.022+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:31.024198+0000) 2024-02-20T09:15:31.148 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:15:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:15:30.841+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:15:31.149 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:15:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:15:30.841+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:15:31.149 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:15:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:15:30.842+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:15:31.149 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:15:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:15:30.888+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:30.889578+0000) 2024-02-20T09:15:31.149 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:15:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:15:31.023+0000 7f8a5f86e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:31.024571+0000) 2024-02-20T09:15:31.149 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:15:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:15:30.927+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:15:31.149 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:15:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:15:30.927+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:15:31.149 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:15:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:15:30.927+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:15:31.149 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:15:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:15:30.927+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:15:31.150 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:15:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:15:31.025+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:31.026703+0000) 2024-02-20T09:15:31.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:15:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:15:31.277+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:31.278949+0000) 2024-02-20T09:15:31.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:15:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:15:31.283+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:15:31.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:15:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:15:31.283+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:15:31.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:15:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:15:31.283+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:15:32.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:15:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:15:32.005+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:32.006694+0000) 2024-02-20T09:15:32.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:15:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:15:31.820+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:15:32.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:15:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:15:31.820+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:15:32.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:15:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:15:31.820+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:15:32.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:15:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:15:31.888+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:31.889723+0000) 2024-02-20T09:15:32.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:15:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:15:31.960+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:15:32.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:15:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:15:31.960+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:15:32.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:15:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:15:31.960+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:15:32.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:15:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:15:31.960+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:15:32.534 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:15:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:15:32.279+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:32.281108+0000) 2024-02-20T09:15:32.534 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:15:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:15:32.289+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:15:32.535 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:15:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:15:32.289+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:15:32.535 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:15:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:15:32.289+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:15:33.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:15:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:15:33.005+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:33.006788+0000) 2024-02-20T09:15:33.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:15:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:15:32.958+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:15:33.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:15:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:15:32.958+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:15:33.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:15:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:15:32.958+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:15:33.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:15:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:15:32.958+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:15:33.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:15:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:15:32.827+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:15:33.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:15:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:15:32.827+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:15:33.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:15:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:15:32.827+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:15:33.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:15:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:15:32.888+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:32.889834+0000) 2024-02-20T09:15:33.534 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:15:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:15:33.281+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:33.283293+0000) 2024-02-20T09:15:33.534 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:15:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:15:33.258+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:15:33.535 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:15:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:15:33.258+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:15:33.535 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:15:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:15:33.258+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:15:34.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:15:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:15:34.005+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:34.006897+0000) 2024-02-20T09:15:34.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:15:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:15:33.862+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:15:34.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:15:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:15:33.862+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:15:34.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:15:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:15:33.862+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:15:34.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:15:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:15:33.888+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:33.889982+0000) 2024-02-20T09:15:34.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:15:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:15:33.995+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:15:34.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:15:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:15:33.995+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:15:34.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:15:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:15:33.995+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:15:34.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:15:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:15:33.995+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:15:34.534 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:15:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:15:34.284+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:34.285457+0000) 2024-02-20T09:15:34.535 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:15:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:15:34.240+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:15:34.535 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:15:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:15:34.240+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:15:34.535 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:15:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:15:34.240+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:15:35.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:15:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:15:35.005+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:35.007023+0000) 2024-02-20T09:15:35.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:15:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:15:34.825+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:15:35.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:15:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:15:34.825+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:15:35.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:15:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:15:34.825+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:15:35.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:15:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:15:34.888+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:34.890074+0000) 2024-02-20T09:15:35.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:15:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:15:35.038+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:15:35.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:15:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:15:35.038+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:15:35.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:15:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:15:35.038+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:15:35.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:15:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:15:35.038+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:15:35.535 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:15:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:15:35.285+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:35.287086+0000) 2024-02-20T09:15:35.535 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:15:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:15:35.255+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:15:35.535 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:15:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:15:35.255+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:15:35.535 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:15:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:15:35.255+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:15:36.155 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:15:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:15:36.005+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:36.007174+0000) 2024-02-20T09:15:36.155 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:15:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:15:36.030+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:36.031763+0000) 2024-02-20T09:15:36.155 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:15:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:15:36.002+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:15:36.155 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:15:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:15:36.002+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:15:36.156 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:15:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:15:36.002+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:15:36.156 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:15:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:15:36.002+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:15:36.156 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:15:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:15:36.032+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:36.034025+0000) 2024-02-20T09:15:36.156 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:15:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:15:35.812+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:15:36.156 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:15:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:15:35.812+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:15:36.156 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:15:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:15:35.812+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:15:36.157 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:15:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:15:35.888+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:35.890185+0000) 2024-02-20T09:15:36.157 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:15:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:15:36.030+0000 7f8a5f86e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:36.031626+0000) 2024-02-20T09:15:36.157 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:15:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:15:36.035+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 64513 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:15:36.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:15:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:15:36.285+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:36.287269+0000) 2024-02-20T09:15:36.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:15:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:15:36.299+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:15:36.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:15:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:15:36.299+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:15:36.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:15:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:15:36.299+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:15:37.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:15:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:15:37.005+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:37.007288+0000) 2024-02-20T09:15:37.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:15:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:15:36.859+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:15:37.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:15:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:15:36.859+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:15:37.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:15:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:15:36.859+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:15:37.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:15:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:15:36.888+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:36.890291+0000) 2024-02-20T09:15:37.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:15:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:15:37.012+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:15:37.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:15:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:15:37.012+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:15:37.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:15:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:15:37.012+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:15:37.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:15:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:15:37.013+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:15:37.534 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:15:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:15:37.289+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:37.290676+0000) 2024-02-20T09:15:37.535 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:15:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:15:37.263+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:15:37.535 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:15:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:15:37.263+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:15:37.535 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:15:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:15:37.263+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:15:38.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:15:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:15:38.006+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:38.007389+0000) 2024-02-20T09:15:38.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:15:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:15:37.964+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:15:38.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:15:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:15:37.964+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:15:38.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:15:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:15:37.964+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:15:38.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:15:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:15:37.965+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:15:38.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:15:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:15:37.885+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:15:38.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:15:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:15:37.885+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:15:38.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:15:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:15:37.885+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:15:38.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:15:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:15:37.888+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:37.890370+0000) 2024-02-20T09:15:38.640 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:15:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:15:38.292+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:38.294122+0000) 2024-02-20T09:15:38.640 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:15:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:15:38.310+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:15:38.640 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:15:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:15:38.310+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:15:38.641 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:15:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:15:38.310+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:15:38.891 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:15:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:15:38.871+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:15:38.891 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:15:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:15:38.871+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:15:38.892 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:15:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:15:38.871+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:15:39.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:15:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:15:38.889+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:38.890489+0000) 2024-02-20T09:15:39.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:15:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:15:39.006+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:39.007508+0000) 2024-02-20T09:15:39.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:15:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:15:38.943+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:15:39.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:15:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:15:38.943+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:15:39.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:15:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:15:38.943+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:15:39.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:15:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:15:38.943+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:15:39.535 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:15:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:15:39.296+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:39.297570+0000) 2024-02-20T09:15:39.535 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:15:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:15:39.273+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:15:39.535 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:15:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:15:39.273+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:15:39.535 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:15:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:15:39.273+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:15:40.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:15:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:15:40.006+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:40.007625+0000) 2024-02-20T09:15:40.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:15:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:15:39.889+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:39.890580+0000) 2024-02-20T09:15:40.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:15:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:15:39.898+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:15:40.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:15:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:15:39.898+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:15:40.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:15:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:15:39.898+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:15:40.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:15:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:15:39.982+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:15:40.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:15:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:15:39.982+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:15:40.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:15:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:15:39.982+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:15:40.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:15:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:15:39.982+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:15:40.643 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:15:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:15:40.298+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:40.299592+0000) 2024-02-20T09:15:40.644 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:15:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:15:40.309+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:15:40.644 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:15:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:15:40.309+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:15:40.644 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:15:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:15:40.309+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:15:41.163 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:15:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:15:41.043+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 64555 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:15:41.164 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:15:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:15:41.006+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:41.007690+0000) 2024-02-20T09:15:41.164 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:15:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:15:41.037+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:41.038464+0000) 2024-02-20T09:15:41.164 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:15:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:15:40.975+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:15:41.165 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:15:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:15:40.975+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:15:41.165 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:15:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:15:40.975+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:15:41.165 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:15:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:15:40.975+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:15:41.165 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:15:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:15:41.039+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:41.040451+0000) 2024-02-20T09:15:41.165 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:15:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:15:40.889+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:40.890692+0000) 2024-02-20T09:15:41.165 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:15:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:15:40.898+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:15:41.165 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:15:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:15:40.898+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:15:41.165 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:15:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:15:40.898+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:15:41.166 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:15:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:15:41.037+0000 7f8a5f86e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:41.038733+0000) 2024-02-20T09:15:41.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:15:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:15:41.298+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:41.299854+0000) 2024-02-20T09:15:41.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:15:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:15:41.272+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:15:41.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:15:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:15:41.272+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:15:41.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:15:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:15:41.272+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:15:42.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:15:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:15:42.006+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:42.007779+0000) 2024-02-20T09:15:42.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:15:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:15:42.006+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:15:42.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:15:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:15:42.006+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:15:42.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:15:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:15:42.006+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:15:42.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:15:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:15:42.006+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:15:42.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:15:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:15:41.889+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:41.890783+0000) 2024-02-20T09:15:42.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:15:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:15:41.936+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:15:42.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:15:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:15:41.936+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:15:42.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:15:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:15:41.936+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:15:42.535 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:15:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:15:42.300+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:42.301975+0000) 2024-02-20T09:15:42.535 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:15:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:15:42.266+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:15:42.536 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:15:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:15:42.266+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:15:42.536 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:15:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:15:42.266+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:15:43.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:15:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:15:43.006+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:43.007874+0000) 2024-02-20T09:15:43.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:15:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:15:42.889+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:42.890883+0000) 2024-02-20T09:15:43.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:15:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:15:42.960+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:15:43.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:15:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:15:42.960+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:15:43.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:15:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:15:42.960+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:15:43.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:15:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:15:43.018+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:15:43.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:15:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:15:43.018+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:15:43.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:15:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:15:43.018+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:15:43.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:15:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:15:43.019+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:15:43.535 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:15:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:15:43.302+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:43.304224+0000) 2024-02-20T09:15:43.536 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:15:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:15:43.254+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:15:43.536 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:15:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:15:43.254+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:15:43.536 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:15:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:15:43.254+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:15:44.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:15:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:15:44.006+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:44.008025+0000) 2024-02-20T09:15:44.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:15:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:15:43.889+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:43.891016+0000) 2024-02-20T09:15:44.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:15:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:15:43.983+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:15:44.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:15:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:15:43.983+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:15:44.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:15:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:15:43.983+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:15:44.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:15:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:15:44.046+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:15:44.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:15:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:15:44.046+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:15:44.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:15:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:15:44.046+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:15:44.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:15:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:15:44.046+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:15:44.640 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:15:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:15:44.304+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:44.306215+0000) 2024-02-20T09:15:44.641 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:15:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:15:44.288+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:15:44.641 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:15:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:15:44.288+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:15:44.641 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:15:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:15:44.288+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:15:45.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:15:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:15:45.006+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:45.008119+0000) 2024-02-20T09:15:45.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:15:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:15:45.045+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:15:45.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:15:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:15:45.045+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:15:45.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:15:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:15:45.045+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:15:45.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:15:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:15:45.046+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:15:45.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:15:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:15:44.889+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:44.891194+0000) 2024-02-20T09:15:45.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:15:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:15:44.995+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:15:45.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:15:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:15:44.995+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:15:45.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:15:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:15:44.995+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:15:45.641 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:15:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:15:45.306+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:45.307549+0000) 2024-02-20T09:15:45.641 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:15:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:15:45.295+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:15:45.641 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:15:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:15:45.295+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:15:45.641 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:15:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:15:45.295+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:15:46.170 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:15:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:15:46.050+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 64594 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:15:46.170 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:15:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:15:46.006+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:46.008250+0000) 2024-02-20T09:15:46.171 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:15:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:15:46.044+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:46.046102+0000) 2024-02-20T09:15:46.171 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:15:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:15:45.889+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:45.891370+0000) 2024-02-20T09:15:46.171 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:15:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:15:45.958+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:15:46.171 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:15:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:15:45.958+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:15:46.171 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:15:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:15:45.958+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:15:46.172 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:15:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:15:46.044+0000 7f8a5f86e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:46.046081+0000) 2024-02-20T09:15:46.172 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:15:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:15:46.047+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:46.048716+0000) 2024-02-20T09:15:46.172 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:15:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:15:46.053+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:15:46.172 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:15:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:15:46.053+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:15:46.172 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:15:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:15:46.053+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:15:46.172 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:15:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:15:46.053+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:15:46.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:15:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:15:46.306+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:46.307812+0000) 2024-02-20T09:15:46.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:15:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:15:46.246+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:15:46.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:15:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:15:46.246+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:15:46.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:15:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:15:46.246+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:15:47.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:15:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:15:47.006+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:47.008366+0000) 2024-02-20T09:15:47.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:15:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:15:46.890+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:46.891507+0000) 2024-02-20T09:15:47.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:15:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:15:46.988+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:15:47.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:15:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:15:46.988+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:15:47.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:15:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:15:46.988+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:15:47.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:15:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:15:47.010+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:15:47.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:15:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:15:47.010+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:15:47.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:15:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:15:47.010+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:15:47.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:15:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:15:47.010+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:15:47.536 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:15:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:15:47.308+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:47.309840+0000) 2024-02-20T09:15:47.536 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:15:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:15:47.253+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:15:47.536 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:15:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:15:47.253+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:15:47.536 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:15:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:15:47.253+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:15:48.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:15:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:15:48.007+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:48.008436+0000) 2024-02-20T09:15:48.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:15:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:15:47.890+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:47.891585+0000) 2024-02-20T09:15:48.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:15:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:15:48.019+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:15:48.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:15:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:15:48.019+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:15:48.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:15:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:15:48.019+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:15:48.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:15:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:15:48.009+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:15:48.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:15:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:15:48.009+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:15:48.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:15:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:15:48.009+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:15:48.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:15:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:15:48.010+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:15:48.536 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:15:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:15:48.310+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:48.311922+0000) 2024-02-20T09:15:48.536 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:15:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:15:48.274+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:15:48.537 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:15:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:15:48.274+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:15:48.537 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:15:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:15:48.274+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:15:49.232 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:15:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:15:49.007+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:49.008538+0000) 2024-02-20T09:15:49.232 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:15:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:15:49.229+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:15:49.232 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:15:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:15:49.229+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:15:49.233 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:15:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:15:48.890+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:48.891680+0000) 2024-02-20T09:15:49.233 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:15:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:15:48.975+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:15:49.233 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:15:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:15:48.975+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:15:49.233 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:15:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:15:48.975+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:15:49.234 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:15:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:15:48.975+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:15:49.234 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:15:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:15:48.975+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:15:49.234 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:15:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:15:48.975+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:15:49.234 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:15:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:15:48.975+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:15:49.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:15:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:15:49.312+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:49.314008+0000) 2024-02-20T09:15:49.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:15:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:15:49.229+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:15:50.201 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:15:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:15:49.942+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:15:50.202 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:15:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:15:49.942+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:15:50.202 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:15:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:15:49.942+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:15:50.202 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:15:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:15:49.942+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:15:50.202 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:15:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:15:49.890+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:49.891819+0000) 2024-02-20T09:15:50.202 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:15:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:15:50.015+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:15:50.203 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:15:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:15:50.015+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:15:50.203 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:15:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:15:50.015+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:15:50.203 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:15:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:15:50.007+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:50.008635+0000) 2024-02-20T09:15:50.203 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:15:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:15:50.198+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:15:50.203 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:15:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:15:50.198+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:15:50.203 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:15:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:15:50.198+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:15:50.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:15:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:15:50.314+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:50.315682+0000) 2024-02-20T09:15:51.177 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:15:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:15:51.058+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 64630 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:15:51.178 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:15:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:15:51.007+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:51.008721+0000) 2024-02-20T09:15:51.178 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:15:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:15:51.052+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:51.052633+0000) 2024-02-20T09:15:51.178 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:15:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:15:50.891+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:50.892034+0000) 2024-02-20T09:15:51.178 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:15:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:15:51.020+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:15:51.179 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:15:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:15:51.020+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:15:51.179 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:15:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:15:51.020+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:15:51.179 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:15:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:15:51.052+0000 7f8a5f86e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:51.052957+0000) 2024-02-20T09:15:51.179 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:15:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:15:50.926+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:15:51.179 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:15:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:15:50.926+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:15:51.179 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:15:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:15:50.926+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:15:51.179 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:15:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:15:50.926+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:15:51.180 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:15:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:15:51.056+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:51.056730+0000) 2024-02-20T09:15:51.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:15:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:15:51.315+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:51.315877+0000) 2024-02-20T09:15:51.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:15:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:15:51.244+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:15:51.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:15:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:15:51.244+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:15:51.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:15:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:15:51.244+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:15:52.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:15:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:15:52.008+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:52.008821+0000) 2024-02-20T09:15:52.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:15:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:15:51.891+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:51.892122+0000) 2024-02-20T09:15:52.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:15:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:15:52.043+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:15:52.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:15:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:15:52.043+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:15:52.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:15:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:15:52.043+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:15:52.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:15:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:15:51.916+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:15:52.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:15:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:15:51.916+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:15:52.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:15:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:15:51.916+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:15:52.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:15:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:15:51.916+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:15:52.537 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:15:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:15:52.317+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:52.317950+0000) 2024-02-20T09:15:52.537 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:15:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:15:52.259+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:15:52.537 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:15:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:15:52.259+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:15:52.537 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:15:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:15:52.259+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:15:53.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:15:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:15:53.008+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:53.008898+0000) 2024-02-20T09:15:53.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:15:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:15:52.891+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:52.892270+0000) 2024-02-20T09:15:53.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:15:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:15:53.024+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:15:53.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:15:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:15:53.024+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:15:53.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:15:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:15:53.024+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:15:53.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:15:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:15:52.884+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:15:53.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:15:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:15:52.884+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:15:53.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:15:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:15:52.884+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:15:53.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:15:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:15:52.884+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:15:53.537 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:15:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:15:53.320+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:53.320736+0000) 2024-02-20T09:15:53.538 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:15:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:15:53.244+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:15:53.538 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:15:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:15:53.244+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:15:53.538 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:15:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:15:53.244+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:15:54.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:15:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:15:54.008+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:54.009021+0000) 2024-02-20T09:15:54.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:15:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:15:53.892+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:53.892421+0000) 2024-02-20T09:15:54.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:15:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:15:54.019+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:15:54.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:15:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:15:54.020+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:15:54.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:15:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:15:54.020+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:15:54.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:15:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:15:53.900+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:15:54.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:15:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:15:53.900+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:15:54.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:15:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:15:53.900+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:15:54.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:15:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:15:53.900+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:15:54.537 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:15:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:15:54.322+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:54.322893+0000) 2024-02-20T09:15:54.537 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:15:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:15:54.260+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:15:54.537 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:15:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:15:54.260+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:15:54.538 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:15:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:15:54.260+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:15:55.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:15:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:15:55.008+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:55.009143+0000) 2024-02-20T09:15:55.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:15:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:15:54.892+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:54.892543+0000) 2024-02-20T09:15:55.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:15:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:15:54.975+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:15:55.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:15:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:15:54.975+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:15:55.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:15:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:15:54.975+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:15:55.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:15:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:15:54.924+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:15:55.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:15:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:15:54.924+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:15:55.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:15:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:15:54.924+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:15:55.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:15:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:15:54.924+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:15:55.642 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:15:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:15:55.323+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:55.324213+0000) 2024-02-20T09:15:55.642 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:15:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:15:55.302+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:15:55.642 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:15:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:15:55.302+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:15:55.642 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:15:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:15:55.302+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:15:56.186 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:15:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:15:56.066+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 64669 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:15:56.186 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:15:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:15:56.008+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:56.009235+0000) 2024-02-20T09:15:56.187 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:15:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:15:56.059+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:56.060112+0000) 2024-02-20T09:15:56.187 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:15:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:15:55.892+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:55.892651+0000) 2024-02-20T09:15:56.187 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:15:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:15:55.942+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:15:56.187 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:15:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:15:55.942+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:15:56.187 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:15:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:15:55.942+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:15:56.187 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:15:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:15:56.059+0000 7f8a5f86e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:56.060036+0000) 2024-02-20T09:15:56.188 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:15:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:15:55.953+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:15:56.188 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:15:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:15:55.953+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:15:56.188 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:15:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:15:55.953+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:15:56.188 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:15:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:15:55.953+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:15:56.188 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:15:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:15:56.063+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:56.063906+0000) 2024-02-20T09:15:56.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:15:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:15:56.324+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:56.324445+0000) 2024-02-20T09:15:56.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:15:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:15:56.266+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:15:56.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:15:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:15:56.266+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:15:56.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:15:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:15:56.266+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:15:57.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:15:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:15:57.009+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:57.009394+0000) 2024-02-20T09:15:57.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:15:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:15:56.917+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:15:57.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:15:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:15:56.917+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:15:57.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:15:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:15:56.917+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:15:57.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:15:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:15:56.917+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:15:57.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:15:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:15:56.892+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:56.892772+0000) 2024-02-20T09:15:57.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:15:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:15:56.918+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:15:57.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:15:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:15:56.918+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:15:57.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:15:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:15:56.918+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:15:57.537 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:15:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:15:57.326+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:57.326529+0000) 2024-02-20T09:15:57.537 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:15:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:15:57.273+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:15:57.537 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:15:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:15:57.273+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:15:57.538 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:15:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:15:57.273+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:15:58.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:15:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:15:58.009+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:58.009497+0000) 2024-02-20T09:15:58.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:15:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:15:57.892+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:57.892847+0000) 2024-02-20T09:15:58.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:15:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:15:57.903+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:15:58.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:15:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:15:57.903+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:15:58.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:15:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:15:57.903+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:15:58.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:15:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:15:57.915+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:15:58.245 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:15:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:15:57.915+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:15:58.245 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:15:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:15:57.915+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:15:58.245 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:15:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:15:57.915+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:15:58.537 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:15:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:15:58.329+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:58.330024+0000) 2024-02-20T09:15:58.537 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:15:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:15:58.274+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:15:58.538 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:15:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:15:58.274+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:15:58.538 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:15:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:15:58.274+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:15:59.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:15:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:15:59.009+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:59.009584+0000) 2024-02-20T09:15:59.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:15:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:15:58.892+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:58.892996+0000) 2024-02-20T09:15:59.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:15:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:15:58.904+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:15:59.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:15:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:15:58.904+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:15:59.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:15:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:15:58.904+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:15:59.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:15:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:15:58.884+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:15:59.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:15:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:15:58.884+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:15:59.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:15:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:15:58.884+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:15:59.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:15:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:15:58.884+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:15:59.643 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:15:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:15:59.332+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:59.332589+0000) 2024-02-20T09:15:59.643 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:15:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:15:59.305+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:15:59.643 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:15:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:15:59.305+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:15:59.643 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:15:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:15:59.305+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:15:59.894 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:15:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:15:59.879+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:15:59.894 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:15:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:15:59.879+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:15:59.894 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:15:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:15:59.879+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:15:59.894 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:15:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:15:59.879+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:16:00.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:16:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:16:00.009+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:00.009675+0000) 2024-02-20T09:16:00.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:15:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:15:59.892+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:15:59.893171+0000) 2024-02-20T09:16:00.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:15:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:15:59.953+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:16:00.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:15:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:15:59.953+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:16:00.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:15:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:15:59.953+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:16:00.642 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:16:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:16:00.333+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:00.334034+0000) 2024-02-20T09:16:00.643 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:16:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:16:00.292+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:16:00.643 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:16:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:16:00.292+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:16:00.643 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:16:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:16:00.292+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:16:00.894 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:16:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:16:00.872+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:16:00.894 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:16:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:16:00.872+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:16:00.894 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:16:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:16:00.872+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:16:00.895 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:16:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:16:00.873+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:16:01.193 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:16:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:16:01.074+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 64709 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:16:01.194 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:16:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:16:01.070+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:01.071345+0000) 2024-02-20T09:16:01.194 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:16:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:16:01.009+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:01.009794+0000) 2024-02-20T09:16:01.194 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:16:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:16:01.067+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:01.068257+0000) 2024-02-20T09:16:01.194 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:16:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:16:00.892+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:00.893331+0000) 2024-02-20T09:16:01.195 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:16:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:16:00.975+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:16:01.195 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:16:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:16:00.975+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:16:01.195 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:16:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:16:00.975+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:16:01.195 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:16:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:16:01.068+0000 7f8a5f86e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:01.068538+0000) 2024-02-20T09:16:01.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:16:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:16:01.333+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:01.334257+0000) 2024-02-20T09:16:01.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:16:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:16:01.291+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:16:01.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:16:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:16:01.291+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:16:01.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:16:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:16:01.291+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:16:02.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:16:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:16:02.009+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:02.009962+0000) 2024-02-20T09:16:02.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:16:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:16:01.836+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:16:02.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:16:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:16:01.836+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:16:02.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:16:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:16:01.836+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:16:02.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:16:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:16:01.836+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:16:02.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:16:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:16:01.893+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:01.893481+0000) 2024-02-20T09:16:02.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:16:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:16:01.978+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:16:02.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:16:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:16:01.978+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:16:02.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:16:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:16:01.978+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:16:02.643 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:16:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:16:02.336+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:02.336655+0000) 2024-02-20T09:16:02.643 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:16:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:16:02.307+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:16:02.643 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:16:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:16:02.307+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:16:02.644 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:16:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:16:02.307+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:16:02.894 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:16:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:16:02.838+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:16:02.894 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:16:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:16:02.838+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:16:02.895 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:16:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:16:02.838+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:16:02.895 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:16:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:16:02.838+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:16:03.193 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:16:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:16:03.009+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:03.010045+0000) 2024-02-20T09:16:03.193 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:16:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:16:02.893+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:02.893579+0000) 2024-02-20T09:16:03.193 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:16:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:16:03.022+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:16:03.193 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:16:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:16:03.022+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:16:03.194 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:16:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:16:03.022+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:16:03.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:16:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:16:03.338+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:03.339113+0000) 2024-02-20T09:16:03.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:16:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:16:03.322+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:16:03.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:16:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:16:03.322+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:16:03.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:16:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:16:03.322+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:16:04.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:16:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:16:04.009+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:04.010145+0000) 2024-02-20T09:16:04.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:16:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:16:03.828+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:16:04.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:16:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:16:03.828+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:16:04.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:16:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:16:03.828+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:16:04.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:16:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:16:03.828+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:16:04.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:16:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:16:03.893+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:03.893669+0000) 2024-02-20T09:16:04.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:16:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:16:04.046+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:16:04.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:16:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:16:04.046+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:16:04.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:16:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:16:04.046+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:16:04.538 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:16:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:16:04.340+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:04.341189+0000) 2024-02-20T09:16:04.538 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:16:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:16:04.278+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:16:04.538 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:16:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:16:04.278+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:16:04.538 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:16:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:16:04.278+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:16:05.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:16:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:16:05.009+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:05.010244+0000) 2024-02-20T09:16:05.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:16:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:16:04.833+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:16:05.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:16:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:16:04.833+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:16:05.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:16:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:16:04.833+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:16:05.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:16:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:16:04.833+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:16:05.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:16:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:16:04.893+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:04.893769+0000) 2024-02-20T09:16:05.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:16:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:16:05.006+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:16:05.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:16:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:16:05.006+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:16:05.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:16:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:16:05.006+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:16:05.643 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:16:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:16:05.342+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:05.342431+0000) 2024-02-20T09:16:05.644 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:16:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:16:05.304+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:16:05.644 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:16:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:16:05.304+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:16:05.644 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:16:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:16:05.304+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:16:05.885 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:16:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:16:05.834+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:16:05.885 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:16:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:16:05.834+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:16:05.885 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:16:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:16:05.834+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:16:05.885 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:16:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:16:05.834+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:16:06.200 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:16:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:16:06.081+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 64748 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:16:06.200 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:16:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:16:06.078+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:06.078527+0000) 2024-02-20T09:16:06.201 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:16:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:16:06.009+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:06.010355+0000) 2024-02-20T09:16:06.201 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:16:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:16:06.075+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:06.076106+0000) 2024-02-20T09:16:06.201 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:16:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:16:05.893+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:05.893860+0000) 2024-02-20T09:16:06.201 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:16:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:16:06.012+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:16:06.201 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:16:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:16:06.012+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:16:06.201 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:16:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:16:06.012+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:16:06.202 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:16:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:16:06.075+0000 7f8a5f86e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:06.076281+0000) 2024-02-20T09:16:06.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:16:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:16:06.342+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:06.342607+0000) 2024-02-20T09:16:06.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:16:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:16:06.351+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:16:06.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:16:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:16:06.351+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:16:06.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:16:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:16:06.352+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:16:07.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:16:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:16:07.010+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:07.010419+0000) 2024-02-20T09:16:07.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:16:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:16:06.880+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:16:07.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:16:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:16:06.880+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:16:07.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:16:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:16:06.880+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:16:07.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:16:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:16:06.881+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:16:07.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:16:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:16:06.893+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:06.893960+0000) 2024-02-20T09:16:07.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:16:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:16:07.005+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:16:07.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:16:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:16:07.005+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:16:07.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:16:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:16:07.005+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:16:07.643 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:16:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:16:07.344+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:07.344644+0000) 2024-02-20T09:16:07.643 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:16:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:16:07.378+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:16:07.644 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:16:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:16:07.378+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:16:07.644 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:16:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:16:07.378+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:16:07.895 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:16:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:16:07.855+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:16:07.895 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:16:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:16:07.855+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:16:07.895 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:16:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:16:07.855+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:16:07.895 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:16:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:16:07.855+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:16:08.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:16:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:16:08.010+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:08.010495+0000) 2024-02-20T09:16:08.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:16:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:16:07.893+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:07.894062+0000) 2024-02-20T09:16:08.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:16:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:16:08.004+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:16:08.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:16:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:16:08.004+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:16:08.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:16:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:16:08.004+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:16:08.644 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:16:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:16:08.346+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:08.347002+0000) 2024-02-20T09:16:08.644 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:16:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:16:08.403+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:16:08.644 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:16:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:16:08.403+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:16:08.645 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:16:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:16:08.403+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:16:08.895 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:16:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:16:08.820+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:16:08.895 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:16:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:16:08.820+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:16:08.895 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:16:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:16:08.820+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:16:08.896 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:16:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:16:08.820+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:16:09.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:16:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:16:09.010+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:09.010585+0000) 2024-02-20T09:16:09.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:16:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:16:08.893+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:08.894206+0000) 2024-02-20T09:16:09.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:16:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:16:08.960+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:16:09.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:16:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:16:08.960+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:16:09.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:16:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:16:08.960+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:16:09.643 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:16:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:16:09.348+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:09.349244+0000) 2024-02-20T09:16:09.644 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:16:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:16:09.423+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:16:09.644 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:16:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:16:09.423+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:16:09.644 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:16:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:16:09.423+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:16:09.895 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:16:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:16:09.788+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:16:09.895 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:16:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:16:09.788+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:16:09.895 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:16:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:16:09.788+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:16:09.896 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:16:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:16:09.788+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:16:10.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:16:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:16:10.010+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:10.010679+0000) 2024-02-20T09:16:10.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:16:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:16:09.893+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:09.894321+0000) 2024-02-20T09:16:10.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:16:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:16:09.929+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:16:10.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:16:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:16:09.929+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:16:10.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:16:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:16:09.929+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:16:10.644 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:16:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:16:10.350+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:10.350561+0000) 2024-02-20T09:16:10.644 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:16:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:16:10.383+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:16:10.644 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:16:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:16:10.383+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:16:10.644 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:16:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:16:10.383+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:16:10.895 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:16:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:16:10.832+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:16:10.895 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:16:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:16:10.832+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:16:10.895 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:16:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:16:10.832+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:16:10.896 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:16:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:16:10.832+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:16:11.209 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:16:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:16:11.089+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 64788 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:16:11.209 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:16:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:16:11.085+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:11.085858+0000) 2024-02-20T09:16:11.210 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:16:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:16:11.010+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:11.010776+0000) 2024-02-20T09:16:11.210 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:16:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:16:11.083+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:11.083665+0000) 2024-02-20T09:16:11.210 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:16:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:16:10.894+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:10.894452+0000) 2024-02-20T09:16:11.210 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:16:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:16:10.968+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:16:11.210 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:16:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:16:10.968+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:16:11.211 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:16:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:16:10.968+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:16:11.211 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:16:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:16:11.082+0000 7f8a5f86e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:11.083296+0000) 2024-02-20T09:16:11.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:16:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:16:11.350+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:11.350732+0000) 2024-02-20T09:16:11.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:16:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:16:11.399+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:16:11.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:16:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:16:11.399+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:16:11.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:16:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:16:11.399+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:16:12.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:16:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:16:12.010+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:12.010890+0000) 2024-02-20T09:16:12.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:16:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:16:11.894+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:11.894551+0000) 2024-02-20T09:16:12.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:16:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:16:11.950+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:16:12.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:16:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:16:11.950+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:16:12.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:16:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:16:11.950+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:16:12.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:16:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:16:11.873+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:16:12.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:16:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:16:11.873+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:16:12.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:16:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:16:11.873+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:16:12.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:16:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:16:11.874+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:16:12.644 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:16:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:16:12.352+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:12.354177+0000) 2024-02-20T09:16:12.644 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:16:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:16:12.368+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:16:12.644 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:16:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:16:12.368+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:16:12.644 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:16:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:16:12.368+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:16:12.895 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:16:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:16:12.882+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:16:12.896 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:16:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:16:12.882+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:16:12.896 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:16:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:16:12.882+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:16:12.896 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:16:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:16:12.882+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:16:13.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:16:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:16:13.009+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:13.011003+0000) 2024-02-20T09:16:13.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:16:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:16:12.893+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:12.894639+0000) 2024-02-20T09:16:13.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:16:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:16:12.919+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:16:13.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:16:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:16:12.919+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:16:13.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:16:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:16:12.919+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:16:13.644 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:16:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:16:13.356+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:13.357637+0000) 2024-02-20T09:16:13.644 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:16:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:16:13.333+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:16:13.645 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:16:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:16:13.333+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:16:13.645 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:16:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:16:13.333+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:16:13.896 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:16:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:16:13.860+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:16:13.896 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:16:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:16:13.860+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:16:13.896 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:16:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:16:13.860+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:16:13.896 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:16:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:16:13.860+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:16:14.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:16:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:16:14.009+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:14.011132+0000) 2024-02-20T09:16:14.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:16:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:16:13.893+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:13.894771+0000) 2024-02-20T09:16:14.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:16:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:16:13.960+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:16:14.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:16:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:16:13.960+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:16:14.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:16:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:16:13.960+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:16:14.645 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:16:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:16:14.359+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:14.361124+0000) 2024-02-20T09:16:14.645 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:16:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:16:14.368+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:16:14.645 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:16:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:16:14.368+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:16:14.645 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:16:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:16:14.368+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:16:14.896 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:16:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:16:14.813+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:16:14.896 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:16:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:16:14.813+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:16:14.896 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:16:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:16:14.813+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:16:14.896 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:16:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:16:14.813+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:16:15.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:16:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:16:15.009+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:15.011220+0000) 2024-02-20T09:16:15.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:16:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:16:14.893+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:14.894882+0000) 2024-02-20T09:16:15.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:16:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:16:15.002+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:16:15.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:16:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:16:15.002+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:16:15.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:16:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:16:15.002+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:16:15.644 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:16:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:16:15.361+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:15.362591+0000) 2024-02-20T09:16:15.644 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:16:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:16:15.335+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:16:15.645 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:16:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:16:15.335+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:16:15.645 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:16:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:16:15.335+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:16:15.896 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:16:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:16:15.791+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:16:15.896 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:16:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:16:15.791+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:16:15.896 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:16:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:16:15.791+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:16:15.896 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:16:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:16:15.791+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:16:16.219 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:16:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:16:16.096+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 64827 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:16:16.219 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:16:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:16:16.091+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:16.093138+0000) 2024-02-20T09:16:16.220 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:16:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:16:16.009+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:16.011343+0000) 2024-02-20T09:16:16.220 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:16:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:16:16.089+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:16.091190+0000) 2024-02-20T09:16:16.220 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:16:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:16:15.893+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:15.894998+0000) 2024-02-20T09:16:16.220 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:16:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:16:15.988+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:16:16.220 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:16:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:16:15.988+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:16:16.221 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:16:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:16:15.988+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:16:16.221 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:16:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:16:16.090+0000 7f8a5f86e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:16.091466+0000) 2024-02-20T09:16:16.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:16:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:16:16.361+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:16.362794+0000) 2024-02-20T09:16:16.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:16:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:16:16.382+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:16:16.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:16:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:16:16.382+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:16:16.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:16:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:16:16.382+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:16:16.852 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:16:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:16:16.775+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:16:16.852 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:16:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:16:16.775+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:16:16.853 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:16:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:16:16.775+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:16:16.853 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:16:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:16:16.775+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:16:17.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:16:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:16:17.010+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:17.011464+0000) 2024-02-20T09:16:17.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:16:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:16:16.893+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:16.895100+0000) 2024-02-20T09:16:17.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:16:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:16:16.975+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:16:17.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:16:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:16:16.975+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:16:17.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:16:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:16:16.975+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:16:17.644 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:16:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:16:17.363+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:17.364870+0000) 2024-02-20T09:16:17.645 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:16:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:16:17.358+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:16:17.645 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:16:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:16:17.358+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:16:17.645 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:16:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:16:17.358+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:16:17.896 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:16:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:16:17.731+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:16:17.896 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:16:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:16:17.731+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:16:17.896 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:16:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:16:17.731+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:16:17.896 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:16:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:16:17.731+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:16:18.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:16:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:16:18.010+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:18.011567+0000) 2024-02-20T09:16:18.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:16:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:16:17.893+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:17.895185+0000) 2024-02-20T09:16:18.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:16:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:16:17.966+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:16:18.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:16:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:16:17.966+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:16:18.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:16:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:16:17.966+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:16:18.644 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:16:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:16:18.365+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:18.367086+0000) 2024-02-20T09:16:18.645 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:16:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:16:18.332+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:16:18.645 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:16:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:16:18.332+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:16:18.645 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:16:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:16:18.332+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:16:18.896 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:16:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:16:18.698+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:16:18.896 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:16:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:16:18.698+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:16:18.896 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:16:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:16:18.698+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:16:18.897 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:16:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:16:18.698+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:16:19.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:16:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:16:19.010+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:19.011676+0000) 2024-02-20T09:16:19.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:16:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:16:18.893+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:18.895320+0000) 2024-02-20T09:16:19.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:16:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:16:18.938+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:16:19.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:16:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:16:18.938+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:16:19.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:16:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:16:18.938+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:16:19.644 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:16:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:16:19.367+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:19.369117+0000) 2024-02-20T09:16:19.645 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:16:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:16:19.377+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:16:19.645 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:16:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:16:19.377+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:16:19.645 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:16:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:16:19.377+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:16:19.896 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:16:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:16:19.699+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:16:19.896 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:16:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:16:19.699+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:16:19.897 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:16:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:16:19.699+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:16:19.897 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:16:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:16:19.699+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:16:20.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:16:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:16:20.010+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:20.011796+0000) 2024-02-20T09:16:20.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:16:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:16:19.894+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:19.895439+0000) 2024-02-20T09:16:20.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:16:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:16:19.941+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:16:20.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:16:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:16:19.941+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:16:20.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:16:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:16:19.941+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:16:20.645 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:16:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:16:20.369+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:20.370554+0000) 2024-02-20T09:16:20.645 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:16:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:16:20.381+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:16:20.645 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:16:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:16:20.381+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:16:20.645 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:16:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:16:20.381+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:16:20.896 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:16:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:16:20.727+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:16:20.897 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:16:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:16:20.727+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:16:20.897 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:16:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:16:20.727+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:16:20.897 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:16:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:16:20.727+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:16:21.225 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:16:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:16:21.103+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 64866 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:16:21.225 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:16:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:16:21.100+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:21.101824+0000) 2024-02-20T09:16:21.226 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:16:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:16:21.010+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:21.011907+0000) 2024-02-20T09:16:21.226 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:16:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:16:21.097+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:21.099263+0000) 2024-02-20T09:16:21.226 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:16:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:16:20.894+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:20.895554+0000) 2024-02-20T09:16:21.226 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:16:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:16:20.981+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:16:21.226 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:16:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:16:20.981+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:16:21.227 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:16:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:16:20.981+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:16:21.227 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:16:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:16:21.097+0000 7f8a5f86e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:21.098595+0000) 2024-02-20T09:16:21.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:16:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:16:21.369+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:21.370778+0000) 2024-02-20T09:16:21.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:16:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:16:21.335+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:16:21.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:16:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:16:21.335+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:16:21.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:16:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:16:21.335+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:16:21.856 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:16:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:16:21.766+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:16:21.856 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:16:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:16:21.766+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:16:21.856 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:16:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:16:21.766+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:16:21.856 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:16:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:16:21.766+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:16:22.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:16:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:16:22.010+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:22.012017+0000) 2024-02-20T09:16:22.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:16:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:16:21.894+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:21.895631+0000) 2024-02-20T09:16:22.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:16:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:16:21.976+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:16:22.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:16:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:16:21.976+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:16:22.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:16:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:16:21.976+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:16:22.645 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:16:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:16:22.371+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:22.372785+0000) 2024-02-20T09:16:22.645 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:16:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:16:22.375+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:16:22.645 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:16:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:16:22.375+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:16:22.645 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:16:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:16:22.375+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:16:22.896 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:16:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:16:22.816+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:16:22.897 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:16:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:16:22.816+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:16:22.897 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:16:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:16:22.816+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:16:22.897 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:16:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:16:22.816+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:16:23.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:16:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:16:23.010+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:23.012115+0000) 2024-02-20T09:16:23.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:16:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:16:22.894+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:22.895723+0000) 2024-02-20T09:16:23.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:16:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:16:22.932+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:16:23.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:16:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:16:22.932+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:16:23.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:16:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:16:22.932+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:16:23.645 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:16:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:16:23.373+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:23.374745+0000) 2024-02-20T09:16:23.645 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:16:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:16:23.333+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:16:23.645 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:16:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:16:23.333+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:16:23.646 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:16:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:16:23.333+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:16:23.896 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:16:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:16:23.780+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:16:23.897 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:16:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:16:23.780+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:16:23.897 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:16:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:16:23.780+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:16:23.897 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:16:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:16:23.781+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:16:24.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:16:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:16:24.010+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:24.012245+0000) 2024-02-20T09:16:24.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:16:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:16:23.894+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:23.895813+0000) 2024-02-20T09:16:24.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:16:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:16:23.973+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:16:24.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:16:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:16:23.973+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:16:24.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:16:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:16:23.973+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:16:24.645 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:16:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:16:24.375+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:24.376805+0000) 2024-02-20T09:16:24.645 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:16:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:16:24.354+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:16:24.646 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:16:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:16:24.354+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:16:24.646 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:16:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:16:24.354+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:16:24.897 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:16:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:16:24.757+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:16:24.897 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:16:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:16:24.757+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:16:24.897 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:16:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:16:24.757+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:16:24.897 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:16:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:16:24.757+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:16:25.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:16:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:16:25.011+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:25.012387+0000) 2024-02-20T09:16:25.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:16:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:16:24.894+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:24.895922+0000) 2024-02-20T09:16:25.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:16:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:16:24.930+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:16:25.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:16:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:16:24.930+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:16:25.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:16:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:16:24.930+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:16:25.645 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:16:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:16:25.376+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:25.378342+0000) 2024-02-20T09:16:25.645 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:16:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:16:25.312+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:16:25.646 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:16:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:16:25.312+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:16:25.646 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:16:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:16:25.312+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:16:25.897 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:16:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:16:25.747+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:16:25.898 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:16:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:16:25.747+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:16:25.898 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:16:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:16:25.747+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:16:25.898 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:16:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:16:25.747+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:16:26.230 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:16:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:16:26.111+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 64905 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:16:26.231 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:16:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:16:26.107+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:26.108660+0000) 2024-02-20T09:16:26.231 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:16:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:16:26.011+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:26.012489+0000) 2024-02-20T09:16:26.231 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:16:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:16:26.104+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:26.105753+0000) 2024-02-20T09:16:26.232 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:16:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:16:25.894+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:25.896053+0000) 2024-02-20T09:16:26.232 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:16:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:16:25.905+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:16:26.232 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:16:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:16:25.905+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:16:26.232 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:16:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:16:25.905+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:16:26.232 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:16:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:16:26.104+0000 7f8a5f86e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:26.105826+0000) 2024-02-20T09:16:26.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:16:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:16:26.377+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:26.378544+0000) 2024-02-20T09:16:26.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:16:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:16:26.292+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:16:26.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:16:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:16:26.292+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:16:26.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:16:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:16:26.292+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:16:27.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:16:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:16:27.011+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:27.012581+0000) 2024-02-20T09:16:27.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:16:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:16:26.866+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:16:27.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:16:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:16:26.866+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:16:27.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:16:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:16:26.866+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:16:27.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:16:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:16:26.894+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:26.896181+0000) 2024-02-20T09:16:27.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:16:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:16:26.789+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:16:27.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:16:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:16:26.789+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:16:27.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:16:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:16:26.789+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:16:27.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:16:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:16:26.789+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:16:27.541 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:16:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:16:27.379+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:27.380517+0000) 2024-02-20T09:16:27.541 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:16:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:16:27.274+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:16:27.541 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:16:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:16:27.274+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:16:27.541 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:16:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:16:27.274+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:16:28.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:16:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:16:28.011+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:28.012660+0000) 2024-02-20T09:16:28.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:16:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:16:27.882+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:16:28.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:16:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:16:27.882+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:16:28.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:16:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:16:27.882+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:16:28.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:16:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:16:27.894+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:27.896289+0000) 2024-02-20T09:16:28.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:16:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:16:27.827+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:16:28.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:16:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:16:27.827+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:16:28.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:16:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:16:27.827+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:16:28.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:16:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:16:27.827+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:16:28.541 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:16:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:16:28.381+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:28.382675+0000) 2024-02-20T09:16:28.541 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:16:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:16:28.278+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:16:28.541 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:16:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:16:28.278+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:16:28.541 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:16:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:16:28.278+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:16:28.863 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:16:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:16:28.784+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:16:28.863 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:16:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:16:28.784+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:16:28.863 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:16:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:16:28.784+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:16:28.863 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:16:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:16:28.784+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:16:29.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:16:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:16:29.011+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:29.012776+0000) 2024-02-20T09:16:29.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:16:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:16:28.874+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:16:29.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:16:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:16:28.874+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:16:29.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:16:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:16:28.874+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:16:29.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:16:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:16:28.894+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:28.896378+0000) 2024-02-20T09:16:29.646 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:16:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:16:29.383+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:29.384855+0000) 2024-02-20T09:16:29.646 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:16:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:16:29.322+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:16:29.646 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:16:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:16:29.322+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:16:29.646 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:16:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:16:29.322+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:16:29.897 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:16:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:16:29.830+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:16:29.897 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:16:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:16:29.830+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:16:29.898 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:16:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:16:29.830+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:16:29.898 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:16:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:16:29.753+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:16:29.898 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:16:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:16:29.753+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:16:29.898 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:16:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:16:29.753+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:16:29.898 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:16:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:16:29.753+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:16:30.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:16:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:16:29.895+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:29.896501+0000) 2024-02-20T09:16:30.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:16:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:16:30.011+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:30.012904+0000) 2024-02-20T09:16:30.646 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:16:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:16:30.384+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:30.386302+0000) 2024-02-20T09:16:30.646 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:16:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:16:30.333+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:16:30.646 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:16:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:16:30.333+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:16:30.646 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:16:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:16:30.333+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:16:30.897 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:16:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:16:30.814+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:16:30.898 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:16:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:16:30.814+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:16:30.898 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:16:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:16:30.814+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:16:30.898 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:16:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:16:30.761+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:16:30.898 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:16:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:16:30.761+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:16:30.898 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:16:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:16:30.761+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:16:30.898 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:16:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:16:30.761+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:16:31.239 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:16:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:16:31.114+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:31.116096+0000) 2024-02-20T09:16:31.239 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:16:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:16:31.118+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 64945 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:16:31.239 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:16:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:16:30.895+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:30.896596+0000) 2024-02-20T09:16:31.239 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:16:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:16:31.112+0000 7f8a5f86e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:31.113709+0000) 2024-02-20T09:16:31.240 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:16:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:16:31.011+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:31.013032+0000) 2024-02-20T09:16:31.240 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:16:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:16:31.112+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:31.113890+0000) 2024-02-20T09:16:31.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:16:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:16:31.385+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:31.386516+0000) 2024-02-20T09:16:31.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:16:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:16:31.378+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:16:31.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:16:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:16:31.378+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:16:31.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:16:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:16:31.378+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:16:32.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:16:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:16:32.011+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:32.013164+0000) 2024-02-20T09:16:32.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:16:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:16:31.833+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:16:32.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:16:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:16:31.833+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:16:32.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:16:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:16:31.833+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:16:32.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:16:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:16:31.895+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:31.896732+0000) 2024-02-20T09:16:32.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:16:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:16:31.811+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:16:32.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:16:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:16:31.811+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:16:32.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:16:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:16:31.811+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:16:32.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:16:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:16:31.811+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:16:32.646 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:16:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:16:32.387+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:32.388453+0000) 2024-02-20T09:16:32.646 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:16:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:16:32.350+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:16:32.646 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:16:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:16:32.350+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:16:32.646 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:16:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:16:32.350+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:16:32.898 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:16:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:16:32.775+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:16:32.898 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:16:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:16:32.775+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:16:32.898 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:16:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:16:32.775+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:16:32.898 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:16:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:16:32.775+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:16:32.898 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:16:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:16:32.863+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:16:32.899 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:16:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:16:32.863+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:16:32.899 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:16:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:16:32.863+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:16:33.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:16:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:16:33.011+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:33.013263+0000) 2024-02-20T09:16:33.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:16:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:16:32.895+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:32.896832+0000) 2024-02-20T09:16:33.646 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:16:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:16:33.389+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:33.390420+0000) 2024-02-20T09:16:33.647 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:16:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:16:33.304+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:16:33.647 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:16:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:16:33.304+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:16:33.647 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:16:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:16:33.304+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:16:33.898 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:16:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:16:33.885+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:16:33.898 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:16:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:16:33.885+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:16:33.898 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:16:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:16:33.885+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:16:33.898 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:16:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:16:33.761+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:16:33.898 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:16:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:16:33.761+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:16:33.899 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:16:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:16:33.761+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:16:33.899 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:16:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:16:33.761+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:16:34.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:16:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:16:33.895+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:33.896960+0000) 2024-02-20T09:16:34.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:16:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:16:34.012+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:34.013421+0000) 2024-02-20T09:16:34.646 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:16:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:16:34.391+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:34.392391+0000) 2024-02-20T09:16:34.647 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:16:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:16:34.305+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:16:34.647 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:16:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:16:34.305+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:16:34.647 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:16:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:16:34.305+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:16:34.898 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:16:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:16:34.888+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:16:34.898 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:16:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:16:34.888+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:16:34.898 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:16:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:16:34.888+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:16:34.898 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:16:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:16:34.786+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:16:34.899 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:16:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:16:34.786+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:16:34.899 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:16:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:16:34.786+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:16:34.899 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:16:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:16:34.786+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:16:35.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:16:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:16:35.012+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:35.013537+0000) 2024-02-20T09:16:35.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:16:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:16:34.895+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:34.897094+0000) 2024-02-20T09:16:35.542 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:16:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:16:35.392+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:35.393731+0000) 2024-02-20T09:16:35.542 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:16:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:16:35.262+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:16:35.542 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:16:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:16:35.262+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:16:35.542 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:16:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:16:35.262+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:16:36.121 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:16:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:16:36.012+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:36.013650+0000) 2024-02-20T09:16:36.122 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:16:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:16:35.834+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:16:36.122 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:16:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:16:35.834+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:16:36.122 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:16:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:16:35.834+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:16:36.122 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:16:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:16:35.834+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:16:36.123 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:16:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:16:35.895+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:35.897185+0000) 2024-02-20T09:16:36.123 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:16:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:16:35.924+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:16:36.123 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:16:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:16:35.924+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:16:36.123 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:16:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:16:35.924+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:16:36.395 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:16:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:16:36.125+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 64984 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:16:36.395 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:16:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:16:36.119+0000 7f8a5f86e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:36.120618+0000) 2024-02-20T09:16:36.396 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:16:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:16:36.121+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:36.123149+0000) 2024-02-20T09:16:36.396 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:16:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:16:36.120+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:36.121707+0000) 2024-02-20T09:16:36.396 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:16:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:16:36.301+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:16:36.396 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:16:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:16:36.301+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:16:36.396 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:16:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:16:36.301+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:16:36.646 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:16:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:16:36.392+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:36.394005+0000) 2024-02-20T09:16:36.898 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:16:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:16:36.823+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:16:36.898 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:16:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:16:36.823+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:16:36.898 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:16:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:16:36.823+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:16:36.898 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:16:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:16:36.824+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:16:37.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:16:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:16:37.012+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:37.013758+0000) 2024-02-20T09:16:37.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:16:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:16:36.895+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:36.897270+0000) 2024-02-20T09:16:37.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:16:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:16:36.972+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:16:37.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:16:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:16:36.972+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:16:37.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:16:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:16:36.972+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:16:37.542 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:16:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:16:37.395+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:37.396747+0000) 2024-02-20T09:16:37.542 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:16:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:16:37.277+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:16:37.542 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:16:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:16:37.277+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:16:37.543 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:16:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:16:37.277+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:16:37.869 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:16:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:16:37.775+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:16:37.869 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:16:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:16:37.775+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:16:37.870 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:16:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:16:37.775+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:16:37.870 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:16:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:16:37.775+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:16:38.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:16:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:16:38.012+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:38.013843+0000) 2024-02-20T09:16:38.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:16:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:16:37.895+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:37.897373+0000) 2024-02-20T09:16:38.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:16:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:16:38.020+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:16:38.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:16:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:16:38.020+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:16:38.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:16:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:16:38.020+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:16:38.542 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:16:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:16:38.397+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:38.398914+0000) 2024-02-20T09:16:38.542 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:16:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:16:38.248+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:16:38.543 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:16:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:16:38.248+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:16:38.543 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:16:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:16:38.248+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:16:39.240 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:16:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:16:39.012+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:39.014007+0000) 2024-02-20T09:16:39.240 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:16:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:16:39.237+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:16:39.240 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:16:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:16:39.237+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:16:39.240 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:16:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:16:39.237+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:16:39.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:16:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:16:38.896+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:38.897492+0000) 2024-02-20T09:16:39.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:16:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:16:38.980+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:16:39.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:16:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:16:38.980+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:16:39.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:16:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:16:38.980+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:16:39.241 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:16:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:16:38.792+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:16:39.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:16:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:16:38.792+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:16:39.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:16:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:16:38.792+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:16:39.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:16:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:16:38.792+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:16:39.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:16:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:16:39.399+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:39.400993+0000) 2024-02-20T09:16:40.200 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:16:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:16:40.012+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:40.014180+0000) 2024-02-20T09:16:40.201 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:16:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:16:39.814+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:16:40.201 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:16:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:16:39.814+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:16:40.201 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:16:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:16:39.814+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:16:40.201 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:16:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:16:39.814+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:16:40.201 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:16:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:16:39.896+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:39.897585+0000) 2024-02-20T09:16:40.201 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:16:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:16:39.987+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:16:40.201 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:16:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:16:39.987+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:16:40.202 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:16:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:16:39.987+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:16:40.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:16:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:16:40.400+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:40.402394+0000) 2024-02-20T09:16:40.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:16:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:16:40.197+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:16:40.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:16:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:16:40.197+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:16:40.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:16:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:16:40.197+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:16:40.871 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:16:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:16:40.785+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:16:40.871 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:16:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:16:40.785+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:16:40.871 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:16:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:16:40.785+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:16:40.871 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:16:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:16:40.785+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:16:41.129 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:16:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:16:41.012+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:41.014317+0000) 2024-02-20T09:16:41.129 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:16:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:16:40.896+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:40.897679+0000) 2024-02-20T09:16:41.129 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:16:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:16:41.024+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:16:41.130 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:16:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:16:41.024+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:16:41.130 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:16:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:16:41.024+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:16:41.403 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:16:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:16:41.133+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 65024 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:16:41.404 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:16:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:16:41.126+0000 7f8a5f86e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:41.128261+0000) 2024-02-20T09:16:41.404 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:16:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:16:41.131+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:41.132572+0000) 2024-02-20T09:16:41.404 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:16:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:16:41.127+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:41.128431+0000) 2024-02-20T09:16:41.405 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:16:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:16:41.212+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:16:41.405 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:16:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:16:41.212+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:16:41.405 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:16:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:16:41.212+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:16:41.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:16:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:16:41.401+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:41.402604+0000) 2024-02-20T09:16:42.015 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:16:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:16:41.896+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:41.897769+0000) 2024-02-20T09:16:42.015 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:16:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:16:41.745+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:16:42.016 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:16:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:16:41.745+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:16:42.016 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:16:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:16:41.745+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:16:42.016 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:16:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:16:41.745+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:16:42.406 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:16:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:16:42.069+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:16:42.406 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:16:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:16:42.069+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:16:42.406 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:16:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:16:42.069+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:16:42.406 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:16:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:16:42.013+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:42.014428+0000) 2024-02-20T09:16:42.406 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:16:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:16:42.172+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:16:42.407 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:16:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:16:42.172+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:16:42.407 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:16:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:16:42.172+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:16:42.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:16:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:16:42.404+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:42.404779+0000) 2024-02-20T09:16:43.015 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:16:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:16:42.896+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:42.897877+0000) 2024-02-20T09:16:43.015 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:16:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:16:42.742+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:16:43.016 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:16:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:16:42.742+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:16:43.016 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:16:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:16:42.742+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:16:43.016 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:16:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:16:42.742+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:16:43.408 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:16:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:16:43.096+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:16:43.408 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:16:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:16:43.096+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:16:43.408 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:16:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:16:43.096+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:16:43.409 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:16:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:16:43.014+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:43.014520+0000) 2024-02-20T09:16:43.409 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:16:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:16:43.192+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:16:43.409 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:16:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:16:43.192+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:16:43.409 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:16:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:16:43.192+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:16:43.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:16:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:16:43.406+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:43.407145+0000) 2024-02-20T09:16:44.015 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:16:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:16:43.897+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:43.898019+0000) 2024-02-20T09:16:44.016 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:16:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:16:43.756+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:16:44.016 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:16:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:16:43.756+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:16:44.016 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:16:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:16:43.756+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:16:44.016 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:16:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:16:43.756+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:16:44.410 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:16:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:16:44.055+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:16:44.411 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:16:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:16:44.055+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:16:44.411 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:16:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:16:44.055+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:16:44.411 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:16:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:16:44.014+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:44.014679+0000) 2024-02-20T09:16:44.411 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:16:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:16:44.158+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:16:44.411 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:16:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:16:44.158+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:16:44.412 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:16:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:16:44.158+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:16:44.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:16:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:16:44.409+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:44.409622+0000) 2024-02-20T09:16:45.172 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:16:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:16:45.014+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:45.014792+0000) 2024-02-20T09:16:45.172 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:16:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:16:44.897+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:44.898208+0000) 2024-02-20T09:16:45.172 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:16:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:16:45.033+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:16:45.173 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:16:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:16:45.033+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:16:45.173 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:16:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:16:45.033+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:16:45.173 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:16:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:16:44.804+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:16:45.173 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:16:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:16:44.805+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:16:45.173 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:16:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:16:44.805+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:16:45.173 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:16:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:16:44.805+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:16:45.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:16:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:16:45.411+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:45.411670+0000) 2024-02-20T09:16:45.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:16:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:16:45.170+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:16:45.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:16:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:16:45.170+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:16:45.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:16:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:16:45.170+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:16:46.137 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:16:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:16:46.014+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:46.014906+0000) 2024-02-20T09:16:46.137 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:16:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:16:45.805+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:16:46.137 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:16:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:16:45.805+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:16:46.137 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:16:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:16:45.805+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:16:46.138 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:16:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:16:45.805+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:16:46.138 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:16:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:16:45.897+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:45.898378+0000) 2024-02-20T09:16:46.138 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:16:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:16:46.015+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:16:46.138 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:16:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:16:46.015+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:16:46.138 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:16:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:16:46.015+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:16:46.138 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:16:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:16:46.135+0000 7f8a5f86e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:46.136066+0000) 2024-02-20T09:16:46.413 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:16:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:16:46.141+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 65063 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:16:46.413 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:16:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:16:46.138+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:46.139053+0000) 2024-02-20T09:16:46.414 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:16:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:16:46.136+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:46.136977+0000) 2024-02-20T09:16:46.414 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:16:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:16:46.175+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:16:46.414 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:16:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:16:46.175+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:16:46.414 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:16:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:16:46.175+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:16:46.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:16:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:16:46.411+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:46.411893+0000) 2024-02-20T09:16:47.166 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:16:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:16:46.898+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:46.898513+0000) 2024-02-20T09:16:47.166 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:16:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:16:47.052+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:16:47.166 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:16:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:16:47.052+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:16:47.167 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:16:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:16:47.052+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:16:47.167 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:16:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:16:46.824+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:16:47.167 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:16:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:16:46.824+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:16:47.167 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:16:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:16:46.824+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:16:47.167 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:16:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:16:46.824+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:16:47.167 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:16:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:16:47.014+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:47.014996+0000) 2024-02-20T09:16:47.168 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:16:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:16:47.164+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:16:47.168 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:16:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:16:47.164+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:16:47.168 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:16:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:16:47.164+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:16:47.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:16:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:16:47.414+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:47.414441+0000) 2024-02-20T09:16:48.076 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:16:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:16:48.014+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:48.015091+0000) 2024-02-20T09:16:48.076 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:16:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:16:47.898+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:47.898592+0000) 2024-02-20T09:16:48.077 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:16:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:16:48.074+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:16:48.077 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:16:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:16:48.074+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:16:48.077 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:16:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:16:48.074+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:16:48.077 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:16:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:16:47.812+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:16:48.077 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:16:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:16:47.812+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:16:48.077 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:16:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:16:47.812+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:16:48.077 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:16:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:16:47.812+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:16:48.418 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:16:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:16:48.136+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:16:48.418 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:16:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:16:48.136+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:16:48.418 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:16:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:16:48.136+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:16:48.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:16:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:16:48.416+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:48.416898+0000) 2024-02-20T09:16:49.068 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:16:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:16:49.014+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:49.015208+0000) 2024-02-20T09:16:49.068 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:16:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:16:48.767+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:16:49.069 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:16:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:16:48.767+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:16:49.069 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:16:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:16:48.767+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:16:49.069 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:16:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:16:48.767+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:16:49.069 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:16:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:16:48.898+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:48.898684+0000) 2024-02-20T09:16:49.069 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:16:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:16:49.066+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:16:49.069 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:16:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:16:49.066+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:16:49.069 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:16:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:16:49.066+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:16:49.421 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:16:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:16:49.111+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:16:49.421 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:16:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:16:49.111+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:16:49.421 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:16:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:16:49.111+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:16:49.729 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:16:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:16:49.418+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:49.419117+0000) 2024-02-20T09:16:49.729 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:16:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:16:49.727+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:16:49.730 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:16:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:16:49.727+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:16:49.730 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:16:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:16:49.727+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:16:49.730 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:16:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:16:49.728+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:16:49.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:16:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:16:49.898+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:49.898781+0000) 2024-02-20T09:16:50.422 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:16:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:16:50.079+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:16:50.423 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:16:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:16:50.079+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:16:50.423 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:16:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:16:50.079+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:16:50.423 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:16:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:16:50.014+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:50.015327+0000) 2024-02-20T09:16:50.423 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:16:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:16:50.097+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:16:50.423 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:16:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:16:50.097+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:16:50.423 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:16:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:16:50.097+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:16:50.711 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:16:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:16:50.421+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:50.421431+0000) 2024-02-20T09:16:50.711 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:16:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:16:50.709+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:16:50.711 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:16:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:16:50.709+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:16:50.711 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:16:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:16:50.709+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:16:50.712 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:16:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:16:50.709+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:16:50.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:16:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:16:50.898+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:50.898887+0000) 2024-02-20T09:16:51.267 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:16:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:16:51.148+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 65103 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:16:51.268 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:16:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:16:51.146+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:51.146407+0000) 2024-02-20T09:16:51.268 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:16:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:16:51.015+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:51.015460+0000) 2024-02-20T09:16:51.268 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:16:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:16:51.137+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:16:51.269 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:16:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:16:51.137+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:16:51.269 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:16:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:16:51.137+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:16:51.269 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:16:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:16:51.143+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:51.143775+0000) 2024-02-20T09:16:51.269 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:16:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:16:51.039+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:16:51.269 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:16:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:16:51.039+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:16:51.269 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:16:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:16:51.039+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:16:51.269 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:16:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:16:51.142+0000 7f8a5f86e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:51.143073+0000) 2024-02-20T09:16:51.544 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:16:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:16:51.421+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:51.421635+0000) 2024-02-20T09:16:51.880 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:16:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:16:51.672+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:16:51.880 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:16:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:16:51.672+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:16:51.880 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:16:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:16:51.672+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:16:51.880 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:16:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:16:51.672+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:16:52.138 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:16:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:16:52.015+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:52.015554+0000) 2024-02-20T09:16:52.138 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:16:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:16:51.898+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:51.898982+0000) 2024-02-20T09:16:52.138 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:16:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:16:51.999+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:16:52.139 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:16:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:16:51.999+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:16:52.139 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:16:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:16:51.999+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:16:52.424 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:16:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:16:52.136+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:16:52.424 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:16:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:16:52.136+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:16:52.425 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:16:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:16:52.136+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:16:52.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:16:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:16:52.423+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:52.423558+0000) 2024-02-20T09:16:52.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:16:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:16:52.672+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:16:52.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:16:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:16:52.672+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:16:52.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:16:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:16:52.672+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:16:52.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:16:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:16:52.672+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:16:53.136 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:16:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:16:52.898+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:52.899076+0000) 2024-02-20T09:16:53.136 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:16:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:16:53.049+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:16:53.136 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:16:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:16:53.049+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:16:53.136 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:16:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:16:53.049+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:16:53.137 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:16:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:16:53.015+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:53.015629+0000) 2024-02-20T09:16:53.137 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:16:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:16:53.134+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:16:53.137 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:16:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:16:53.134+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:16:53.137 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:16:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:16:53.134+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:16:53.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:16:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:16:53.425+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:53.425607+0000) 2024-02-20T09:16:53.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:16:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:16:53.624+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:16:53.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:16:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:16:53.624+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:16:53.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:16:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:16:53.624+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:16:53.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:16:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:16:53.624+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:16:54.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:16:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:16:53.898+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:53.899189+0000) 2024-02-20T09:16:54.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:16:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:16:54.085+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:16:54.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:16:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:16:54.085+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:16:54.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:16:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:16:54.085+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:16:54.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:16:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:16:54.015+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:54.015752+0000) 2024-02-20T09:16:54.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:16:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:16:54.098+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:16:54.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:16:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:16:54.098+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:16:54.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:16:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:16:54.098+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:16:54.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:16:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:16:54.427+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:54.427608+0000) 2024-02-20T09:16:54.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:16:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:16:54.580+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:16:54.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:16:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:16:54.580+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:16:54.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:16:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:16:54.580+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:16:54.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:16:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:16:54.580+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:16:55.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:16:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:16:54.898+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:54.899312+0000) 2024-02-20T09:16:55.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:16:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:16:55.058+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:16:55.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:16:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:16:55.058+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:16:55.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:16:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:16:55.058+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:16:55.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:16:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:16:55.015+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:55.015882+0000) 2024-02-20T09:16:55.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:16:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:16:55.064+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:16:55.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:16:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:16:55.064+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:16:55.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:16:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:16:55.064+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:16:55.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:16:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:16:55.428+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:55.428919+0000) 2024-02-20T09:16:55.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:16:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:16:55.614+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:16:55.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:16:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:16:55.614+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:16:55.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:16:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:16:55.614+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:16:55.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:16:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:16:55.614+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:16:56.151 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:16:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:16:55.899+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:55.899432+0000) 2024-02-20T09:16:56.151 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:16:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:16:56.064+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:16:56.152 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:16:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:16:56.064+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:16:56.152 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:16:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:16:56.064+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:16:56.152 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:16:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:16:56.015+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:56.016018+0000) 2024-02-20T09:16:56.152 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:16:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:16:56.107+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:16:56.152 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:16:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:16:56.107+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:16:56.152 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:16:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:16:56.107+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:16:56.430 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:16:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:16:56.150+0000 7f8a5f86e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:56.151093+0000) 2024-02-20T09:16:56.430 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:16:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:16:56.153+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:56.153993+0000) 2024-02-20T09:16:56.431 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:16:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:16:56.150+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:56.150867+0000) 2024-02-20T09:16:56.431 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:16:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:16:56.156+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 65142 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:16:56.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:16:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:16:56.428+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:56.429188+0000) 2024-02-20T09:16:56.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:16:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:16:56.641+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:16:56.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:16:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:16:56.641+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:16:56.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:16:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:16:56.641+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:16:56.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:16:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:16:56.641+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:16:57.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:16:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:16:56.899+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:56.899505+0000) 2024-02-20T09:16:57.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:16:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:16:57.095+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:16:57.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:16:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:16:57.095+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:16:57.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:16:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:16:57.095+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:16:57.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:16:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:16:57.015+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:57.016145+0000) 2024-02-20T09:16:57.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:16:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:16:57.116+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:16:57.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:16:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:16:57.116+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:16:57.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:16:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:16:57.116+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:16:57.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:16:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:16:57.431+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:57.431482+0000) 2024-02-20T09:16:57.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:16:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:16:57.649+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:16:57.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:16:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:16:57.649+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:16:57.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:16:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:16:57.649+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:16:57.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:16:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:16:57.649+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:16:58.162 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:16:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:16:58.015+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:58.016219+0000) 2024-02-20T09:16:58.163 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:16:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:16:57.899+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:57.899614+0000) 2024-02-20T09:16:58.163 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:16:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:16:58.072+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:16:58.163 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:16:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:16:58.072+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:16:58.163 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:16:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:16:58.072+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:16:58.434 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:16:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:16:58.161+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:16:58.435 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:16:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:16:58.161+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:16:58.435 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:16:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:16:58.161+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:16:58.695 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:16:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:16:58.433+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:58.433709+0000) 2024-02-20T09:16:58.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:16:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:16:58.899+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:58.899759+0000) 2024-02-20T09:16:58.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:16:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:16:58.693+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:16:58.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:16:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:16:58.693+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:16:58.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:16:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:16:58.693+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:16:58.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:16:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:16:58.694+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:16:59.437 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:16:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:16:59.074+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:16:59.437 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:16:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:16:59.074+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:16:59.437 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:16:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:16:59.074+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:16:59.438 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:16:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:16:59.015+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:59.016335+0000) 2024-02-20T09:16:59.438 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:16:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:16:59.140+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:16:59.438 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:16:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:16:59.140+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:16:59.438 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:16:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:16:59.140+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:16:59.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:16:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:16:59.435+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:59.435964+0000) 2024-02-20T09:16:59.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:16:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:16:59.660+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:16:59.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:16:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:16:59.660+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:16:59.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:16:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:16:59.660+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:16:59.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:16:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:16:59.660+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:17:00.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:17:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:17:00.016+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:00.016419+0000) 2024-02-20T09:17:00.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:17:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:17:00.104+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:17:00.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:17:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:17:00.104+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:17:00.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:17:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:17:00.104+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:17:00.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:16:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:16:59.899+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:16:59.899904+0000) 2024-02-20T09:17:00.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:17:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:17:00.078+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:17:00.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:17:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:17:00.078+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:17:00.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:17:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:17:00.078+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:17:00.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:17:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:17:00.437+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:00.437563+0000) 2024-02-20T09:17:00.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:17:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:17:00.644+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:17:00.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:17:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:17:00.644+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:17:00.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:17:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:17:00.644+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:17:00.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:17:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:17:00.644+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:17:01.159 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:17:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:17:00.899+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:00.900008+0000) 2024-02-20T09:17:01.159 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:17:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:17:01.128+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:17:01.159 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:17:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:17:01.128+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:17:01.159 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:17:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:17:01.128+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:17:01.160 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:17:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:17:01.016+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:01.016530+0000) 2024-02-20T09:17:01.160 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:17:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:17:01.090+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:17:01.160 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:17:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:17:01.090+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:17:01.160 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:17:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:17:01.090+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:17:01.160 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:17:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:17:01.157+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:01.158072+0000) 2024-02-20T09:17:01.438 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:17:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:17:01.165+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 65182 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:17:01.439 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:17:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:17:01.160+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:01.161139+0000) 2024-02-20T09:17:01.439 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:17:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:17:01.158+0000 7f8a5f86e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:01.159105+0000) 2024-02-20T09:17:01.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:17:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:17:01.437+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:01.437826+0000) 2024-02-20T09:17:01.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:17:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:17:01.617+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:17:01.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:17:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:17:01.617+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:17:01.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:17:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:17:01.617+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:17:01.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:17:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:17:01.617+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:17:02.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:17:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:17:01.899+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:01.900115+0000) 2024-02-20T09:17:02.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:17:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:17:02.097+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:17:02.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:17:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:17:02.098+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:17:02.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:17:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:17:02.098+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:17:02.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:17:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:17:02.016+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:02.016626+0000) 2024-02-20T09:17:02.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:17:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:17:02.117+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:17:02.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:17:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:17:02.117+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:17:02.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:17:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:17:02.117+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:17:02.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:17:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:17:02.439+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:02.439893+0000) 2024-02-20T09:17:02.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:17:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:17:02.642+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:17:02.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:17:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:17:02.642+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:17:02.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:17:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:17:02.642+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:17:02.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:17:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:17:02.642+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:17:03.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:17:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:17:02.899+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:02.900238+0000) 2024-02-20T09:17:03.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:17:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:17:03.069+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:17:03.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:17:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:17:03.069+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:17:03.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:17:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:17:03.069+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:17:03.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:17:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:17:03.016+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:03.016704+0000) 2024-02-20T09:17:03.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:17:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:17:03.091+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:17:03.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:17:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:17:03.091+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:17:03.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:17:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:17:03.091+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:17:03.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:17:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:17:03.441+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:03.441900+0000) 2024-02-20T09:17:03.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:17:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:17:03.680+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:17:03.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:17:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:17:03.680+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:17:03.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:17:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:17:03.680+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:17:03.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:17:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:17:03.680+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:17:04.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:17:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:17:03.899+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:03.900330+0000) 2024-02-20T09:17:04.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:17:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:17:04.023+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:17:04.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:17:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:17:04.023+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:17:04.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:17:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:17:04.023+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:17:04.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:17:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:17:04.016+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:04.016794+0000) 2024-02-20T09:17:04.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:17:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:17:04.106+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:17:04.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:17:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:17:04.106+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:17:04.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:17:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:17:04.106+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:17:04.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:17:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:17:04.444+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:04.445141+0000) 2024-02-20T09:17:04.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:17:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:17:04.680+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:17:04.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:17:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:17:04.680+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:17:04.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:17:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:17:04.680+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:17:04.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:17:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:17:04.680+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:17:05.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:17:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:17:05.016+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:05.016887+0000) 2024-02-20T09:17:05.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:17:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:17:05.102+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:17:05.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:17:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:17:05.102+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:17:05.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:17:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:17:05.102+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:17:05.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:17:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:17:04.900+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:04.900443+0000) 2024-02-20T09:17:05.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:17:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:17:05.019+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:17:05.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:17:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:17:05.019+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:17:05.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:17:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:17:05.019+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:17:05.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:17:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:17:05.446+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:05.446560+0000) 2024-02-20T09:17:05.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:17:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:17:05.637+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:17:05.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:17:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:17:05.637+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:17:05.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:17:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:17:05.637+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:17:05.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:17:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:17:05.637+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:17:06.167 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:17:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:17:06.015+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:06.016975+0000) 2024-02-20T09:17:06.168 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:17:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:17:06.114+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:17:06.168 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:17:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:17:06.114+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:17:06.168 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:17:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:17:06.114+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:17:06.168 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:17:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:17:05.899+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:05.900534+0000) 2024-02-20T09:17:06.168 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:17:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:17:06.034+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:17:06.168 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:17:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:17:06.034+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:17:06.169 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:17:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:17:06.034+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:17:06.448 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:17:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:17:06.172+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 65221 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:17:06.448 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:17:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:17:06.168+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:06.169663+0000) 2024-02-20T09:17:06.448 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:17:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:17:06.165+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:06.166834+0000) 2024-02-20T09:17:06.449 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:17:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:17:06.165+0000 7f8a5f86e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:06.166506+0000) 2024-02-20T09:17:06.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:17:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:17:06.445+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:06.446740+0000) 2024-02-20T09:17:06.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:17:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:17:06.686+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:17:06.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:17:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:17:06.686+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:17:06.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:17:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:17:06.686+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:17:06.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:17:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:17:06.686+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:17:07.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:17:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:17:06.899+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:06.900625+0000) 2024-02-20T09:17:07.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:17:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:17:07.017+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:17:07.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:17:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:17:07.017+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:17:07.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:17:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:17:07.017+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:17:07.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:17:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:17:07.015+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:07.017064+0000) 2024-02-20T09:17:07.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:17:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:17:07.133+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:17:07.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:17:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:17:07.133+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:17:07.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:17:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:17:07.133+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:17:07.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:17:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:17:07.447+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:07.448845+0000) 2024-02-20T09:17:07.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:17:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:17:07.686+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:17:07.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:17:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:17:07.686+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:17:07.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:17:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:17:07.686+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:17:07.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:17:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:17:07.686+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:17:08.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:17:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:17:08.015+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:08.017167+0000) 2024-02-20T09:17:08.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:17:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:17:08.084+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:17:08.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:17:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:17:08.084+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:17:08.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:17:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:17:08.084+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:17:08.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:17:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:17:07.899+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:07.900716+0000) 2024-02-20T09:17:08.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:17:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:17:07.980+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:17:08.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:17:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:17:07.980+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:17:08.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:17:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:17:07.980+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:17:08.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:17:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:17:08.449+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:08.450851+0000) 2024-02-20T09:17:08.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:17:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:17:08.695+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:17:08.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:17:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:17:08.695+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:17:08.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:17:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:17:08.695+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:17:08.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:17:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:17:08.695+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:17:09.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:17:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:17:08.899+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:08.900830+0000) 2024-02-20T09:17:09.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:17:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:17:09.001+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:17:09.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:17:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:17:09.001+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:17:09.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:17:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:17:09.001+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:17:09.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:17:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:17:09.015+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:09.017282+0000) 2024-02-20T09:17:09.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:17:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:17:09.076+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:17:09.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:17:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:17:09.076+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:17:09.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:17:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:17:09.076+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:17:09.735 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:17:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:17:09.453+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:09.454421+0000) 2024-02-20T09:17:09.735 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:17:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:17:09.733+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:17:09.736 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:17:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:17:09.733+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:17:09.995 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:17:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:17:09.733+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:17:09.995 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:17:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:17:09.733+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:17:09.996 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:17:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:17:09.899+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:09.900920+0000) 2024-02-20T09:17:09.996 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:17:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:17:09.953+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:17:09.996 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:17:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:17:09.953+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:17:09.996 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:17:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:17:09.953+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:17:10.433 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:17:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:17:10.016+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:10.017392+0000) 2024-02-20T09:17:10.434 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:17:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:17:10.088+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:17:10.434 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:17:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:17:10.088+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:17:10.434 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:17:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:17:10.088+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:17:10.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:17:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:17:10.454+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:10.455996+0000) 2024-02-20T09:17:11.065 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:17:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:17:10.899+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:10.901041+0000) 2024-02-20T09:17:11.065 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:17:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:17:10.921+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:17:11.065 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:17:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:17:10.921+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:17:11.065 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:17:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:17:10.921+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:17:11.066 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:17:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:17:10.771+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:17:11.066 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:17:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:17:10.771+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:17:11.066 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:17:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:17:10.771+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:17:11.066 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:17:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:17:10.771+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:17:11.066 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:17:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:17:11.016+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:11.017508+0000) 2024-02-20T09:17:11.066 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:17:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:17:11.062+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:17:11.067 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:17:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:17:11.062+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:17:11.067 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:17:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:17:11.062+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:17:11.457 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:17:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:17:11.179+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 65261 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:17:11.457 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:17:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:17:11.173+0000 7f8a5f86e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:11.175212+0000) 2024-02-20T09:17:11.458 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:17:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:17:11.176+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:11.177901+0000) 2024-02-20T09:17:11.458 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:17:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:17:11.173+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:11.174559+0000) 2024-02-20T09:17:11.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:17:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:17:11.454+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:11.456221+0000) 2024-02-20T09:17:12.070 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:17:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:17:12.016+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:12.017601+0000) 2024-02-20T09:17:12.070 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:17:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:17:12.067+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:17:12.070 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:17:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:17:12.067+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:17:12.071 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:17:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:17:12.067+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:17:12.071 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:17:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:17:11.899+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:11.901176+0000) 2024-02-20T09:17:12.071 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:17:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:17:11.963+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:17:12.071 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:17:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:17:11.963+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:17:12.071 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:17:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:17:11.963+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:17:12.072 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:17:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:17:11.782+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:17:12.072 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:17:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:17:11.782+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:17:12.072 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:17:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:17:11.782+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:17:12.072 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:17:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:17:11.782+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:17:12.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:17:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:17:12.458+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:12.459712+0000) 2024-02-20T09:17:13.018 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:17:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:17:12.899+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:12.901296+0000) 2024-02-20T09:17:13.019 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:17:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:17:12.969+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:17:13.019 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:17:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:17:12.969+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:17:13.019 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:17:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:17:12.969+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:17:13.019 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:17:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:17:12.752+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:17:13.019 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:17:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:17:12.752+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:17:13.020 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:17:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:17:12.752+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:17:13.020 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:17:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:17:12.752+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:17:13.464 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:17:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:17:13.016+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:13.017700+0000) 2024-02-20T09:17:13.464 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:17:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:17:13.081+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:17:13.464 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:17:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:17:13.081+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:17:13.465 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:17:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:17:13.081+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:17:13.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:17:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:17:13.461+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:13.463235+0000) 2024-02-20T09:17:13.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:17:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:17:13.706+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:17:13.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:17:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:17:13.706+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:17:13.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:17:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:17:13.706+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:17:13.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:17:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:17:13.706+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:17:14.108 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:17:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:17:14.016+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:14.017790+0000) 2024-02-20T09:17:14.108 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:17:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:17:14.105+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:17:14.108 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:17:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:17:13.900+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:13.901422+0000) 2024-02-20T09:17:14.109 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:17:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:17:14.000+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:17:14.109 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:17:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:17:14.000+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:17:14.109 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:17:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:17:14.000+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:17:14.466 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:17:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:17:14.105+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:17:14.467 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:17:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:17:14.105+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:17:14.732 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:17:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:17:14.464+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:14.465606+0000) 2024-02-20T09:17:14.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:17:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:17:14.729+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:17:14.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:17:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:17:14.729+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:17:14.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:17:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:17:14.729+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:17:14.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:17:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:17:14.729+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:17:14.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:17:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:17:14.900+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:14.901541+0000) 2024-02-20T09:17:14.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:17:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:17:14.968+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:17:14.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:17:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:17:14.968+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:17:14.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:17:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:17:14.968+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:17:15.468 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:17:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:17:15.016+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:15.017904+0000) 2024-02-20T09:17:15.469 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:17:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:17:15.057+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:17:15.469 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:17:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:17:15.057+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:17:15.469 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:17:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:17:15.057+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:17:15.732 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:17:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:17:15.466+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:15.467444+0000) 2024-02-20T09:17:15.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:17:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:17:15.900+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:15.901662+0000) 2024-02-20T09:17:15.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:17:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:17:15.964+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:17:15.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:17:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:17:15.964+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:17:15.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:17:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:17:15.964+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:17:15.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:17:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:17:15.729+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:17:15.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:17:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:17:15.729+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:17:15.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:17:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:17:15.729+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:17:15.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:17:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:17:15.729+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:17:16.306 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:17:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:17:16.181+0000 7f8a5f86e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:16.182468+0000) 2024-02-20T09:17:16.307 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:17:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:17:16.184+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:16.185815+0000) 2024-02-20T09:17:16.307 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:17:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:17:16.016+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:16.018026+0000) 2024-02-20T09:17:16.307 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:17:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:17:16.019+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:17:16.307 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:17:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:17:16.019+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:17:16.307 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:17:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:17:16.019+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:17:16.308 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:17:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:17:16.181+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:16.182590+0000) 2024-02-20T09:17:16.308 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:17:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:17:16.187+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 65300 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:17:16.651 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:17:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:17:16.466+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:16.467633+0000) 2024-02-20T09:17:16.903 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:17:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:17:16.755+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:17:16.903 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:17:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:17:16.755+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:17:16.903 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:17:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:17:16.755+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:17:16.903 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:17:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:17:16.755+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:17:17.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:17:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:17:16.900+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:16.901772+0000) 2024-02-20T09:17:17.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:17:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:17:16.922+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:17:17.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:17:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:17:16.922+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:17:17.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:17:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:17:16.922+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:17:17.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:17:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:17:16.999+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:17:17.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:17:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:17:16.999+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:17:17.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:17:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:17:16.999+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:17:17.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:17:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:17:17.016+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:17.018136+0000) 2024-02-20T09:17:17.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:17:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:17:17.468+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:17.469927+0000) 2024-02-20T09:17:18.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:17:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:17:17.787+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:17:18.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:17:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:17:17.787+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:17:18.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:17:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:17:17.787+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:17:18.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:17:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:17:17.788+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:17:18.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:17:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:17:17.874+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:17:18.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:17:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:17:17.874+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:17:18.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:17:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:17:17.874+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:17:18.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:17:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:17:17.900+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:17.901873+0000) 2024-02-20T09:17:18.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:17:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:17:18.006+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:17:18.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:17:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:17:18.006+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:17:18.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:17:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:17:18.006+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:17:18.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:17:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:17:18.016+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:18.018217+0000) 2024-02-20T09:17:18.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:17:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:17:18.470+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:18.472210+0000) 2024-02-20T09:17:19.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:17:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:17:18.828+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:17:19.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:17:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:17:18.828+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:17:19.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:17:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:17:18.828+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:17:19.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:17:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:17:18.900+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:18.901973+0000) 2024-02-20T09:17:19.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:17:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:17:18.805+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:17:19.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:17:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:17:18.805+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:17:19.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:17:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:17:18.805+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:17:19.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:17:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:17:18.805+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:17:19.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:17:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:17:18.996+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:17:19.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:17:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:17:18.996+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:17:19.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:17:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:17:18.996+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:17:19.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:17:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:17:19.016+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:19.018313+0000) 2024-02-20T09:17:19.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:17:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:17:19.473+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:19.474469+0000) 2024-02-20T09:17:20.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:17:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:17:19.786+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:17:20.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:17:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:17:19.786+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:17:20.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:17:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:17:19.786+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:17:20.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:17:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:17:19.786+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:17:20.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:17:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:17:19.833+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:17:20.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:17:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:17:19.833+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:17:20.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:17:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:17:19.833+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:17:20.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:17:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:17:19.900+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:19.902078+0000) 2024-02-20T09:17:20.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:17:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:17:20.017+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:20.018438+0000) 2024-02-20T09:17:20.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:17:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:17:20.027+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:17:20.245 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:17:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:17:20.027+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:17:20.245 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:17:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:17:20.027+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:17:20.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:17:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:17:20.474+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:20.475911+0000) 2024-02-20T09:17:21.019 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:17:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:17:20.983+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:17:21.020 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:17:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:17:20.983+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:17:21.020 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:17:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:17:20.983+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:17:21.020 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:17:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:17:20.806+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:17:21.020 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:17:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:17:20.807+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:17:21.020 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:17:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:17:20.807+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:17:21.020 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:17:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:17:20.900+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:20.902205+0000) 2024-02-20T09:17:21.021 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:17:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:17:20.767+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:17:21.021 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:17:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:17:20.767+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:17:21.021 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:17:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:17:20.767+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:17:21.021 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:17:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:17:20.767+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:17:21.316 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:17:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:17:21.194+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 65340 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:17:21.317 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:17:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:17:21.188+0000 7f8a5f86e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:21.189438+0000) 2024-02-20T09:17:21.317 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:17:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:17:21.192+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:21.193426+0000) 2024-02-20T09:17:21.317 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:17:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:17:21.017+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:21.018526+0000) 2024-02-20T09:17:21.318 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:17:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:17:21.188+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:21.189587+0000) 2024-02-20T09:17:21.651 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:17:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:17:21.474+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:21.476142+0000) 2024-02-20T09:17:21.903 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:17:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:17:21.785+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:17:21.903 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:17:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:17:21.785+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:17:21.903 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:17:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:17:21.785+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:17:21.904 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:17:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:17:21.756+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:17:21.904 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:17:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:17:21.756+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:17:21.904 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:17:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:17:21.756+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:17:21.904 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:17:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:17:21.757+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:17:22.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:17:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:17:21.900+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:21.902323+0000) 2024-02-20T09:17:22.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:17:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:17:22.003+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:17:22.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:17:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:17:22.003+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:17:22.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:17:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:17:22.003+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:17:22.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:17:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:17:22.017+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:22.018619+0000) 2024-02-20T09:17:22.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:17:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:17:22.476+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:22.478058+0000) 2024-02-20T09:17:23.019 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:17:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:17:22.982+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:17:23.020 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:17:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:17:22.982+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:17:23.020 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:17:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:17:22.982+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:17:23.020 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:17:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:17:22.762+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:17:23.021 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:17:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:17:22.762+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:17:23.021 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:17:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:17:22.762+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:17:23.021 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:17:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:17:22.762+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:17:23.021 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:17:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:17:22.807+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:17:23.021 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:17:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:17:22.807+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:17:23.021 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:17:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:17:22.807+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:17:23.022 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:17:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:17:22.901+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:22.902462+0000) 2024-02-20T09:17:23.483 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:17:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:17:23.017+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:23.018709+0000) 2024-02-20T09:17:23.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:17:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:17:23.480+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:23.481596+0000) 2024-02-20T09:17:24.020 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:17:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:17:23.940+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:17:24.020 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:17:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:17:23.940+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:17:24.020 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:17:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:17:23.940+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:17:24.021 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:17:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:17:23.765+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:17:24.021 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:17:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:17:23.765+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:17:24.021 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:17:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:17:23.765+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:17:24.021 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:17:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:17:23.901+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:23.902581+0000) 2024-02-20T09:17:24.021 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:17:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:17:23.794+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:17:24.022 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:17:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:17:23.794+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:17:24.022 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:17:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:17:23.794+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:17:24.022 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:17:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:17:23.794+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:17:24.486 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:17:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:17:24.017+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:24.018814+0000) 2024-02-20T09:17:24.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:17:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:17:24.483+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:24.485143+0000) 2024-02-20T09:17:24.742 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:17:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:17:24.733+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:17:24.742 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:17:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:17:24.733+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:17:24.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:17:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:17:24.733+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:17:25.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:17:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:17:24.901+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:24.902671+0000) 2024-02-20T09:17:25.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:17:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:17:24.952+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:17:25.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:17:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:17:24.952+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:17:25.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:17:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:17:24.952+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:17:25.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:17:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:17:25.017+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:25.018925+0000) 2024-02-20T09:17:25.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:17:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:17:24.813+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:17:25.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:17:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:17:24.813+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:17:25.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:17:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:17:24.813+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:17:25.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:17:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:17:24.813+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:17:25.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:17:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:17:25.486+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:25.487546+0000) 2024-02-20T09:17:26.198 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:17:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:17:25.859+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:17:26.198 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:17:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:17:25.859+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:17:26.198 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:17:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:17:25.859+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:17:26.198 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:17:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:17:25.859+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:17:26.199 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:17:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:17:25.917+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:17:26.199 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:17:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:17:25.917+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:17:26.199 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:17:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:17:25.917+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:17:26.199 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:17:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:17:26.017+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:26.019030+0000) 2024-02-20T09:17:26.200 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:17:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:17:25.778+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:17:26.200 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:17:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:17:25.778+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:17:26.200 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:17:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:17:25.778+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:17:26.200 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:17:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:17:25.901+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:25.902760+0000) 2024-02-20T09:17:26.488 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:17:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:17:26.201+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 65379 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:17:26.489 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:17:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:17:26.195+0000 7f8a5f86e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:26.197221+0000) 2024-02-20T09:17:26.489 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:17:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:17:26.198+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:26.199807+0000) 2024-02-20T09:17:26.490 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:17:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:17:26.195+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:26.196963+0000) 2024-02-20T09:17:26.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:17:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:17:26.486+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:26.487718+0000) 2024-02-20T09:17:27.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:17:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:17:26.916+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:17:27.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:17:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:17:26.916+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:17:27.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:17:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:17:26.916+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:17:27.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:17:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:17:27.017+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:27.019146+0000) 2024-02-20T09:17:27.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:17:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:17:26.806+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:17:27.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:17:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:17:26.806+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:17:27.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:17:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:17:26.806+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:17:27.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:17:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:17:26.901+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:26.902898+0000) 2024-02-20T09:17:27.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:17:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:17:26.839+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:17:27.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:17:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:17:26.839+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:17:27.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:17:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:17:26.839+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:17:27.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:17:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:17:26.839+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:17:27.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:17:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:17:27.489+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:27.491081+0000) 2024-02-20T09:17:28.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:17:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:17:27.798+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:17:28.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:17:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:17:27.798+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:17:28.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:17:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:17:27.798+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:17:28.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:17:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:17:27.901+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:27.902977+0000) 2024-02-20T09:17:28.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:17:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:17:27.810+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:17:28.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:17:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:17:27.810+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:17:28.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:17:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:17:27.810+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:17:28.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:17:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:17:27.810+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:17:28.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:17:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:17:27.918+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:17:28.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:17:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:17:27.918+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:17:28.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:17:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:17:27.918+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:17:28.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:17:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:17:28.017+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:28.019248+0000) 2024-02-20T09:17:28.799 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:17:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:17:28.492+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:28.493593+0000) 2024-02-20T09:17:29.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:17:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:17:28.807+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:17:29.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:17:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:17:28.807+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:17:29.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:17:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:17:28.807+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:17:29.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:17:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:17:28.901+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:28.903095+0000) 2024-02-20T09:17:29.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:17:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:17:28.797+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:17:29.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:17:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:17:28.797+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:17:29.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:17:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:17:28.797+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:17:29.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:17:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:17:28.797+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:17:29.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:17:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:17:28.928+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:17:29.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:17:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:17:28.928+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:17:29.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:17:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:17:28.928+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:17:29.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:17:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:17:29.017+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:29.019363+0000) 2024-02-20T09:17:29.828 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:17:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:17:29.495+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:29.497150+0000) 2024-02-20T09:17:29.828 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:17:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:17:29.825+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:17:29.828 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:17:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:17:29.825+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:17:29.828 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:17:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:17:29.825+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:17:29.829 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:17:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:17:29.825+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:17:30.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:17:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:17:29.923+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:17:30.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:17:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:17:29.923+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:17:30.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:17:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:17:29.923+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:17:30.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:17:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:17:30.018+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:30.019439+0000) 2024-02-20T09:17:30.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:17:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:17:29.841+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:17:30.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:17:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:17:29.841+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:17:30.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:17:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:17:29.841+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:17:30.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:17:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:17:29.901+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:29.903212+0000) 2024-02-20T09:17:30.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:17:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:17:30.497+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:30.498708+0000) 2024-02-20T09:17:31.205 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:17:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:17:30.866+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:17:31.205 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:17:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:17:30.866+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:17:31.206 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:17:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:17:30.866+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:17:31.206 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:17:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:17:30.901+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:30.903337+0000) 2024-02-20T09:17:31.206 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:17:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:17:30.800+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:17:31.206 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:17:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:17:30.800+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:17:31.206 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:17:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:17:30.800+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:17:31.207 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:17:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:17:30.800+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:17:31.207 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:17:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:17:30.939+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:17:31.207 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:17:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:17:30.939+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:17:31.207 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:17:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:17:30.939+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:17:31.207 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:17:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:17:31.018+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:31.019529+0000) 2024-02-20T09:17:31.207 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:17:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:17:31.202+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:31.204196+0000) 2024-02-20T09:17:31.492 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:17:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:17:31.202+0000 7f8a5f86e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:31.204306+0000) 2024-02-20T09:17:31.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:17:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:17:31.206+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:31.207816+0000) 2024-02-20T09:17:31.493 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:17:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:17:31.209+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 65419 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:17:31.828 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:17:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:17:31.497+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:31.498913+0000) 2024-02-20T09:17:32.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:17:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:17:31.900+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:17:32.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:17:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:17:31.900+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:17:32.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:17:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:17:31.900+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:17:32.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:17:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:17:32.018+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:32.019641+0000) 2024-02-20T09:17:32.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:17:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:17:31.825+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:17:32.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:17:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:17:31.825+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:17:32.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:17:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:17:31.825+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:17:32.245 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:17:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:17:31.902+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:31.903430+0000) 2024-02-20T09:17:32.245 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:17:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:17:31.846+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:17:32.245 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:17:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:17:31.846+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:17:32.246 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:17:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:17:31.846+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:17:32.246 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:17:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:17:31.846+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:17:32.548 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:17:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:17:32.500+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:32.501391+0000) 2024-02-20T09:17:33.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:17:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:17:32.849+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:17:33.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:17:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:17:32.849+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:17:33.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:17:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:17:32.849+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:17:33.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:17:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:17:32.902+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:32.903489+0000) 2024-02-20T09:17:33.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:17:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:17:32.820+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:17:33.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:17:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:17:32.820+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:17:33.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:17:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:17:32.820+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:17:33.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:17:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:17:32.820+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:17:33.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:17:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:17:32.903+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:17:33.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:17:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:17:32.903+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:17:33.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:17:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:17:32.903+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:17:33.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:17:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:17:33.018+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:33.019715+0000) 2024-02-20T09:17:33.816 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:17:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:17:33.502+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:33.503628+0000) 2024-02-20T09:17:34.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:17:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:17:33.814+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:17:34.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:17:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:17:33.814+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:17:34.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:17:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:17:33.814+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:17:34.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:17:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:17:33.902+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:33.903580+0000) 2024-02-20T09:17:34.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:17:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:17:33.830+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:17:34.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:17:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:17:33.830+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:17:34.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:17:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:17:33.830+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:17:34.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:17:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:17:33.830+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:17:34.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:17:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:17:33.890+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:17:34.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:17:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:17:33.890+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:17:34.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:17:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:17:33.890+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:17:34.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:17:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:17:34.018+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:34.019831+0000) 2024-02-20T09:17:34.835 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:17:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:17:34.504+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:34.505862+0000) 2024-02-20T09:17:34.836 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:17:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:17:34.833+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:17:34.836 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:17:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:17:34.833+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:17:34.836 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:17:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:17:34.833+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:17:34.836 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:17:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:17:34.833+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:17:35.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:17:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:17:34.857+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:17:35.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:17:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:17:34.857+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:17:35.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:17:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:17:34.857+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:17:35.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:17:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:17:34.902+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:34.903674+0000) 2024-02-20T09:17:35.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:17:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:17:34.871+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:17:35.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:17:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:17:34.871+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:17:35.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:17:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:17:34.871+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:17:35.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:17:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:17:35.018+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:35.019943+0000) 2024-02-20T09:17:35.860 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:17:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:17:35.506+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:35.507436+0000) 2024-02-20T09:17:35.860 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:17:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:17:35.857+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:17:35.860 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:17:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:17:35.857+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:17:36.213 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:17:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:17:35.857+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:17:36.214 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:17:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:17:35.857+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:17:36.214 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:17:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:17:35.863+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:17:36.214 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:17:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:17:35.863+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:17:36.214 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:17:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:17:35.863+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:17:36.214 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:17:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:17:36.018+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:36.020059+0000) 2024-02-20T09:17:36.215 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:17:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:17:35.898+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:17:36.215 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:17:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:17:35.898+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:17:36.215 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:17:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:17:35.898+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:17:36.215 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:17:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:17:35.902+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:35.903773+0000) 2024-02-20T09:17:36.215 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:17:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:17:36.210+0000 7f8a5f86e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:36.212386+0000) 2024-02-20T09:17:36.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:17:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:17:36.215+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:36.216558+0000) 2024-02-20T09:17:36.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:17:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:17:36.211+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:36.212434+0000) 2024-02-20T09:17:36.493 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:17:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:17:36.217+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 65458 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:17:36.874 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:17:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:17:36.507+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:36.507601+0000) 2024-02-20T09:17:36.874 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:17:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:17:36.871+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:17:36.874 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:17:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:17:36.871+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:17:36.875 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:17:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:17:36.871+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:17:36.875 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:17:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:17:36.871+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:17:37.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:17:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:17:36.902+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:36.903912+0000) 2024-02-20T09:17:37.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:17:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:17:36.947+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:17:37.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:17:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:17:36.947+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:17:37.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:17:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:17:36.947+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:17:37.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:17:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:17:36.907+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:17:37.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:17:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:17:36.907+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:17:37.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:17:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:17:36.907+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:17:37.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:17:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:17:37.019+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:37.020229+0000) 2024-02-20T09:17:37.842 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:17:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:17:37.508+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:37.509589+0000) 2024-02-20T09:17:37.842 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:17:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:17:37.839+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:17:37.843 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:17:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:17:37.839+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:17:38.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:17:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:17:37.839+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:17:38.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:17:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:17:37.839+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:17:38.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:17:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:17:37.867+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:17:38.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:17:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:17:37.867+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:17:38.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:17:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:17:37.867+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:17:38.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:17:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:17:38.020+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:38.020427+0000) 2024-02-20T09:17:38.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:17:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:17:37.903+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:37.904075+0000) 2024-02-20T09:17:38.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:17:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:17:37.918+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:17:38.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:17:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:17:37.918+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:17:38.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:17:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:17:37.918+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:17:38.815 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:17:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:17:38.511+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:38.512323+0000) 2024-02-20T09:17:38.816 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:17:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:17:38.814+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:17:38.816 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:17:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:17:38.814+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:17:38.816 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:17:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:17:38.814+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:17:38.816 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:17:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:17:38.814+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:17:39.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:17:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:17:38.903+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:38.904278+0000) 2024-02-20T09:17:39.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:17:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:17:38.955+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:17:39.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:17:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:17:38.955+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:17:39.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:17:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:17:38.955+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:17:39.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:17:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:17:38.882+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:17:39.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:17:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:17:38.882+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:17:39.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:17:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:17:38.882+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:17:39.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:17:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:17:39.020+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:39.020551+0000) 2024-02-20T09:17:39.863 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:17:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:17:39.514+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:39.514465+0000) 2024-02-20T09:17:39.864 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:17:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:17:39.862+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:17:39.864 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:17:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:17:39.862+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:17:39.864 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:17:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:17:39.862+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:17:39.864 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:17:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:17:39.862+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:17:40.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:17:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:17:39.904+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:39.904446+0000) 2024-02-20T09:17:40.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:17:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:17:39.995+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:17:40.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:17:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:17:39.995+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:17:40.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:17:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:17:39.995+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:17:40.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:17:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:17:39.931+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:17:40.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:17:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:17:39.931+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:17:40.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:17:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:17:39.931+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:17:40.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:17:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:17:40.020+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:40.020663+0000) 2024-02-20T09:17:40.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:17:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:17:40.516+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:40.517171+0000) 2024-02-20T09:17:41.221 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:17:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:17:40.874+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:17:41.221 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:17:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:17:40.874+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:17:41.222 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:17:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:17:40.874+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:17:41.222 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:17:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:17:40.874+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:17:41.222 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:17:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:17:40.978+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:17:41.222 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:17:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:17:40.978+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:17:41.223 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:17:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:17:40.978+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:17:41.223 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:17:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:17:41.020+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:41.020785+0000) 2024-02-20T09:17:41.223 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:17:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:17:40.904+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:40.904582+0000) 2024-02-20T09:17:41.223 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:17:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:17:41.025+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:17:41.223 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:17:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:17:41.025+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:17:41.224 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:17:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:17:41.025+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:17:41.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:17:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:17:41.219+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:41.220342+0000) 2024-02-20T09:17:41.493 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:17:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:17:41.225+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 65498 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:17:41.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:17:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:17:41.219+0000 7f8a5f86e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:41.220288+0000) 2024-02-20T09:17:41.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:17:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:17:41.223+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:41.224059+0000) 2024-02-20T09:17:41.895 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:17:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:17:41.517+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:41.517388+0000) 2024-02-20T09:17:42.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:17:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:17:41.899+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:17:42.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:17:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:17:41.899+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:17:42.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:17:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:17:41.899+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:17:42.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:17:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:17:41.899+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:17:42.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:17:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:17:41.968+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:17:42.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:17:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:17:41.968+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:17:42.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:17:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:17:41.968+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:17:42.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:17:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:17:42.020+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:42.020947+0000) 2024-02-20T09:17:42.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:17:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:17:41.904+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:41.904688+0000) 2024-02-20T09:17:42.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:17:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:17:42.042+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:17:42.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:17:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:17:42.042+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:17:42.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:17:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:17:42.042+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:17:42.881 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:17:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:17:42.519+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:42.519436+0000) 2024-02-20T09:17:42.882 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:17:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:17:42.880+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:17:42.882 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:17:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:17:42.880+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:17:42.882 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:17:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:17:42.880+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:17:42.882 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:17:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:17:42.880+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:17:43.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:17:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:17:42.950+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:17:43.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:17:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:17:42.950+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:17:43.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:17:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:17:42.950+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:17:43.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:17:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:17:43.020+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:43.021033+0000) 2024-02-20T09:17:43.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:17:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:17:42.904+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:42.904803+0000) 2024-02-20T09:17:43.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:17:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:17:43.028+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:17:43.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:17:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:17:43.028+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:17:43.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:17:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:17:43.028+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:17:43.895 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:17:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:17:43.521+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:43.521497+0000) 2024-02-20T09:17:44.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:17:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:17:43.923+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:17:44.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:17:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:17:43.923+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:17:44.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:17:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:17:43.923+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:17:44.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:17:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:17:43.923+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:17:44.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:17:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:17:43.948+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:17:44.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:17:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:17:43.948+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:17:44.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:17:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:17:43.948+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:17:44.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:17:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:17:44.020+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:44.021205+0000) 2024-02-20T09:17:44.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:17:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:17:43.904+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:43.904910+0000) 2024-02-20T09:17:44.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:17:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:17:43.989+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:17:44.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:17:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:17:43.989+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:17:44.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:17:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:17:43.989+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:17:44.895 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:17:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:17:44.523+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:44.523502+0000) 2024-02-20T09:17:45.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:17:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:17:44.998+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:17:45.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:17:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:17:44.998+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:17:45.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:17:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:17:44.998+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:17:45.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:17:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:17:45.021+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:45.021386+0000) 2024-02-20T09:17:45.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:17:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:17:44.904+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:44.905114+0000) 2024-02-20T09:17:45.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:17:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:17:44.949+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:17:45.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:17:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:17:44.949+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:17:45.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:17:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:17:44.949+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:17:45.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:17:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:17:44.913+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:17:45.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:17:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:17:44.913+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:17:45.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:17:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:17:44.913+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:17:45.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:17:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:17:44.913+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:17:45.895 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:17:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:17:45.524+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:45.525015+0000) 2024-02-20T09:17:46.228 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:17:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:17:45.894+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:17:46.229 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:17:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:17:45.894+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:17:46.229 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:17:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:17:45.894+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:17:46.229 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:17:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:17:45.894+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:17:46.229 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:17:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:17:46.021+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:46.021534+0000) 2024-02-20T09:17:46.229 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:17:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:17:46.023+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:17:46.230 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:17:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:17:46.023+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:17:46.230 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:17:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:17:46.023+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:17:46.230 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:17:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:17:45.904+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:45.905297+0000) 2024-02-20T09:17:46.230 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:17:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:17:45.956+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:17:46.230 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:17:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:17:45.956+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:17:46.231 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:17:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:17:45.956+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:17:46.492 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:17:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:17:46.233+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 65537 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:17:46.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:17:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:17:46.230+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:46.230718+0000) 2024-02-20T09:17:46.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:17:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:17:46.227+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:46.227425+0000) 2024-02-20T09:17:46.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:17:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:17:46.227+0000 7f8a5f86e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:46.227666+0000) 2024-02-20T09:17:46.859 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:17:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:17:46.524+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:46.525298+0000) 2024-02-20T09:17:46.860 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:17:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:17:46.857+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:17:46.860 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:17:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:17:46.857+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:17:47.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:17:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:17:46.857+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:17:47.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:17:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:17:46.857+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:17:47.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:17:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:17:47.004+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:17:47.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:17:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:17:47.004+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:17:47.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:17:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:17:47.004+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:17:47.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:17:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:17:47.021+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:47.021691+0000) 2024-02-20T09:17:47.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:17:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:17:46.905+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:46.905421+0000) 2024-02-20T09:17:47.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:17:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:17:47.001+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:17:47.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:17:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:17:47.001+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:17:47.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:17:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:17:47.001+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:17:47.856 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:17:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:17:47.527+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:47.527387+0000) 2024-02-20T09:17:47.856 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:17:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:17:47.854+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:17:48.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:17:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:17:48.021+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:48.021770+0000) 2024-02-20T09:17:48.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:17:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:17:48.037+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:17:48.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:17:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:17:48.037+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:17:48.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:17:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:17:48.037+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:17:48.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:17:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:17:47.854+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:17:48.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:17:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:17:47.854+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:17:48.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:17:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:17:47.855+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:17:48.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:17:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:17:47.905+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:47.905497+0000) 2024-02-20T09:17:48.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:17:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:17:47.985+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:17:48.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:17:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:17:47.985+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:17:48.245 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:17:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:17:47.985+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:17:48.895 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:17:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:17:48.530+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:48.531040+0000) 2024-02-20T09:17:49.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:17:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:17:49.021+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:49.021876+0000) 2024-02-20T09:17:49.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:17:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:17:49.038+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:17:49.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:17:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:17:49.038+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:17:49.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:17:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:17:49.038+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:17:49.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:17:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:17:48.905+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:48.905629+0000) 2024-02-20T09:17:49.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:17:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:17:49.020+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:17:49.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:17:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:17:49.020+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:17:49.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:17:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:17:49.020+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:17:49.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:17:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:17:48.901+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:17:49.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:17:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:17:48.901+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:17:49.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:17:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:17:48.901+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:17:49.245 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:17:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:17:48.901+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:17:49.855 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:17:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:17:49.532+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:49.533140+0000) 2024-02-20T09:17:49.855 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:17:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:17:49.853+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:17:49.855 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:17:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:17:49.853+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:17:49.855 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:17:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:17:49.853+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:17:49.855 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:17:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:17:49.853+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:17:50.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:17:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:17:50.021+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:50.021963+0000) 2024-02-20T09:17:50.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:17:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:17:50.086+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:17:50.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:17:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:17:50.086+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:17:50.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:17:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:17:50.086+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:17:50.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:17:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:17:49.905+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:49.905729+0000) 2024-02-20T09:17:50.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:17:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:17:50.046+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:17:50.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:17:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:17:50.046+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:17:50.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:17:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:17:50.046+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:17:50.712 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:17:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:17:50.534+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:50.534917+0000) 2024-02-20T09:17:50.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:17:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:17:50.905+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:50.905837+0000) 2024-02-20T09:17:50.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:17:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:17:50.816+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:17:50.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:17:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:17:50.816+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:17:50.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:17:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:17:50.816+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:17:50.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:17:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:17:50.816+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:17:51.359 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:17:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:17:51.237+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:51.238254+0000) 2024-02-20T09:17:51.360 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:17:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:17:51.240+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 65577 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:17:51.360 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:17:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:17:51.088+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:17:51.360 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:17:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:17:51.088+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:17:51.361 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:17:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:17:51.088+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:17:51.361 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:17:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:17:51.234+0000 7f8a5f86e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:51.235097+0000) 2024-02-20T09:17:51.361 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:17:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:17:51.021+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:51.022073+0000) 2024-02-20T09:17:51.361 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:17:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:17:51.061+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:17:51.361 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:17:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:17:51.061+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:17:51.361 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:17:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:17:51.061+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:17:51.361 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:17:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:17:51.235+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:51.235444+0000) 2024-02-20T09:17:51.655 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:17:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:17:51.534+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:51.535103+0000) 2024-02-20T09:17:51.907 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:17:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:17:51.838+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:17:51.907 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:17:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:17:51.838+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:17:51.907 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:17:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:17:51.838+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:17:51.907 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:17:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:17:51.838+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:17:52.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:17:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:17:51.905+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:51.906013+0000) 2024-02-20T09:17:52.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:17:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:17:52.064+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:17:52.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:17:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:17:52.064+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:17:52.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:17:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:17:52.064+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:17:52.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:17:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:17:52.021+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:52.022270+0000) 2024-02-20T09:17:52.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:17:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:17:52.104+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:17:52.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:17:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:17:52.104+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:17:52.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:17:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:17:52.104+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:17:52.859 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:17:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:17:52.537+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:52.537688+0000) 2024-02-20T09:17:52.859 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:17:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:17:52.857+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:17:52.859 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:17:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:17:52.857+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:17:52.860 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:17:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:17:52.857+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:17:52.860 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:17:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:17:52.857+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:17:53.146 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:17:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:17:53.022+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:53.022415+0000) 2024-02-20T09:17:53.146 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:17:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:17:53.144+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:17:53.146 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:17:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:17:52.905+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:52.906180+0000) 2024-02-20T09:17:53.146 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:17:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:17:53.090+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:17:53.147 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:17:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:17:53.090+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:17:53.147 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:17:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:17:53.090+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:17:53.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:17:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:17:53.144+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:17:53.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:17:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:17:53.144+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:17:53.815 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:17:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:17:53.539+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:53.539664+0000) 2024-02-20T09:17:53.815 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:17:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:17:53.813+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:17:53.815 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:17:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:17:53.813+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:17:53.815 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:17:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:17:53.813+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:17:53.816 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:17:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:17:53.813+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:17:54.138 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:17:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:17:54.022+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:54.022566+0000) 2024-02-20T09:17:54.138 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:17:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:17:53.905+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:53.906330+0000) 2024-02-20T09:17:54.138 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:17:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:17:54.136+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:17:54.139 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:17:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:17:54.136+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:17:54.492 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:17:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:17:54.136+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:17:54.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:17:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:17:54.139+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:17:54.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:17:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:17:54.139+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:17:54.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:17:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:17:54.139+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:17:54.896 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:17:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:17:54.541+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:54.541820+0000) 2024-02-20T09:17:54.896 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:17:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:17:54.775+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:17:54.897 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:17:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:17:54.775+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:17:54.897 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:17:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:17:54.775+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:17:54.897 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:17:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:17:54.775+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:17:55.147 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:17:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:17:55.022+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:55.022708+0000) 2024-02-20T09:17:55.147 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:17:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:17:54.906+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:54.906500+0000) 2024-02-20T09:17:55.147 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:17:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:17:55.145+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:17:55.148 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:17:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:17:55.145+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:17:55.148 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:17:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:17:55.145+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:17:55.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:17:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:17:55.160+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:17:55.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:17:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:17:55.160+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:17:55.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:17:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:17:55.160+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:17:55.808 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:17:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:17:55.542+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:55.543274+0000) 2024-02-20T09:17:55.808 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:17:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:17:55.806+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:17:55.808 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:17:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:17:55.806+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:17:55.808 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:17:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:17:55.806+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:17:55.808 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:17:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:17:55.806+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:17:56.191 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:17:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:17:56.022+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:56.022834+0000) 2024-02-20T09:17:56.191 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:17:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:17:55.906+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:55.906644+0000) 2024-02-20T09:17:56.191 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:17:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:17:56.189+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:17:56.191 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:17:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:17:56.189+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:17:56.191 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:17:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:17:56.189+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:17:56.492 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:17:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:17:56.248+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 65616 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:17:56.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:17:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:17:56.241+0000 7f8a5f86e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:56.242257+0000) 2024-02-20T09:17:56.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:17:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:17:56.244+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:56.244475+0000) 2024-02-20T09:17:56.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:17:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:17:56.202+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:17:56.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:17:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:17:56.202+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:17:56.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:17:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:17:56.202+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:17:56.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:17:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:17:56.242+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:56.242404+0000) 2024-02-20T09:17:56.847 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:17:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:17:56.543+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:56.543444+0000) 2024-02-20T09:17:56.848 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:17:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:17:56.846+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:17:56.848 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:17:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:17:56.846+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:17:56.848 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:17:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:17:56.846+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:17:56.848 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:17:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:17:56.846+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:17:57.217 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:17:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:17:57.022+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:57.022955+0000) 2024-02-20T09:17:57.217 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:17:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:17:56.906+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:56.906775+0000) 2024-02-20T09:17:57.217 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:17:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:17:57.215+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:17:57.218 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:17:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:17:57.215+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:17:57.218 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:17:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:17:57.215+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:17:57.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:17:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:17:57.252+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:17:57.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:17:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:17:57.252+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:17:57.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:17:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:17:57.252+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:17:57.880 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:17:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:17:57.545+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:57.545544+0000) 2024-02-20T09:17:57.880 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:17:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:17:57.878+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:17:57.880 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:17:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:17:57.878+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:17:57.881 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:17:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:17:57.878+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:17:58.226 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:17:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:17:57.906+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:57.906885+0000) 2024-02-20T09:17:58.227 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:17:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:17:57.878+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:17:58.227 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:17:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:17:58.022+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:58.023034+0000) 2024-02-20T09:17:58.227 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:17:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:17:58.225+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:17:58.227 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:17:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:17:58.225+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:17:58.228 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:17:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:17:58.225+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:17:58.492 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:17:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:17:58.263+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:17:58.492 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:17:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:17:58.263+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:17:58.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:17:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:17:58.263+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:17:58.896 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:17:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:17:58.548+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:58.549062+0000) 2024-02-20T09:17:59.221 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:17:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:17:58.906+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:58.906999+0000) 2024-02-20T09:17:59.221 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:17:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:17:59.022+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:59.023123+0000) 2024-02-20T09:17:59.221 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:17:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:17:59.218+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:17:59.221 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:17:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:17:59.218+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:17:59.222 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:17:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:17:59.218+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:17:59.222 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:17:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:17:58.903+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:17:59.222 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:17:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:17:58.903+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:17:59.222 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:17:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:17:58.903+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:17:59.222 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:17:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:17:58.903+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:17:59.492 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:17:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:17:59.256+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:17:59.492 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:17:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:17:59.256+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:17:59.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:17:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:17:59.256+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:17:59.869 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:17:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:17:59.549+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:59.551214+0000) 2024-02-20T09:17:59.869 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:17:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:17:59.866+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:17:59.869 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:17:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:17:59.866+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:17:59.869 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:17:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:17:59.866+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:17:59.869 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:17:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:17:59.866+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:18:00.176 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:18:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:18:00.021+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:00.023264+0000) 2024-02-20T09:18:00.177 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:17:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:17:59.905+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:17:59.907135+0000) 2024-02-20T09:18:00.438 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:18:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:18:00.174+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:18:00.438 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:18:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:18:00.174+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:18:00.438 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:18:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:18:00.174+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:18:00.438 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:18:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:18:00.264+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:18:00.439 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:18:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:18:00.264+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:18:00.439 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:18:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:18:00.264+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:18:00.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:18:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:18:00.551+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:00.553223+0000) 2024-02-20T09:18:01.189 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:18:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:18:00.905+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:00.907270+0000) 2024-02-20T09:18:01.190 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:18:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:18:01.021+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:01.023375+0000) 2024-02-20T09:18:01.190 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:18:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:18:01.187+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:18:01.190 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:18:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:18:01.187+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:18:01.190 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:18:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:18:00.872+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:18:01.190 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:18:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:18:00.872+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:18:01.191 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:18:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:18:00.872+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:18:01.191 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:18:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:18:00.872+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:18:01.492 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:18:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:18:01.254+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 65656 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:18:01.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:18:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:18:01.251+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:01.252730+0000) 2024-02-20T09:18:01.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:18:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:18:01.187+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:18:01.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:18:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:18:01.248+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:01.250260+0000) 2024-02-20T09:18:01.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:18:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:18:01.248+0000 7f8a5f86e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:01.250194+0000) 2024-02-20T09:18:01.494 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:18:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:18:01.285+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:18:01.494 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:18:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:18:01.285+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:18:01.494 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:18:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:18:01.285+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:18:01.841 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:18:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:18:01.552+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:01.553437+0000) 2024-02-20T09:18:01.842 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:18:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:18:01.838+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:18:01.842 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:18:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:18:01.838+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:18:01.842 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:18:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:18:01.839+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:18:01.842 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:18:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:18:01.839+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:18:02.162 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:18:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:18:01.906+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:01.907427+0000) 2024-02-20T09:18:02.162 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:18:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:18:02.022+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:02.023520+0000) 2024-02-20T09:18:02.162 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:18:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:18:02.159+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:18:02.162 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:18:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:18:02.159+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:18:02.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:18:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:18:02.159+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:18:02.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:18:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:18:02.268+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:18:02.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:18:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:18:02.268+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:18:02.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:18:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:18:02.268+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:18:02.851 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:18:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:18:02.554+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:02.555495+0000) 2024-02-20T09:18:03.125 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:18:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:18:02.906+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:02.907551+0000) 2024-02-20T09:18:03.126 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:18:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:18:02.848+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:18:03.126 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:18:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:18:02.848+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:18:03.126 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:18:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:18:02.848+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:18:03.126 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:18:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:18:02.848+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:18:03.126 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:18:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:18:03.022+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:03.023598+0000) 2024-02-20T09:18:03.127 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:18:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:18:03.123+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:18:03.127 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:18:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:18:03.123+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:18:03.127 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:18:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:18:03.123+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:18:03.492 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:18:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:18:03.269+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:18:03.492 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:18:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:18:03.269+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:18:03.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:18:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:18:03.269+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:18:03.868 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:18:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:18:03.557+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:03.559058+0000) 2024-02-20T09:18:03.868 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:18:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:18:03.865+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:18:03.868 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:18:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:18:03.865+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:18:03.868 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:18:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:18:03.865+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:18:03.869 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:18:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:18:03.865+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:18:04.121 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:18:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:18:03.906+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:03.907645+0000) 2024-02-20T09:18:04.122 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:18:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:18:04.022+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:04.023692+0000) 2024-02-20T09:18:04.122 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:18:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:18:04.119+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:18:04.122 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:18:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:18:04.119+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:18:04.122 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:18:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:18:04.119+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:18:04.492 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:18:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:18:04.287+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:18:04.492 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:18:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:18:04.287+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:18:04.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:18:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:18:04.287+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:18:04.880 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:18:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:18:04.559+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:04.561105+0000) 2024-02-20T09:18:04.881 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:18:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:18:04.878+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:18:05.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:18:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:18:04.906+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:04.907767+0000) 2024-02-20T09:18:05.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:18:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:18:04.878+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:18:05.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:18:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:18:04.878+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:18:05.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:18:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:18:04.878+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:18:05.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:18:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:18:05.022+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:05.023793+0000) 2024-02-20T09:18:05.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:18:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:18:05.106+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:18:05.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:18:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:18:05.106+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:18:05.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:18:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:18:05.106+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:18:05.552 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:18:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:18:05.263+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:18:05.553 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:18:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:18:05.263+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:18:05.553 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:18:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:18:05.263+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:18:05.897 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:18:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:18:05.561+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:05.562605+0000) 2024-02-20T09:18:06.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:18:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:18:05.906+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:05.907851+0000) 2024-02-20T09:18:06.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:18:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:18:05.920+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:18:06.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:18:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:18:05.920+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:18:06.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:18:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:18:05.920+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:18:06.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:18:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:18:05.920+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:18:06.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:18:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:18:06.022+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:06.023911+0000) 2024-02-20T09:18:06.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:18:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:18:06.107+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:18:06.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:18:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:18:06.107+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:18:06.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:18:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:18:06.107+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:18:06.553 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:18:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:18:06.256+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:06.257734+0000) 2024-02-20T09:18:06.553 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:18:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:18:06.261+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 65695 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:18:06.553 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:18:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:18:06.258+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:06.259520+0000) 2024-02-20T09:18:06.554 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:18:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:18:06.256+0000 7f8a5f86e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:06.258003+0000) 2024-02-20T09:18:06.554 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:18:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:18:06.263+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:18:06.554 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:18:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:18:06.263+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:18:06.554 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:18:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:18:06.263+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:18:06.897 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:18:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:18:06.561+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:06.562781+0000) 2024-02-20T09:18:07.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:18:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:18:06.906+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:06.907959+0000) 2024-02-20T09:18:07.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:18:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:18:06.934+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:18:07.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:18:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:18:06.934+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:18:07.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:18:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:18:06.934+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:18:07.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:18:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:18:06.934+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:18:07.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:18:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:18:07.022+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:07.024030+0000) 2024-02-20T09:18:07.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:18:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:18:07.078+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:18:07.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:18:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:18:07.078+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:18:07.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:18:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:18:07.078+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:18:07.553 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:18:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:18:07.264+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:18:07.553 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:18:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:18:07.264+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:18:07.553 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:18:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:18:07.264+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:18:07.897 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:18:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:18:07.564+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:07.565451+0000) 2024-02-20T09:18:08.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:18:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:18:07.906+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:07.908052+0000) 2024-02-20T09:18:08.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:18:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:18:07.956+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:18:08.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:18:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:18:07.956+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:18:08.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:18:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:18:07.956+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:18:08.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:18:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:18:07.956+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:18:08.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:18:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:18:08.022+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:08.024135+0000) 2024-02-20T09:18:08.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:18:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:18:08.090+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:18:08.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:18:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:18:08.090+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:18:08.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:18:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:18:08.090+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:18:08.553 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:18:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:18:08.274+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:18:08.553 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:18:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:18:08.274+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:18:08.553 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:18:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:18:08.274+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:18:08.897 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:18:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:18:08.566+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:08.567440+0000) 2024-02-20T09:18:09.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:18:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:18:08.906+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:08.908160+0000) 2024-02-20T09:18:09.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:18:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:18:09.022+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:09.024248+0000) 2024-02-20T09:18:09.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:18:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:18:09.050+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:18:09.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:18:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:18:09.050+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:18:09.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:18:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:18:09.050+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:18:09.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:18:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:18:08.920+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:18:09.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:18:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:18:08.920+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:18:09.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:18:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:18:08.920+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:18:09.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:18:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:18:08.920+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:18:09.553 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:18:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:18:09.244+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:18:09.553 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:18:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:18:09.244+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:18:09.553 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:18:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:18:09.244+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:18:09.885 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:18:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:18:09.568+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:09.569423+0000) 2024-02-20T09:18:10.230 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:18:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:18:09.906+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:09.908259+0000) 2024-02-20T09:18:10.230 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:18:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:18:10.227+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:18:10.231 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:18:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:18:10.022+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:10.024365+0000) 2024-02-20T09:18:10.231 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:18:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:18:10.039+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:18:10.231 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:18:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:18:10.039+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:18:10.231 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:18:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:18:10.039+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:18:10.231 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:18:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:18:09.950+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:18:10.231 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:18:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:18:09.950+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:18:10.232 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:18:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:18:09.950+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:18:10.232 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:18:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:18:09.950+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:18:10.492 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:18:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:18:10.227+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:18:10.492 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:18:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:18:10.227+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:18:10.897 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:18:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:18:10.569+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:10.570904+0000) 2024-02-20T09:18:11.205 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:18:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:18:10.932+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:18:11.206 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:18:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:18:10.932+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:18:11.206 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:18:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:18:10.932+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:18:11.206 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:18:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:18:10.932+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:18:11.206 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:18:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:18:10.907+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:10.908385+0000) 2024-02-20T09:18:11.206 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:18:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:18:11.203+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:18:11.207 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:18:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:18:11.203+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:18:11.207 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:18:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:18:11.203+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:18:11.207 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:18:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:18:10.993+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:18:11.207 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:18:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:18:10.993+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:18:11.207 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:18:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:18:10.993+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:18:11.207 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:18:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:18:11.023+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:11.024431+0000) 2024-02-20T09:18:11.492 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:18:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:18:11.269+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 65735 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:18:11.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:18:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:18:11.266+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:11.268073+0000) 2024-02-20T09:18:11.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:18:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:18:11.263+0000 7f8a5f86e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:11.264699+0000) 2024-02-20T09:18:11.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:18:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:18:11.263+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:11.264541+0000) 2024-02-20T09:18:11.898 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:18:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:18:11.569+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:11.571164+0000) 2024-02-20T09:18:12.216 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:18:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:18:11.907+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:11.908497+0000) 2024-02-20T09:18:12.217 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:18:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:18:11.946+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:18:12.217 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:18:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:18:11.946+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:18:12.217 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:18:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:18:11.946+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:18:12.217 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:18:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:18:12.023+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:12.024523+0000) 2024-02-20T09:18:12.218 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:18:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:18:11.963+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:18:12.218 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:18:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:18:11.963+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:18:12.218 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:18:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:18:11.963+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:18:12.218 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:18:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:18:11.963+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:18:12.492 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:18:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:18:12.230+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:18:12.492 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:18:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:18:12.230+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:18:12.492 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:18:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:18:12.230+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:18:12.898 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:18:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:18:12.573+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:12.574656+0000) 2024-02-20T09:18:13.205 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:18:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:18:12.907+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:12.908610+0000) 2024-02-20T09:18:13.206 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:18:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:18:13.203+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:18:13.206 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:18:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:18:13.203+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:18:13.206 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:18:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:18:12.983+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:18:13.206 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:18:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:18:12.983+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:18:13.206 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:18:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:18:12.983+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:18:13.207 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:18:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:18:13.023+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:13.024604+0000) 2024-02-20T09:18:13.207 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:18:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:18:12.916+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:18:13.207 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:18:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:18:12.916+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:18:13.207 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:18:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:18:12.916+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:18:13.207 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:18:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:18:12.916+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:18:13.492 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:18:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:18:13.203+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:18:13.898 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:18:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:18:13.576+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:13.578234+0000) 2024-02-20T09:18:14.159 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:18:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:18:13.907+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:13.908699+0000) 2024-02-20T09:18:14.160 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:18:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:18:14.157+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:18:14.160 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:18:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:18:13.902+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:18:14.160 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:18:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:18:13.902+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:18:14.160 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:18:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:18:13.902+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:18:14.160 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:18:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:18:13.902+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:18:14.161 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:18:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:18:14.005+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:18:14.161 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:18:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:18:14.005+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:18:14.161 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:18:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:18:14.005+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:18:14.161 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:18:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:18:14.023+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:14.024706+0000) 2024-02-20T09:18:14.492 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:18:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:18:14.157+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:18:14.492 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:18:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:18:14.157+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:18:14.873 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:18:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:18:14.579+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:14.580829+0000) 2024-02-20T09:18:14.874 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:18:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:18:14.871+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:18:14.874 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:18:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:18:14.871+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:18:14.874 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:18:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:18:14.871+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:18:14.874 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:18:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:18:14.871+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:18:15.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:18:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:18:14.907+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:14.908815+0000) 2024-02-20T09:18:15.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:18:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:18:15.118+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:18:15.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:18:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:18:15.118+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:18:15.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:18:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:18:15.118+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:18:15.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:18:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:18:14.977+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:18:15.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:18:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:18:14.977+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:18:15.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:18:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:18:14.977+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:18:15.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:18:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:18:15.023+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:15.024813+0000) 2024-02-20T09:18:15.898 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:18:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:18:15.581+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:15.582394+0000) 2024-02-20T09:18:16.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:18:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:18:15.907+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:18:16.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:18:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:18:15.907+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:18:16.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:18:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:18:15.907+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:18:16.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:18:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:18:15.907+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:18:16.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:18:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:18:15.907+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:15.908933+0000) 2024-02-20T09:18:16.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:18:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:18:16.105+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:18:16.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:18:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:18:16.105+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:18:16.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:18:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:18:16.105+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:18:16.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:18:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:18:15.929+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:18:16.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:18:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:18:15.929+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:18:16.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:18:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:18:15.929+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:18:16.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:18:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:18:16.023+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:16.024921+0000) 2024-02-20T09:18:16.554 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:18:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:18:16.271+0000 7f8a5f86e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:16.272424+0000) 2024-02-20T09:18:16.554 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:18:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:18:16.276+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 65774 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:18:16.555 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:18:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:18:16.272+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:16.274162+0000) 2024-02-20T09:18:16.555 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:18:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:18:16.270+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:16.272235+0000) 2024-02-20T09:18:16.898 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:18:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:18:16.581+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:16.582595+0000) 2024-02-20T09:18:17.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:18:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:18:16.907+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:16.909085+0000) 2024-02-20T09:18:17.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:18:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:18:17.128+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:18:17.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:18:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:18:17.128+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:18:17.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:18:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:18:17.128+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:18:17.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:18:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:18:16.914+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:18:17.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:18:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:18:16.914+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:18:17.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:18:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:18:16.914+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:18:17.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:18:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:18:17.023+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:17.025113+0000) 2024-02-20T09:18:17.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:18:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:18:16.898+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:18:17.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:18:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:18:16.898+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:18:17.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:18:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:18:16.898+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:18:17.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:18:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:18:16.898+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:18:17.899 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:18:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:18:17.583+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:17.584560+0000) 2024-02-20T09:18:18.155 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:18:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:18:17.907+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:17.909222+0000) 2024-02-20T09:18:18.155 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:18:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:18:17.903+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:18:18.156 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:18:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:18:17.903+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:18:18.156 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:18:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:18:17.903+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:18:18.156 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:18:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:18:17.903+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:18:18.156 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:18:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:18:17.952+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:18:18.156 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:18:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:18:17.952+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:18:18.156 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:18:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:18:17.952+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:18:18.156 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:18:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:18:18.023+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:18.025282+0000) 2024-02-20T09:18:18.492 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:18:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:18:18.152+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:18:18.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:18:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:18:18.152+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:18:18.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:18:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:18:18.152+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:18:18.898 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:18:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:18:18.585+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:18.586559+0000) 2024-02-20T09:18:19.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:18:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:18:18.908+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:18.909394+0000) 2024-02-20T09:18:19.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:18:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:18:19.106+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:18:19.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:18:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:18:19.106+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:18:19.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:18:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:18:19.106+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:18:19.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:18:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:18:18.947+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:18:19.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:18:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:18:18.947+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:18:19.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:18:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:18:18.947+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:18:19.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:18:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:18:18.948+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:18:19.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:18:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:18:19.001+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:18:19.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:18:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:18:19.001+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:18:19.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:18:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:18:19.001+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:18:19.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:18:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:18:19.024+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:19.025444+0000) 2024-02-20T09:18:19.885 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:18:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:18:19.587+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:19.588631+0000) 2024-02-20T09:18:20.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:18:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:18:19.970+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:18:20.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:18:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:18:19.970+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:18:20.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:18:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:18:19.970+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:18:20.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:18:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:18:19.970+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:18:20.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:18:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:18:20.024+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:20.025562+0000) 2024-02-20T09:18:20.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:18:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:18:20.043+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:18:20.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:18:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:18:20.043+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:18:20.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:18:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:18:20.043+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:18:20.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:18:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:18:19.908+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:19.909556+0000) 2024-02-20T09:18:20.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:18:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:18:20.100+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:18:20.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:18:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:18:20.100+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:18:20.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:18:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:18:20.100+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:18:20.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:18:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:18:20.588+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:20.590348+0000) 2024-02-20T09:18:21.152 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:18:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:18:20.908+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:20.909651+0000) 2024-02-20T09:18:21.153 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:18:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:18:20.995+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:18:21.153 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:18:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:18:20.995+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:18:21.153 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:18:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:18:20.995+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:18:21.153 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:18:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:18:21.024+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:21.025681+0000) 2024-02-20T09:18:21.153 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:18:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:18:20.961+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:18:21.154 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:18:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:18:20.961+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:18:21.154 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:18:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:18:20.961+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:18:21.154 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:18:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:18:20.961+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:18:21.403 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:18:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:18:21.284+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 65817 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:18:21.404 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:18:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:18:21.281+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:21.282613+0000) 2024-02-20T09:18:21.404 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:18:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:18:21.277+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:21.279389+0000) 2024-02-20T09:18:21.405 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:18:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:18:21.150+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:18:21.405 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:18:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:18:21.150+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:18:21.405 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:18:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:18:21.150+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:18:21.405 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:18:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:18:21.277+0000 7f8a5f86e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:21.279278+0000) 2024-02-20T09:18:21.658 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:18:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:18:21.589+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:21.590593+0000) 2024-02-20T09:18:22.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:18:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:18:21.908+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:21.909760+0000) 2024-02-20T09:18:22.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:18:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:18:22.108+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:18:22.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:18:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:18:22.108+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:18:22.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:18:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:18:22.108+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:18:22.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:18:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:18:22.010+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:18:22.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:18:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:18:22.010+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:18:22.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:18:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:18:22.010+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:18:22.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:18:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:18:22.010+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:18:22.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:18:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:18:22.005+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:18:22.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:18:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:18:22.005+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:18:22.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:18:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:18:22.005+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:18:22.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:18:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:18:22.024+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:22.025778+0000) 2024-02-20T09:18:22.899 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:18:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:18:22.591+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:22.592683+0000) 2024-02-20T09:18:23.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:18:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:18:22.908+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:22.909863+0000) 2024-02-20T09:18:23.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:18:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:18:23.116+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:18:23.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:18:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:18:23.116+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:18:23.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:18:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:18:23.116+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:18:23.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:18:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:18:23.024+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:23.025868+0000) 2024-02-20T09:18:23.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:18:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:18:23.039+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:18:23.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:18:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:18:23.039+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:18:23.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:18:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:18:23.039+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:18:23.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:18:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:18:23.007+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:18:23.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:18:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:18:23.007+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:18:23.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:18:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:18:23.007+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:18:23.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:18:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:18:23.007+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:18:23.899 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:18:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:18:23.594+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:23.596261+0000) 2024-02-20T09:18:24.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:18:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:18:23.975+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:18:24.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:18:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:18:23.975+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:18:24.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:18:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:18:23.975+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:18:24.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:18:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:18:23.975+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:18:24.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:18:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:18:24.011+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:18:24.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:18:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:18:24.011+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:18:24.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:18:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:18:24.011+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:18:24.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:18:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:18:24.024+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:24.026003+0000) 2024-02-20T09:18:24.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:18:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:18:23.908+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:23.910032+0000) 2024-02-20T09:18:24.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:18:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:18:24.121+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:18:24.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:18:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:18:24.121+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:18:24.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:18:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:18:24.121+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:18:24.899 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:18:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:18:24.596+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:24.598321+0000) 2024-02-20T09:18:25.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:18:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:18:24.908+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:24.910146+0000) 2024-02-20T09:18:25.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:18:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:18:25.140+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:18:25.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:18:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:18:25.140+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:18:25.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:18:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:18:25.140+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:18:25.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:18:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:18:24.942+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:18:25.245 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:18:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:18:24.942+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:18:25.245 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:18:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:18:24.942+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:18:25.245 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:18:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:18:24.942+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:18:25.246 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:18:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:18:24.991+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:18:25.246 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:18:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:18:24.991+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:18:25.246 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:18:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:18:24.991+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:18:25.246 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:18:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:18:25.024+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:25.026139+0000) 2024-02-20T09:18:25.899 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:18:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:18:25.598+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:25.599935+0000) 2024-02-20T09:18:26.192 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:18:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:18:25.949+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:18:26.192 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:18:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:18:25.949+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:18:26.192 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:18:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:18:25.949+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:18:26.193 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:18:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:18:26.024+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:26.026241+0000) 2024-02-20T09:18:26.193 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:18:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:18:25.913+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:18:26.193 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:18:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:18:25.913+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:18:26.193 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:18:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:18:25.913+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:18:26.193 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:18:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:18:25.913+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:18:26.194 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:18:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:18:25.908+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:25.910276+0000) 2024-02-20T09:18:26.194 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:18:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:18:26.189+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:18:26.194 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:18:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:18:26.189+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:18:26.194 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:18:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:18:26.189+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:18:26.492 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:18:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:18:26.285+0000 7f8a5f86e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:26.286489+0000) 2024-02-20T09:18:26.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:18:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:18:26.285+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:26.286744+0000) 2024-02-20T09:18:26.493 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:18:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:18:26.291+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 65856 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:18:26.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:18:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:18:26.287+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:26.288956+0000) 2024-02-20T09:18:26.899 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:18:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:18:26.598+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:26.600188+0000) 2024-02-20T09:18:27.195 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:18:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:18:26.909+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:26.910416+0000) 2024-02-20T09:18:27.196 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:18:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:18:27.193+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:18:27.196 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:18:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:18:26.941+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:18:27.196 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:18:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:18:26.941+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:18:27.196 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:18:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:18:26.941+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:18:27.197 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:18:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:18:26.941+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:18:27.197 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:18:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:18:26.963+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:18:27.197 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:18:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:18:26.963+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:18:27.197 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:18:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:18:26.963+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:18:27.197 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:18:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:18:27.024+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:27.026353+0000) 2024-02-20T09:18:27.492 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:18:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:18:27.193+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:18:27.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:18:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:18:27.193+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:18:27.899 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:18:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:18:27.600+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:27.602364+0000) 2024-02-20T09:18:28.153 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:18:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:18:27.909+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:27.910498+0000) 2024-02-20T09:18:28.153 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:18:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:18:28.150+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:18:28.154 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:18:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:18:27.985+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:18:28.154 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:18:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:18:27.985+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:18:28.154 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:18:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:18:27.985+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:18:28.154 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:18:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:18:27.985+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:18:28.154 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:18:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:18:28.004+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:18:28.154 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:18:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:18:28.004+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:18:28.155 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:18:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:18:28.004+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:18:28.155 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:18:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:18:28.025+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:28.026441+0000) 2024-02-20T09:18:28.492 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:18:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:18:28.150+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:18:28.492 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:18:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:18:28.150+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:18:28.899 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:18:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:18:28.603+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:28.605030+0000) 2024-02-20T09:18:29.186 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:18:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:18:29.025+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:29.026529+0000) 2024-02-20T09:18:29.187 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:18:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:18:29.044+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:18:29.187 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:18:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:18:29.044+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:18:29.187 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:18:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:18:29.044+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:18:29.187 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:18:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:18:28.909+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:28.910611+0000) 2024-02-20T09:18:29.187 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:18:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:18:29.184+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:18:29.188 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:18:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:18:29.184+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:18:29.188 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:18:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:18:29.184+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:18:29.188 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:18:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:18:28.964+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:18:29.188 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:18:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:18:28.964+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:18:29.188 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:18:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:18:28.964+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:18:29.189 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:18:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:18:28.964+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:18:29.886 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:18:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:18:29.607+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:29.608577+0000) 2024-02-20T09:18:30.223 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:18:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:18:29.909+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:29.910697+0000) 2024-02-20T09:18:30.224 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:18:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:18:30.221+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:18:30.224 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:18:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:18:30.010+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:18:30.224 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:18:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:18:30.010+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:18:30.224 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:18:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:18:30.010+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:18:30.225 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:18:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:18:30.010+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:18:30.225 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:18:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:18:30.003+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:18:30.225 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:18:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:18:30.003+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:18:30.225 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:18:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:18:30.003+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:18:30.225 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:18:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:18:30.025+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:30.026620+0000) 2024-02-20T09:18:30.492 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:18:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:18:30.221+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:18:30.492 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:18:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:18:30.221+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:18:30.900 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:18:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:18:30.608+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:30.610295+0000) 2024-02-20T09:18:31.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:18:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:18:30.910+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:30.910813+0000) 2024-02-20T09:18:31.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:18:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:18:31.025+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:31.026718+0000) 2024-02-20T09:18:31.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:18:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:18:31.039+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:18:31.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:18:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:18:31.039+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:18:31.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:18:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:18:31.039+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:18:31.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:18:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:18:31.029+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:18:31.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:18:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:18:31.029+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:18:31.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:18:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:18:31.029+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:18:31.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:18:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:18:31.029+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:18:31.556 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:18:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:18:31.299+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 65896 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:18:31.557 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:18:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:18:31.296+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:31.296433+0000) 2024-02-20T09:18:31.557 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:18:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:18:31.293+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:31.294391+0000) 2024-02-20T09:18:31.557 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:18:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:18:31.260+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:18:31.557 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:18:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:18:31.260+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:18:31.557 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:18:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:18:31.260+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:18:31.558 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:18:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:18:31.292+0000 7f8a5f86e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:31.294222+0000) 2024-02-20T09:18:31.900 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:18:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:18:31.610+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:31.610514+0000) 2024-02-20T09:18:32.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:18:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:18:31.910+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:31.910951+0000) 2024-02-20T09:18:32.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:18:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:18:32.016+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:18:32.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:18:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:18:32.016+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:18:32.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:18:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:18:32.016+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:18:32.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:18:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:18:32.026+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:32.026847+0000) 2024-02-20T09:18:32.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:18:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:18:32.079+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:18:32.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:18:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:18:32.079+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:18:32.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:18:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:18:32.079+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:18:32.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:18:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:18:32.079+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:18:32.556 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:18:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:18:32.303+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:18:32.556 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:18:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:18:32.303+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:18:32.556 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:18:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:18:32.303+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:18:32.900 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:18:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:18:32.612+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:32.612678+0000) 2024-02-20T09:18:33.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:18:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:18:32.910+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:32.911064+0000) 2024-02-20T09:18:33.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:18:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:18:33.118+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:18:33.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:18:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:18:33.118+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:18:33.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:18:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:18:33.118+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:18:33.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:18:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:18:33.118+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:18:33.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:18:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:18:32.998+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:18:33.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:18:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:18:32.998+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:18:33.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:18:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:18:32.998+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:18:33.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:18:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:18:33.026+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:33.026924+0000) 2024-02-20T09:18:33.556 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:18:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:18:33.279+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:18:33.556 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:18:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:18:33.279+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:18:33.557 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:18:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:18:33.279+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:18:33.900 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:18:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:18:33.614+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:33.614983+0000) 2024-02-20T09:18:34.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:18:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:18:33.910+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:33.911184+0000) 2024-02-20T09:18:34.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:18:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:18:34.001+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:18:34.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:18:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:18:34.001+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:18:34.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:18:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:18:34.001+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:18:34.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:18:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:18:34.026+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:34.027033+0000) 2024-02-20T09:18:34.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:18:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:18:34.068+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:18:34.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:18:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:18:34.068+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:18:34.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:18:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:18:34.068+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:18:34.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:18:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:18:34.069+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:18:34.556 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:18:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:18:34.281+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:18:34.556 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:18:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:18:34.281+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:18:34.557 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:18:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:18:34.281+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:18:34.900 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:18:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:18:34.616+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:34.616991+0000) 2024-02-20T09:18:35.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:18:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:18:34.910+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:34.911303+0000) 2024-02-20T09:18:35.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:18:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:18:35.110+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:18:35.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:18:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:18:35.110+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:18:35.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:18:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:18:35.110+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:18:35.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:18:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:18:35.110+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:18:35.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:18:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:18:35.020+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:18:35.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:18:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:18:35.020+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:18:35.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:18:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:18:35.020+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:18:35.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:18:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:18:35.026+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:35.027164+0000) 2024-02-20T09:18:35.620 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:18:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:18:35.316+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:18:35.620 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:18:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:18:35.316+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:18:35.620 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:18:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:18:35.316+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:18:35.900 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:18:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:18:35.618+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:35.618877+0000) 2024-02-20T09:18:36.152 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:18:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:18:35.911+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:35.911423+0000) 2024-02-20T09:18:36.153 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:18:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:18:36.026+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:36.027253+0000) 2024-02-20T09:18:36.153 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:18:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:18:36.064+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:18:36.153 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:18:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:18:36.064+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:18:36.153 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:18:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:18:36.064+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:18:36.426 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:18:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:18:36.301+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:36.301652+0000) 2024-02-20T09:18:36.426 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:18:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:18:36.307+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 65935 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:18:36.426 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:18:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:18:36.300+0000 7f8a5f86e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:36.301052+0000) 2024-02-20T09:18:36.427 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:18:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:18:36.328+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:18:36.427 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:18:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:18:36.328+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:18:36.427 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:18:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:18:36.328+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:18:36.427 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:18:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:18:36.151+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:18:36.427 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:18:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:18:36.151+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:18:36.427 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:18:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:18:36.151+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:18:36.428 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:18:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:18:36.151+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:18:36.428 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:18:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:18:36.304+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:36.304512+0000) 2024-02-20T09:18:36.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:18:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:18:36.618+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:36.619090+0000) 2024-02-20T09:18:37.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:18:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:18:36.911+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:36.911509+0000) 2024-02-20T09:18:37.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:18:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:18:37.121+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:18:37.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:18:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:18:37.121+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:18:37.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:18:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:18:37.121+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:18:37.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:18:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:18:37.121+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:18:37.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:18:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:18:37.027+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:37.027385+0000) 2024-02-20T09:18:37.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:18:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:18:37.072+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:18:37.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:18:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:18:37.072+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:18:37.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:18:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:18:37.072+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:18:37.660 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:18:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:18:37.620+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:37.621091+0000) 2024-02-20T09:18:37.683 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:18:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:18:37.371+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:18:37.683 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:18:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:18:37.371+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:18:37.683 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:18:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:18:37.371+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:18:38.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:18:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:18:37.911+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:37.911605+0000) 2024-02-20T09:18:38.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:18:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:18:38.098+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:18:38.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:18:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:18:38.098+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:18:38.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:18:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:18:38.098+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:18:38.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:18:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:18:38.099+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:18:38.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:18:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:18:38.027+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:38.027501+0000) 2024-02-20T09:18:38.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:18:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:18:38.107+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:18:38.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:18:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:18:38.107+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:18:38.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:18:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:18:38.107+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:18:38.624 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:18:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:18:38.328+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:18:38.625 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:18:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:18:38.328+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:18:38.625 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:18:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:18:38.328+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:18:38.901 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:18:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:18:38.623+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:38.623659+0000) 2024-02-20T09:18:39.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:18:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:18:38.911+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:38.911726+0000) 2024-02-20T09:18:39.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:18:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:18:39.027+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:39.027637+0000) 2024-02-20T09:18:39.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:18:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:18:39.071+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:18:39.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:18:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:18:39.071+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:18:39.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:18:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:18:39.071+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:18:39.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:18:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:18:39.147+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:18:39.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:18:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:18:39.147+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:18:39.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:18:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:18:39.147+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:18:39.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:18:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:18:39.147+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:18:39.627 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:18:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:18:39.356+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:18:39.627 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:18:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:18:39.356+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:18:39.628 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:18:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:18:39.356+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:18:39.885 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:18:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:18:39.625+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:39.626367+0000) 2024-02-20T09:18:40.178 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:18:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:18:39.911+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:39.911875+0000) 2024-02-20T09:18:40.179 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:18:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:18:40.027+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:40.027778+0000) 2024-02-20T09:18:40.179 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:18:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:18:40.048+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:18:40.179 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:18:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:18:40.048+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:18:40.179 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:18:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:18:40.048+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:18:40.440 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:18:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:18:40.331+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:18:40.440 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:18:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:18:40.331+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:18:40.441 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:18:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:18:40.331+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:18:40.441 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:18:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:18:40.177+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:18:40.441 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:18:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:18:40.177+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:18:40.441 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:18:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:18:40.177+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:18:40.441 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:18:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:18:40.177+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:18:40.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:18:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:18:40.627+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:40.628127+0000) 2024-02-20T09:18:41.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:18:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:18:40.911+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:40.912064+0000) 2024-02-20T09:18:41.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:18:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:18:41.142+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:18:41.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:18:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:18:41.142+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:18:41.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:18:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:18:41.142+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:18:41.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:18:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:18:41.142+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:18:41.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:18:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:18:41.018+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:18:41.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:18:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:18:41.018+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:18:41.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:18:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:18:41.018+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:18:41.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:18:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:18:41.027+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:41.027952+0000) 2024-02-20T09:18:41.629 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:18:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:18:41.314+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 65975 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:18:41.630 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:18:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:18:41.311+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:41.311841+0000) 2024-02-20T09:18:41.630 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:18:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:18:41.308+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:41.309238+0000) 2024-02-20T09:18:41.630 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:18:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:18:41.308+0000 7f8a5f86e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:41.309106+0000) 2024-02-20T09:18:41.631 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:18:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:18:41.364+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:18:41.631 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:18:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:18:41.364+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:18:41.631 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:18:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:18:41.364+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:18:41.901 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:18:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:18:41.628+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:41.628426+0000) 2024-02-20T09:18:42.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:18:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:18:41.911+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:41.912207+0000) 2024-02-20T09:18:42.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:18:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:18:41.977+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:18:42.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:18:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:18:41.977+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:18:42.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:18:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:18:41.977+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:18:42.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:18:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:18:42.027+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:42.028080+0000) 2024-02-20T09:18:42.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:18:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:18:42.147+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:18:42.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:18:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:18:42.147+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:18:42.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:18:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:18:42.147+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:18:42.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:18:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:18:42.147+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:18:42.661 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:18:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:18:42.630+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:42.631297+0000) 2024-02-20T09:18:42.661 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:18:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:18:42.401+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:18:42.662 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:18:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:18:42.401+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:18:42.662 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:18:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:18:42.401+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:18:43.183 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:18:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:18:42.911+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:42.912336+0000) 2024-02-20T09:18:43.184 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:18:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:18:42.951+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:18:43.184 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:18:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:18:42.951+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:18:43.184 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:18:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:18:42.951+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:18:43.184 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:18:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:18:43.027+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:43.028193+0000) 2024-02-20T09:18:43.492 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:18:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:18:43.422+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:18:43.492 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:18:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:18:43.422+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:18:43.492 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:18:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:18:43.422+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:18:43.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:18:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:18:43.182+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:18:43.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:18:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:18:43.182+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:18:43.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:18:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:18:43.182+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:18:43.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:18:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:18:43.182+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:18:43.901 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:18:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:18:43.634+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:43.634842+0000) 2024-02-20T09:18:44.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:18:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:18:43.912+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:43.912443+0000) 2024-02-20T09:18:44.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:18:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:18:44.143+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:18:44.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:18:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:18:44.143+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:18:44.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:18:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:18:44.143+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:18:44.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:18:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:18:44.143+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:18:44.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:18:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:18:43.956+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:18:44.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:18:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:18:43.956+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:18:44.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:18:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:18:43.956+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:18:44.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:18:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:18:44.027+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:44.028312+0000) 2024-02-20T09:18:44.661 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:18:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:18:44.636+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:44.636857+0000) 2024-02-20T09:18:44.662 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:18:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:18:44.403+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:18:44.662 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:18:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:18:44.403+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:18:44.663 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:18:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:18:44.403+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:18:45.182 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:18:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:18:44.912+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:44.912540+0000) 2024-02-20T09:18:45.182 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:18:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:18:44.988+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:18:45.182 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:18:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:18:44.988+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:18:45.183 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:18:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:18:44.988+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:18:45.183 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:18:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:18:45.028+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:45.028472+0000) 2024-02-20T09:18:45.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:18:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:18:45.180+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:18:45.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:18:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:18:45.180+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:18:45.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:18:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:18:45.180+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:18:45.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:18:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:18:45.180+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:18:45.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:18:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:18:45.385+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:18:45.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:18:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:18:45.385+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:18:45.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:18:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:18:45.385+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:18:45.901 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:18:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:18:45.638+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:45.638374+0000) 2024-02-20T09:18:46.207 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:18:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:18:45.912+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:45.912663+0000) 2024-02-20T09:18:46.207 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:18:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:18:45.995+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:18:46.208 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:18:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:18:45.995+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:18:46.208 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:18:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:18:45.995+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:18:46.208 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:18:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:18:46.028+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:46.028568+0000) 2024-02-20T09:18:46.492 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:18:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:18:46.321+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 66014 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:18:46.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:18:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:18:46.315+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:46.315676+0000) 2024-02-20T09:18:46.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:18:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:18:46.315+0000 7f8a5f86e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:46.315617+0000) 2024-02-20T09:18:46.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:18:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:18:46.367+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:18:46.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:18:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:18:46.367+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:18:46.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:18:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:18:46.367+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:18:46.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:18:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:18:46.206+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:18:46.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:18:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:18:46.206+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:18:46.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:18:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:18:46.206+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:18:46.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:18:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:18:46.206+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:18:46.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:18:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:18:46.318+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:46.318913+0000) 2024-02-20T09:18:46.901 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:18:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:18:46.638+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:46.638572+0000) 2024-02-20T09:18:47.233 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:18:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:18:46.912+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:46.912784+0000) 2024-02-20T09:18:47.234 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:18:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:18:46.981+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:18:47.234 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:18:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:18:46.981+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:18:47.234 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:18:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:18:46.981+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:18:47.234 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:18:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:18:47.028+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:47.028713+0000) 2024-02-20T09:18:47.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:18:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:18:47.232+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:18:47.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:18:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:18:47.232+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:18:47.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:18:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:18:47.232+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:18:47.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:18:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:18:47.232+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:18:47.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:18:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:18:47.387+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:18:47.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:18:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:18:47.387+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:18:47.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:18:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:18:47.387+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:18:47.902 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:18:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:18:47.640+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:47.640562+0000) 2024-02-20T09:18:48.210 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:18:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:18:47.912+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:47.912890+0000) 2024-02-20T09:18:48.211 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:18:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:18:48.018+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:18:48.211 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:18:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:18:48.018+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:18:48.211 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:18:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:18:48.018+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:18:48.212 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:18:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:18:48.028+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:48.028819+0000) 2024-02-20T09:18:48.492 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:18:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:18:48.414+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:18:48.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:18:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:18:48.414+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:18:48.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:18:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:18:48.414+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:18:48.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:18:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:18:48.208+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:18:48.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:18:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:18:48.208+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:18:48.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:18:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:18:48.208+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:18:48.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:18:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:18:48.208+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:18:48.902 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:18:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:18:48.642+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:48.642971+0000) 2024-02-20T09:18:49.239 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:18:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:18:48.912+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:48.913005+0000) 2024-02-20T09:18:49.239 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:18:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:18:49.028+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:49.028938+0000) 2024-02-20T09:18:49.239 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:18:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:18:49.054+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:18:49.239 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:18:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:18:49.054+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:18:49.240 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:18:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:18:49.054+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:18:49.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:18:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:18:49.237+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:18:49.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:18:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:18:49.237+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:18:49.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:18:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:18:49.237+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:18:49.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:18:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:18:49.237+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:18:49.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:18:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:18:49.406+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:18:49.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:18:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:18:49.406+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:18:49.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:18:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:18:49.406+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:18:49.885 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:18:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:18:49.644+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:49.644987+0000) 2024-02-20T09:18:50.225 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:18:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:18:49.912+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:49.913131+0000) 2024-02-20T09:18:50.226 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:18:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:18:50.028+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:50.029057+0000) 2024-02-20T09:18:50.226 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:18:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:18:50.073+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:18:50.226 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:18:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:18:50.073+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:18:50.226 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:18:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:18:50.073+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:18:50.492 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:18:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:18:50.394+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:18:50.492 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:18:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:18:50.394+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:18:50.492 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:18:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:18:50.394+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:18:50.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:18:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:18:50.224+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:18:50.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:18:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:18:50.224+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:18:50.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:18:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:18:50.224+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:18:50.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:18:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:18:50.224+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:18:50.902 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:18:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:18:50.646+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:50.646846+0000) 2024-02-20T09:18:51.220 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:18:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:18:50.912+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:50.913234+0000) 2024-02-20T09:18:51.221 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:18:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:18:51.028+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:51.029185+0000) 2024-02-20T09:18:51.221 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:18:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:18:51.102+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:18:51.221 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:18:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:18:51.102+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:18:51.221 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:18:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:18:51.102+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:18:51.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:18:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:18:51.322+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:51.323231+0000) 2024-02-20T09:18:51.492 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:18:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:18:51.329+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 66054 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:18:51.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:18:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:18:51.323+0000 7f8a5f86e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:51.323451+0000) 2024-02-20T09:18:51.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:18:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:18:51.384+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:18:51.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:18:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:18:51.384+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:18:51.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:18:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:18:51.384+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:18:51.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:18:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:18:51.219+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:18:51.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:18:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:18:51.219+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:18:51.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:18:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:18:51.219+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:18:51.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:18:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:18:51.219+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:18:51.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:18:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:18:51.327+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:51.327544+0000) 2024-02-20T09:18:51.902 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:18:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:18:51.646+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:51.647035+0000) 2024-02-20T09:18:52.152 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:18:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:18:51.912+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:51.913358+0000) 2024-02-20T09:18:52.153 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:18:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:18:52.028+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:52.029299+0000) 2024-02-20T09:18:52.153 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:18:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:18:52.146+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:18:52.153 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:18:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:18:52.146+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:18:52.153 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:18:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:18:52.146+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:18:52.405 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:18:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:18:52.173+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:18:52.405 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:18:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:18:52.173+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:18:52.405 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:18:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:18:52.173+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:18:52.406 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:18:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:18:52.173+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:18:52.662 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:18:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:18:52.647+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:52.649081+0000) 2024-02-20T09:18:52.662 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:18:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:18:52.403+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:18:52.662 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:18:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:18:52.403+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:18:52.662 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:18:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:18:52.403+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:18:53.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:18:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:18:52.912+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:52.913472+0000) 2024-02-20T09:18:53.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:18:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:18:53.156+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:18:53.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:18:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:18:53.156+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:18:53.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:18:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:18:53.156+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:18:53.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:18:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:18:53.156+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:18:53.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:18:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:18:53.028+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:53.029430+0000) 2024-02-20T09:18:53.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:18:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:18:53.150+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:18:53.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:18:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:18:53.150+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:18:53.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:18:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:18:53.150+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:18:53.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:18:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:18:53.650+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:53.651696+0000) 2024-02-20T09:18:53.742 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:18:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:18:53.425+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:18:53.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:18:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:18:53.425+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:18:53.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:18:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:18:53.425+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:18:54.154 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:18:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:18:53.912+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:53.913582+0000) 2024-02-20T09:18:54.155 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:18:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:18:54.028+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:54.029547+0000) 2024-02-20T09:18:54.492 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:18:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:18:54.380+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:18:54.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:18:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:18:54.380+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:18:54.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:18:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:18:54.380+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:18:54.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:18:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:18:54.152+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:18:54.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:18:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:18:54.152+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:18:54.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:18:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:18:54.152+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:18:54.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:18:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:18:54.152+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:18:54.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:18:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:18:54.192+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:18:54.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:18:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:18:54.192+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:18:54.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:18:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:18:54.192+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:18:54.902 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:18:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:18:54.652+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:54.654378+0000) 2024-02-20T09:18:55.188 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:18:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:18:54.912+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:54.913676+0000) 2024-02-20T09:18:55.188 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:18:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:18:55.028+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:55.029635+0000) 2024-02-20T09:18:55.189 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:18:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:18:55.185+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:18:55.189 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:18:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:18:55.185+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:18:55.189 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:18:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:18:55.143+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:18:55.189 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:18:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:18:55.143+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:18:55.189 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:18:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:18:55.143+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:18:55.189 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:18:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:18:55.143+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:18:55.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:18:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:18:55.185+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:18:55.492 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:18:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:18:55.351+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:18:55.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:18:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:18:55.351+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:18:55.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:18:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:18:55.351+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:18:55.902 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:18:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:18:55.654+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:55.656041+0000) 2024-02-20T09:18:56.166 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:18:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:18:55.912+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:55.913774+0000) 2024-02-20T09:18:56.166 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:18:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:18:56.028+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:56.029746+0000) 2024-02-20T09:18:56.167 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:18:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:18:56.164+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:18:56.167 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:18:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:18:56.164+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:18:56.167 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:18:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:18:56.164+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:18:56.456 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:18:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:18:56.336+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 66093 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:18:56.457 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:18:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:18:56.330+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:56.331546+0000) 2024-02-20T09:18:56.457 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:18:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:18:56.329+0000 7f8a5f86e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:56.331099+0000) 2024-02-20T09:18:56.457 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:18:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:18:56.375+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:18:56.457 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:18:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:18:56.375+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:18:56.458 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:18:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:18:56.375+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:18:56.458 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:18:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:18:56.193+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:18:56.458 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:18:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:18:56.193+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:18:56.458 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:18:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:18:56.193+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:18:56.458 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:18:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:18:56.193+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:18:56.458 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:18:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:18:56.332+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:56.333617+0000) 2024-02-20T09:18:56.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:18:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:18:56.654+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:56.656326+0000) 2024-02-20T09:18:57.155 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:18:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:18:56.912+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:56.913952+0000) 2024-02-20T09:18:57.155 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:18:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:18:57.145+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:18:57.155 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:18:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:18:57.145+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:18:57.156 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:18:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:18:57.145+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:18:57.156 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:18:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:18:57.145+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:18:57.156 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:18:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:18:57.028+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:57.029847+0000) 2024-02-20T09:18:57.156 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:18:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:18:57.152+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:18:57.156 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:18:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:18:57.152+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:18:57.156 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:18:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:18:57.152+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:18:57.492 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:18:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:18:57.386+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:18:57.492 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:18:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:18:57.386+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:18:57.492 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:18:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:18:57.386+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:18:57.903 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:18:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:18:57.657+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:57.658511+0000) 2024-02-20T09:18:58.158 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:18:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:18:57.912+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:57.914098+0000) 2024-02-20T09:18:58.158 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:18:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:18:58.028+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:58.029992+0000) 2024-02-20T09:18:58.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:18:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:18:58.182+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:18:58.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:18:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:18:58.182+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:18:58.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:18:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:18:58.182+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:18:58.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:18:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:18:58.155+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:18:58.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:18:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:18:58.155+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:18:58.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:18:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:18:58.155+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:18:58.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:18:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:18:58.155+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:18:58.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:18:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:18:58.402+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:18:58.494 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:18:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:18:58.402+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:18:58.494 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:18:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:18:58.402+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:18:58.903 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:18:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:18:58.659+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:58.660550+0000) 2024-02-20T09:18:59.173 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:18:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:18:59.028+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:59.030164+0000) 2024-02-20T09:18:59.174 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:18:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:18:58.912+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:58.914295+0000) 2024-02-20T09:18:59.492 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:18:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:18:59.405+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:18:59.492 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:18:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:18:59.405+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:18:59.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:18:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:18:59.405+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:18:59.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:18:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:18:59.215+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:18:59.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:18:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:18:59.215+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:18:59.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:18:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:18:59.215+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:18:59.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:18:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:18:59.171+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:18:59.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:18:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:18:59.171+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:18:59.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:18:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:18:59.171+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:18:59.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:18:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:18:59.171+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:18:59.885 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:18:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:18:59.661+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:59.662592+0000) 2024-02-20T09:19:00.142 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:18:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:18:59.913+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:18:59.914512+0000) 2024-02-20T09:19:00.142 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:19:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:19:00.028+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:00.030367+0000) 2024-02-20T09:19:00.442 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:19:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:19:00.139+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:19:00.442 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:19:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:19:00.139+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:19:00.442 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:19:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:19:00.139+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:19:00.442 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:19:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:19:00.139+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:19:00.443 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:19:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:19:00.233+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:19:00.443 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:19:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:19:00.233+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:19:00.443 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:19:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:19:00.233+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:19:00.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:19:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:19:00.662+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:00.664368+0000) 2024-02-20T09:19:00.742 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:19:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:19:00.447+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:19:00.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:19:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:19:00.447+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:19:00.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:19:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:19:00.447+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:19:01.197 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:19:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:19:00.913+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:00.914643+0000) 2024-02-20T09:19:01.198 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:19:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:19:01.125+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:19:01.198 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:19:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:19:01.125+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:19:01.198 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:19:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:19:01.125+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:19:01.198 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:19:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:19:01.125+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:19:01.198 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:19:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:19:01.029+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:01.030509+0000) 2024-02-20T09:19:01.198 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:19:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:19:01.195+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:19:01.198 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:19:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:19:01.195+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:19:01.199 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:19:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:19:01.195+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:19:01.464 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:19:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:19:01.343+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 66133 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:19:01.465 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:19:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:19:01.341+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:01.342959+0000) 2024-02-20T09:19:01.465 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:19:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:19:01.337+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:01.339065+0000) 2024-02-20T09:19:01.465 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:19:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:19:01.337+0000 7f8a5f86e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:01.339015+0000) 2024-02-20T09:19:01.465 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:19:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:19:01.407+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:19:01.466 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:19:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:19:01.407+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:19:01.466 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:19:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:19:01.407+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:19:01.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:19:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:19:01.663+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:01.664570+0000) 2024-02-20T09:19:02.185 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:19:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:19:01.913+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:01.914782+0000) 2024-02-20T09:19:02.186 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:19:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:19:02.147+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:19:02.186 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:19:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:19:02.147+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:19:02.186 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:19:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:19:02.147+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:19:02.186 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:19:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:19:02.147+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:19:02.186 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:19:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:19:02.029+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:02.030636+0000) 2024-02-20T09:19:02.187 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:19:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:19:02.182+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:19:02.187 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:19:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:19:02.183+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:19:02.187 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:19:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:19:02.183+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:19:02.492 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:19:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:19:02.419+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:19:02.492 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:19:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:19:02.419+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:19:02.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:19:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:19:02.419+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:19:02.903 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:19:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:19:02.665+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:02.667189+0000) 2024-02-20T09:19:03.183 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:19:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:19:02.913+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:02.914867+0000) 2024-02-20T09:19:03.183 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:19:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:19:03.029+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:03.030718+0000) 2024-02-20T09:19:03.447 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:19:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:19:03.198+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:19:03.447 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:19:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:19:03.198+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:19:03.447 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:19:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:19:03.198+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:19:03.448 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:19:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:19:03.180+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:19:03.448 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:19:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:19:03.180+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:19:03.448 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:19:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:19:03.180+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:19:03.448 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:19:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:19:03.180+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:19:03.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:19:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:19:03.667+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:03.669286+0000) 2024-02-20T09:19:03.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:19:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:19:03.444+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:19:03.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:19:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:19:03.444+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:19:03.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:19:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:19:03.444+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:19:04.156 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:19:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:19:03.913+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:03.914979+0000) 2024-02-20T09:19:04.157 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:19:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:19:04.029+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:04.030853+0000) 2024-02-20T09:19:04.157 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:19:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:19:04.153+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:19:04.157 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:19:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:19:04.153+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:19:04.440 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:19:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:19:04.153+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:19:04.441 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:19:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:19:04.155+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:19:04.441 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:19:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:19:04.155+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:19:04.441 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:19:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:19:04.155+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:19:04.441 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:19:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:19:04.155+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:19:04.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:19:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:19:04.669+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:04.671369+0000) 2024-02-20T09:19:04.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:19:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:19:04.488+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:19:04.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:19:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:19:04.488+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:19:04.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:19:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:19:04.488+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:19:05.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:19:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:19:04.913+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:04.915122+0000) 2024-02-20T09:19:05.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:19:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:19:05.029+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:05.030945+0000) 2024-02-20T09:19:05.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:19:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:19:05.128+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:19:05.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:19:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:19:05.128+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:19:05.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:19:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:19:05.128+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:19:05.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:19:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:19:05.142+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:19:05.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:19:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:19:05.142+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:19:05.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:19:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:19:05.142+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:19:05.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:19:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:19:05.142+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:19:05.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:19:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:19:05.672+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:05.674075+0000) 2024-02-20T09:19:05.742 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:19:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:19:05.448+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:19:05.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:19:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:19:05.448+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:19:05.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:19:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:19:05.448+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:19:06.176 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:19:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:19:05.913+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:05.915219+0000) 2024-02-20T09:19:06.176 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:19:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:19:06.029+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:06.031086+0000) 2024-02-20T09:19:06.176 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:19:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:19:06.173+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:19:06.447 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:19:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:19:06.351+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 66172 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:19:06.447 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:19:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:19:06.173+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:19:06.448 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:19:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:19:06.173+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:19:06.448 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:19:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:19:06.345+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:06.346799+0000) 2024-02-20T09:19:06.448 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:19:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:19:06.183+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:19:06.448 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:19:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:19:06.183+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:19:06.448 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:19:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:19:06.183+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:19:06.449 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:19:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:19:06.183+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:19:06.449 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:19:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:19:06.348+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:06.349866+0000) 2024-02-20T09:19:06.449 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:19:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:19:06.345+0000 7f8a5f86e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:06.346605+0000) 2024-02-20T09:19:06.449 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:19:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:19:06.444+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:19:06.449 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:19:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:19:06.444+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:19:06.449 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:19:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:19:06.444+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:19:06.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:19:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:19:06.672+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:06.674270+0000) 2024-02-20T09:19:07.207 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:19:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:19:06.913+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:06.915401+0000) 2024-02-20T09:19:07.208 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:19:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:19:07.029+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:07.031171+0000) 2024-02-20T09:19:07.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:19:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:19:07.208+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:19:07.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:19:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:19:07.208+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:19:07.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:19:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:19:07.208+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:19:07.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:19:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:19:07.402+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:19:07.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:19:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:19:07.402+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:19:07.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:19:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:19:07.402+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:19:07.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:19:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:19:07.204+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:19:07.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:19:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:19:07.204+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:19:07.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:19:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:19:07.204+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:19:07.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:19:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:19:07.205+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:19:07.904 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:19:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:19:07.675+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:07.676742+0000) 2024-02-20T09:19:08.191 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:19:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:19:08.029+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:08.031345+0000) 2024-02-20T09:19:08.192 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:19:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:19:07.914+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:07.915473+0000) 2024-02-20T09:19:08.492 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:19:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:19:08.372+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:19:08.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:19:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:19:08.372+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:19:08.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:19:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:19:08.372+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:19:08.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:19:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:19:08.230+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:19:08.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:19:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:19:08.230+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:19:08.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:19:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:19:08.230+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:19:08.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:19:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:19:08.188+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:19:08.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:19:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:19:08.188+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:19:08.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:19:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:19:08.188+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:19:08.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:19:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:19:08.189+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:19:08.904 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:19:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:19:08.678+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:08.680237+0000) 2024-02-20T09:19:09.183 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:19:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:19:08.914+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:08.915567+0000) 2024-02-20T09:19:09.184 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:19:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:19:09.030+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:09.031459+0000) 2024-02-20T09:19:09.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:19:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:19:09.181+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:19:09.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:19:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:19:09.181+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:19:09.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:19:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:19:09.181+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:19:09.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:19:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:19:09.181+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:19:09.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:19:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:19:09.402+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:19:09.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:19:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:19:09.402+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:19:09.494 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:19:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:19:09.402+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:19:09.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:19:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:19:09.200+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:19:09.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:19:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:19:09.200+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:19:09.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:19:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:19:09.200+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:19:09.886 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:19:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:19:09.681+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:09.682734+0000) 2024-02-20T09:19:10.216 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:19:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:19:09.914+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:09.915674+0000) 2024-02-20T09:19:10.217 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:19:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:19:10.030+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:10.031591+0000) 2024-02-20T09:19:10.492 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:19:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:19:10.395+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:19:10.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:19:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:19:10.395+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:19:10.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:19:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:19:10.395+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:19:10.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:19:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:19:10.217+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:19:10.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:19:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:19:10.217+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:19:10.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:19:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:19:10.217+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:19:10.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:19:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:19:10.214+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:19:10.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:19:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:19:10.214+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:19:10.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:19:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:19:10.214+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:19:10.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:19:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:19:10.214+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:19:10.905 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:19:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:19:10.684+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:10.685541+0000) 2024-02-20T09:19:11.218 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:19:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:19:10.914+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:10.915787+0000) 2024-02-20T09:19:11.219 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:19:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:19:11.030+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:11.031702+0000) 2024-02-20T09:19:11.219 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:19:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:19:11.216+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:19:11.219 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:19:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:19:11.216+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:19:11.219 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:19:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:19:11.216+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:19:11.479 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:19:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:19:11.358+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 66212 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:19:11.479 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:19:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:19:11.353+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:11.354489+0000) 2024-02-20T09:19:11.479 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:19:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:19:11.352+0000 7f8a5f86e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:11.354239+0000) 2024-02-20T09:19:11.480 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:19:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:19:11.422+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:19:11.480 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:19:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:19:11.422+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:19:11.480 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:19:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:19:11.422+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:19:11.480 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:19:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:19:11.252+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:19:11.480 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:19:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:19:11.252+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:19:11.480 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:19:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:19:11.252+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:19:11.481 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:19:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:19:11.252+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:19:11.481 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:19:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:19:11.356+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:11.357881+0000) 2024-02-20T09:19:11.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:19:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:19:11.684+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:11.685753+0000) 2024-02-20T09:19:12.185 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:19:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:19:11.914+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:11.915879+0000) 2024-02-20T09:19:12.185 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:19:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:19:12.030+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:12.031816+0000) 2024-02-20T09:19:12.185 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:19:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:19:12.182+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:19:12.492 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:19:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:19:12.428+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:19:12.492 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:19:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:19:12.428+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:19:12.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:19:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:19:12.428+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:19:12.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:19:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:19:12.182+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:19:12.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:19:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:19:12.182+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:19:12.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:19:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:19:12.290+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:19:12.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:19:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:19:12.290+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:19:12.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:19:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:19:12.290+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:19:12.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:19:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:19:12.290+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:19:12.905 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:19:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:19:12.687+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:12.689162+0000) 2024-02-20T09:19:13.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:19:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:19:12.914+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:12.916035+0000) 2024-02-20T09:19:13.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:19:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:19:13.030+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:13.031910+0000) 2024-02-20T09:19:13.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:19:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:19:13.139+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:19:13.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:19:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:19:13.139+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:19:13.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:19:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:19:13.139+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:19:13.656 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:19:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:19:13.478+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:19:13.657 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:19:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:19:13.478+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:19:13.657 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:19:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:19:13.478+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:19:13.657 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:19:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:19:13.326+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:19:13.657 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:19:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:19:13.326+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:19:13.657 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:19:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:19:13.326+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:19:13.658 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:19:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:19:13.326+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:19:13.917 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:19:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:19:13.690+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:13.691666+0000) 2024-02-20T09:19:14.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:19:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:19:13.914+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:13.916243+0000) 2024-02-20T09:19:14.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:19:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:19:14.030+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:14.032111+0000) 2024-02-20T09:19:14.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:19:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:19:14.145+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:19:14.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:19:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:19:14.145+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:19:14.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:19:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:19:14.145+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:19:14.665 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:19:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:19:14.344+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:19:14.665 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:19:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:19:14.344+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:19:14.665 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:19:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:19:14.344+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:19:14.665 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:19:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:19:14.344+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:19:14.666 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:19:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:19:14.510+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:19:14.666 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:19:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:19:14.510+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:19:14.666 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:19:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:19:14.510+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:19:14.917 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:19:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:19:14.692+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:14.694086+0000) 2024-02-20T09:19:15.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:19:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:19:14.915+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:14.916425+0000) 2024-02-20T09:19:15.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:19:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:19:15.030+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:15.032315+0000) 2024-02-20T09:19:15.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:19:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:19:15.151+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:19:15.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:19:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:19:15.151+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:19:15.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:19:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:19:15.151+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:19:15.664 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:19:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:19:15.387+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:19:15.664 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:19:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:19:15.387+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:19:15.665 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:19:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:19:15.387+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:19:15.665 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:19:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:19:15.387+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:19:15.665 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:19:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:19:15.506+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:19:15.665 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:19:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:19:15.506+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:19:15.665 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:19:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:19:15.506+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:19:15.917 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:19:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:19:15.694+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:15.696208+0000) 2024-02-20T09:19:16.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:19:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:19:15.915+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:15.916556+0000) 2024-02-20T09:19:16.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:19:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:19:16.031+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:16.032513+0000) 2024-02-20T09:19:16.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:19:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:19:16.121+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:19:16.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:19:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:19:16.121+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:19:16.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:19:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:19:16.121+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:19:16.664 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:19:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:19:16.366+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 66251 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:19:16.665 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:19:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:19:16.360+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:16.361901+0000) 2024-02-20T09:19:16.665 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:19:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:19:16.360+0000 7f8a5f86e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:16.361757+0000) 2024-02-20T09:19:16.665 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:19:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:19:16.469+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:19:16.665 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:19:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:19:16.469+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:19:16.665 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:19:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:19:16.469+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:19:16.666 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:19:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:19:16.354+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:19:16.666 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:19:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:19:16.354+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:19:16.666 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:19:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:19:16.354+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:19:16.666 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:19:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:19:16.354+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:19:16.666 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:19:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:19:16.362+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:16.364178+0000) 2024-02-20T09:19:16.917 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:19:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:19:16.695+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:16.696457+0000) 2024-02-20T09:19:17.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:19:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:19:16.915+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:16.916703+0000) 2024-02-20T09:19:17.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:19:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:19:17.031+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:17.032705+0000) 2024-02-20T09:19:17.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:19:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:19:17.078+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:19:17.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:19:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:19:17.078+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:19:17.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:19:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:19:17.078+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:19:17.664 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:19:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:19:17.430+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:19:17.665 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:19:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:19:17.430+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:19:17.665 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:19:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:19:17.430+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:19:17.665 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:19:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:19:17.386+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:19:17.665 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:19:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:19:17.386+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:19:17.665 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:19:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:19:17.386+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:19:17.665 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:19:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:19:17.386+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:19:17.917 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:19:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:19:17.698+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:17.699992+0000) 2024-02-20T09:19:18.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:19:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:19:17.915+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:17.916784+0000) 2024-02-20T09:19:18.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:19:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:19:18.029+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:19:18.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:19:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:19:18.029+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:19:18.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:19:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:19:18.029+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:19:18.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:19:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:19:18.031+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:18.032819+0000) 2024-02-20T09:19:18.664 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:19:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:19:18.391+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:19:18.665 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:19:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:19:18.391+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:19:18.665 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:19:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:19:18.391+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:19:18.665 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:19:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:19:18.352+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:19:18.665 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:19:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:19:18.352+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:19:18.665 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:19:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:19:18.352+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:19:18.666 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:19:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:19:18.352+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:19:18.917 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:19:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:19:18.702+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:18.703536+0000) 2024-02-20T09:19:19.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:19:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:19:18.915+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:18.916890+0000) 2024-02-20T09:19:19.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:19:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:19:18.994+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:19:19.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:19:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:19:18.994+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:19:19.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:19:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:19:18.994+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:19:19.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:19:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:19:19.031+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:19.032944+0000) 2024-02-20T09:19:19.665 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:19:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:19:19.432+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:19:19.665 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:19:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:19:19.432+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:19:19.665 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:19:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:19:19.432+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:19:19.665 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:19:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:19:19.375+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:19:19.665 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:19:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:19:19.375+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:19:19.666 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:19:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:19:19.375+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:19:19.666 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:19:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:19:19.375+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:19:19.918 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:19:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:19:19.705+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:19.707154+0000) 2024-02-20T09:19:20.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:19:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:19:19.915+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:19.917003+0000) 2024-02-20T09:19:20.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:19:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:19:20.031+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:20.033070+0000) 2024-02-20T09:19:20.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:19:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:19:20.034+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:19:20.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:19:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:19:20.034+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:19:20.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:19:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:19:20.034+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:19:20.665 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:19:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:19:20.425+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:19:20.665 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:19:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:19:20.425+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:19:20.665 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:19:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:19:20.425+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:19:20.665 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:19:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:19:20.358+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:19:20.666 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:19:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:19:20.358+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:19:20.666 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:19:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:19:20.358+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:19:20.666 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:19:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:19:20.358+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:19:20.918 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:19:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:19:20.708+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:20.710001+0000) 2024-02-20T09:19:21.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:19:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:19:20.915+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:20.917135+0000) 2024-02-20T09:19:21.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:19:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:19:21.031+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:21.033203+0000) 2024-02-20T09:19:21.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:19:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:19:21.055+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:19:21.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:19:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:19:21.055+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:19:21.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:19:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:19:21.055+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:19:21.665 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:19:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:19:21.373+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 66291 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:19:21.665 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:19:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:19:21.367+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:21.369293+0000) 2024-02-20T09:19:21.666 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:19:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:19:21.367+0000 7f8a5f86e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:21.369092+0000) 2024-02-20T09:19:21.666 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:19:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:19:21.389+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:19:21.666 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:19:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:19:21.390+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:19:21.666 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:19:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:19:21.390+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:19:21.666 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:19:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:19:21.325+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:19:21.666 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:19:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:19:21.325+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:19:21.667 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:19:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:19:21.325+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:19:21.667 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:19:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:19:21.325+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:19:21.667 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:19:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:19:21.371+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:21.372658+0000) 2024-02-20T09:19:21.918 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:19:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:19:21.709+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:21.710203+0000) 2024-02-20T09:19:22.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:19:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:19:21.916+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:21.917274+0000) 2024-02-20T09:19:22.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:19:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:19:22.031+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:22.033346+0000) 2024-02-20T09:19:22.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:19:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:19:22.033+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:19:22.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:19:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:19:22.033+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:19:22.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:19:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:19:22.033+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:19:22.665 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:19:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:19:22.417+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:19:22.665 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:19:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:19:22.417+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:19:22.665 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:19:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:19:22.417+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:19:22.666 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:19:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:19:22.342+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:19:22.666 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:19:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:19:22.342+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:19:22.666 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:19:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:19:22.342+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:19:22.666 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:19:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:19:22.342+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:19:22.918 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:19:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:19:22.710+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:22.712266+0000) 2024-02-20T09:19:23.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:19:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:19:22.917+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:22.917476+0000) 2024-02-20T09:19:23.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:19:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:19:23.022+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:19:23.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:19:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:19:23.022+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:19:23.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:19:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:19:23.022+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:19:23.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:19:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:19:23.032+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:23.033460+0000) 2024-02-20T09:19:23.665 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:19:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:19:23.367+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:19:23.666 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:19:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:19:23.367+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:19:23.666 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:19:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:19:23.367+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:19:23.666 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:19:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:19:23.376+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:19:23.666 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:19:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:19:23.376+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:19:23.666 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:19:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:19:23.376+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:19:23.666 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:19:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:19:23.376+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:19:23.918 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:19:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:19:23.712+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:23.714261+0000) 2024-02-20T09:19:24.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:19:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:19:23.916+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:23.917615+0000) 2024-02-20T09:19:24.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:19:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:19:23.976+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:19:24.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:19:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:19:23.976+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:19:24.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:19:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:19:23.976+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:19:24.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:19:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:19:24.032+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:24.033554+0000) 2024-02-20T09:19:24.665 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:19:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:19:24.322+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:19:24.665 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:19:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:19:24.322+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:19:24.666 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:19:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:19:24.322+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:19:24.666 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:19:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:19:24.420+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:19:24.666 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:19:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:19:24.420+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:19:24.666 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:19:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:19:24.420+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:19:24.666 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:19:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:19:24.420+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:19:24.918 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:19:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:19:24.714+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:24.716360+0000) 2024-02-20T09:19:25.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:19:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:19:24.917+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:24.917726+0000) 2024-02-20T09:19:25.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:19:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:19:24.959+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:19:25.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:19:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:19:24.959+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:19:25.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:19:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:19:24.959+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:19:25.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:19:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:19:25.033+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:25.033689+0000) 2024-02-20T09:19:25.665 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:19:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:19:25.352+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:19:25.665 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:19:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:19:25.352+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:19:25.666 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:19:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:19:25.352+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:19:25.666 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:19:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:19:25.466+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:19:25.666 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:19:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:19:25.466+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:19:25.666 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:19:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:19:25.466+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:19:25.666 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:19:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:19:25.467+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:19:25.919 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:19:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:19:25.717+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:25.718116+0000) 2024-02-20T09:19:25.919 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:19:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:19:25.910+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:19:25.919 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:19:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:19:25.910+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:19:25.919 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:19:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:19:25.910+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:19:26.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:19:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:19:25.917+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:25.917842+0000) 2024-02-20T09:19:26.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:19:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:19:26.033+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:26.033805+0000) 2024-02-20T09:19:26.665 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:19:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:19:26.382+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 66330 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:19:26.666 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:19:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:19:26.376+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:26.376578+0000) 2024-02-20T09:19:26.666 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:19:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:19:26.379+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:26.379418+0000) 2024-02-20T09:19:26.666 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:19:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:19:26.452+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:19:26.667 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:19:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:19:26.452+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:19:26.667 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:19:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:19:26.452+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:19:26.667 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:19:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:19:26.452+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:19:26.667 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:19:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:19:26.315+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:19:26.667 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:19:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:19:26.315+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:19:26.667 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:19:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:19:26.315+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:19:26.667 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:19:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:19:26.376+0000 7f8a5f86e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:26.376523+0000) 2024-02-20T09:19:26.919 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:19:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:19:26.717+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:26.718336+0000) 2024-02-20T09:19:27.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:19:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:19:26.917+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:26.917929+0000) 2024-02-20T09:19:27.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:19:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:19:26.934+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:19:27.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:19:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:19:26.934+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:19:27.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:19:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:19:26.934+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:19:27.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:19:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:19:27.033+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:27.033908+0000) 2024-02-20T09:19:27.563 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:19:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:19:27.272+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:19:27.563 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:19:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:19:27.273+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:19:27.564 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:19:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:19:27.273+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:19:27.564 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:19:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:19:27.410+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:19:27.564 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:19:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:19:27.410+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:19:27.564 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:19:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:19:27.410+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:19:27.564 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:19:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:19:27.410+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:19:27.904 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:19:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:19:27.721+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:27.721814+0000) 2024-02-20T09:19:28.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:19:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:19:27.917+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:27.918005+0000) 2024-02-20T09:19:28.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:19:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:19:27.903+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:19:28.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:19:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:19:27.903+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:19:28.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:19:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:19:27.903+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:19:28.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:19:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:19:28.033+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:28.034023+0000) 2024-02-20T09:19:28.563 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:19:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:19:28.249+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:19:28.563 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:19:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:19:28.249+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:19:28.564 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:19:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:19:28.249+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:19:28.564 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:19:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:19:28.434+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:19:28.564 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:19:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:19:28.434+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:19:28.564 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:19:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:19:28.434+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:19:28.564 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:19:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:19:28.434+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:19:28.907 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:19:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:19:28.724+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:28.725347+0000) 2024-02-20T09:19:29.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:19:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:19:28.917+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:28.918139+0000) 2024-02-20T09:19:29.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:19:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:19:28.905+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:19:29.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:19:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:19:28.905+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:19:29.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:19:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:19:28.905+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:19:29.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:19:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:19:29.033+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:29.034115+0000) 2024-02-20T09:19:29.563 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:19:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:19:29.483+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:19:29.564 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:19:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:19:29.483+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:19:29.564 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:19:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:19:29.483+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:19:29.564 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:19:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:19:29.483+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:19:29.564 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:19:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:19:29.297+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:19:29.564 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:19:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:19:29.297+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:19:29.565 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:19:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:19:29.297+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:19:29.888 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:19:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:19:29.728+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:29.728927+0000) 2024-02-20T09:19:30.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:19:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:19:29.917+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:29.918254+0000) 2024-02-20T09:19:30.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:19:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:19:29.913+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:19:30.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:19:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:19:29.913+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:19:30.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:19:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:19:29.913+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:19:30.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:19:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:19:30.033+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:30.034199+0000) 2024-02-20T09:19:30.511 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:19:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:19:30.253+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:19:30.511 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:19:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:19:30.253+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:19:30.511 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:19:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:19:30.253+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:19:30.907 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:19:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:19:30.509+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:19:30.907 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:19:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:19:30.509+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:19:30.907 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:19:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:19:30.509+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:19:30.908 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:19:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:19:30.509+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:19:30.908 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:19:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:19:30.731+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:30.731791+0000) 2024-02-20T09:19:31.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:19:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:19:30.917+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:30.918387+0000) 2024-02-20T09:19:31.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:19:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:19:30.931+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:19:31.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:19:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:19:30.931+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:19:31.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:19:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:19:30.931+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:19:31.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:19:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:19:31.033+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:31.034325+0000) 2024-02-20T09:19:31.564 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:19:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:19:31.383+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:31.384272+0000) 2024-02-20T09:19:31.564 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:19:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:19:31.389+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 66370 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:19:31.564 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:19:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:19:31.387+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:31.387416+0000) 2024-02-20T09:19:31.565 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:19:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:19:31.504+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:19:31.565 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:19:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:19:31.504+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:19:31.565 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:19:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:19:31.504+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:19:31.565 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:19:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:19:31.504+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:19:31.565 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:19:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:19:31.300+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:19:31.565 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:19:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:19:31.300+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:19:31.565 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:19:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:19:31.300+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:19:31.566 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:19:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:19:31.383+0000 7f8a5f86e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:31.383635+0000) 2024-02-20T09:19:31.901 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:19:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:19:31.731+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:31.732021+0000) 2024-02-20T09:19:32.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:19:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:19:31.918+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:31.918500+0000) 2024-02-20T09:19:32.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:19:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:19:31.899+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:19:32.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:19:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:19:31.899+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:19:32.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:19:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:19:31.899+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:19:32.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:19:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:19:32.034+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:32.034435+0000) 2024-02-20T09:19:32.564 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:19:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:19:32.512+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:19:32.564 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:19:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:19:32.512+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:19:32.564 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:19:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:19:32.512+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:19:32.564 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:19:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:19:32.512+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:19:32.565 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:19:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:19:32.291+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:19:32.565 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:19:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:19:32.291+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:19:32.565 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:19:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:19:32.291+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:19:32.907 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:19:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:19:32.733+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:32.733981+0000) 2024-02-20T09:19:33.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:19:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:19:32.918+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:32.918597+0000) 2024-02-20T09:19:33.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:19:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:19:32.939+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:19:33.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:19:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:19:32.939+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:19:33.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:19:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:19:32.939+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:19:33.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:19:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:19:33.034+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:33.034515+0000) 2024-02-20T09:19:33.564 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:19:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:19:33.309+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:19:33.564 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:19:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:19:33.309+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:19:33.564 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:19:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:19:33.309+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:19:33.565 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:19:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:19:33.549+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:19:33.565 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:19:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:19:33.549+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:19:33.565 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:19:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:19:33.549+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:19:33.565 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:19:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:19:33.549+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:19:33.907 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:19:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:19:33.735+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:33.736174+0000) 2024-02-20T09:19:34.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:19:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:19:33.918+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:33.918704+0000) 2024-02-20T09:19:34.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:19:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:19:33.915+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:19:34.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:19:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:19:33.915+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:19:34.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:19:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:19:33.915+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:19:34.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:19:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:19:34.034+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:34.034604+0000) 2024-02-20T09:19:34.530 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:19:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:19:34.275+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:19:34.530 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:19:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:19:34.275+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:19:34.530 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:19:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:19:34.275+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:19:34.907 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:19:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:19:34.528+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:19:34.908 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:19:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:19:34.528+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:19:34.908 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:19:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:19:34.528+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:19:34.908 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:19:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:19:34.528+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:19:34.908 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:19:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:19:34.737+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:34.738175+0000) 2024-02-20T09:19:35.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:19:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:19:34.918+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:34.918814+0000) 2024-02-20T09:19:35.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:19:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:19:34.915+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:19:35.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:19:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:19:34.915+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:19:35.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:19:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:19:34.915+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:19:35.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:19:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:19:35.034+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:35.034689+0000) 2024-02-20T09:19:35.564 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:19:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:19:35.274+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:19:35.564 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:19:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:19:35.274+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:19:35.564 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:19:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:19:35.274+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:19:35.907 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:19:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:19:35.574+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:19:35.908 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:19:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:19:35.574+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:19:35.908 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:19:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:19:35.574+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:19:35.908 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:19:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:19:35.574+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:19:35.908 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:19:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:19:35.739+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:35.739861+0000) 2024-02-20T09:19:36.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:19:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:19:35.918+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:35.918926+0000) 2024-02-20T09:19:36.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:19:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:19:35.954+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:19:36.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:19:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:19:35.954+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:19:36.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:19:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:19:35.954+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:19:36.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:19:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:19:36.034+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:36.034778+0000) 2024-02-20T09:19:36.517 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:19:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:19:36.394+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:36.394991+0000) 2024-02-20T09:19:36.518 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:19:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:19:36.391+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:36.391791+0000) 2024-02-20T09:19:36.518 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:19:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:19:36.397+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 66409 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:19:36.519 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:19:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:19:36.250+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:19:36.519 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:19:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:19:36.250+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:19:36.519 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:19:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:19:36.250+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:19:36.519 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:19:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:19:36.391+0000 7f8a5f86e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:36.391709+0000) 2024-02-20T09:19:36.908 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:19:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:19:36.535+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:19:36.908 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:19:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:19:36.535+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:19:36.908 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:19:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:19:36.535+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:19:36.908 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:19:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:19:36.535+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:19:36.908 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:19:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:19:36.739+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:36.740106+0000) 2024-02-20T09:19:37.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:19:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:19:36.918+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:36.919046+0000) 2024-02-20T09:19:37.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:19:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:19:36.941+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:19:37.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:19:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:19:36.941+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:19:37.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:19:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:19:36.941+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:19:37.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:19:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:19:37.034+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:37.034868+0000) 2024-02-20T09:19:37.564 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:19:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:19:37.293+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:19:37.564 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:19:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:19:37.293+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:19:37.565 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:19:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:19:37.293+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:19:37.565 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:19:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:19:37.541+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:19:37.565 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:19:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:19:37.541+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:19:37.565 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:19:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:19:37.541+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:19:37.565 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:19:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:19:37.541+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:19:37.908 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:19:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:19:37.742+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:37.742533+0000) 2024-02-20T09:19:38.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:19:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:19:37.918+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:37.919126+0000) 2024-02-20T09:19:38.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:19:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:19:37.977+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:19:38.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:19:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:19:37.977+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:19:38.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:19:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:19:37.977+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:19:38.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:19:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:19:38.034+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:38.034986+0000) 2024-02-20T09:19:38.564 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:19:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:19:38.255+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:19:38.564 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:19:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:19:38.255+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:19:38.565 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:19:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:19:38.255+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:19:38.908 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:19:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:19:38.586+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:19:38.908 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:19:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:19:38.586+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:19:38.908 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:19:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:19:38.586+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:19:38.908 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:19:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:19:38.586+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:19:38.909 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:19:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:19:38.744+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:38.744632+0000) 2024-02-20T09:19:39.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:19:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:19:38.918+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:38.919244+0000) 2024-02-20T09:19:39.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:19:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:19:38.952+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:19:39.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:19:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:19:38.952+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:19:39.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:19:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:19:38.952+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:19:39.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:19:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:19:39.034+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:39.035092+0000) 2024-02-20T09:19:39.565 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:19:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:19:39.284+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:19:39.565 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:19:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:19:39.284+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:19:39.565 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:19:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:19:39.284+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:19:39.885 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:19:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:19:39.575+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:19:39.886 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:19:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:19:39.575+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:19:39.886 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:19:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:19:39.575+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:19:39.886 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:19:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:19:39.575+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:19:39.886 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:19:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:19:39.746+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:39.746786+0000) 2024-02-20T09:19:40.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:19:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:19:39.918+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:39.919365+0000) 2024-02-20T09:19:40.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:19:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:19:39.927+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:19:40.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:19:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:19:39.927+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:19:40.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:19:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:19:39.927+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:19:40.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:19:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:19:40.034+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:40.035234+0000) 2024-02-20T09:19:40.564 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:19:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:19:40.273+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:19:40.565 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:19:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:19:40.273+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:19:40.565 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:19:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:19:40.273+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:19:40.908 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:19:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:19:40.592+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:19:40.908 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:19:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:19:40.592+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:19:40.909 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:19:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:19:40.592+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:19:40.909 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:19:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:19:40.592+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:19:40.909 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:19:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:19:40.748+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:40.748729+0000) 2024-02-20T09:19:41.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:19:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:19:40.919+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:40.919419+0000) 2024-02-20T09:19:41.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:19:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:19:40.944+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:19:41.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:19:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:19:40.944+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:19:41.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:19:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:19:40.944+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:19:41.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:19:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:19:41.034+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:41.035320+0000) 2024-02-20T09:19:41.565 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:19:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:19:41.405+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 66449 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:19:41.568 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:19:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:19:41.401+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:41.402043+0000) 2024-02-20T09:19:41.568 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:19:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:19:41.398+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:41.399103+0000) 2024-02-20T09:19:41.569 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:19:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:19:41.297+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:19:41.569 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:19:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:19:41.297+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:19:41.569 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:19:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:19:41.297+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:19:41.569 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:19:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:19:41.399+0000 7f8a5f86e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:41.399722+0000) 2024-02-20T09:19:41.908 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:19:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:19:41.607+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:19:41.908 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:19:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:19:41.607+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:19:41.909 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:19:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:19:41.607+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:19:41.909 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:19:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:19:41.607+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:19:41.909 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:19:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:19:41.748+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:41.748945+0000) 2024-02-20T09:19:42.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:19:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:19:41.919+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:41.919533+0000) 2024-02-20T09:19:42.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:19:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:19:41.936+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:19:42.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:19:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:19:41.936+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:19:42.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:19:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:19:41.936+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:19:42.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:19:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:19:42.035+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:42.035412+0000) 2024-02-20T09:19:42.565 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:19:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:19:42.268+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:19:42.565 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:19:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:19:42.268+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:19:42.565 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:19:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:19:42.268+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:19:42.908 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:19:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:19:42.617+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:19:42.909 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:19:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:19:42.617+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:19:42.909 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:19:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:19:42.617+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:19:42.909 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:19:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:19:42.617+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:19:42.909 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:19:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:19:42.750+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:42.750942+0000) 2024-02-20T09:19:43.236 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:19:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:19:42.919+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:42.919655+0000) 2024-02-20T09:19:43.237 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:19:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:19:43.235+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:19:43.237 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:19:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:19:43.235+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:19:43.237 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:19:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:19:42.916+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:19:43.237 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:19:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:19:42.916+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:19:43.237 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:19:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:19:42.916+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:19:43.238 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:19:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:19:43.035+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:43.035490+0000) 2024-02-20T09:19:43.908 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:19:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:19:43.235+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:19:43.909 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:19:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:19:43.576+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:19:43.909 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:19:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:19:43.576+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:19:43.909 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:19:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:19:43.576+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:19:43.909 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:19:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:19:43.576+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:19:43.909 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:19:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:19:43.752+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:43.752977+0000) 2024-02-20T09:19:44.220 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:19:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:19:43.912+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:19:44.220 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:19:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:19:43.912+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:19:44.220 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:19:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:19:43.912+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:19:44.221 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:19:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:19:44.035+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:44.035596+0000) 2024-02-20T09:19:44.221 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:19:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:19:43.919+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:43.919747+0000) 2024-02-20T09:19:44.221 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:19:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:19:44.218+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:19:44.221 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:19:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:19:44.218+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:19:44.221 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:19:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:19:44.218+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:19:44.909 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:19:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:19:44.583+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:19:44.909 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:19:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:19:44.583+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:19:44.909 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:19:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:19:44.583+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:19:44.909 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:19:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:19:44.583+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:19:44.909 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:19:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:19:44.754+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:44.755042+0000) 2024-02-20T09:19:45.195 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:19:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:19:44.919+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:44.919857+0000) 2024-02-20T09:19:45.196 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:19:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:19:44.954+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:19:45.196 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:19:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:19:44.954+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:19:45.196 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:19:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:19:44.954+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:19:45.196 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:19:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:19:45.035+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:45.035689+0000) 2024-02-20T09:19:45.492 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:19:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:19:45.194+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:19:45.492 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:19:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:19:45.194+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:19:45.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:19:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:19:45.194+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:19:45.909 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:19:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:19:45.537+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:19:45.909 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:19:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:19:45.537+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:19:45.909 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:19:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:19:45.537+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:19:45.909 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:19:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:19:45.537+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:19:45.909 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:19:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:19:45.756+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:45.757128+0000) 2024-02-20T09:19:46.198 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:19:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:19:45.919+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:45.919974+0000) 2024-02-20T09:19:46.199 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:19:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:19:46.196+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:19:46.199 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:19:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:19:46.196+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:19:46.199 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:19:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:19:46.196+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:19:46.199 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:19:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:19:45.967+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:19:46.199 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:19:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:19:45.967+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:19:46.200 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:19:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:19:45.967+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:19:46.200 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:19:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:19:46.034+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:46.035776+0000) 2024-02-20T09:19:46.492 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:19:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:19:46.405+0000 7f8a5f86e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:46.407074+0000) 2024-02-20T09:19:46.493 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:19:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:19:46.411+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 66488 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:19:46.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:19:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:19:46.408+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:46.410160+0000) 2024-02-20T09:19:46.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:19:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:19:46.405+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:46.407072+0000) 2024-02-20T09:19:46.909 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:19:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:19:46.547+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:19:46.910 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:19:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:19:46.547+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:19:46.910 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:19:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:19:46.547+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:19:46.910 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:19:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:19:46.547+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:19:46.910 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:19:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:19:46.755+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:46.757319+0000) 2024-02-20T09:19:47.220 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:19:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:19:46.918+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:46.920076+0000) 2024-02-20T09:19:47.221 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:19:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:19:47.217+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:19:47.221 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:19:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:19:46.976+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:19:47.221 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:19:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:19:46.976+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:19:47.221 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:19:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:19:46.976+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:19:47.221 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:19:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:19:47.034+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:47.035859+0000) 2024-02-20T09:19:47.492 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:19:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:19:47.217+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:19:47.492 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:19:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:19:47.217+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:19:47.909 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:19:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:19:47.527+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:19:47.909 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:19:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:19:47.527+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:19:47.910 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:19:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:19:47.527+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:19:47.910 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:19:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:19:47.527+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:19:47.910 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:19:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:19:47.758+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:47.759724+0000) 2024-02-20T09:19:48.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:19:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:19:47.918+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:47.920197+0000) 2024-02-20T09:19:48.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:19:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:19:47.927+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:19:48.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:19:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:19:47.927+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:19:48.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:19:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:19:47.927+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:19:48.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:19:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:19:48.034+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:48.036016+0000) 2024-02-20T09:19:48.552 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:19:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:19:48.253+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:19:48.552 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:19:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:19:48.253+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:19:48.552 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:19:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:19:48.253+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:19:48.896 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:19:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:19:48.549+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:19:48.896 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:19:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:19:48.549+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:19:48.896 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:19:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:19:48.549+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:19:48.897 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:19:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:19:48.549+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:19:48.897 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:19:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:19:48.760+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:48.762197+0000) 2024-02-20T09:19:49.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:19:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:19:48.918+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:48.920301+0000) 2024-02-20T09:19:49.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:19:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:19:48.893+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:19:49.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:19:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:19:48.893+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:19:49.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:19:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:19:48.893+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:19:49.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:19:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:19:49.034+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:49.036217+0000) 2024-02-20T09:19:49.530 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:19:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:19:49.264+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:19:49.530 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:19:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:19:49.264+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:19:49.530 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:19:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:19:49.264+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:19:49.886 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:19:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:19:49.527+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:19:49.886 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:19:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:19:49.527+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:19:49.886 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:19:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:19:49.527+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:19:49.886 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:19:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:19:49.527+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:19:49.886 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:19:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:19:49.763+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:49.764989+0000) 2024-02-20T09:19:50.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:19:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:19:49.919+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:49.920451+0000) 2024-02-20T09:19:50.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:19:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:19:49.888+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:19:50.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:19:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:19:49.888+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:19:50.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:19:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:19:49.888+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:19:50.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:19:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:19:50.035+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:50.036391+0000) 2024-02-20T09:19:50.566 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:19:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:19:50.294+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:19:50.566 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:19:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:19:50.294+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:19:50.566 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:19:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:19:50.294+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:19:50.567 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:19:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:19:50.535+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:19:50.567 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:19:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:19:50.535+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:19:50.567 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:19:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:19:50.535+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:19:50.567 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:19:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:19:50.535+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:19:50.846 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:19:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:19:50.765+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:50.767004+0000) 2024-02-20T09:19:51.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:19:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:19:50.919+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:50.920569+0000) 2024-02-20T09:19:51.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:19:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:19:50.844+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:19:51.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:19:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:19:50.844+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:19:51.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:19:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:19:50.844+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:19:51.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:19:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:19:51.035+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:51.036524+0000) 2024-02-20T09:19:51.540 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:19:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:19:51.419+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 66528 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:19:51.541 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:19:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:19:51.415+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:51.416878+0000) 2024-02-20T09:19:51.541 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:19:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:19:51.412+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:51.414272+0000) 2024-02-20T09:19:51.541 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:19:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:19:51.261+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:19:51.542 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:19:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:19:51.261+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:19:51.542 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:19:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:19:51.261+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:19:51.542 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:19:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:19:51.413+0000 7f8a5f86e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:51.415062+0000) 2024-02-20T09:19:51.814 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:19:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:19:51.583+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:19:51.814 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:19:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:19:51.583+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:19:51.815 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:19:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:19:51.583+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:19:51.815 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:19:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:19:51.583+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:19:51.815 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:19:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:19:51.765+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:51.767251+0000) 2024-02-20T09:19:52.232 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:19:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:19:51.919+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:51.920667+0000) 2024-02-20T09:19:52.232 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:19:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:19:52.229+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:19:52.233 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:19:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:19:51.811+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:19:52.233 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:19:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:19:51.811+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:19:52.233 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:19:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:19:51.811+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:19:52.233 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:19:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:19:52.035+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:52.036618+0000) 2024-02-20T09:19:52.492 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:19:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:19:52.229+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:19:52.492 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:19:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:19:52.229+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:19:52.910 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:19:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:19:52.770+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:19:52.910 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:19:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:19:52.770+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:19:52.910 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:19:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:19:52.770+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:19:52.910 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:19:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:19:52.588+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:19:52.911 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:19:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:19:52.588+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:19:52.911 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:19:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:19:52.588+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:19:52.911 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:19:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:19:52.588+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:19:52.911 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:19:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:19:52.767+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:52.769290+0000) 2024-02-20T09:19:53.184 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:19:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:19:53.035+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:53.036698+0000) 2024-02-20T09:19:53.184 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:19:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:19:52.919+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:52.920787+0000) 2024-02-20T09:19:53.185 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:19:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:19:53.181+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:19:53.185 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:19:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:19:53.181+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:19:53.185 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:19:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:19:53.181+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:19:53.910 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:19:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:19:53.797+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:19:53.910 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:19:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:19:53.797+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:19:53.910 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:19:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:19:53.797+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:19:53.911 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:19:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:19:53.605+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:19:53.911 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:19:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:19:53.605+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:19:53.911 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:19:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:19:53.605+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:19:53.911 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:19:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:19:53.605+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:19:53.911 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:19:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:19:53.770+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:53.771381+0000) 2024-02-20T09:19:54.209 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:19:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:19:54.035+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:54.036794+0000) 2024-02-20T09:19:54.210 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:19:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:19:53.919+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:53.920898+0000) 2024-02-20T09:19:54.210 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:19:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:19:54.207+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:19:54.210 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:19:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:19:54.207+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:19:54.843 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:19:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:19:54.207+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:19:54.844 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:19:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:19:54.571+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:19:54.844 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:19:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:19:54.571+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:19:54.844 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:19:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:19:54.571+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:19:54.844 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:19:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:19:54.571+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:19:54.844 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:19:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:19:54.772+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:54.773446+0000) 2024-02-20T09:19:55.236 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:19:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:19:54.841+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:19:55.236 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:19:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:19:54.841+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:19:55.236 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:19:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:19:54.841+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:19:55.237 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:19:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:19:55.035+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:55.036888+0000) 2024-02-20T09:19:55.237 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:19:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:19:54.919+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:54.921009+0000) 2024-02-20T09:19:55.237 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:19:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:19:55.233+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:19:55.237 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:19:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:19:55.233+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:19:55.237 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:19:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:19:55.233+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:19:55.846 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:19:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:19:55.606+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:19:55.846 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:19:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:19:55.606+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:19:55.846 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:19:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:19:55.606+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:19:55.847 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:19:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:19:55.606+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:19:55.847 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:19:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:19:55.773+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:55.775113+0000) 2024-02-20T09:19:56.226 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:19:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:19:55.843+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:19:56.226 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:19:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:19:55.843+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:19:56.226 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:19:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:19:55.843+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:19:56.227 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:19:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:19:56.035+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:56.037017+0000) 2024-02-20T09:19:56.227 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:19:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:19:55.919+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:55.921107+0000) 2024-02-20T09:19:56.227 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:19:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:19:56.223+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:19:56.227 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:19:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:19:56.223+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:19:56.227 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:19:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:19:56.223+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:19:56.492 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:19:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:19:56.427+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 66567 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:19:56.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:19:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:19:56.420+0000 7f8a5f86e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:56.422167+0000) 2024-02-20T09:19:56.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:19:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:19:56.423+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:56.424984+0000) 2024-02-20T09:19:56.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:19:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:19:56.420+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:56.421693+0000) 2024-02-20T09:19:56.800 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:19:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:19:56.595+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:19:56.800 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:19:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:19:56.595+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:19:56.800 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:19:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:19:56.595+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:19:56.800 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:19:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:19:56.595+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:19:56.801 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:19:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:19:56.773+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:56.775323+0000) 2024-02-20T09:19:57.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:19:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:19:56.919+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:56.921209+0000) 2024-02-20T09:19:57.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:19:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:19:56.797+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:19:57.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:19:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:19:56.797+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:19:57.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:19:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:19:56.797+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:19:57.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:19:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:19:57.035+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:57.037134+0000) 2024-02-20T09:19:57.566 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:19:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:19:57.248+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:19:57.567 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:19:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:19:57.248+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:19:57.567 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:19:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:19:57.248+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:19:57.910 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:19:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:19:57.790+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:19:57.911 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:19:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:19:57.790+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:19:57.911 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:19:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:19:57.790+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:19:57.911 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:19:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:19:57.618+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:19:57.911 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:19:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:19:57.618+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:19:57.911 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:19:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:19:57.618+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:19:57.911 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:19:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:19:57.618+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:19:57.912 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:19:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:19:57.775+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:57.777276+0000) 2024-02-20T09:19:58.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:19:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:19:57.919+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:57.921288+0000) 2024-02-20T09:19:58.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:19:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:19:58.035+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:58.037247+0000) 2024-02-20T09:19:58.567 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:19:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:19:58.245+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:19:58.567 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:19:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:19:58.245+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:19:58.567 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:19:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:19:58.245+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:19:58.910 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:19:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:19:58.786+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:19:58.911 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:19:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:19:58.786+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:19:58.911 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:19:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:19:58.786+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:19:58.911 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:19:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:19:58.605+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:19:58.911 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:19:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:19:58.605+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:19:58.911 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:19:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:19:58.605+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:19:58.911 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:19:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:19:58.605+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:19:58.912 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:19:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:19:58.777+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:58.779286+0000) 2024-02-20T09:19:59.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:19:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:19:59.036+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:59.037374+0000) 2024-02-20T09:19:59.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:19:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:19:58.919+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:58.921378+0000) 2024-02-20T09:19:59.567 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:19:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:19:59.285+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:19:59.567 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:19:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:19:59.285+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:19:59.567 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:19:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:19:59.285+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:19:59.885 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:19:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:19:59.782+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:19:59.885 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:19:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:19:59.782+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:19:59.885 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:19:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:19:59.782+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:19:59.886 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:19:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:19:59.624+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:19:59.886 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:19:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:19:59.624+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:19:59.886 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:19:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:19:59.624+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:19:59.886 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:19:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:19:59.624+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:19:59.886 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:19:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:19:59.780+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:59.781473+0000) 2024-02-20T09:20:00.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:19:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:19:59.920+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:19:59.921498+0000) 2024-02-20T09:20:00.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:20:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:20:00.036+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:00.037442+0000) 2024-02-20T09:20:00.667 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:20:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:20:00.334+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:20:00.667 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:20:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:20:00.334+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:20:00.668 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:20:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:20:00.334+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:20:00.922 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:20:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:20:00.812+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:20:00.923 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:20:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:20:00.812+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:20:00.923 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:20:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:20:00.812+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:20:00.923 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:20:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:20:00.665+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:20:00.923 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:20:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:20:00.665+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:20:00.923 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:20:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:20:00.665+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:20:00.923 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:20:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:20:00.665+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:20:00.924 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:20:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:20:00.783+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:00.784448+0000) 2024-02-20T09:20:01.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:20:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:20:00.920+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:00.921614+0000) 2024-02-20T09:20:01.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:20:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:20:01.036+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:01.037534+0000) 2024-02-20T09:20:01.669 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:20:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:20:01.431+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:01.432696+0000) 2024-02-20T09:20:01.670 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:20:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:20:01.428+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:01.429981+0000) 2024-02-20T09:20:01.670 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:20:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:20:01.434+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 66607 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:20:01.670 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:20:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:20:01.360+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:20:01.671 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:20:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:20:01.360+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:20:01.671 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:20:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:20:01.360+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:20:01.671 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:20:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:20:01.428+0000 7f8a5f86e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:01.429937+0000) 2024-02-20T09:20:01.923 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:20:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:20:01.787+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:20:01.923 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:20:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:20:01.787+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:20:01.923 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:20:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:20:01.787+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:20:01.923 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:20:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:20:01.701+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:20:01.924 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:20:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:20:01.701+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:20:01.924 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:20:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:20:01.701+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:20:01.924 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:20:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:20:01.701+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:20:01.924 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:20:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:20:01.783+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:01.784624+0000) 2024-02-20T09:20:02.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:20:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:20:01.920+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:01.921714+0000) 2024-02-20T09:20:02.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:20:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:20:02.036+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:02.037651+0000) 2024-02-20T09:20:02.669 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:20:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:20:02.379+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:20:02.670 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:20:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:20:02.379+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:20:02.670 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:20:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:20:02.379+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:20:02.923 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:20:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:20:02.835+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:20:02.923 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:20:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:20:02.835+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:20:02.923 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:20:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:20:02.835+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:20:02.923 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:20:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:20:02.669+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:20:02.924 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:20:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:20:02.669+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:20:02.924 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:20:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:20:02.669+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:20:02.924 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:20:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:20:02.669+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:20:02.924 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:20:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:20:02.786+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:02.788177+0000) 2024-02-20T09:20:03.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:20:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:20:02.920+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:02.921805+0000) 2024-02-20T09:20:03.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:20:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:20:03.036+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:03.037742+0000) 2024-02-20T09:20:03.655 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:20:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:20:03.379+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:20:03.656 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:20:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:20:03.379+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:20:03.656 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:20:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:20:03.379+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:20:03.911 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:20:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:20:03.879+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:20:03.911 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:20:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:20:03.879+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:20:03.912 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:20:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:20:03.879+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:20:03.912 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:20:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:20:03.653+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:20:03.912 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:20:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:20:03.653+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:20:03.912 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:20:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:20:03.653+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:20:03.912 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:20:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:20:03.653+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:20:03.913 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:20:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:20:03.788+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:03.790152+0000) 2024-02-20T09:20:04.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:20:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:20:04.036+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:04.037845+0000) 2024-02-20T09:20:04.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:20:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:20:03.920+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:03.921897+0000) 2024-02-20T09:20:04.670 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:20:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:20:04.628+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:20:04.670 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:20:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:20:04.628+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:20:04.670 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:20:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:20:04.628+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:20:04.670 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:20:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:20:04.628+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:20:04.670 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:20:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:20:04.404+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:20:04.671 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:20:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:20:04.404+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:20:04.671 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:20:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:20:04.404+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:20:04.923 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:20:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:20:04.790+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:04.792163+0000) 2024-02-20T09:20:04.923 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:20:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:20:04.834+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:20:04.923 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:20:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:20:04.834+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:20:04.923 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:20:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:20:04.834+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:20:05.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:20:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:20:05.036+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:05.037973+0000) 2024-02-20T09:20:05.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:20:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:20:04.920+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:04.921990+0000) 2024-02-20T09:20:05.648 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:20:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:20:05.389+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:20:05.649 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:20:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:20:05.389+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:20:05.649 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:20:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:20:05.390+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:20:05.911 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:20:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:20:05.791+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:20:05.912 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:20:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:20:05.791+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:20:05.912 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:20:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:20:05.791+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:20:05.912 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:20:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:20:05.646+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:20:05.912 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:20:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:20:05.646+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:20:05.913 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:20:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:20:05.646+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:20:05.913 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:20:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:20:05.646+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:20:05.913 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:20:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:20:05.792+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:05.793832+0000) 2024-02-20T09:20:06.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:20:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:20:06.036+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:06.038080+0000) 2024-02-20T09:20:06.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:20:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:20:05.920+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:05.922097+0000) 2024-02-20T09:20:06.564 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:20:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:20:06.439+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:06.440662+0000) 2024-02-20T09:20:06.565 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:20:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:20:06.435+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:06.437043+0000) 2024-02-20T09:20:06.565 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:20:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:20:06.442+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 66645 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:20:06.565 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:20:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:20:06.395+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:20:06.565 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:20:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:20:06.395+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:20:06.565 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:20:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:20:06.395+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:20:06.566 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:20:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:20:06.436+0000 7f8a5f86e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:06.437651+0000) 2024-02-20T09:20:06.911 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:20:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:20:06.758+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:20:06.912 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:20:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:20:06.758+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:20:06.912 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:20:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:20:06.758+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:20:06.912 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:20:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:20:06.676+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:20:06.912 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:20:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:20:06.676+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:20:06.912 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:20:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:20:06.676+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:20:06.912 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:20:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:20:06.676+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:20:06.912 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:20:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:20:06.792+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:06.794027+0000) 2024-02-20T09:20:07.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:20:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:20:06.920+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:06.922245+0000) 2024-02-20T09:20:07.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:20:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:20:07.036+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:07.038268+0000) 2024-02-20T09:20:07.652 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:20:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:20:07.353+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:20:07.652 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:20:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:20:07.353+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:20:07.653 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:20:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:20:07.353+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:20:07.912 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:20:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:20:07.776+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:20:07.912 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:20:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:20:07.776+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:20:07.912 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:20:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:20:07.776+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:20:07.913 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:20:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:20:07.649+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:20:07.913 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:20:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:20:07.649+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:20:07.913 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:20:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:20:07.649+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:20:07.913 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:20:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:20:07.650+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:20:07.913 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:20:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:20:07.796+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:07.797582+0000) 2024-02-20T09:20:08.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:20:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:20:07.921+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:07.922437+0000) 2024-02-20T09:20:08.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:20:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:20:08.037+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:08.038380+0000) 2024-02-20T09:20:08.640 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:20:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:20:08.333+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:20:08.640 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:20:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:20:08.333+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:20:08.641 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:20:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:20:08.333+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:20:08.912 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:20:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:20:08.755+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:20:08.912 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:20:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:20:08.755+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:20:08.912 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:20:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:20:08.756+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:20:08.912 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:20:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:20:08.637+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:20:08.913 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:20:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:20:08.637+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:20:08.913 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:20:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:20:08.637+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:20:08.913 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:20:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:20:08.637+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:20:08.913 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:20:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:20:08.799+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:08.801178+0000) 2024-02-20T09:20:09.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:20:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:20:08.921+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:08.922529+0000) 2024-02-20T09:20:09.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:20:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:20:09.037+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:09.038519+0000) 2024-02-20T09:20:09.630 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:20:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:20:09.324+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:20:09.631 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:20:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:20:09.324+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:20:09.631 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:20:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:20:09.324+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:20:09.886 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:20:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:20:09.722+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:20:09.886 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:20:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:20:09.722+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:20:09.886 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:20:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:20:09.722+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:20:09.887 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:20:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:20:09.627+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:20:09.887 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:20:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:20:09.627+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:20:09.887 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:20:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:20:09.627+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:20:09.887 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:20:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:20:09.628+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:20:09.887 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:20:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:20:09.803+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:09.804695+0000) 2024-02-20T09:20:10.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:20:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:20:10.037+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:10.038683+0000) 2024-02-20T09:20:10.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:20:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:20:09.921+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:09.922640+0000) 2024-02-20T09:20:10.671 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:20:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:20:10.581+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:20:10.671 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:20:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:20:10.581+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:20:10.672 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:20:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:20:10.581+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:20:10.672 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:20:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:20:10.581+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:20:10.672 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:20:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:20:10.365+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:20:10.672 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:20:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:20:10.365+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:20:10.672 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:20:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:20:10.365+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:20:10.924 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:20:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:20:10.805+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:10.806830+0000) 2024-02-20T09:20:10.925 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:20:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:20:10.717+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:20:10.925 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:20:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:20:10.717+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:20:10.925 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:20:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:20:10.717+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:20:11.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:20:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:20:11.037+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:11.038819+0000) 2024-02-20T09:20:11.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:20:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:20:10.921+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:10.922749+0000) 2024-02-20T09:20:11.670 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:20:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:20:11.449+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 66686 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:20:11.671 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:20:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:20:11.443+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:11.444680+0000) 2024-02-20T09:20:11.671 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:20:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:20:11.414+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:20:11.671 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:20:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:20:11.414+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:20:11.672 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:20:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:20:11.414+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:20:11.672 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:20:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:20:11.443+0000 7f8a5f86e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:11.444927+0000) 2024-02-20T09:20:11.672 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:20:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:20:11.447+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:11.448629+0000) 2024-02-20T09:20:11.672 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:20:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:20:11.572+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:20:11.672 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:20:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:20:11.572+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:20:11.672 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:20:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:20:11.572+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:20:11.672 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:20:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:20:11.572+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:20:11.924 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:20:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:20:11.805+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:11.807046+0000) 2024-02-20T09:20:11.924 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:20:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:20:11.670+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:20:11.924 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:20:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:20:11.670+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:20:11.924 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:20:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:20:11.670+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:20:12.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:20:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:20:11.921+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:11.922909+0000) 2024-02-20T09:20:12.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:20:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:20:12.037+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:12.038956+0000) 2024-02-20T09:20:12.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:20:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:20:12.666+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:20:12.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:20:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:20:12.666+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:20:12.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:20:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:20:12.666+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:20:12.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:20:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:20:12.440+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:20:12.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:20:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:20:12.440+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:20:12.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:20:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:20:12.440+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:20:12.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:20:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:20:12.562+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:20:12.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:20:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:20:12.562+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:20:12.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:20:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:20:12.562+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:20:12.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:20:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:20:12.562+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:20:13.061 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:20:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:20:12.921+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:12.923039+0000) 2024-02-20T09:20:13.062 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:20:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:20:12.807+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:12.809102+0000) 2024-02-20T09:20:13.062 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:20:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:20:13.037+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:13.039064+0000) 2024-02-20T09:20:13.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:20:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:20:13.646+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:20:13.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:20:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:20:13.646+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:20:13.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:20:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:20:13.646+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:20:13.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:20:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:20:13.470+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:20:13.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:20:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:20:13.470+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:20:13.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:20:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:20:13.470+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:20:13.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:20:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:20:13.555+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:20:13.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:20:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:20:13.555+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:20:13.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:20:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:20:13.555+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:20:13.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:20:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:20:13.555+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:20:14.063 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:20:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:20:13.809+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:13.811250+0000) 2024-02-20T09:20:14.063 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:20:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:20:13.921+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:13.923152+0000) 2024-02-20T09:20:14.064 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:20:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:20:14.037+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:14.039173+0000) 2024-02-20T09:20:14.814 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:20:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:20:14.500+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:20:14.815 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:20:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:20:14.500+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:20:14.815 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:20:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:20:14.500+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:20:14.815 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:20:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:20:14.690+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:20:14.815 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:20:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:20:14.690+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:20:14.815 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:20:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:20:14.690+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:20:14.816 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:20:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:20:14.507+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:20:14.816 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:20:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:20:14.507+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:20:14.816 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:20:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:20:14.507+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:20:14.816 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:20:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:20:14.507+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:20:15.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:20:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:20:14.921+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:14.923278+0000) 2024-02-20T09:20:15.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:20:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:20:14.811+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:14.813381+0000) 2024-02-20T09:20:15.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:20:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:20:15.037+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:15.039299+0000) 2024-02-20T09:20:15.816 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:20:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:20:15.641+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:20:15.816 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:20:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:20:15.641+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:20:15.816 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:20:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:20:15.641+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:20:15.817 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:20:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:20:15.496+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:20:15.817 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:20:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:20:15.496+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:20:15.817 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:20:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:20:15.496+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:20:15.817 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:20:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:20:15.501+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:20:15.817 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:20:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:20:15.501+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:20:15.818 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:20:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:20:15.501+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:20:15.818 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:20:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:20:15.501+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:20:15.818 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:20:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:20:15.813+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:15.815179+0000) 2024-02-20T09:20:16.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:20:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:20:15.922+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:15.923427+0000) 2024-02-20T09:20:16.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:20:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:20:16.038+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:16.039427+0000) 2024-02-20T09:20:16.742 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:20:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:20:16.457+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 66724 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:20:16.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:20:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:20:16.450+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:16.452285+0000) 2024-02-20T09:20:16.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:20:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:20:16.686+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:20:16.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:20:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:20:16.686+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:20:16.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:20:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:20:16.686+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:20:16.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:20:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:20:16.451+0000 7f8a5f86e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:16.452431+0000) 2024-02-20T09:20:16.744 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:20:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:20:16.544+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:20:16.744 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:20:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:20:16.544+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:20:16.744 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:20:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:20:16.544+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:20:16.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:20:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:20:16.453+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:16.454742+0000) 2024-02-20T09:20:16.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:20:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:20:16.532+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:20:16.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:20:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:20:16.532+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:20:16.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:20:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:20:16.532+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:20:16.745 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:20:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:20:16.532+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:20:17.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:20:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:20:16.814+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:16.815481+0000) 2024-02-20T09:20:17.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:20:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:20:17.038+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:17.039531+0000) 2024-02-20T09:20:17.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:20:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:20:16.922+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:16.923551+0000) 2024-02-20T09:20:17.818 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:20:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:20:17.574+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:20:17.819 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:20:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:20:17.574+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:20:17.819 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:20:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:20:17.574+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:20:17.819 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:20:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:20:17.722+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:20:17.819 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:20:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:20:17.722+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:20:17.820 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:20:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:20:17.722+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:20:17.820 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:20:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:20:17.526+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:20:17.820 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:20:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:20:17.526+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:20:17.820 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:20:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:20:17.526+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:20:17.820 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:20:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:20:17.526+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:20:18.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:20:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:20:17.922+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:17.923639+0000) 2024-02-20T09:20:18.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:20:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:20:17.815+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:17.817363+0000) 2024-02-20T09:20:18.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:20:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:20:18.038+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:18.039607+0000) 2024-02-20T09:20:18.821 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:20:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:20:18.593+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:20:18.821 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:20:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:20:18.593+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:20:18.821 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:20:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:20:18.593+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:20:18.821 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:20:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:20:18.690+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:20:18.822 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:20:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:20:18.690+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:20:18.822 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:20:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:20:18.690+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:20:18.822 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:20:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:20:18.543+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:20:18.822 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:20:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:20:18.543+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:20:18.823 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:20:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:20:18.543+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:20:18.823 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:20:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:20:18.543+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:20:19.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:20:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:20:18.923+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:18.923815+0000) 2024-02-20T09:20:19.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:20:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:20:18.819+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:18.819918+0000) 2024-02-20T09:20:19.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:20:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:20:19.039+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:19.039715+0000) 2024-02-20T09:20:19.823 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:20:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:20:19.551+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:20:19.824 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:20:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:20:19.551+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:20:19.824 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:20:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:20:19.551+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:20:19.824 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:20:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:20:19.653+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:20:19.824 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:20:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:20:19.653+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:20:19.824 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:20:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:20:19.653+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:20:19.825 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:20:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:20:19.493+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:20:19.825 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:20:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:20:19.493+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:20:19.825 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:20:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:20:19.493+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:20:19.825 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:20:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:20:19.494+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:20:20.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:20:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:20:19.923+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:19.923970+0000) 2024-02-20T09:20:20.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:20:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:20:19.822+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:19.822530+0000) 2024-02-20T09:20:20.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:20:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:20:20.039+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:20.039855+0000) 2024-02-20T09:20:20.742 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:20:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:20:20.568+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:20:20.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:20:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:20:20.568+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:20:20.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:20:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:20:20.568+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:20:20.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:20:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:20:20.655+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:20:20.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:20:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:20:20.655+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:20:20.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:20:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:20:20.655+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:20:20.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:20:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:20:20.505+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:20:20.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:20:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:20:20.505+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:20:20.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:20:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:20:20.505+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:20:20.745 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:20:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:20:20.505+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:20:21.240 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:20:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:20:20.923+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:20.924086+0000) 2024-02-20T09:20:21.241 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:20:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:20:20.824+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:20.824884+0000) 2024-02-20T09:20:21.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:20:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:20:21.039+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:21.040031+0000) 2024-02-20T09:20:21.492 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:20:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:20:21.459+0000 7f8a5f86e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:21.460093+0000) 2024-02-20T09:20:21.493 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:20:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:20:21.465+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 66765 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:20:21.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:20:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:20:21.461+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:21.462386+0000) 2024-02-20T09:20:21.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:20:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:20:21.459+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:21.459736+0000) 2024-02-20T09:20:21.826 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:20:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:20:21.530+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:20:21.827 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:20:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:20:21.530+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:20:21.827 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:20:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:20:21.530+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:20:21.827 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:20:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:20:21.530+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:20:21.827 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:20:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:20:21.534+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:20:21.827 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:20:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:20:21.534+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:20:21.827 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:20:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:20:21.534+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:20:21.828 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:20:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:20:21.698+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:20:21.828 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:20:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:20:21.698+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:20:21.828 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:20:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:20:21.698+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:20:22.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:20:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:20:22.039+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:22.040200+0000) 2024-02-20T09:20:22.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:20:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:20:21.923+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:21.924227+0000) 2024-02-20T09:20:22.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:20:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:20:21.824+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:21.825148+0000) 2024-02-20T09:20:22.828 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:20:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:20:22.510+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:20:22.829 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:20:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:20:22.510+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:20:22.829 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:20:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:20:22.510+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:20:22.830 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:20:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:20:22.700+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:20:22.830 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:20:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:20:22.700+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:20:22.830 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:20:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:20:22.700+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:20:22.831 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:20:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:20:22.519+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:20:22.831 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:20:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:20:22.519+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:20:22.831 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:20:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:20:22.519+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:20:22.831 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:20:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:20:22.519+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:20:23.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:20:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:20:22.826+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:22.827357+0000) 2024-02-20T09:20:23.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:20:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:20:22.923+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:22.924349+0000) 2024-02-20T09:20:23.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:20:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:20:23.039+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:23.040302+0000) 2024-02-20T09:20:23.831 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:20:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:20:23.693+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:20:23.831 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:20:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:20:23.693+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:20:23.831 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:20:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:20:23.693+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:20:23.832 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:20:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:20:23.515+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:20:23.832 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:20:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:20:23.515+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:20:23.832 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:20:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:20:23.515+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:20:23.832 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:20:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:20:23.555+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:20:23.832 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:20:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:20:23.555+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:20:23.832 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:20:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:20:23.555+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:20:23.833 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:20:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:20:23.555+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:20:24.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:20:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:20:23.924+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:23.924442+0000) 2024-02-20T09:20:24.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:20:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:20:23.829+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:23.829907+0000) 2024-02-20T09:20:24.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:20:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:20:24.040+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:24.040390+0000) 2024-02-20T09:20:24.570 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:20:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:20:24.530+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:20:24.570 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:20:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:20:24.530+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:20:24.570 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:20:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:20:24.530+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:20:24.571 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:20:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:20:24.529+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:20:24.571 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:20:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:20:24.529+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:20:24.571 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:20:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:20:24.529+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:20:24.571 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:20:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:20:24.529+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:20:24.833 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:20:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:20:24.661+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:20:24.834 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:20:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:20:24.661+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:20:24.834 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:20:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:20:24.661+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:20:25.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:20:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:20:24.924+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:24.924532+0000) 2024-02-20T09:20:25.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:20:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:20:25.040+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:25.040537+0000) 2024-02-20T09:20:25.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:20:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:20:24.832+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:24.832441+0000) 2024-02-20T09:20:25.836 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:20:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:20:25.575+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:20:25.836 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:20:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:20:25.575+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:20:25.836 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:20:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:20:25.575+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:20:25.836 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:20:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:20:25.579+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:20:25.836 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:20:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:20:25.579+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:20:25.837 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:20:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:20:25.579+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:20:25.837 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:20:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:20:25.579+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:20:25.837 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:20:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:20:25.662+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:20:25.837 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:20:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:20:25.662+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:20:25.837 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:20:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:20:25.662+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:20:26.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:20:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:20:25.924+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:25.924622+0000) 2024-02-20T09:20:26.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:20:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:20:25.834+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:25.834652+0000) 2024-02-20T09:20:26.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:20:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:20:26.040+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:26.040654+0000) 2024-02-20T09:20:26.742 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:20:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:20:26.472+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 66803 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:20:26.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:20:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:20:26.466+0000 7f8a5f86e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:26.466869+0000) 2024-02-20T09:20:26.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:20:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:20:26.599+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:20:26.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:20:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:20:26.599+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:20:26.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:20:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:20:26.599+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:20:26.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:20:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:20:26.469+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:26.470297+0000) 2024-02-20T09:20:26.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:20:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:20:26.557+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:20:26.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:20:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:20:26.557+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:20:26.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:20:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:20:26.557+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:20:26.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:20:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:20:26.557+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:20:26.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:20:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:20:26.466+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:26.466817+0000) 2024-02-20T09:20:26.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:20:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:20:26.643+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:20:26.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:20:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:20:26.643+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:20:26.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:20:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:20:26.643+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:20:27.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:20:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:20:26.924+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:26.924725+0000) 2024-02-20T09:20:27.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:20:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:20:26.834+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:26.834955+0000) 2024-02-20T09:20:27.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:20:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:20:27.040+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:27.040769+0000) 2024-02-20T09:20:27.838 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:20:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:20:27.666+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:20:27.839 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:20:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:20:27.666+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:20:27.839 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:20:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:20:27.666+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:20:27.839 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:20:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:20:27.587+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:20:27.839 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:20:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:20:27.587+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:20:27.839 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:20:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:20:27.587+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:20:27.840 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:20:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:20:27.527+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:20:27.840 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:20:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:20:27.528+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:20:27.840 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:20:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:20:27.528+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:20:27.840 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:20:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:20:27.528+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:20:28.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:20:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:20:27.924+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:27.924841+0000) 2024-02-20T09:20:28.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:20:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:20:27.836+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:27.837349+0000) 2024-02-20T09:20:28.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:20:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:20:28.040+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:28.040871+0000) 2024-02-20T09:20:28.638 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:20:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:20:28.487+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:20:28.639 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:20:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:20:28.487+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:20:28.639 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:20:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:20:28.487+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:20:28.639 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:20:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:20:28.487+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:20:28.914 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:20:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:20:28.840+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:28.840901+0000) 2024-02-20T09:20:28.915 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:20:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:20:28.637+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:20:28.915 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:20:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:20:28.637+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:20:28.915 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:20:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:20:28.637+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:20:28.916 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:20:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:20:28.692+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:20:28.916 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:20:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:20:28.692+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:20:28.916 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:20:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:20:28.692+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:20:29.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:20:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:20:28.924+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:28.924974+0000) 2024-02-20T09:20:29.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:20:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:20:29.040+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:29.041033+0000) 2024-02-20T09:20:29.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:20:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:20:29.480+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:20:29.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:20:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:20:29.480+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:20:29.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:20:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:20:29.480+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:20:29.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:20:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:20:29.480+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:20:29.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:20:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:20:29.633+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:20:29.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:20:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:20:29.633+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:20:29.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:20:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:20:29.633+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:20:29.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:20:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:20:29.715+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:20:29.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:20:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:20:29.715+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:20:29.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:20:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:20:29.715+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:20:30.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:20:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:20:29.924+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:29.925089+0000) 2024-02-20T09:20:30.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:20:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:20:29.844+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:29.844531+0000) 2024-02-20T09:20:30.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:20:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:20:30.040+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:30.041159+0000) 2024-02-20T09:20:30.742 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:20:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:20:30.623+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:20:30.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:20:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:20:30.623+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:20:30.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:20:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:20:30.623+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:20:30.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:20:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:20:30.433+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:20:30.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:20:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:20:30.433+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:20:30.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:20:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:20:30.433+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:20:30.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:20:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:20:30.433+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:20:31.042 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:20:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:20:30.924+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:30.925200+0000) 2024-02-20T09:20:31.042 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:20:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:20:30.846+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:30.846681+0000) 2024-02-20T09:20:31.043 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:20:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:20:30.742+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:20:31.043 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:20:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:20:30.742+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:20:31.043 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:20:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:20:30.742+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:20:31.475 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:20:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:20:31.040+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:31.041264+0000) 2024-02-20T09:20:31.730 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:20:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:20:31.473+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:31.474125+0000) 2024-02-20T09:20:31.731 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:20:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:20:31.729+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:20:31.731 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:20:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:20:31.729+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:20:31.731 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:20:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:20:31.473+0000 7f8a5f86e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:31.474049+0000) 2024-02-20T09:20:31.731 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:20:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:20:31.584+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:20:31.731 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:20:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:20:31.584+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:20:31.732 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:20:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:20:31.584+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:20:31.732 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:20:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:20:31.481+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 66844 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:20:31.732 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:20:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:20:31.475+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:20:31.732 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:20:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:20:31.475+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:20:31.732 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:20:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:20:31.475+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:20:31.733 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:20:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:20:31.475+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:20:31.733 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:20:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:20:31.477+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:31.478356+0000) 2024-02-20T09:20:31.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:20:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:20:31.846+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:31.846856+0000) 2024-02-20T09:20:31.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:20:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:20:31.729+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:20:31.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:20:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:20:31.924+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:31.925327+0000) 2024-02-20T09:20:32.487 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:20:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:20:32.041+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:32.041421+0000) 2024-02-20T09:20:32.742 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:20:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:20:32.603+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:20:32.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:20:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:20:32.603+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:20:32.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:20:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:20:32.603+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:20:32.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:20:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:20:32.486+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:20:32.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:20:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:20:32.486+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:20:32.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:20:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:20:32.486+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:20:32.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:20:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:20:32.486+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:20:33.042 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:20:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:20:32.925+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:32.925437+0000) 2024-02-20T09:20:33.043 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:20:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:20:32.848+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:32.848916+0000) 2024-02-20T09:20:33.043 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:20:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:20:32.777+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:20:33.043 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:20:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:20:32.777+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:20:33.043 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:20:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:20:32.777+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:20:33.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:20:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:20:33.041+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:33.041522+0000) 2024-02-20T09:20:33.763 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:20:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:20:33.561+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:20:33.764 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:20:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:20:33.561+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:20:33.764 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:20:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:20:33.561+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:20:33.765 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:20:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:20:33.505+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:20:33.765 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:20:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:20:33.505+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:20:33.766 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:20:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:20:33.505+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:20:33.766 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:20:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:20:33.505+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:20:34.042 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:20:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:20:33.925+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:33.925547+0000) 2024-02-20T09:20:34.043 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:20:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:20:33.850+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:33.851006+0000) 2024-02-20T09:20:34.043 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:20:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:20:33.761+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:20:34.044 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:20:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:20:33.761+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:20:34.044 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:20:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:20:33.761+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:20:34.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:20:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:20:34.041+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:34.041619+0000) 2024-02-20T09:20:34.809 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:20:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:20:34.549+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:20:34.809 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:20:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:20:34.549+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:20:34.809 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:20:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:20:34.549+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:20:34.809 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:20:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:20:34.497+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:20:34.809 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:20:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:20:34.497+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:20:34.810 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:20:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:20:34.497+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:20:34.810 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:20:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:20:34.497+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:20:35.081 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:20:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:20:34.852+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:34.853089+0000) 2024-02-20T09:20:35.081 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:20:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:20:34.925+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:34.925642+0000) 2024-02-20T09:20:35.082 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:20:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:20:34.807+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:20:35.082 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:20:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:20:34.807+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:20:35.082 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:20:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:20:34.807+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:20:35.082 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:20:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:20:35.041+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:35.041734+0000) 2024-02-20T09:20:35.841 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:20:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:20:35.544+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:20:35.842 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:20:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:20:35.544+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:20:35.842 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:20:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:20:35.544+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:20:35.842 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:20:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:20:35.544+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:20:35.842 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:20:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:20:35.501+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:20:35.842 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:20:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:20:35.501+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:20:35.842 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:20:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:20:35.501+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:20:36.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:20:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:20:35.925+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:35.925755+0000) 2024-02-20T09:20:36.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:20:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:20:35.855+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:35.856070+0000) 2024-02-20T09:20:36.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:20:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:20:35.840+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:20:36.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:20:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:20:35.840+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:20:36.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:20:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:20:35.840+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:20:36.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:20:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:20:36.041+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:36.041821+0000) 2024-02-20T09:20:36.496 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:20:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:20:36.488+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 66882 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:20:36.497 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:20:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:20:36.486+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:36.486687+0000) 2024-02-20T09:20:36.497 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:20:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:20:36.482+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:36.483095+0000) 2024-02-20T09:20:36.497 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:20:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:20:36.460+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:20:36.498 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:20:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:20:36.460+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:20:36.498 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:20:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:20:36.460+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:20:36.498 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:20:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:20:36.482+0000 7f8a5f86e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:36.482825+0000) 2024-02-20T09:20:36.800 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:20:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:20:36.516+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:20:36.800 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:20:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:20:36.516+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:20:36.800 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:20:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:20:36.516+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:20:36.800 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:20:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:20:36.516+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:20:37.081 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:20:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:20:36.925+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:36.925879+0000) 2024-02-20T09:20:37.082 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:20:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:20:36.855+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:36.856347+0000) 2024-02-20T09:20:37.082 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:20:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:20:36.798+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:20:37.082 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:20:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:20:36.798+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:20:37.082 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:20:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:20:36.798+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:20:37.082 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:20:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:20:37.041+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:37.041949+0000) 2024-02-20T09:20:37.846 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:20:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:20:37.519+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:20:37.846 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:20:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:20:37.519+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:20:37.846 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:20:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:20:37.519+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:20:37.847 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:20:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:20:37.519+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:20:37.847 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:20:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:20:37.505+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:20:37.847 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:20:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:20:37.505+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:20:37.847 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:20:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:20:37.505+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:20:38.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:20:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:20:37.925+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:37.926001+0000) 2024-02-20T09:20:38.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:20:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:20:37.859+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:37.859913+0000) 2024-02-20T09:20:38.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:20:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:20:37.844+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:20:38.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:20:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:20:37.844+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:20:38.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:20:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:20:37.844+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:20:38.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:20:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:20:38.041+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:38.042084+0000) 2024-02-20T09:20:38.863 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:20:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:20:38.516+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:20:38.864 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:20:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:20:38.516+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:20:38.864 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:20:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:20:38.516+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:20:38.864 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:20:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:20:38.559+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:20:38.864 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:20:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:20:38.559+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:20:38.864 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:20:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:20:38.559+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:20:38.864 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:20:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:20:38.560+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:20:39.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:20:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:20:38.925+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:38.926181+0000) 2024-02-20T09:20:39.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:20:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:20:38.861+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:38.862179+0000) 2024-02-20T09:20:39.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:20:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:20:38.867+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:20:39.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:20:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:20:38.867+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:20:39.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:20:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:20:38.867+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:20:39.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:20:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:20:39.041+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:39.042197+0000) 2024-02-20T09:20:39.742 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:20:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:20:39.468+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:20:39.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:20:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:20:39.468+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:20:39.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:20:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:20:39.468+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:20:39.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:20:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:20:39.553+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:20:39.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:20:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:20:39.553+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:20:39.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:20:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:20:39.553+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:20:39.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:20:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:20:39.553+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:20:40.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:20:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:20:39.924+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:39.926333+0000) 2024-02-20T09:20:40.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:20:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:20:39.863+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:39.864707+0000) 2024-02-20T09:20:40.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:20:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:20:39.875+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:20:40.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:20:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:20:39.875+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:20:40.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:20:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:20:39.875+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:20:40.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:20:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:20:40.040+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:40.042310+0000) 2024-02-20T09:20:40.674 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:20:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:20:40.418+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:20:40.674 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:20:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:20:40.418+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:20:40.674 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:20:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:20:40.418+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:20:40.675 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:20:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:20:40.563+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:20:40.675 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:20:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:20:40.563+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:20:40.675 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:20:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:20:40.563+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:20:40.675 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:20:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:20:40.563+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:20:40.928 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:20:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:20:40.865+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:40.866658+0000) 2024-02-20T09:20:40.928 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:20:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:20:40.896+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:20:40.928 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:20:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:20:40.896+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:20:40.928 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:20:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:20:40.896+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:20:41.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:20:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:20:40.925+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:40.926502+0000) 2024-02-20T09:20:41.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:20:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:20:41.041+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:41.042465+0000) 2024-02-20T09:20:41.742 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:20:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:20:41.495+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 66923 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:20:41.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:20:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:20:41.489+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:41.490573+0000) 2024-02-20T09:20:41.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:20:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:20:41.427+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:20:41.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:20:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:20:41.427+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:20:41.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:20:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:20:41.427+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:20:41.744 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:20:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:20:41.488+0000 7f8a5f86e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:41.490345+0000) 2024-02-20T09:20:41.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:20:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:20:41.492+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:41.494011+0000) 2024-02-20T09:20:41.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:20:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:20:41.517+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:20:41.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:20:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:20:41.517+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:20:41.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:20:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:20:41.517+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:20:41.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:20:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:20:41.517+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:20:42.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:20:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:20:41.925+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:41.926630+0000) 2024-02-20T09:20:42.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:20:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:20:41.865+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:41.866932+0000) 2024-02-20T09:20:42.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:20:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:20:41.864+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:20:42.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:20:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:20:41.864+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:20:42.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:20:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:20:41.864+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:20:42.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:20:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:20:42.041+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:42.042572+0000) 2024-02-20T09:20:42.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:20:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:20:42.461+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:20:42.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:20:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:20:42.461+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:20:42.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:20:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:20:42.461+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:20:42.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:20:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:20:42.515+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:20:42.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:20:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:20:42.515+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:20:42.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:20:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:20:42.515+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:20:42.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:20:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:20:42.515+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:20:43.043 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:20:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:20:42.867+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:42.869061+0000) 2024-02-20T09:20:43.044 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:20:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:20:42.925+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:42.926718+0000) 2024-02-20T09:20:43.044 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:20:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:20:42.891+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:20:43.044 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:20:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:20:42.891+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:20:43.045 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:20:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:20:42.891+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:20:43.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:20:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:20:43.041+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:43.042649+0000) 2024-02-20T09:20:43.864 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:20:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:20:43.492+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:20:43.864 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:20:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:20:43.492+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:20:43.864 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:20:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:20:43.492+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:20:43.865 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:20:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:20:43.493+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:20:43.865 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:20:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:20:43.493+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:20:43.865 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:20:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:20:43.493+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:20:43.865 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:20:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:20:43.494+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:20:44.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:20:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:20:43.925+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:43.926826+0000) 2024-02-20T09:20:44.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:20:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:20:43.870+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:43.871783+0000) 2024-02-20T09:20:44.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:20:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:20:43.861+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:20:44.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:20:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:20:43.861+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:20:44.245 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:20:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:20:43.861+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:20:44.245 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:20:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:20:44.041+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:44.042770+0000) 2024-02-20T09:20:44.855 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:20:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:20:44.511+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:20:44.855 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:20:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:20:44.511+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:20:44.855 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:20:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:20:44.511+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:20:44.855 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:20:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:20:44.519+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:20:44.856 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:20:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:20:44.519+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:20:44.856 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:20:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:20:44.519+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:20:44.856 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:20:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:20:44.519+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:20:45.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:20:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:20:44.872+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:44.873797+0000) 2024-02-20T09:20:45.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:20:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:20:44.925+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:44.926924+0000) 2024-02-20T09:20:45.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:20:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:20:44.852+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:20:45.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:20:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:20:44.852+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:20:45.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:20:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:20:44.852+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:20:45.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:20:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:20:45.041+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:45.042852+0000) 2024-02-20T09:20:45.838 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:20:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:20:45.537+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:20:45.838 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:20:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:20:45.537+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:20:45.838 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:20:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:20:45.537+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:20:45.838 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:20:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:20:45.501+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:20:45.839 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:20:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:20:45.501+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:20:45.839 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:20:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:20:45.501+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:20:45.839 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:20:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:20:45.501+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:20:46.091 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:20:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:20:45.874+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:45.876009+0000) 2024-02-20T09:20:46.091 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:20:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:20:45.925+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:45.927087+0000) 2024-02-20T09:20:46.091 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:20:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:20:45.834+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:20:46.092 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:20:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:20:45.834+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:20:46.092 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:20:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:20:45.834+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:20:46.092 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:20:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:20:46.041+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:46.042998+0000) 2024-02-20T09:20:46.847 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:20:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:20:46.502+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 66961 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:20:46.847 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:20:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:20:46.497+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:46.498768+0000) 2024-02-20T09:20:46.848 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:20:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:20:46.496+0000 7f8a5f86e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:46.498167+0000) 2024-02-20T09:20:46.848 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:20:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:20:46.522+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:20:46.848 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:20:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:20:46.522+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:20:46.848 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:20:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:20:46.522+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:20:46.848 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:20:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:20:46.499+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:46.500999+0000) 2024-02-20T09:20:46.848 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:20:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:20:46.547+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:20:46.849 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:20:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:20:46.547+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:20:46.849 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:20:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:20:46.547+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:20:46.849 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:20:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:20:46.547+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:20:47.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:20:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:20:46.925+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:46.927260+0000) 2024-02-20T09:20:47.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:20:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:20:46.874+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:46.876195+0000) 2024-02-20T09:20:47.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:20:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:20:46.844+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:20:47.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:20:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:20:46.844+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:20:47.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:20:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:20:46.844+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:20:47.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:20:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:20:47.041+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:47.043106+0000) 2024-02-20T09:20:47.799 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:20:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:20:47.544+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:20:47.799 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:20:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:20:47.544+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:20:47.799 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:20:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:20:47.544+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:20:47.800 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:20:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:20:47.506+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:20:47.800 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:20:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:20:47.506+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:20:47.800 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:20:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:20:47.506+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:20:47.800 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:20:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:20:47.506+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:20:48.096 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:20:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:20:47.925+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:47.927368+0000) 2024-02-20T09:20:48.096 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:20:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:20:47.877+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:47.878648+0000) 2024-02-20T09:20:48.097 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:20:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:20:47.796+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:20:48.097 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:20:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:20:47.796+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:20:48.097 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:20:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:20:47.796+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:20:48.097 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:20:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:20:48.041+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:48.043203+0000) 2024-02-20T09:20:48.823 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:20:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:20:48.571+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:20:48.823 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:20:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:20:48.571+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:20:48.824 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:20:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:20:48.571+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:20:48.824 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:20:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:20:48.493+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:20:48.824 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:20:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:20:48.493+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:20:48.824 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:20:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:20:48.493+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:20:48.824 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:20:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:20:48.493+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:20:49.096 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:20:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:20:48.926+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:48.927484+0000) 2024-02-20T09:20:49.097 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:20:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:20:48.879+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:48.881053+0000) 2024-02-20T09:20:49.097 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:20:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:20:48.820+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:20:49.097 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:20:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:20:48.820+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:20:49.097 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:20:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:20:48.820+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:20:49.098 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:20:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:20:49.041+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:49.043332+0000) 2024-02-20T09:20:49.742 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:20:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:20:49.567+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:20:49.742 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:20:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:20:49.567+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:20:49.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:20:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:20:49.567+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:20:49.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:20:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:20:49.457+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:20:49.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:20:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:20:49.457+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:20:49.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:20:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:20:49.457+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:20:49.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:20:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:20:49.457+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:20:50.044 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:20:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:20:49.882+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:49.883463+0000) 2024-02-20T09:20:50.045 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:20:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:20:49.926+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:49.927572+0000) 2024-02-20T09:20:50.045 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:20:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:20:49.772+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:20:50.045 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:20:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:20:49.772+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:20:50.045 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:20:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:20:49.772+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:20:50.416 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:20:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:20:50.042+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:50.043465+0000) 2024-02-20T09:20:50.675 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:20:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:20:50.413+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:20:50.675 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:20:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:20:50.413+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:20:50.676 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:20:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:20:50.413+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:20:50.676 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:20:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:20:50.413+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:20:50.676 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:20:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:20:50.583+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:20:50.676 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:20:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:20:50.583+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:20:50.676 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:20:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:20:50.583+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:20:50.928 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:20:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:20:50.884+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:50.885671+0000) 2024-02-20T09:20:50.929 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:20:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:20:50.778+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:20:50.929 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:20:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:20:50.778+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:20:50.929 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:20:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:20:50.778+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:20:51.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:20:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:20:50.926+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:50.927664+0000) 2024-02-20T09:20:51.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:20:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:20:51.042+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:51.043561+0000) 2024-02-20T09:20:51.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:20:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:20:51.504+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:51.505579+0000) 2024-02-20T09:20:51.743 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:20:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:20:51.510+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 67002 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:20:51.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:20:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:20:51.443+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:20:51.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:20:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:20:51.443+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:20:51.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:20:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:20:51.443+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:20:51.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:20:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:20:51.443+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:20:51.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:20:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:20:51.507+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:51.508979+0000) 2024-02-20T09:20:51.744 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:20:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:20:51.504+0000 7f8a5f86e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:51.505511+0000) 2024-02-20T09:20:51.744 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:20:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:20:51.574+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:20:51.744 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:20:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:20:51.574+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:20:51.744 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:20:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:20:51.574+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:20:52.045 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:20:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:20:51.884+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:51.885873+0000) 2024-02-20T09:20:52.045 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:20:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:20:51.926+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:51.927750+0000) 2024-02-20T09:20:52.046 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:20:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:20:51.774+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:20:52.046 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:20:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:20:51.774+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:20:52.046 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:20:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:20:51.774+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:20:52.422 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:20:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:20:52.042+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:52.043657+0000) 2024-02-20T09:20:52.675 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:20:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:20:52.601+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:20:52.676 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:20:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:20:52.601+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:20:52.676 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:20:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:20:52.601+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:20:52.676 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:20:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:20:52.419+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:20:52.676 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:20:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:20:52.419+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:20:52.677 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:20:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:20:52.419+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:20:52.677 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:20:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:20:52.420+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:20:52.929 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:20:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:20:52.886+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:52.888109+0000) 2024-02-20T09:20:52.929 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:20:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:20:52.815+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:20:52.929 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:20:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:20:52.815+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:20:52.929 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:20:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:20:52.815+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:20:53.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:20:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:20:52.926+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:52.927847+0000) 2024-02-20T09:20:53.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:20:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:20:53.042+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:53.043749+0000) 2024-02-20T09:20:53.742 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:20:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:20:53.636+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:20:53.742 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:20:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:20:53.636+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:20:53.742 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:20:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:20:53.636+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:20:53.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:20:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:20:53.444+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:20:53.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:20:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:20:53.444+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:20:53.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:20:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:20:53.444+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:20:53.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:20:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:20:53.444+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:20:54.101 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:20:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:20:53.926+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:53.928019+0000) 2024-02-20T09:20:54.101 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:20:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:20:53.889+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:53.890760+0000) 2024-02-20T09:20:54.102 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:20:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:20:53.824+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:20:54.102 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:20:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:20:53.824+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:20:54.102 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:20:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:20:53.824+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:20:54.102 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:20:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:20:54.042+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:54.043864+0000) 2024-02-20T09:20:54.673 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:20:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:20:54.397+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:20:54.674 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:20:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:20:54.397+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:20:54.674 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:20:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:20:54.397+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:20:54.674 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:20:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:20:54.397+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:20:54.929 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:20:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:20:54.892+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:54.893467+0000) 2024-02-20T09:20:54.929 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:20:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:20:54.671+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:20:54.930 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:20:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:20:54.671+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:20:54.930 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:20:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:20:54.671+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:20:54.930 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:20:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:20:54.806+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:20:54.930 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:20:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:20:54.806+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:20:54.930 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:20:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:20:54.806+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:20:55.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:20:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:20:54.926+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:54.928165+0000) 2024-02-20T09:20:55.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:20:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:20:55.042+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:55.043947+0000) 2024-02-20T09:20:55.742 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:20:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:20:55.669+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:20:55.742 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:20:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:20:55.669+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:20:55.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:20:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:20:55.669+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:20:55.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:20:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:20:55.432+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:20:55.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:20:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:20:55.432+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:20:55.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:20:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:20:55.432+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:20:55.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:20:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:20:55.433+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:20:56.101 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:20:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:20:55.926+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:55.928315+0000) 2024-02-20T09:20:56.102 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:20:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:20:55.894+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:55.895639+0000) 2024-02-20T09:20:56.102 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:20:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:20:55.797+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:20:56.102 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:20:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:20:55.797+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:20:56.102 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:20:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:20:55.797+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:20:56.102 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:20:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:20:56.042+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:56.044091+0000) 2024-02-20T09:20:56.686 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:20:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:20:56.518+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 67040 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:20:56.687 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:20:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:20:56.512+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:56.513414+0000) 2024-02-20T09:20:56.687 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:20:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:20:56.511+0000 7f8a5f86e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:56.513143+0000) 2024-02-20T09:20:56.687 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:20:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:20:56.683+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:20:56.687 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:20:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:20:56.683+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:20:56.688 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:20:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:20:56.683+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:20:56.688 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:20:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:20:56.430+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:20:56.688 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:20:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:20:56.430+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:20:56.688 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:20:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:20:56.430+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:20:56.688 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:20:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:20:56.430+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:20:56.688 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:20:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:20:56.516+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:56.517535+0000) 2024-02-20T09:20:56.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:20:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:20:56.927+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:56.928440+0000) 2024-02-20T09:20:56.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:20:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:20:56.894+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:56.895884+0000) 2024-02-20T09:20:56.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:20:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:20:56.772+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:20:56.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:20:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:20:56.772+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:20:56.994 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:20:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:20:56.772+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:20:57.395 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:20:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:20:57.042+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:57.044176+0000) 2024-02-20T09:20:57.676 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:20:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:20:57.393+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:20:57.676 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:20:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:20:57.393+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:20:57.677 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:20:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:20:57.393+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:20:57.677 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:20:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:20:57.393+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:20:57.929 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:20:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:20:57.896+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:57.898122+0000) 2024-02-20T09:20:57.930 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:20:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:20:57.723+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:20:57.930 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:20:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:20:57.723+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:20:57.930 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:20:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:20:57.723+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:20:57.930 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:20:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:20:57.730+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:20:57.930 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:20:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:20:57.730+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:20:57.931 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:20:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:20:57.730+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:20:58.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:20:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:20:57.927+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:57.928525+0000) 2024-02-20T09:20:58.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:20:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:20:58.042+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:58.044275+0000) 2024-02-20T09:20:58.676 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:20:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:20:58.364+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:20:58.676 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:20:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:20:58.364+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:20:58.677 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:20:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:20:58.364+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:20:58.677 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:20:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:20:58.364+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:20:58.929 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:20:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:20:58.899+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:58.900436+0000) 2024-02-20T09:20:58.930 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:20:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:20:58.708+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:20:58.930 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:20:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:20:58.708+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:20:58.930 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:20:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:20:58.708+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:20:58.930 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:20:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:20:58.699+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:20:58.931 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:20:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:20:58.699+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:20:58.931 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:20:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:20:58.699+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:20:59.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:20:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:20:58.927+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:58.928620+0000) 2024-02-20T09:20:59.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:20:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:20:59.043+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:59.044426+0000) 2024-02-20T09:20:59.676 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:20:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:20:59.358+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:20:59.676 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:20:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:20:59.358+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:20:59.677 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:20:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:20:59.358+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:20:59.677 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:20:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:20:59.359+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:20:59.930 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:20:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:20:59.901+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:59.902693+0000) 2024-02-20T09:20:59.930 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:20:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:20:59.744+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:20:59.930 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:20:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:20:59.744+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:20:59.930 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:20:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:20:59.744+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:20:59.931 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:20:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:20:59.696+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:20:59.931 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:20:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:20:59.696+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:20:59.931 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:20:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:20:59.696+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:21:00.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:20:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:20:59.927+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:20:59.928712+0000) 2024-02-20T09:21:00.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:21:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:21:00.043+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:00.044499+0000) 2024-02-20T09:21:00.574 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:21:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:21:00.313+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:21:00.574 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:21:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:21:00.313+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:21:00.575 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:21:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:21:00.313+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:21:00.575 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:21:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:21:00.313+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:21:00.907 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:21:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:21:00.784+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:21:00.907 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:21:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:21:00.784+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:21:00.907 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:21:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:21:00.784+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:21:00.907 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:21:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:21:00.736+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:21:00.908 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:21:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:21:00.736+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:21:00.908 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:21:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:21:00.736+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:21:01.176 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:21:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:21:00.927+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:00.928831+0000) 2024-02-20T09:21:01.177 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:21:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:21:00.904+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:00.905861+0000) 2024-02-20T09:21:01.177 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:21:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:21:01.043+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:01.044612+0000) 2024-02-20T09:21:01.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:21:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:21:01.300+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:21:01.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:21:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:21:01.300+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:21:01.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:21:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:21:01.300+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:21:01.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:21:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:21:01.300+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:21:01.907 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:21:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:21:01.525+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 67081 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:21:01.908 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:21:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:21:01.523+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:01.524725+0000) 2024-02-20T09:21:01.908 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:21:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:21:01.519+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:01.520895+0000) 2024-02-20T09:21:01.908 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:21:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:21:01.710+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:21:01.908 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:21:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:21:01.710+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:21:01.909 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:21:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:21:01.710+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:21:01.909 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:21:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:21:01.519+0000 7f8a5f86e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:01.521183+0000) 2024-02-20T09:21:01.909 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:21:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:21:01.741+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:21:01.909 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:21:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:21:01.741+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:21:01.910 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:21:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:21:01.741+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:21:02.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:21:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:21:01.927+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:01.928925+0000) 2024-02-20T09:21:02.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:21:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:21:01.904+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:01.906079+0000) 2024-02-20T09:21:02.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:21:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:21:02.043+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:02.044707+0000) 2024-02-20T09:21:02.574 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:21:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:21:02.312+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:21:02.575 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:21:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:21:02.312+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:21:02.575 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:21:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:21:02.312+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:21:02.575 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:21:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:21:02.312+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:21:02.909 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:21:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:21:02.746+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:21:02.909 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:21:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:21:02.746+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:21:02.910 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:21:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:21:02.746+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:21:02.910 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:21:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:21:02.785+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:21:02.910 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:21:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:21:02.785+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:21:02.910 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:21:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:21:02.785+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:21:03.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:21:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:21:02.927+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:02.929029+0000) 2024-02-20T09:21:03.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:21:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:21:02.906+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:02.908299+0000) 2024-02-20T09:21:03.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:21:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:21:03.043+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:03.044807+0000) 2024-02-20T09:21:03.575 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:21:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:21:03.320+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:21:03.575 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:21:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:21:03.320+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:21:03.575 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:21:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:21:03.320+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:21:03.575 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:21:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:21:03.320+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:21:03.912 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:21:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:21:03.779+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:21:03.912 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:21:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:21:03.779+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:21:03.912 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:21:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:21:03.779+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:21:03.912 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:21:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:21:03.728+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:21:03.913 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:21:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:21:03.728+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:21:03.913 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:21:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:21:03.728+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:21:04.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:21:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:21:03.909+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:03.910875+0000) 2024-02-20T09:21:04.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:21:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:21:03.927+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:03.929144+0000) 2024-02-20T09:21:04.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:21:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:21:04.043+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:04.044928+0000) 2024-02-20T09:21:04.575 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:21:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:21:04.369+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:21:04.575 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:21:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:21:04.369+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:21:04.575 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:21:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:21:04.369+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:21:04.575 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:21:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:21:04.370+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:21:04.914 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:21:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:21:04.810+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:21:04.914 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:21:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:21:04.810+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:21:04.914 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:21:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:21:04.810+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:21:04.915 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:21:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:21:04.775+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:21:04.915 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:21:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:21:04.775+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:21:04.915 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:21:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:21:04.775+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:21:05.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:21:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:21:04.927+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:04.929264+0000) 2024-02-20T09:21:05.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:21:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:21:04.911+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:04.913019+0000) 2024-02-20T09:21:05.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:21:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:21:05.043+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:05.045046+0000) 2024-02-20T09:21:05.677 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:21:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:21:05.363+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:21:05.677 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:21:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:21:05.363+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:21:05.677 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:21:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:21:05.363+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:21:05.677 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:21:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:21:05.363+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:21:05.930 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:21:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:21:05.913+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:05.915020+0000) 2024-02-20T09:21:05.931 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:21:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:21:05.835+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:21:05.931 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:21:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:21:05.835+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:21:05.931 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:21:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:21:05.835+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:21:05.931 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:21:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:21:05.805+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:21:05.931 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:21:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:21:05.805+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:21:05.931 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:21:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:21:05.805+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:21:06.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:21:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:21:05.927+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:05.929379+0000) 2024-02-20T09:21:06.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:21:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:21:06.043+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:06.045136+0000) 2024-02-20T09:21:06.531 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:21:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:21:06.397+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:21:06.531 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:21:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:21:06.397+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:21:06.531 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:21:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:21:06.397+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:21:06.531 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:21:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:21:06.397+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:21:06.832 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:21:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:21:06.526+0000 7f8a5f86e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:06.528228+0000) 2024-02-20T09:21:06.832 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:21:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:21:06.530+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:06.532122+0000) 2024-02-20T09:21:06.833 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:21:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:21:06.533+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 67119 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:21:06.833 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:21:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:21:06.527+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:06.528419+0000) 2024-02-20T09:21:06.833 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:21:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:21:06.829+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:21:06.833 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:21:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:21:06.829+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:21:06.833 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:21:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:21:06.829+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:21:07.112 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:21:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:21:06.913+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:06.915294+0000) 2024-02-20T09:21:07.112 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:21:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:21:07.043+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:07.045308+0000) 2024-02-20T09:21:07.112 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:21:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:21:06.871+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:21:07.113 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:21:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:21:06.871+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:21:07.113 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:21:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:21:06.871+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:21:07.113 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:21:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:21:06.928+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:06.929500+0000) 2024-02-20T09:21:07.677 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:21:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:21:07.363+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:21:07.677 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:21:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:21:07.363+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:21:07.677 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:21:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:21:07.363+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:21:07.678 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:21:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:21:07.363+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:21:07.930 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:21:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:21:07.916+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:07.917812+0000) 2024-02-20T09:21:07.931 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:21:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:21:07.854+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:21:07.931 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:21:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:21:07.854+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:21:07.931 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:21:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:21:07.854+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:21:07.931 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:21:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:21:07.894+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:21:07.931 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:21:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:21:07.894+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:21:07.932 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:21:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:21:07.894+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:21:08.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:21:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:21:07.928+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:07.929579+0000) 2024-02-20T09:21:08.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:21:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:21:08.044+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:08.045474+0000) 2024-02-20T09:21:08.677 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:21:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:21:08.390+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:21:08.677 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:21:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:21:08.390+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:21:08.678 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:21:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:21:08.390+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:21:08.678 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:21:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:21:08.390+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:21:08.931 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:21:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:21:08.919+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:08.920436+0000) 2024-02-20T09:21:08.931 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:21:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:21:08.870+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:21:08.931 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:21:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:21:08.870+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:21:08.931 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:21:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:21:08.870+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:21:08.932 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:21:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:21:08.854+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:21:08.932 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:21:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:21:08.854+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:21:08.932 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:21:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:21:08.854+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:21:09.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:21:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:21:08.929+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:08.929665+0000) 2024-02-20T09:21:09.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:21:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:21:09.044+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:09.045592+0000) 2024-02-20T09:21:09.677 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:21:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:21:09.416+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:21:09.678 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:21:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:21:09.416+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:21:09.678 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:21:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:21:09.416+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:21:09.678 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:21:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:21:09.416+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:21:09.930 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:21:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:21:09.921+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:09.922732+0000) 2024-02-20T09:21:09.931 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:21:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:21:09.861+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:21:09.931 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:21:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:21:09.861+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:21:09.931 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:21:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:21:09.861+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:37.764623+0000 (oldest deadline 2024-02-20T09:10:57.764623+0000) 2024-02-20T09:21:09.931 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:21:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:21:09.835+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:21:09.932 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:21:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:21:09.835+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:21:09.932 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:21:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:21:09.835+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:21:10.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:21:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:21:09.929+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:09.929754+0000) 2024-02-20T09:21:10.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:21:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:21:10.044+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:10.045755+0000) 2024-02-20T09:21:10.677 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:21:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:21:10.418+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:21:10.678 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:21:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:21:10.418+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:21:10.678 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:21:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:21:10.418+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:21:10.678 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:21:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:21:10.418+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:21:10.931 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:21:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:21:10.923+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:10.924659+0000) 2024-02-20T09:21:10.931 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:21:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:21:10.884+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:21:10.931 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:21:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:21:10.884+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:21:10.931 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:21:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:21:10.884+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:21:11.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:21:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:21:11.044+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:11.045860+0000) 2024-02-20T09:21:11.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:21:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:21:10.928+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:10.929837+0000) 2024-02-20T09:21:11.670 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:21:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:21:11.535+0000 7f8a5f86e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:11.536492+0000) 2024-02-20T09:21:11.671 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:21:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:21:11.534+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:11.536007+0000) 2024-02-20T09:21:11.671 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:21:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:21:11.541+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 67161 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:21:11.672 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:21:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:21:11.401+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:21:11.672 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:21:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:21:11.401+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:21:11.672 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:21:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:21:11.401+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:21:11.672 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:21:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:21:11.401+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:21:11.672 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:21:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:21:11.538+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:11.539616+0000) 2024-02-20T09:21:11.926 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:21:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:21:11.883+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:21:11.926 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:21:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:21:11.883+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:21:11.926 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:21:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:21:11.883+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:21:12.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:21:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:21:11.928+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:11.929945+0000) 2024-02-20T09:21:12.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:21:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:21:11.924+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:11.924928+0000) 2024-02-20T09:21:12.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:21:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:21:12.045+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:12.045978+0000) 2024-02-20T09:21:12.678 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:21:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:21:12.417+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:21:12.678 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:21:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:21:12.417+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:21:12.678 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:21:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:21:12.417+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:21:12.678 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:21:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:21:12.417+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:21:12.928 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:21:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:21:12.925+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:21:12.929 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:21:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:21:12.925+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:21:12.929 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:21:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:21:12.925+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:21:13.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:21:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:21:12.929+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:12.930077+0000) 2024-02-20T09:21:13.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:21:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:21:13.045+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:13.046072+0000) 2024-02-20T09:21:13.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:21:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:21:12.927+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:12.927479+0000) 2024-02-20T09:21:13.678 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:21:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:21:13.377+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:21:13.678 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:21:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:21:13.377+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:21:13.678 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:21:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:21:13.377+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:21:13.678 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:21:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:21:13.377+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:21:14.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:21:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:21:13.929+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:13.929502+0000) 2024-02-20T09:21:14.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:21:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:21:13.929+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:13.930175+0000) 2024-02-20T09:21:14.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:21:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:21:13.962+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:21:14.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:21:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:21:13.962+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:21:14.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:21:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:21:13.962+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:21:14.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:21:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:21:14.045+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:14.046182+0000) 2024-02-20T09:21:14.678 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:21:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:21:14.357+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:21:14.678 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:21:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:21:14.357+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:21:14.678 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:21:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:21:14.357+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:21:14.678 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:21:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:21:14.357+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:21:15.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:21:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:21:14.929+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:14.930303+0000) 2024-02-20T09:21:15.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:21:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:21:14.931+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:14.931824+0000) 2024-02-20T09:21:15.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:21:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:21:14.964+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:21:15.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:21:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:21:14.964+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:21:15.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:21:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:21:14.964+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:21:15.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:21:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:21:15.045+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:15.046299+0000) 2024-02-20T09:21:15.678 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:21:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:21:15.331+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:21:15.678 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:21:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:21:15.331+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:21:15.679 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:21:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:21:15.331+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:21:15.679 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:21:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:21:15.331+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:21:16.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:21:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:21:15.933+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:15.934120+0000) 2024-02-20T09:21:16.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:21:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:21:15.930+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:15.930421+0000) 2024-02-20T09:21:16.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:21:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:21:15.997+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:21:16.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:21:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:21:15.997+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:21:16.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:21:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:21:15.997+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:21:16.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:21:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:21:16.046+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:16.046428+0000) 2024-02-20T09:21:16.671 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:21:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:21:16.543+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:16.543837+0000) 2024-02-20T09:21:16.671 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:21:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:21:16.550+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 67199 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:21:16.672 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:21:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:21:16.543+0000 7f8a5f86e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:16.543729+0000) 2024-02-20T09:21:16.672 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:21:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:21:16.345+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:21:16.672 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:21:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:21:16.345+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:21:16.672 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:21:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:21:16.345+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:21:16.672 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:21:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:21:16.345+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:21:16.673 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:21:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:21:16.546+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:16.547266+0000) 2024-02-20T09:21:17.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:21:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:21:16.930+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:16.930519+0000) 2024-02-20T09:21:17.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:21:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:21:16.933+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:16.934348+0000) 2024-02-20T09:21:17.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:21:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:21:17.032+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:21:17.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:21:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:21:17.032+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:21:17.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:21:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:21:17.032+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:21:17.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:21:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:21:17.046+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:17.046548+0000) 2024-02-20T09:21:17.678 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:21:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:21:17.380+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:21:17.679 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:21:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:21:17.380+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:21:17.679 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:21:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:21:17.380+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:21:17.679 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:21:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:21:17.380+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:21:18.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:21:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:21:17.930+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:17.930598+0000) 2024-02-20T09:21:18.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:21:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:21:17.936+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:17.936395+0000) 2024-02-20T09:21:18.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:21:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:21:18.046+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:18.046652+0000) 2024-02-20T09:21:18.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:21:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:21:18.050+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:21:18.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:21:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:21:18.050+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:21:18.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:21:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:21:18.050+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:21:18.679 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:21:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:21:18.359+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:21:18.679 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:21:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:21:18.359+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:21:18.679 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:21:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:21:18.359+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:21:18.680 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:21:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:21:18.359+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:21:19.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:21:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:21:18.930+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:18.930711+0000) 2024-02-20T09:21:19.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:21:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:21:18.937+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:18.938323+0000) 2024-02-20T09:21:19.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:21:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:21:19.046+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:19.046743+0000) 2024-02-20T09:21:19.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:21:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:21:19.065+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:21:19.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:21:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:21:19.065+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:21:19.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:21:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:21:19.065+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:21:19.679 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:21:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:21:19.401+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:21:19.679 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:21:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:21:19.401+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:21:19.679 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:21:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:21:19.401+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:21:19.679 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:21:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:21:19.401+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:21:20.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:21:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:21:19.930+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:19.930826+0000) 2024-02-20T09:21:20.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:21:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:21:19.939+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:19.940292+0000) 2024-02-20T09:21:20.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:21:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:21:20.022+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:21:20.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:21:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:21:20.022+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:21:20.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:21:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:21:20.022+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:21:20.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:21:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:21:20.046+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:20.046955+0000) 2024-02-20T09:21:20.679 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:21:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:21:20.396+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:21:20.679 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:21:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:21:20.396+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:21:20.679 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:21:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:21:20.396+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:21:20.679 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:21:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:21:20.396+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:21:21.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:21:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:21:20.930+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:20.931062+0000) 2024-02-20T09:21:21.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:21:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:21:20.943+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:20.943483+0000) 2024-02-20T09:21:21.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:21:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:21:20.995+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:21:21.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:21:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:21:20.995+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:21:21.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:21:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:21:20.995+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:21:21.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:21:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:21:21.046+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:21.047147+0000) 2024-02-20T09:21:21.677 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:21:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:21:21.557+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 67240 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:21:21.677 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:21:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:21:21.551+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:21.552190+0000) 2024-02-20T09:21:21.678 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:21:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:21:21.551+0000 7f8a5f86e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:21.551845+0000) 2024-02-20T09:21:21.678 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:21:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:21:21.410+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:21:21.678 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:21:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:21:21.410+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:21:21.678 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:21:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:21:21.410+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:21:21.678 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:21:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:21:21.410+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:21:21.678 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:21:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:21:21.553+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:21.554304+0000) 2024-02-20T09:21:22.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:21:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:21:21.943+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:21.943697+0000) 2024-02-20T09:21:22.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:21:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:21:21.930+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:21.931291+0000) 2024-02-20T09:21:22.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:21:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:21:22.009+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:21:22.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:21:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:21:22.009+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:21:22.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:21:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:21:22.009+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:21:22.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:21:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:21:22.046+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:22.047236+0000) 2024-02-20T09:21:22.679 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:21:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:21:22.401+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:21:22.679 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:21:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:21:22.401+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:21:22.679 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:21:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:21:22.401+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:21:22.680 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:21:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:21:22.401+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:21:23.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:21:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:21:22.931+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:22.931494+0000) 2024-02-20T09:21:23.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:21:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:21:22.945+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:22.945997+0000) 2024-02-20T09:21:23.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:21:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:21:22.998+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:21:23.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:21:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:21:22.998+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:21:23.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:21:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:21:22.998+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:21:23.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:21:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:21:23.046+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:23.047312+0000) 2024-02-20T09:21:23.679 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:21:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:21:23.362+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:21:23.679 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:21:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:21:23.362+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:21:23.680 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:21:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:21:23.362+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:21:23.680 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:21:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:21:23.362+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:21:24.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:21:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:21:23.931+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:23.931615+0000) 2024-02-20T09:21:24.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:21:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:21:23.947+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:23.948093+0000) 2024-02-20T09:21:24.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:21:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:21:24.018+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:21:24.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:21:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:21:24.018+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:21:24.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:21:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:21:24.018+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:10:50.682520+0000 (oldest deadline 2024-02-20T09:11:10.682520+0000) 2024-02-20T09:21:24.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:21:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:21:24.047+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:24.047426+0000) 2024-02-20T09:21:24.679 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:21:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:21:24.404+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:21:24.679 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:21:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:21:24.404+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:21:24.680 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:21:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:21:24.404+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:21:24.680 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:21:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:21:24.405+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:21:25.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:21:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:21:25.047+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:25.047504+0000) 2024-02-20T09:21:25.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:21:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:21:24.931+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:24.931707+0000) 2024-02-20T09:21:25.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:21:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:21:24.949+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:24.950153+0000) 2024-02-20T09:21:25.679 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:21:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:21:25.405+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:21:25.680 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:21:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:21:25.405+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:21:25.680 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:21:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:21:25.405+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:21:25.680 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:21:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:21:25.405+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:21:26.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:21:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:21:25.951+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:25.952206+0000) 2024-02-20T09:21:26.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:21:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:21:25.931+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:25.931850+0000) 2024-02-20T09:21:26.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:21:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:21:26.047+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:26.047637+0000) 2024-02-20T09:21:26.679 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:21:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:21:26.565+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 67278 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:21:26.680 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:21:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:21:26.559+0000 7f8a5f86e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:26.559539+0000) 2024-02-20T09:21:26.680 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:21:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:21:26.558+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:26.559356+0000) 2024-02-20T09:21:26.681 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:21:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:21:26.356+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:21:26.681 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:21:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:21:26.356+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:21:26.681 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:21:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:21:26.356+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:21:26.681 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:21:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:21:26.356+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:21:26.681 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:21:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:21:26.562+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:26.563286+0000) 2024-02-20T09:21:27.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:21:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:21:26.931+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:26.931956+0000) 2024-02-20T09:21:27.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:21:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:21:27.047+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:27.047755+0000) 2024-02-20T09:21:27.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:21:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:21:26.952+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:26.952471+0000) 2024-02-20T09:21:27.679 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:21:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:21:27.386+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:21:27.680 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:21:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:21:27.386+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:21:27.680 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:21:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:21:27.386+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:21:27.680 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:21:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:21:27.386+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:21:28.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:21:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:21:27.931+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:27.932151+0000) 2024-02-20T09:21:28.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:21:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:21:27.954+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:27.955172+0000) 2024-02-20T09:21:28.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:21:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:21:28.047+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:28.047842+0000) 2024-02-20T09:21:28.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:21:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:21:28.434+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:21:28.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:21:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:21:28.434+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:21:28.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:21:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:21:28.434+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:21:28.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:21:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:21:28.435+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:21:29.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:21:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:21:28.931+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:28.932319+0000) 2024-02-20T09:21:29.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:21:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:21:29.047+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:29.047953+0000) 2024-02-20T09:21:29.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:21:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:21:28.956+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:28.957293+0000) 2024-02-20T09:21:29.680 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:21:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:21:29.415+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:21:29.680 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:21:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:21:29.415+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:21:29.680 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:21:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:21:29.415+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:21:29.680 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:21:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:21:29.415+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:21:30.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:21:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:21:29.932+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:29.932510+0000) 2024-02-20T09:21:30.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:21:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:21:29.959+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:29.959442+0000) 2024-02-20T09:21:30.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:21:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:21:30.047+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:30.048056+0000) 2024-02-20T09:21:30.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:21:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:21:30.446+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:21:30.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:21:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:21:30.446+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:21:30.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:21:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:21:30.446+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:21:30.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:21:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:21:30.447+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:21:31.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:21:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:21:30.961+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:30.961777+0000) 2024-02-20T09:21:31.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:21:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:21:31.047+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:31.048229+0000) 2024-02-20T09:21:31.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:21:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:21:30.932+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:30.932631+0000) 2024-02-20T09:21:31.680 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:21:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:21:31.573+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 67319 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:21:31.681 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:21:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:21:31.566+0000 7f8a5f86e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:31.567377+0000) 2024-02-20T09:21:31.681 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:21:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:21:31.566+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:31.567179+0000) 2024-02-20T09:21:31.681 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:21:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:21:31.419+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:21:31.681 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:21:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:21:31.419+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:21:31.682 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:21:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:21:31.419+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:21:31.682 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:21:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:21:31.419+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:21:31.682 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:21:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:21:31.570+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:31.571040+0000) 2024-02-20T09:21:32.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:21:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:21:31.961+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:31.961990+0000) 2024-02-20T09:21:32.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:21:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:21:32.048+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:32.048433+0000) 2024-02-20T09:21:32.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:21:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:21:31.932+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:31.932726+0000) 2024-02-20T09:21:32.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:21:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:21:32.447+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:21:32.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:21:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:21:32.447+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:21:32.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:21:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:21:32.447+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:21:32.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:21:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:21:32.448+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:21:32.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:21:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:21:32.684+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:21:32.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:21:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:21:32.684+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:21:32.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:21:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:21:32.684+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:21:33.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:21:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:21:32.963+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:32.964172+0000) 2024-02-20T09:21:33.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:21:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:21:33.048+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:33.048517+0000) 2024-02-20T09:21:33.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:21:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:21:32.932+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:32.932838+0000) 2024-02-20T09:21:33.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:21:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:21:33.456+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:21:33.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:21:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:21:33.456+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:21:33.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:21:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:21:33.456+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:21:33.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:21:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:21:33.456+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:21:33.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:21:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:21:33.634+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:21:33.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:21:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:21:33.634+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:21:33.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:21:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:21:33.634+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:21:34.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:21:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:21:33.965+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:33.966625+0000) 2024-02-20T09:21:34.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:21:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:21:33.931+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:33.932937+0000) 2024-02-20T09:21:34.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:21:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:21:34.047+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:34.048636+0000) 2024-02-20T09:21:34.742 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:21:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:21:34.672+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:21:34.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:21:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:21:34.672+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:21:34.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:21:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:21:34.672+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:21:34.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:21:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:21:34.458+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:21:34.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:21:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:21:34.458+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:21:34.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:21:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:21:34.458+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:21:34.744 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:21:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:21:34.458+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:21:35.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:21:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:21:34.931+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:34.933050+0000) 2024-02-20T09:21:35.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:21:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:21:35.047+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:35.048737+0000) 2024-02-20T09:21:35.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:21:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:21:34.967+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:34.968680+0000) 2024-02-20T09:21:35.922 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:21:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:21:35.657+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:21:35.923 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:21:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:21:35.657+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:21:35.923 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:21:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:21:35.657+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:21:35.923 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:21:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:21:35.499+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:21:35.923 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:21:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:21:35.499+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:21:35.923 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:21:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:21:35.499+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:21:35.923 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:21:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:21:35.499+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:21:36.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:21:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:21:35.931+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:35.933192+0000) 2024-02-20T09:21:36.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:21:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:21:36.047+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:36.048875+0000) 2024-02-20T09:21:36.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:21:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:21:35.969+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:35.970560+0000) 2024-02-20T09:21:36.576 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:21:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:21:36.459+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:21:36.577 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:21:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:21:36.459+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:21:36.577 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:21:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:21:36.459+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:21:36.577 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:21:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:21:36.459+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:21:36.923 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:21:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:21:36.574+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:36.575531+0000) 2024-02-20T09:21:36.923 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:21:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:21:36.580+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 67357 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:21:36.923 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:21:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:21:36.577+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:36.578787+0000) 2024-02-20T09:21:36.924 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:21:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:21:36.575+0000 7f8a5f86e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:36.576457+0000) 2024-02-20T09:21:36.924 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:21:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:21:36.616+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:21:36.924 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:21:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:21:36.616+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:21:36.924 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:21:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:21:36.616+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:21:37.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:21:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:21:36.931+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:36.933282+0000) 2024-02-20T09:21:37.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:21:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:21:36.969+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:36.970765+0000) 2024-02-20T09:21:37.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:21:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:21:37.047+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:37.048964+0000) 2024-02-20T09:21:37.742 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:21:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:21:37.615+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:21:37.742 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:21:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:21:37.615+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:21:37.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:21:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:21:37.615+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:21:37.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:21:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:21:37.429+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:21:37.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:21:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:21:37.429+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:21:37.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:21:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:21:37.429+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:21:37.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:21:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:21:37.429+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:21:38.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:21:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:21:38.047+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:38.049066+0000) 2024-02-20T09:21:38.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:21:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:21:37.932+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:37.933403+0000) 2024-02-20T09:21:38.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:21:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:21:37.972+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:37.974265+0000) 2024-02-20T09:21:38.681 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:21:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:21:38.573+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:21:38.681 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:21:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:21:38.573+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:21:38.681 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:21:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:21:38.573+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:21:38.682 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:21:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:21:38.421+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:21:38.682 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:21:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:21:38.421+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:21:38.682 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:21:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:21:38.421+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:11:03.575066+0000 (oldest deadline 2024-02-20T09:11:23.575066+0000) 2024-02-20T09:21:38.682 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:21:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:21:38.421+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:21:39.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:21:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:21:38.932+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:38.933501+0000) 2024-02-20T09:21:39.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:21:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:21:38.975+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:38.976800+0000) 2024-02-20T09:21:39.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:21:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:21:39.047+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:39.049152+0000) 2024-02-20T09:21:39.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:21:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:21:39.431+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:21:39.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:21:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:21:39.533+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:21:39.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:21:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:21:39.533+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:21:39.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:21:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:21:39.533+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:21:40.143 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:21:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:21:39.932+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:39.933617+0000) 2024-02-20T09:21:40.144 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:21:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:21:39.977+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:39.978799+0000) 2024-02-20T09:21:40.144 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:21:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:21:40.047+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:40.049268+0000) 2024-02-20T09:21:40.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:21:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:21:40.450+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:21:40.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:21:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:21:40.548+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:21:40.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:21:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:21:40.548+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:21:40.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:21:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:21:40.548+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:21:41.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:21:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:21:41.047+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:41.049380+0000) 2024-02-20T09:21:41.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:21:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:21:40.932+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:40.933732+0000) 2024-02-20T09:21:41.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:21:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:21:40.979+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:40.981157+0000) 2024-02-20T09:21:41.710 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:21:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:21:41.582+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:41.583544+0000) 2024-02-20T09:21:41.711 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:21:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:21:41.442+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:21:41.711 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:21:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:21:41.585+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:41.586592+0000) 2024-02-20T09:21:41.711 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:21:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:21:41.520+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:21:41.712 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:21:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:21:41.520+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:21:41.712 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:21:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:21:41.520+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:21:41.712 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:21:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:21:41.582+0000 7f8a5f86e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:41.583531+0000) 2024-02-20T09:21:41.712 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:21:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:21:41.588+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 67398 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:21:41.978 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:21:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:21:41.932+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:41.933862+0000) 2024-02-20T09:21:42.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:21:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:21:41.979+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:41.981360+0000) 2024-02-20T09:21:42.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:21:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:21:42.048+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:42.049478+0000) 2024-02-20T09:21:42.681 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:21:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:21:42.415+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:21:42.682 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:21:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:21:42.564+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:21:42.682 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:21:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:21:42.564+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:21:42.682 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:21:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:21:42.564+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:21:43.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:21:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:21:42.932+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:42.934056+0000) 2024-02-20T09:21:43.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:21:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:21:42.981+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:42.983372+0000) 2024-02-20T09:21:43.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:21:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:21:43.048+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:43.049553+0000) 2024-02-20T09:21:43.681 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:21:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:21:43.422+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:21:43.682 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:21:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:21:43.536+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:21:43.682 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:21:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:21:43.536+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:21:43.682 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:21:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:21:43.536+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:21:44.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:21:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:21:44.048+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:44.049639+0000) 2024-02-20T09:21:44.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:21:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:21:43.932+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:43.934205+0000) 2024-02-20T09:21:44.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:21:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:21:43.984+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:43.985588+0000) 2024-02-20T09:21:44.682 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:21:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:21:44.384+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:21:44.682 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:21:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:21:44.520+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:21:44.682 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:21:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:21:44.520+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:21:44.682 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:21:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:21:44.520+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:21:45.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:21:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:21:44.932+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:44.934356+0000) 2024-02-20T09:21:45.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:21:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:21:44.986+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:44.987705+0000) 2024-02-20T09:21:45.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:21:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:21:45.048+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:45.049728+0000) 2024-02-20T09:21:45.682 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:21:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:21:45.344+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:21:45.682 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:21:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:21:45.527+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:21:45.682 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:21:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:21:45.527+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:21:45.682 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:21:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:21:45.527+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:21:46.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:21:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:21:45.933+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:45.934508+0000) 2024-02-20T09:21:46.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:21:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:21:45.988+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:45.990074+0000) 2024-02-20T09:21:46.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:21:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:21:46.048+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:46.049813+0000) 2024-02-20T09:21:46.682 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:21:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:21:46.595+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 67436 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:21:46.682 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:21:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:21:46.589+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:46.590763+0000) 2024-02-20T09:21:46.683 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:21:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:21:46.377+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:21:46.683 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:21:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:21:46.591+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:46.593341+0000) 2024-02-20T09:21:46.683 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:21:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:21:46.525+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:21:46.683 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:21:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:21:46.525+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:21:46.684 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:21:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:21:46.525+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:21:46.684 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:21:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:21:46.589+0000 7f8a5f86e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:46.591060+0000) 2024-02-20T09:21:47.205 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:21:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:21:46.988+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:46.990294+0000) 2024-02-20T09:21:47.206 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:21:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:21:46.933+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:46.934602+0000) 2024-02-20T09:21:47.206 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:21:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:21:47.048+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:47.049987+0000) 2024-02-20T09:21:47.206 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:21:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:21:47.202+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:21:47.206 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:21:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:21:47.202+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:21:47.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:21:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:21:47.365+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:21:47.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:21:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:21:47.202+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:21:47.924 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:21:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:21:47.506+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:21:47.924 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:21:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:21:47.506+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:21:47.924 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:21:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:21:47.506+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:21:48.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:21:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:21:47.933+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:47.934723+0000) 2024-02-20T09:21:48.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:21:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:21:47.990+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:47.992271+0000) 2024-02-20T09:21:48.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:21:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:21:48.048+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:48.050138+0000) 2024-02-20T09:21:48.581 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:21:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:21:48.352+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:21:48.581 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:21:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:21:48.483+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:21:48.581 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:21:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:21:48.483+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:21:48.582 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:21:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:21:48.483+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:21:48.582 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:21:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:21:48.253+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:21:48.582 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:21:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:21:48.253+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:21:48.582 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:21:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:21:48.253+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:21:49.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:21:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:21:48.992+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:48.994316+0000) 2024-02-20T09:21:49.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:21:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:21:48.933+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:48.934830+0000) 2024-02-20T09:21:49.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:21:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:21:49.048+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:49.050246+0000) 2024-02-20T09:21:49.581 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:21:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:21:49.320+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:21:49.581 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:21:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:21:49.476+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:21:49.581 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:21:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:21:49.476+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:21:49.581 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:21:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:21:49.476+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:21:49.582 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:21:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:21:49.284+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:21:49.582 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:21:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:21:49.284+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:21:49.582 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:21:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:21:49.284+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:21:50.152 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:21:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:21:49.933+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:49.934927+0000) 2024-02-20T09:21:50.155 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:21:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:21:49.995+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:49.996442+0000) 2024-02-20T09:21:50.155 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:21:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:21:50.049+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:50.050382+0000) 2024-02-20T09:21:50.453 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:21:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:21:50.294+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:21:50.454 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:21:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:21:50.275+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:21:50.454 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:21:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:21:50.275+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:21:50.454 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:21:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:21:50.275+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:21:50.742 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:21:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:21:50.516+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:21:50.742 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:21:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:21:50.516+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:21:50.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:21:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:21:50.516+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:21:51.233 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:21:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:21:50.998+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:50.999479+0000) 2024-02-20T09:21:51.234 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:21:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:21:50.933+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:50.935042+0000) 2024-02-20T09:21:51.234 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:21:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:21:51.049+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:51.050564+0000) 2024-02-20T09:21:51.234 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:21:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:21:51.230+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:21:51.234 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:21:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:21:51.230+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:21:51.234 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:21:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:21:51.230+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:21:51.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:21:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:21:51.271+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:21:51.924 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:21:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:21:51.603+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 67477 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:21:51.925 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:21:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:21:51.599+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:51.601165+0000) 2024-02-20T09:21:51.925 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:21:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:21:51.596+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:51.598386+0000) 2024-02-20T09:21:51.926 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:21:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:21:51.564+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:21:51.926 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:21:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:21:51.564+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:21:51.926 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:21:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:21:51.564+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:21:51.926 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:21:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:21:51.597+0000 7f8a5f86e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:51.598605+0000) 2024-02-20T09:21:52.204 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:21:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:21:51.933+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:51.935180+0000) 2024-02-20T09:21:52.205 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:21:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:21:51.998+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:51.999659+0000) 2024-02-20T09:21:52.205 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:21:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:21:52.049+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:52.050696+0000) 2024-02-20T09:21:52.205 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:21:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:21:52.201+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:21:52.205 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:21:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:21:52.201+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:21:52.205 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:21:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:21:52.201+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:21:52.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:21:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:21:52.277+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:21:52.924 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:21:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:21:52.547+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:21:52.925 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:21:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:21:52.547+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:21:52.925 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:21:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:21:52.547+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:21:53.199 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:21:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:21:52.933+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:52.935296+0000) 2024-02-20T09:21:53.199 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:21:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:21:53.001+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:53.003083+0000) 2024-02-20T09:21:53.199 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:21:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:21:53.049+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:53.050783+0000) 2024-02-20T09:21:53.199 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:21:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:21:53.196+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:21:53.200 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:21:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:21:53.196+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:21:53.200 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:21:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:21:53.196+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:21:53.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:21:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:21:53.298+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:21:53.925 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:21:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:21:53.565+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:21:53.925 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:21:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:21:53.565+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:21:53.925 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:21:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:21:53.565+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:21:54.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:21:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:21:53.934+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:53.935446+0000) 2024-02-20T09:21:54.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:21:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:21:54.005+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:54.006534+0000) 2024-02-20T09:21:54.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:21:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:21:54.049+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:54.050905+0000) 2024-02-20T09:21:54.576 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:21:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:21:54.282+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:21:54.576 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:21:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:21:54.241+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:21:54.576 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:21:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:21:54.241+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:21:54.576 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:21:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:21:54.241+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:21:54.925 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:21:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:21:54.573+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:21:54.925 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:21:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:21:54.573+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:21:54.925 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:21:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:21:54.573+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:21:55.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:21:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:21:54.934+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:54.935538+0000) 2024-02-20T09:21:55.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:21:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:21:55.008+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:55.010012+0000) 2024-02-20T09:21:55.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:21:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:21:55.049+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:55.051015+0000) 2024-02-20T09:21:55.581 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:21:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:21:55.249+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:21:55.582 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:21:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:21:55.263+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:21:55.582 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:21:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:21:55.263+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:21:55.582 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:21:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:21:55.263+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:21:55.925 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:21:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:21:55.584+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:21:55.925 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:21:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:21:55.584+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:21:55.925 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:21:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:21:55.584+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:21:56.226 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:21:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:21:55.934+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:55.935640+0000) 2024-02-20T09:21:56.227 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:21:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:21:56.049+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:56.051144+0000) 2024-02-20T09:21:56.227 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:21:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:21:56.010+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:56.012127+0000) 2024-02-20T09:21:56.227 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:21:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:21:56.224+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:21:56.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:21:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:21:56.227+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:21:56.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:21:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:21:56.227+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:21:56.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:21:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:21:56.227+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:21:56.925 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:21:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:21:56.611+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 67515 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:21:56.926 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:21:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:21:56.607+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:56.609052+0000) 2024-02-20T09:21:56.926 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:21:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:21:56.604+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:56.606047+0000) 2024-02-20T09:21:56.927 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:21:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:21:56.591+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:21:56.927 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:21:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:21:56.591+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:21:56.927 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:21:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:21:56.591+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:21:56.927 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:21:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:21:56.605+0000 7f8a5f86e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:56.606568+0000) 2024-02-20T09:21:57.233 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:21:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:21:56.934+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:56.935765+0000) 2024-02-20T09:21:57.233 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:21:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:21:57.049+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:57.051273+0000) 2024-02-20T09:21:57.234 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:21:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:21:57.010+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:57.012306+0000) 2024-02-20T09:21:57.234 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:21:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:21:57.230+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:21:57.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:21:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:21:57.272+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:21:57.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:21:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:21:57.272+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:21:57.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:21:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:21:57.272+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:21:57.925 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:21:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:21:57.560+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:21:57.925 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:21:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:21:57.560+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:21:57.925 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:21:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:21:57.560+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:21:58.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:21:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:21:57.934+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:57.935846+0000) 2024-02-20T09:21:58.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:21:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:21:58.013+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:58.014675+0000) 2024-02-20T09:21:58.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:21:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:21:58.049+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:58.051387+0000) 2024-02-20T09:21:58.567 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:21:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:21:58.280+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:21:58.567 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:21:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:21:58.299+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:21:58.567 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:21:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:21:58.299+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:21:58.567 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:21:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:21:58.299+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:21:58.925 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:21:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:21:58.564+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:21:58.926 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:21:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:21:58.564+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:21:58.926 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:21:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:21:58.564+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:21:59.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:21:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:21:59.050+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:59.051493+0000) 2024-02-20T09:21:59.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:21:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:21:58.934+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:58.935935+0000) 2024-02-20T09:21:59.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:21:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:21:59.015+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:59.016595+0000) 2024-02-20T09:21:59.582 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:21:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:21:59.295+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:21:59.582 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:21:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:21:59.519+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:21:59.582 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:21:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:21:59.519+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:21:59.583 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:21:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:21:59.519+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:21:59.583 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:21:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:21:59.321+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:21:59.583 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:21:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:21:59.321+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:21:59.583 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:21:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:21:59.321+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:22:00.161 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:22:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:22:00.017+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:00.018769+0000) 2024-02-20T09:22:00.161 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:22:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:22:00.050+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:00.051582+0000) 2024-02-20T09:22:00.162 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:21:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:21:59.934+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:21:59.936058+0000) 2024-02-20T09:22:00.454 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:22:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:22:00.299+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:22:00.455 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:22:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:22:00.305+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:22:00.455 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:22:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:22:00.305+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:22:00.455 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:22:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:22:00.305+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:22:00.742 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:22:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:22:00.561+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:22:00.742 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:22:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:22:00.561+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:22:00.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:22:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:22:00.561+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:22:01.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:22:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:22:00.934+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:00.936180+0000) 2024-02-20T09:22:01.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:22:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:22:01.019+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:01.020818+0000) 2024-02-20T09:22:01.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:22:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:22:01.050+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:01.051673+0000) 2024-02-20T09:22:01.582 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:22:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:22:01.281+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:22:01.582 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:22:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:22:01.281+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:22:01.582 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:22:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:22:01.281+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:22:01.583 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:22:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:22:01.256+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:22:01.583 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:22:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:22:01.256+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:22:01.583 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:22:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:22:01.256+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:22:01.583 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:22:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:22:01.256+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:22:01.926 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:22:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:22:01.615+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:01.617286+0000) 2024-02-20T09:22:01.926 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:22:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:22:01.613+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:01.614610+0000) 2024-02-20T09:22:01.926 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:22:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:22:01.619+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 67556 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:22:01.927 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:22:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:22:01.587+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:22:01.927 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:22:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:22:01.587+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:22:01.927 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:22:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:22:01.587+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:22:01.927 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:22:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:22:01.612+0000 7f8a5f86e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:01.614117+0000) 2024-02-20T09:22:02.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:22:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:22:01.934+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:01.936271+0000) 2024-02-20T09:22:02.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:22:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:22:02.019+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:02.021039+0000) 2024-02-20T09:22:02.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:22:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:22:02.050+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:02.051769+0000) 2024-02-20T09:22:02.582 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:22:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:22:02.257+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:22:02.582 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:22:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:22:02.257+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:22:02.583 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:22:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:22:02.257+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:22:02.583 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:22:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:22:02.287+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:22:02.583 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:22:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:22:02.287+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:22:02.583 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:22:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:22:02.287+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:22:02.583 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:22:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:22:02.287+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:22:02.925 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:22:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:22:02.633+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:22:02.926 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:22:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:22:02.633+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:22:02.926 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:22:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:22:02.633+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:22:03.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:22:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:22:02.935+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:02.936392+0000) 2024-02-20T09:22:03.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:22:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:22:03.022+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:03.024045+0000) 2024-02-20T09:22:03.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:22:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:22:03.050+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:03.051850+0000) 2024-02-20T09:22:03.582 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:22:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:22:03.287+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:22:03.583 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:22:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:22:03.287+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:22:03.583 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:22:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:22:03.287+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:22:03.583 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:22:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:22:03.287+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:22:03.583 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:22:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:22:03.268+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:22:03.583 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:22:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:22:03.268+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:22:03.584 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:22:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:22:03.268+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:22:03.926 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:22:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:22:03.617+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:22:03.926 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:22:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:22:03.617+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:22:03.926 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:22:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:22:03.617+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:22:04.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:22:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:22:03.935+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:03.936487+0000) 2024-02-20T09:22:04.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:22:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:22:04.024+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:04.026259+0000) 2024-02-20T09:22:04.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:22:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:22:04.051+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:04.052014+0000) 2024-02-20T09:22:04.582 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:22:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:22:04.259+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:22:04.583 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:22:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:22:04.259+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:22:04.583 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:22:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:22:04.259+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:22:04.583 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:22:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:22:04.241+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:22:04.583 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:22:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:22:04.241+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:22:04.583 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:22:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:22:04.241+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:22:04.584 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:22:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:22:04.241+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:22:04.927 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:22:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:22:04.609+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:22:04.927 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:22:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:22:04.609+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:22:04.927 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:22:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:22:04.609+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:22:05.227 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:22:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:22:04.936+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:04.936613+0000) 2024-02-20T09:22:05.227 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:22:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:22:05.050+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:05.052107+0000) 2024-02-20T09:22:05.228 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:22:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:22:05.028+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:05.028757+0000) 2024-02-20T09:22:05.228 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:22:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:22:05.225+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:22:05.228 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:22:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:22:05.225+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:22:05.228 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:22:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:22:05.225+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:22:05.228 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:22:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:22:05.225+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:22:05.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:22:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:22:05.274+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:22:05.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:22:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:22:05.274+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:22:05.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:22:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:22:05.274+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:22:05.926 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:22:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:22:05.641+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:22:05.926 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:22:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:22:05.641+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:22:05.926 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:22:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:22:05.641+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:22:06.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:22:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:22:06.030+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:06.031369+0000) 2024-02-20T09:22:06.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:22:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:22:06.051+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:06.052225+0000) 2024-02-20T09:22:06.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:22:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:22:05.936+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:05.936788+0000) 2024-02-20T09:22:06.583 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:22:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:22:06.261+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:22:06.583 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:22:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:22:06.261+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:22:06.583 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:22:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:22:06.261+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:22:06.583 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:22:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:22:06.261+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:22:06.583 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:22:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:22:06.296+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:22:06.584 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:22:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:22:06.296+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:22:06.584 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:22:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:22:06.296+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:22:06.926 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:22:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:22:06.628+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 67594 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:22:06.926 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:22:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:22:06.625+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:06.625734+0000) 2024-02-20T09:22:06.927 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:22:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:22:06.621+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:06.622071+0000) 2024-02-20T09:22:06.927 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:22:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:22:06.611+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:22:06.927 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:22:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:22:06.612+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:22:06.927 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:22:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:22:06.612+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:22:06.928 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:22:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:22:06.621+0000 7f8a5f86e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:06.621950+0000) 2024-02-20T09:22:07.215 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:22:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:22:06.936+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:06.936880+0000) 2024-02-20T09:22:07.215 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:22:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:22:07.051+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:07.052357+0000) 2024-02-20T09:22:07.215 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:22:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:22:07.031+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:07.031567+0000) 2024-02-20T09:22:07.215 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:22:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:22:07.213+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:22:07.216 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:22:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:22:07.213+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:22:07.216 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:22:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:22:07.213+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:22:07.216 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:22:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:22:07.213+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:22:07.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:22:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:22:07.345+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:22:07.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:22:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:22:07.345+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:22:07.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:22:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:22:07.345+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:22:07.926 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:22:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:22:07.655+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:22:07.927 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:22:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:22:07.655+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:22:07.927 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:22:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:22:07.655+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:22:08.186 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:22:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:22:07.936+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:07.936966+0000) 2024-02-20T09:22:08.186 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:22:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:22:08.052+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:08.052443+0000) 2024-02-20T09:22:08.187 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:22:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:22:08.033+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:08.033608+0000) 2024-02-20T09:22:08.187 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:22:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:22:08.184+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:22:08.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:22:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:22:08.303+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:22:08.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:22:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:22:08.303+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:22:08.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:22:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:22:08.303+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:22:08.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:22:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:22:08.184+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:22:08.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:22:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:22:08.184+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:22:08.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:22:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:22:08.184+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:22:08.927 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:22:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:22:08.635+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:22:08.927 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:22:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:22:08.636+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:22:08.927 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:22:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:22:08.636+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:22:09.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:22:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:22:09.052+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:09.052560+0000) 2024-02-20T09:22:09.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:22:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:22:08.936+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:08.937173+0000) 2024-02-20T09:22:09.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:22:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:22:09.035+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:09.035630+0000) 2024-02-20T09:22:09.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:22:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:22:09.154+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:22:09.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:22:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:22:09.154+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:22:09.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:22:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:22:09.154+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:22:09.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:22:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:22:09.154+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:22:09.583 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:22:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:22:09.274+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:22:09.583 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:22:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:22:09.274+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:22:09.583 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:22:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:22:09.274+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:22:09.885 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:22:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:22:09.589+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:22:09.885 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:22:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:22:09.589+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:22:09.886 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:22:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:22:09.589+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:22:10.145 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:22:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:22:09.936+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:09.937378+0000) 2024-02-20T09:22:10.145 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:22:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:22:10.037+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:10.037943+0000) 2024-02-20T09:22:10.145 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:22:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:22:10.052+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:10.052652+0000) 2024-02-20T09:22:10.455 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:22:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:22:10.143+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:22:10.456 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:22:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:22:10.143+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:22:10.456 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:22:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:22:10.143+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:22:10.456 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:22:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:22:10.143+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:22:10.456 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:22:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:22:10.242+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:22:10.456 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:22:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:22:10.242+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:22:10.456 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:22:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:22:10.242+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:22:10.742 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:22:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:22:10.596+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:22:10.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:22:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:22:10.596+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:22:10.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:22:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:22:10.596+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:22:11.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:22:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:22:10.937+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:10.937486+0000) 2024-02-20T09:22:11.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:22:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:22:11.052+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:11.052832+0000) 2024-02-20T09:22:11.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:22:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:22:11.039+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:11.039907+0000) 2024-02-20T09:22:11.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:22:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:22:11.126+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:22:11.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:22:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:22:11.126+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:22:11.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:22:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:22:11.126+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:22:11.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:22:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:22:11.126+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:22:11.548 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:22:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:22:11.252+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:22:11.549 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:22:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:22:11.252+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:22:11.549 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:22:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:22:11.252+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:22:11.833 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:22:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:22:11.633+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:11.633857+0000) 2024-02-20T09:22:11.833 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:22:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:22:11.629+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:11.630333+0000) 2024-02-20T09:22:11.833 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:22:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:22:11.547+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:22:11.834 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:22:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:22:11.547+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:22:11.834 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:22:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:22:11.547+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:22:11.834 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:22:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:22:11.630+0000 7f8a5f86e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:11.630890+0000) 2024-02-20T09:22:11.834 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:22:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:22:11.636+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 67635 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:22:12.150 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:22:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:22:11.937+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:11.937582+0000) 2024-02-20T09:22:12.151 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:22:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:22:12.052+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:12.052996+0000) 2024-02-20T09:22:12.151 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:22:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:22:12.039+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:12.040113+0000) 2024-02-20T09:22:12.151 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:22:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:22:12.148+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:22:12.151 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:22:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:22:12.148+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:22:12.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:22:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:22:12.148+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:22:12.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:22:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:22:12.149+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:22:12.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:22:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:22:12.273+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:22:12.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:22:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:22:12.273+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:22:12.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:22:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:22:12.273+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:22:12.927 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:22:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:22:12.543+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:22:12.927 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:22:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:22:12.543+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:22:12.927 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:22:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:22:12.543+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:22:13.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:22:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:22:12.937+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:12.937695+0000) 2024-02-20T09:22:13.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:22:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:22:13.052+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:13.053140+0000) 2024-02-20T09:22:13.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:22:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:22:13.043+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:13.043602+0000) 2024-02-20T09:22:13.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:22:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:22:13.110+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:22:13.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:22:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:22:13.110+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:22:13.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:22:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:22:13.110+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:22:13.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:22:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:22:13.111+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:22:13.539 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:22:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:22:13.243+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:22:13.539 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:22:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:22:13.243+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:22:13.540 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:22:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:22:13.243+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:22:13.927 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:22:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:22:13.537+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:22:13.928 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:22:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:22:13.537+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:22:13.928 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:22:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:22:13.537+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:22:14.178 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:22:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:22:13.937+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:13.937816+0000) 2024-02-20T09:22:14.178 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:22:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:22:14.052+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:14.053316+0000) 2024-02-20T09:22:14.179 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:22:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:22:14.045+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:14.046172+0000) 2024-02-20T09:22:14.179 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:22:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:22:14.116+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:22:14.179 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:22:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:22:14.116+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:22:14.179 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:22:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:22:14.116+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:22:14.179 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:22:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:22:14.116+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:22:14.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:22:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:22:14.284+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:22:14.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:22:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:22:14.284+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:22:14.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:22:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:22:14.284+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:22:14.904 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:22:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:22:14.507+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:22:14.905 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:22:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:22:14.507+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:22:14.905 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:22:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:22:14.507+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:22:15.178 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:22:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:22:14.937+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:14.937929+0000) 2024-02-20T09:22:15.179 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:22:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:22:15.053+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:15.053499+0000) 2024-02-20T09:22:15.179 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:22:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:22:15.047+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:15.048283+0000) 2024-02-20T09:22:15.179 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:22:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:22:15.085+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:22:15.180 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:22:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:22:15.085+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:22:15.180 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:22:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:22:15.085+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:22:15.180 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:22:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:22:15.085+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:22:15.488 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:22:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:22:15.286+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:22:15.488 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:22:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:22:15.286+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:22:15.489 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:22:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:22:15.286+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:22:15.742 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:22:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:22:15.486+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:22:15.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:22:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:22:15.486+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:22:15.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:22:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:22:15.486+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:22:16.178 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:22:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:22:15.937+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:15.938065+0000) 2024-02-20T09:22:16.179 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:22:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:22:16.053+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:16.053643+0000) 2024-02-20T09:22:16.179 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:22:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:22:16.049+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:16.050369+0000) 2024-02-20T09:22:16.179 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:22:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:22:16.065+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:22:16.179 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:22:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:22:16.065+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:22:16.180 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:22:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:22:16.065+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:22:16.180 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:22:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:22:16.065+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:22:16.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:22:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:22:16.247+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:22:16.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:22:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:22:16.247+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:22:16.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:22:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:22:16.247+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:22:16.927 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:22:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:22:16.644+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 67673 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:22:16.928 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:22:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:22:16.641+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:16.641976+0000) 2024-02-20T09:22:16.928 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:22:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:22:16.637+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:16.638069+0000) 2024-02-20T09:22:16.928 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:22:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:22:16.514+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:22:16.928 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:22:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:22:16.514+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:22:16.929 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:22:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:22:16.514+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:22:16.929 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:22:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:22:16.637+0000 7f8a5f86e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:16.638271+0000) 2024-02-20T09:22:17.179 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:22:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:22:17.053+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:17.053783+0000) 2024-02-20T09:22:17.179 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:22:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:22:16.937+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:16.938161+0000) 2024-02-20T09:22:17.179 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:22:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:22:17.050+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:17.050551+0000) 2024-02-20T09:22:17.179 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:22:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:22:17.068+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:22:17.180 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:22:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:22:17.068+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:22:17.180 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:22:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:22:17.068+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:22:17.180 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:22:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:22:17.068+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:22:17.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:22:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:22:17.282+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:22:17.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:22:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:22:17.282+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:22:17.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:22:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:22:17.282+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:22:17.927 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:22:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:22:17.549+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:22:17.927 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:22:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:22:17.549+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:22:17.928 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:22:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:22:17.549+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:22:18.181 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:22:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:22:17.937+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:17.938260+0000) 2024-02-20T09:22:18.181 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:22:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:22:18.053+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:18.053862+0000) 2024-02-20T09:22:18.182 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:22:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:22:18.024+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:22:18.182 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:22:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:22:18.024+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:22:18.182 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:22:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:22:18.024+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:22:18.182 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:22:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:22:18.024+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:22:18.182 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:22:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:22:18.052+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:18.052643+0000) 2024-02-20T09:22:18.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:22:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:22:18.232+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:22:18.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:22:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:22:18.232+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:22:18.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:22:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:22:18.232+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:22:18.927 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:22:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:22:18.544+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:22:18.928 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:22:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:22:18.544+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:22:18.928 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:22:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:22:18.544+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:22:19.183 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:22:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:22:18.938+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:18.938411+0000) 2024-02-20T09:22:19.184 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:22:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:22:19.053+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:19.053959+0000) 2024-02-20T09:22:19.184 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:22:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:22:19.004+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:22:19.184 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:22:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:22:19.004+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:22:19.184 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:22:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:22:19.004+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:22:19.184 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:22:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:22:19.005+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:22:19.184 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:22:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:22:19.054+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:19.054634+0000) 2024-02-20T09:22:19.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:22:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:22:19.196+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:22:19.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:22:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:22:19.196+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:22:19.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:22:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:22:19.196+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:22:19.885 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:22:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:22:19.564+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:22:19.885 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:22:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:22:19.564+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:22:19.885 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:22:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:22:19.564+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:22:20.162 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:22:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:22:19.938+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:19.938509+0000) 2024-02-20T09:22:20.163 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:22:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:22:20.053+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:20.054072+0000) 2024-02-20T09:22:20.163 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:22:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:22:19.998+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:22:20.163 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:22:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:22:19.998+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:22:20.163 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:22:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:22:19.998+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:22:20.163 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:22:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:22:19.998+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:22:20.164 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:22:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:22:20.056+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:20.056715+0000) 2024-02-20T09:22:20.456 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:22:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:22:20.161+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:22:20.456 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:22:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:22:20.161+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:22:20.456 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:22:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:22:20.161+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:22:20.742 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:22:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:22:20.600+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:22:20.742 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:22:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:22:20.600+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:22:20.742 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:22:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:22:20.600+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:22:21.184 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:22:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:22:20.938+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:20.938603+0000) 2024-02-20T09:22:21.184 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:22:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:22:21.053+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:21.054217+0000) 2024-02-20T09:22:21.184 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:22:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:22:21.152+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:22:21.184 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:22:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:22:21.152+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:22:21.185 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:22:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:22:21.152+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:22:21.185 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:22:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:22:20.954+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:22:21.185 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:22:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:22:20.954+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:22:21.185 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:22:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:22:20.954+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:22:21.185 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:22:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:22:20.954+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:22:21.185 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:22:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:22:21.058+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:21.058651+0000) 2024-02-20T09:22:21.928 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:22:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:22:21.649+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:21.649605+0000) 2024-02-20T09:22:21.928 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:22:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:22:21.652+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 67714 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:22:21.929 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:22:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:22:21.645+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:21.645837+0000) 2024-02-20T09:22:21.929 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:22:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:22:21.646+0000 7f8a5f86e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:21.646545+0000) 2024-02-20T09:22:21.929 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:22:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:22:21.647+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:22:21.929 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:22:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:22:21.647+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:22:21.930 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:22:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:22:21.647+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:22:22.184 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:22:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:22:21.938+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:21.938709+0000) 2024-02-20T09:22:22.184 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:22:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:22:22.053+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:22.054335+0000) 2024-02-20T09:22:22.185 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:22:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:22:21.978+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:22:22.185 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:22:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:22:21.978+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:22:22.185 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:22:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:22:21.978+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:22:22.185 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:22:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:22:21.978+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:22:22.185 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:22:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:22:22.058+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:22.058825+0000) 2024-02-20T09:22:22.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:22:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:22:22.185+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:22:22.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:22:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:22:22.185+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:22:22.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:22:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:22:22.185+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:22:22.928 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:22:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:22:22.660+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:22:22.928 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:22:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:22:22.660+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:22:22.928 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:22:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:22:22.660+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:22:23.187 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:22:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:22:22.938+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:22.938827+0000) 2024-02-20T09:22:23.187 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:22:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:22:23.054+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:23.054437+0000) 2024-02-20T09:22:23.187 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:22:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:22:23.006+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:22:23.188 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:22:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:22:23.006+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:22:23.188 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:22:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:22:23.006+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:22:23.188 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:22:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:22:23.006+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:22:23.188 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:22:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:22:23.060+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:23.061146+0000) 2024-02-20T09:22:23.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:22:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:22:23.222+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:22:23.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:22:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:22:23.222+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:22:23.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:22:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:22:23.222+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:22:23.928 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:22:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:22:23.632+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:22:23.929 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:22:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:22:23.632+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:22:23.929 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:22:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:22:23.632+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:22:24.188 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:22:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:22:23.938+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:23.938929+0000) 2024-02-20T09:22:24.189 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:22:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:22:24.054+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:24.054533+0000) 2024-02-20T09:22:24.189 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:22:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:22:23.970+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:22:24.189 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:22:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:22:23.970+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:22:24.189 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:22:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:22:23.970+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:22:24.190 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:22:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:22:23.970+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:22:24.190 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:22:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:22:24.063+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:24.063618+0000) 2024-02-20T09:22:24.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:22:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:22:24.266+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:22:24.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:22:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:22:24.266+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:22:24.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:22:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:22:24.266+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:22:24.928 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:22:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:22:24.616+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:22:24.928 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:22:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:22:24.616+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:22:24.929 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:22:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:22:24.616+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:22:25.188 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:22:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:22:24.938+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:24.939051+0000) 2024-02-20T09:22:25.189 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:22:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:22:25.054+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:25.054639+0000) 2024-02-20T09:22:25.189 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:22:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:22:25.016+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:22:25.189 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:22:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:22:25.016+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:22:25.189 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:22:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:22:25.016+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:22:25.190 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:22:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:22:25.016+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:22:25.190 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:22:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:22:25.065+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:25.065853+0000) 2024-02-20T09:22:25.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:22:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:22:25.217+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:22:25.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:22:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:22:25.217+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:22:25.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:22:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:22:25.217+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:22:25.928 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:22:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:22:25.585+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:22:25.929 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:22:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:22:25.585+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:22:25.929 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:22:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:22:25.585+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:22:26.189 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:22:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:22:25.938+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:25.939177+0000) 2024-02-20T09:22:26.189 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:22:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:22:26.054+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:26.054735+0000) 2024-02-20T09:22:26.189 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:22:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:22:26.171+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:22:26.189 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:22:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:22:26.171+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:22:26.190 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:22:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:22:26.171+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:22:26.190 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:22:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:22:25.980+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:22:26.190 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:22:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:22:25.980+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:22:26.190 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:22:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:22:25.980+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:22:26.190 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:22:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:22:25.980+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:22:26.190 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:22:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:22:26.067+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:26.068150+0000) 2024-02-20T09:22:26.928 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:22:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:22:26.660+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 67752 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:22:26.929 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:22:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:22:26.656+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:26.656586+0000) 2024-02-20T09:22:26.929 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:22:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:22:26.653+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:26.653897+0000) 2024-02-20T09:22:26.929 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:22:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:22:26.623+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:22:26.930 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:22:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:22:26.623+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:22:26.930 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:22:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:22:26.623+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:22:26.930 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:22:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:22:26.653+0000 7f8a5f86e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:26.654174+0000) 2024-02-20T09:22:27.189 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:22:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:22:26.938+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:26.939337+0000) 2024-02-20T09:22:27.189 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:22:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:22:27.054+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:27.054861+0000) 2024-02-20T09:22:27.189 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:22:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:22:27.173+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:22:27.189 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:22:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:22:27.173+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:22:27.190 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:22:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:22:27.173+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:22:27.190 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:22:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:22:26.934+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:22:27.190 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:22:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:22:26.934+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:22:27.190 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:22:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:22:26.934+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:22:27.190 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:22:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:22:26.934+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:22:27.190 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:22:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:22:27.067+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:27.068378+0000) 2024-02-20T09:22:27.928 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:22:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:22:27.670+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:22:27.929 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:22:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:22:27.670+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:22:27.929 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:22:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:22:27.670+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:22:28.191 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:22:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:22:27.938+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:27.939462+0000) 2024-02-20T09:22:28.192 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:22:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:22:28.053+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:28.054988+0000) 2024-02-20T09:22:28.192 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:22:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:22:27.958+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:22:28.192 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:22:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:22:27.958+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:22:28.192 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:22:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:22:27.958+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:22:28.192 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:22:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:22:27.958+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:22:28.192 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:22:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:22:28.068+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:28.070236+0000) 2024-02-20T09:22:28.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:22:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:22:28.198+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:22:28.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:22:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:22:28.198+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:22:28.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:22:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:22:28.198+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:22:28.929 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:22:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:22:28.672+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:22:28.929 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:22:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:22:28.672+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:22:28.929 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:22:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:22:28.672+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:22:29.192 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:22:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:22:28.938+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:28.939576+0000) 2024-02-20T09:22:29.193 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:22:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:22:29.053+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:29.055197+0000) 2024-02-20T09:22:29.193 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:22:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:22:28.956+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:22:29.193 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:22:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:22:28.956+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:22:29.194 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:22:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:22:28.956+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:22:29.194 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:22:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:22:28.956+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:22:29.194 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:22:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:22:29.070+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:29.072290+0000) 2024-02-20T09:22:29.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:22:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:22:29.236+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:22:29.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:22:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:22:29.236+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:22:29.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:22:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:22:29.236+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:22:29.887 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:22:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:22:29.689+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:22:29.888 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:22:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:22:29.689+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:22:29.888 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:22:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:22:29.689+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:22:30.193 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:22:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:22:29.938+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:29.939718+0000) 2024-02-20T09:22:30.193 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:22:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:22:30.053+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:30.055369+0000) 2024-02-20T09:22:30.193 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:22:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:22:29.984+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:22:30.194 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:22:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:22:29.984+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:22:30.194 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:22:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:22:29.984+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:22:30.194 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:22:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:22:29.984+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:22:30.194 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:22:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:22:30.074+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:30.075834+0000) 2024-02-20T09:22:30.457 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:22:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:22:30.255+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:22:30.457 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:22:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:22:30.255+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:22:30.457 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:22:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:22:30.255+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:22:30.742 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:22:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:22:30.642+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:22:30.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:22:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:22:30.642+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:22:30.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:22:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:22:30.642+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:22:31.193 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:22:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:22:30.938+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:30.939830+0000) 2024-02-20T09:22:31.193 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:22:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:22:31.054+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:31.055541+0000) 2024-02-20T09:22:31.194 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:22:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:22:30.986+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:22:31.194 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:22:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:22:30.986+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:22:31.194 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:22:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:22:30.986+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:22:31.194 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:22:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:22:30.986+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:22:31.194 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:22:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:22:31.076+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:31.077798+0000) 2024-02-20T09:22:31.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:22:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:22:31.290+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:22:31.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:22:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:22:31.290+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:22:31.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:22:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:22:31.290+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:22:31.931 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:22:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:22:31.667+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 67793 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:22:31.932 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:22:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:22:31.663+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:31.665238+0000) 2024-02-20T09:22:31.932 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:22:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:22:31.660+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:31.662330+0000) 2024-02-20T09:22:31.933 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:22:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:22:31.660+0000 7f8a5f86e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:31.662296+0000) 2024-02-20T09:22:31.933 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:22:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:22:31.668+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:22:31.933 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:22:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:22:31.668+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:22:31.933 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:22:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:22:31.668+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:22:32.193 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:22:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:22:32.054+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:32.055692+0000) 2024-02-20T09:22:32.194 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:22:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:22:31.938+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:31.939959+0000) 2024-02-20T09:22:32.194 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:22:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:22:32.011+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:22:32.194 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:22:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:22:32.011+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:22:32.194 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:22:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:22:32.011+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:22:32.194 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:22:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:22:32.012+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:22:32.194 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:22:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:22:32.076+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:32.078003+0000) 2024-02-20T09:22:32.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:22:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:22:32.265+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:22:32.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:22:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:22:32.265+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:22:32.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:22:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:22:32.265+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:22:32.931 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:22:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:22:32.683+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:22:32.932 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:22:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:22:32.683+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:22:32.932 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:22:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:22:32.683+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:22:33.195 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:22:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:22:32.938+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:32.940048+0000) 2024-02-20T09:22:33.196 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:22:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:22:33.054+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:33.055775+0000) 2024-02-20T09:22:33.196 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:22:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:22:32.971+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:22:33.196 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:22:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:22:32.971+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:22:33.197 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:22:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:22:32.971+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:22:33.197 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:22:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:22:32.971+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:22:33.197 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:22:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:22:33.080+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:33.081507+0000) 2024-02-20T09:22:33.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:22:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:22:33.271+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:22:33.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:22:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:22:33.271+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:22:33.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:22:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:22:33.271+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:22:33.929 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:22:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:22:33.675+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:22:33.930 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:22:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:22:33.675+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:22:33.930 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:22:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:22:33.675+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:22:34.198 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:22:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:22:34.054+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:34.055914+0000) 2024-02-20T09:22:34.198 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:22:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:22:33.938+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:33.940168+0000) 2024-02-20T09:22:34.198 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:22:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:22:33.949+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:22:34.198 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:22:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:22:33.949+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:22:34.199 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:22:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:22:33.949+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:22:34.199 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:22:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:22:33.949+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:22:34.199 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:22:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:22:34.082+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:34.084128+0000) 2024-02-20T09:22:34.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:22:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:22:34.284+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:22:34.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:22:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:22:34.284+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:22:34.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:22:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:22:34.284+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:22:34.929 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:22:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:22:34.666+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:22:34.930 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:22:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:22:34.666+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:22:34.930 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:22:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:22:34.666+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:22:35.198 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:22:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:22:34.938+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:34.940266+0000) 2024-02-20T09:22:35.198 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:22:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:22:35.054+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:35.056037+0000) 2024-02-20T09:22:35.199 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:22:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:22:34.996+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:22:35.199 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:22:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:22:34.996+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:22:35.199 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:22:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:22:34.996+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:22:35.199 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:22:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:22:34.996+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:22:35.199 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:22:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:22:35.084+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:35.086259+0000) 2024-02-20T09:22:35.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:22:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:22:35.248+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:22:35.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:22:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:22:35.248+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:22:35.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:22:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:22:35.248+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:22:35.929 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:22:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:22:35.655+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:22:35.930 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:22:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:22:35.655+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:22:35.930 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:22:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:22:35.655+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:22:36.198 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:22:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:22:35.939+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:35.940389+0000) 2024-02-20T09:22:36.198 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:22:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:22:36.054+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:36.056152+0000) 2024-02-20T09:22:36.199 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:22:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:22:35.950+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:22:36.199 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:22:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:22:35.950+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:22:36.199 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:22:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:22:35.950+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:22:36.199 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:22:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:22:35.950+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:22:36.199 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:22:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:22:36.087+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:36.088637+0000) 2024-02-20T09:22:36.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:22:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:22:36.210+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:22:36.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:22:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:22:36.210+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:22:36.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:22:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:22:36.210+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:22:36.920 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:22:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:22:36.675+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 67831 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:22:36.921 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:22:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:22:36.668+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:36.670006+0000) 2024-02-20T09:22:36.921 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:22:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:22:36.649+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:22:36.921 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:22:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:22:36.649+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:22:36.922 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:22:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:22:36.649+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:22:36.922 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:22:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:22:36.669+0000 7f8a5f86e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:36.670393+0000) 2024-02-20T09:22:36.922 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:22:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:22:36.671+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:36.672405+0000) 2024-02-20T09:22:36.922 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:22:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:22:36.917+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:22:36.923 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:22:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:22:36.917+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:22:36.923 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:22:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:22:36.917+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:22:36.923 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:22:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:22:36.917+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:22:37.198 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:22:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:22:36.939+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:36.940505+0000) 2024-02-20T09:22:37.199 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:22:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:22:37.087+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:37.088843+0000) 2024-02-20T09:22:37.199 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:22:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:22:37.054+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:37.056238+0000) 2024-02-20T09:22:37.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:22:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:22:37.253+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:22:37.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:22:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:22:37.253+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:22:37.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:22:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:22:37.253+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:22:37.890 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:22:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:22:37.625+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:22:37.890 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:22:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:22:37.625+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:22:37.890 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:22:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:22:37.625+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:22:38.200 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:22:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:22:37.939+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:37.940633+0000) 2024-02-20T09:22:38.201 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:22:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:22:38.054+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:38.056333+0000) 2024-02-20T09:22:38.201 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:22:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:22:37.887+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:22:38.201 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:22:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:22:37.887+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:22:38.202 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:22:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:22:37.887+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:22:38.202 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:22:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:22:37.887+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:22:38.202 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:22:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:22:38.089+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:38.090823+0000) 2024-02-20T09:22:38.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:22:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:22:38.291+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:22:38.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:22:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:22:38.291+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:22:38.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:22:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:22:38.291+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:22:38.893 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:22:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:22:38.609+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:22:38.894 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:22:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:22:38.609+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:22:38.894 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:22:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:22:38.609+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:22:39.202 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:22:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:22:38.939+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:38.940766+0000) 2024-02-20T09:22:39.203 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:22:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:22:39.055+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:39.056469+0000) 2024-02-20T09:22:39.203 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:22:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:22:38.891+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:22:39.203 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:22:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:22:38.891+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:22:39.203 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:22:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:22:38.891+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:22:39.204 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:22:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:22:38.891+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:22:39.204 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:22:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:22:39.093+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:39.094420+0000) 2024-02-20T09:22:39.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:22:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:22:39.336+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:22:39.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:22:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:22:39.336+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:22:39.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:22:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:22:39.336+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:22:39.885 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:22:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:22:39.562+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:22:39.885 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:22:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:22:39.562+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:22:39.886 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:22:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:22:39.562+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:22:40.203 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:22:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:22:39.939+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:39.940909+0000) 2024-02-20T09:22:40.203 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:22:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:22:40.055+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:40.056585+0000) 2024-02-20T09:22:40.203 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:22:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:22:39.935+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:22:40.203 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:22:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:22:39.935+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:22:40.204 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:22:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:22:39.935+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:22:40.204 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:22:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:22:39.935+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:22:40.204 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:22:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:22:40.096+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:40.097954+0000) 2024-02-20T09:22:40.457 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:22:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:22:40.326+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:22:40.457 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:22:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:22:40.326+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:22:40.457 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:22:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:22:40.326+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:22:40.742 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:22:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:22:40.611+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:22:40.742 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:22:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:22:40.611+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:22:40.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:22:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:22:40.611+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:22:41.203 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:22:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:22:41.055+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:41.056690+0000) 2024-02-20T09:22:41.203 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:22:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:22:40.939+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:40.941076+0000) 2024-02-20T09:22:41.203 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:22:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:22:40.955+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:22:41.203 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:22:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:22:40.955+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:22:41.204 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:22:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:22:40.955+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:22:41.204 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:22:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:22:40.955+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:22:41.204 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:22:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:22:41.099+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:41.100510+0000) 2024-02-20T09:22:41.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:22:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:22:41.340+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:22:41.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:22:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:22:41.340+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:22:41.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:22:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:22:41.340+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:22:41.930 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:22:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:22:41.682+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 67872 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:22:41.931 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:22:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:22:41.678+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:41.680334+0000) 2024-02-20T09:22:41.931 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:22:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:22:41.676+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:41.677732+0000) 2024-02-20T09:22:41.932 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:22:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:22:41.584+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:22:41.932 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:22:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:22:41.585+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:22:41.932 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:22:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:22:41.585+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:22:41.932 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:22:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:22:41.676+0000 7f8a5f86e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:41.677938+0000) 2024-02-20T09:22:42.203 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:22:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:22:41.939+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:41.941166+0000) 2024-02-20T09:22:42.203 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:22:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:22:42.055+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:42.056837+0000) 2024-02-20T09:22:42.203 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:22:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:22:41.987+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:22:42.204 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:22:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:22:41.987+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:22:42.204 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:22:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:22:41.987+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:22:42.204 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:22:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:22:41.987+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:22:42.204 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:22:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:22:42.099+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:42.100721+0000) 2024-02-20T09:22:42.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:22:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:22:42.320+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:22:42.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:22:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:22:42.320+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:22:42.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:22:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:22:42.320+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:22:42.930 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:22:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:22:42.593+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:22:42.931 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:22:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:22:42.593+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:22:42.931 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:22:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:22:42.593+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:22:43.205 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:22:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:22:43.055+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:43.056927+0000) 2024-02-20T09:22:43.206 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:22:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:22:42.939+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:42.941284+0000) 2024-02-20T09:22:43.206 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:22:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:22:42.973+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:22:43.206 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:22:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:22:42.973+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:22:43.206 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:22:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:22:42.973+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:22:43.206 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:22:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:22:42.973+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:22:43.206 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:22:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:22:43.101+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:43.103362+0000) 2024-02-20T09:22:43.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:22:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:22:43.330+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:22:43.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:22:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:22:43.330+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:22:43.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:22:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:22:43.330+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:22:43.931 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:22:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:22:43.626+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:22:43.931 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:22:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:22:43.626+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:22:43.931 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:22:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:22:43.626+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:22:44.207 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:22:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:22:43.940+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:43.941419+0000) 2024-02-20T09:22:44.208 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:22:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:22:44.055+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:44.057018+0000) 2024-02-20T09:22:44.208 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:22:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:22:43.938+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:22:44.208 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:22:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:22:43.938+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:22:44.208 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:22:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:22:43.938+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:22:44.208 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:22:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:22:43.938+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:22:44.209 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:22:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:22:44.104+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:44.105460+0000) 2024-02-20T09:22:44.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:22:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:22:44.282+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:22:44.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:22:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:22:44.282+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:22:44.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:22:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:22:44.282+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:22:44.931 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:22:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:22:44.594+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:22:44.931 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:22:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:22:44.594+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:22:44.931 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:22:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:22:44.594+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:22:45.208 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:22:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:22:44.940+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:44.941510+0000) 2024-02-20T09:22:45.208 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:22:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:22:45.055+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:45.057155+0000) 2024-02-20T09:22:45.208 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:22:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:22:44.929+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:22:45.209 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:22:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:22:44.929+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:22:45.209 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:22:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:22:44.929+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:22:45.209 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:22:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:22:44.929+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:22:45.209 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:22:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:22:45.106+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:45.107434+0000) 2024-02-20T09:22:45.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:22:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:22:45.245+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:22:45.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:22:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:22:45.245+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:22:45.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:22:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:22:45.245+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:22:45.889 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:22:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:22:45.589+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:22:45.889 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:22:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:22:45.589+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:22:45.889 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:22:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:22:45.589+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:22:46.208 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:22:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:22:45.940+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:45.941626+0000) 2024-02-20T09:22:46.208 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:22:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:22:46.055+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:46.057270+0000) 2024-02-20T09:22:46.209 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:22:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:22:45.886+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:22:46.209 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:22:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:22:45.886+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:22:46.209 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:22:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:22:45.886+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:22:46.209 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:22:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:22:45.886+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:22:46.209 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:22:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:22:46.108+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:46.109569+0000) 2024-02-20T09:22:46.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:22:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:22:46.215+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:22:46.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:22:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:22:46.215+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:22:46.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:22:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:22:46.215+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:22:46.859 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:22:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:22:46.690+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 67910 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:22:46.860 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:22:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:22:46.684+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:46.685773+0000) 2024-02-20T09:22:46.860 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:22:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:22:46.595+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:22:46.860 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:22:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:22:46.595+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:22:46.860 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:22:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:22:46.595+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:22:46.860 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:22:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:22:46.684+0000 7f8a5f86e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:46.685903+0000) 2024-02-20T09:22:46.861 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:22:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:22:46.687+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:46.688761+0000) 2024-02-20T09:22:46.861 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:22:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:22:46.857+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:22:46.861 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:22:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:22:46.857+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:22:46.861 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:22:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:22:46.857+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:22:46.861 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:22:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:22:46.857+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:22:47.111 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:22:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:22:46.940+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:46.941737+0000) 2024-02-20T09:22:47.111 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:22:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:22:47.056+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:47.057411+0000) 2024-02-20T09:22:47.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:22:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:22:47.108+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:47.109808+0000) 2024-02-20T09:22:47.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:22:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:22:47.180+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:22:47.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:22:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:22:47.180+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:22:47.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:22:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:22:47.180+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:22:47.890 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:22:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:22:47.641+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:22:47.891 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:22:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:22:47.641+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:22:47.891 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:22:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:22:47.641+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:22:48.164 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:22:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:22:48.056+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:48.057488+0000) 2024-02-20T09:22:48.165 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:22:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:22:47.940+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:47.941860+0000) 2024-02-20T09:22:48.165 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:22:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:22:47.888+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:22:48.165 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:22:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:22:47.888+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:22:48.165 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:22:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:22:47.888+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:22:48.165 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:22:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:22:47.888+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:22:48.165 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:22:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:22:48.110+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:48.111944+0000) 2024-02-20T09:22:48.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:22:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:22:48.161+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:22:48.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:22:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:22:48.161+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:22:48.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:22:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:22:48.161+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:22:48.881 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:22:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:22:48.600+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:22:48.882 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:22:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:22:48.600+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:22:48.882 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:22:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:22:48.600+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:22:49.141 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:22:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:22:48.940+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:48.942038+0000) 2024-02-20T09:22:49.142 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:22:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:22:49.056+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:49.057607+0000) 2024-02-20T09:22:49.142 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:22:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:22:49.138+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:22:49.142 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:22:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:22:49.138+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:22:49.142 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:22:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:22:48.879+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:22:49.142 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:22:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:22:48.879+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:22:49.142 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:22:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:22:48.879+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:22:49.142 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:22:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:22:48.879+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:22:49.143 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:22:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:22:49.112+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:49.114383+0000) 2024-02-20T09:22:49.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:22:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:22:49.138+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:22:49.885 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:22:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:22:49.635+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:22:49.885 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:22:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:22:49.636+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:22:49.885 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:22:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:22:49.636+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:22:50.212 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:22:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:22:49.940+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:49.942235+0000) 2024-02-20T09:22:50.212 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:22:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:22:49.908+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:22:50.212 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:22:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:22:49.908+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:22:50.213 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:22:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:22:49.908+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:22:50.213 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:22:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:22:49.908+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:22:50.213 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:22:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:22:50.115+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:50.117061+0000) 2024-02-20T09:22:50.213 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:22:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:22:50.056+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:50.057714+0000) 2024-02-20T09:22:50.213 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:22:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:22:50.107+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:22:50.213 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:22:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:22:50.107+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:22:50.214 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:22:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:22:50.107+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:22:50.929 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:22:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:22:50.627+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:22:50.929 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:22:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:22:50.627+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:22:50.929 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:22:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:22:50.627+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:22:51.212 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:22:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:22:50.941+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:50.942392+0000) 2024-02-20T09:22:51.212 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:22:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:22:50.926+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:22:51.213 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:22:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:22:50.926+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:22:51.213 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:22:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:22:50.926+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:22:51.213 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:22:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:22:50.926+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:22:51.213 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:22:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:22:51.118+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:51.119589+0000) 2024-02-20T09:22:51.213 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:22:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:22:51.056+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:51.057860+0000) 2024-02-20T09:22:51.214 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:22:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:22:51.103+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:22:51.214 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:22:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:22:51.103+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:22:51.214 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:22:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:22:51.103+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:22:51.909 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:22:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:22:51.698+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 67951 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:22:51.910 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:22:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:22:51.691+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:51.692694+0000) 2024-02-20T09:22:51.910 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:22:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:22:51.693+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:51.694955+0000) 2024-02-20T09:22:51.910 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:22:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:22:51.906+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:22:51.910 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:22:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:22:51.906+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:22:51.910 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:22:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:22:51.906+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:22:51.911 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:22:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:22:51.639+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:22:51.911 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:22:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:22:51.639+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:22:51.911 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:22:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:22:51.639+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:22:51.911 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:22:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:22:51.691+0000 7f8a5f86e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:51.692892+0000) 2024-02-20T09:22:52.212 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:22:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:22:51.941+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:51.942533+0000) 2024-02-20T09:22:52.213 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:22:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:22:51.906+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:22:52.213 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:22:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:22:52.118+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:52.119763+0000) 2024-02-20T09:22:52.213 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:22:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:22:52.056+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:52.057971+0000) 2024-02-20T09:22:52.213 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:22:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:22:52.138+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:22:52.213 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:22:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:22:52.138+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:22:52.214 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:22:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:22:52.138+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:22:52.891 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:22:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:22:52.592+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:22:52.891 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:22:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:22:52.592+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:22:52.891 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:22:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:22:52.592+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:22:53.151 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:22:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:22:52.941+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:52.942622+0000) 2024-02-20T09:22:53.152 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:22:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:22:53.056+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:53.058074+0000) 2024-02-20T09:22:53.152 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:22:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:22:53.149+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:22:53.152 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:22:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:22:53.149+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:22:53.152 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:22:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:22:53.149+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:22:53.153 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:22:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:22:52.888+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:22:53.153 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:22:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:22:52.888+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:22:53.153 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:22:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:22:52.888+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:22:53.153 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:22:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:22:52.888+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:22:53.153 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:22:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:22:53.120+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:53.122257+0000) 2024-02-20T09:22:53.918 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:22:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:22:53.612+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:22:53.918 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:22:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:22:53.612+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:22:53.918 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:22:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:22:53.612+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:22:54.216 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:22:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:22:53.941+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:53.942715+0000) 2024-02-20T09:22:54.217 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:22:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:22:54.056+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:54.058199+0000) 2024-02-20T09:22:54.217 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:22:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:22:54.100+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:22:54.217 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:22:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:22:54.100+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:22:54.217 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:22:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:22:54.100+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:22:54.217 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:22:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:22:53.915+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:22:54.218 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:22:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:22:53.915+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:22:54.218 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:22:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:22:53.915+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:22:54.218 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:22:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:22:53.915+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:22:54.218 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:22:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:22:54.123+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:54.124443+0000) 2024-02-20T09:22:54.840 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:22:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:22:54.643+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:22:54.841 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:22:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:22:54.643+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:22:54.841 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:22:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:22:54.643+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:22:55.127 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:22:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:22:54.941+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:54.942807+0000) 2024-02-20T09:22:55.128 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:22:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:22:54.946+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:22:55.128 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:22:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:22:54.946+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:22:55.128 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:22:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:22:54.946+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:22:55.128 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:22:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:22:54.946+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:22:55.128 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:22:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:22:55.056+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:55.058326+0000) 2024-02-20T09:22:55.129 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:22:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:22:55.065+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:22:55.129 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:22:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:22:55.065+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:22:55.129 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:22:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:22:55.065+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:22:55.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:22:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:22:55.125+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:55.126461+0000) 2024-02-20T09:22:55.916 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:22:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:22:55.622+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:22:55.917 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:22:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:22:55.622+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:22:55.917 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:22:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:22:55.622+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:22:56.216 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:22:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:22:55.941+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:55.942907+0000) 2024-02-20T09:22:56.217 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:22:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:22:56.057+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:56.058439+0000) 2024-02-20T09:22:56.217 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:22:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:22:56.059+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:22:56.217 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:22:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:22:56.059+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:22:56.217 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:22:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:22:56.059+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:22:56.217 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:22:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:22:55.914+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:22:56.218 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:22:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:22:55.914+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:22:56.218 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:22:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:22:55.914+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:22:56.218 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:22:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:22:55.914+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:22:56.218 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:22:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:22:56.127+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:56.128931+0000) 2024-02-20T09:22:56.922 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:22:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:22:56.706+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 67989 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:22:56.923 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:22:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:22:56.699+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:56.700838+0000) 2024-02-20T09:22:56.923 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:22:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:22:56.702+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:56.703493+0000) 2024-02-20T09:22:56.923 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:22:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:22:56.920+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:22:56.924 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:22:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:22:56.920+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:22:56.927 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:22:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:22:56.638+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:22:56.927 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:22:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:22:56.638+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:22:56.927 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:22:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:22:56.638+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:22:56.927 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:22:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:22:56.699+0000 7f8a5f86e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:56.700767+0000) 2024-02-20T09:22:57.217 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:22:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:22:56.941+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:56.943025+0000) 2024-02-20T09:22:57.217 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:22:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:22:56.920+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:22:57.217 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:22:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:22:56.920+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:22:57.217 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:22:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:22:57.127+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:57.129137+0000) 2024-02-20T09:22:57.218 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:22:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:22:57.014+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:22:57.218 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:22:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:22:57.014+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:22:57.218 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:22:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:22:57.014+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:22:57.218 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:22:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:22:57.057+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:57.058556+0000) 2024-02-20T09:22:57.932 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:22:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:22:57.592+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:22:57.933 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:22:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:22:57.592+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:22:57.933 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:22:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:22:57.592+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:22:58.219 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:22:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:22:58.048+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:22:58.219 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:22:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:22:58.048+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:22:58.220 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:22:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:22:58.048+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:22:58.220 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:22:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:22:58.057+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:58.058661+0000) 2024-02-20T09:22:58.220 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:22:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:22:57.962+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:22:58.220 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:22:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:22:57.962+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:22:58.220 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:22:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:22:57.962+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:22:58.220 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:22:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:22:57.962+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:22:58.221 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:22:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:22:58.129+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:58.131240+0000) 2024-02-20T09:22:58.221 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:22:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:22:57.941+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:57.943128+0000) 2024-02-20T09:22:58.925 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:22:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:22:58.603+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:22:58.925 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:22:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:22:58.603+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:22:58.925 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:22:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:22:58.603+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:22:59.221 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:22:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:22:58.941+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:58.943264+0000) 2024-02-20T09:22:59.222 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:22:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:22:58.922+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:22:59.222 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:22:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:22:58.922+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:22:59.222 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:22:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:22:58.922+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:22:59.222 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:22:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:22:58.922+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:22:59.222 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:22:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:22:59.133+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:59.134735+0000) 2024-02-20T09:22:59.222 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:22:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:22:59.057+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:59.058750+0000) 2024-02-20T09:22:59.222 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:22:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:22:59.083+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:22:59.223 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:22:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:22:59.083+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:22:59.223 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:22:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:22:59.083+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:22:59.886 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:22:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:22:59.611+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:22:59.886 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:22:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:22:59.611+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:22:59.886 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:22:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:22:59.611+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:23:00.138 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:22:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:22:59.943+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:22:59.943395+0000) 2024-02-20T09:23:00.138 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:22:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:22:59.972+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:23:00.138 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:22:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:22:59.972+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:23:00.139 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:22:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:22:59.972+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:23:00.139 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:22:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:22:59.972+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:23:00.139 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:23:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:23:00.058+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:00.058840+0000) 2024-02-20T09:23:00.139 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:23:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:23:00.083+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:23:00.139 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:23:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:23:00.083+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:23:00.139 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:23:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:23:00.083+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:23:00.459 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:23:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:23:00.136+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:00.136736+0000) 2024-02-20T09:23:00.742 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:23:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:23:00.633+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:23:00.742 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:23:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:23:00.633+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:23:00.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:23:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:23:00.633+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:23:01.221 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:23:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:23:00.943+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:00.943509+0000) 2024-02-20T09:23:01.222 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:23:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:23:01.058+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:01.058934+0000) 2024-02-20T09:23:01.222 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:23:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:23:01.126+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:23:01.222 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:23:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:23:01.126+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:23:01.222 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:23:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:23:01.126+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:23:01.223 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:23:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:23:00.997+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:23:01.223 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:23:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:23:00.997+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:23:01.223 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:23:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:23:00.997+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:23:01.223 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:23:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:23:00.998+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:23:01.223 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:23:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:23:01.138+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:01.138816+0000) 2024-02-20T09:23:01.933 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:23:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:23:01.712+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:01.712421+0000) 2024-02-20T09:23:01.934 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:23:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:23:01.715+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 68030 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:23:01.934 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:23:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:23:01.708+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:01.708726+0000) 2024-02-20T09:23:01.934 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:23:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:23:01.592+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:23:01.934 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:23:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:23:01.592+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:23:01.935 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:23:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:23:01.592+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:23:01.935 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:23:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:23:01.708+0000 7f8a5f86e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:01.709022+0000) 2024-02-20T09:23:02.221 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:23:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:23:01.943+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:01.943606+0000) 2024-02-20T09:23:02.222 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:23:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:23:02.058+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:02.059029+0000) 2024-02-20T09:23:02.222 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:23:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:23:02.135+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:23:02.222 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:23:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:23:02.135+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:23:02.222 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:23:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:23:02.135+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:23:02.223 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:23:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:23:01.974+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:23:02.223 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:23:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:23:01.974+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:23:02.223 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:23:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:23:01.974+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:23:02.223 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:23:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:23:01.974+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:23:02.223 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:23:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:23:02.138+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:02.139036+0000) 2024-02-20T09:23:02.933 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:23:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:23:02.610+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:23:02.934 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:23:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:23:02.610+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:23:02.934 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:23:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:23:02.610+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:23:03.225 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:23:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:23:02.943+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:02.943756+0000) 2024-02-20T09:23:03.226 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:23:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:23:03.058+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:03.059123+0000) 2024-02-20T09:23:03.226 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:23:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:23:03.097+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:23:03.226 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:23:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:23:03.097+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:23:03.226 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:23:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:23:03.097+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:23:03.226 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:23:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:23:03.011+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:23:03.226 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:23:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:23:03.011+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:23:03.227 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:23:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:23:03.011+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:23:03.227 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:23:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:23:03.011+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:23:03.227 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:23:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:23:03.140+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:03.140981+0000) 2024-02-20T09:23:03.933 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:23:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:23:03.604+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:23:03.934 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:23:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:23:03.604+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:23:03.934 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:23:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:23:03.604+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:23:04.227 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:23:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:23:03.943+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:03.943896+0000) 2024-02-20T09:23:04.227 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:23:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:23:04.058+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:04.059237+0000) 2024-02-20T09:23:04.227 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:23:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:23:04.088+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:23:04.227 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:23:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:23:04.088+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:23:04.228 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:23:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:23:04.088+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:23:04.228 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:23:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:23:04.035+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:23:04.228 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:23:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:23:04.035+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:23:04.228 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:23:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:23:04.035+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:23:04.228 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:23:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:23:04.035+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:23:04.228 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:23:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:23:04.142+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:04.143206+0000) 2024-02-20T09:23:04.934 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:23:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:23:04.587+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:23:04.934 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:23:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:23:04.587+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:23:04.934 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:23:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:23:04.587+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:23:05.227 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:23:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:23:04.943+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:04.944003+0000) 2024-02-20T09:23:05.227 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:23:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:23:05.058+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:05.059363+0000) 2024-02-20T09:23:05.227 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:23:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:23:05.099+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:23:05.228 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:23:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:23:05.099+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:23:05.228 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:23:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:23:05.099+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:23:05.228 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:23:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:23:05.033+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:23:05.228 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:23:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:23:05.033+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:23:05.228 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:23:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:23:05.033+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:23:05.228 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:23:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:23:05.033+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:23:05.229 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:23:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:23:05.145+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:05.145516+0000) 2024-02-20T09:23:05.934 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:23:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:23:05.556+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:23:05.934 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:23:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:23:05.556+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:23:05.934 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:23:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:23:05.556+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:23:06.227 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:23:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:23:05.943+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:05.944112+0000) 2024-02-20T09:23:06.227 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:23:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:23:06.059+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:06.059472+0000) 2024-02-20T09:23:06.228 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:23:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:23:06.088+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:23:06.228 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:23:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:23:06.088+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:23:06.228 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:23:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:23:06.088+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:23:06.228 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:23:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:23:06.042+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:23:06.228 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:23:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:23:06.042+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:23:06.229 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:23:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:23:06.042+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:23:06.229 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:23:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:23:06.042+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:23:06.229 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:23:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:23:06.147+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:06.147795+0000) 2024-02-20T09:23:06.841 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:23:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:23:06.723+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 68068 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:23:06.842 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:23:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:23:06.718+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:06.718969+0000) 2024-02-20T09:23:06.842 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:23:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:23:06.716+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:06.716606+0000) 2024-02-20T09:23:06.842 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:23:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:23:06.599+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:23:06.842 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:23:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:23:06.599+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:23:06.842 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:23:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:23:06.599+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:23:06.843 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:23:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:23:06.716+0000 7f8a5f86e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:06.716661+0000) 2024-02-20T09:23:07.130 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:23:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:23:06.943+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:06.944204+0000) 2024-02-20T09:23:07.131 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:23:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:23:07.073+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:23:07.131 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:23:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:23:07.073+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:23:07.131 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:23:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:23:07.073+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:23:07.131 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:23:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:23:07.073+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:23:07.132 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:23:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:23:07.059+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:07.059563+0000) 2024-02-20T09:23:07.132 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:23:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:23:07.129+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:23:07.132 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:23:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:23:07.129+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:23:07.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:23:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:23:07.129+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:23:07.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:23:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:23:07.147+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:07.147988+0000) 2024-02-20T09:23:07.742 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:23:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:23:07.605+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:23:07.742 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:23:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:23:07.605+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:23:07.742 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:23:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:23:07.605+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:23:08.229 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:23:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:23:07.943+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:07.944303+0000) 2024-02-20T09:23:08.230 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:23:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:23:08.059+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:08.059643+0000) 2024-02-20T09:23:08.230 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:23:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:23:08.163+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:23:08.230 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:23:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:23:08.163+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:23:08.230 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:23:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:23:08.163+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:23:08.230 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:23:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:23:08.092+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:23:08.231 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:23:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:23:08.092+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:23:08.231 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:23:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:23:08.092+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:23:08.231 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:23:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:23:08.092+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:23:08.231 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:23:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:23:08.149+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:08.149971+0000) 2024-02-20T09:23:08.934 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:23:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:23:08.638+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:23:08.934 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:23:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:23:08.638+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:23:08.934 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:23:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:23:08.638+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:23:09.195 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:23:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:23:08.944+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:08.944437+0000) 2024-02-20T09:23:09.195 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:23:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:23:09.059+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:09.059742+0000) 2024-02-20T09:23:09.195 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:23:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:23:09.193+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:23:09.196 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:23:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:23:09.112+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:23:09.196 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:23:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:23:09.112+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:23:09.196 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:23:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:23:09.112+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:23:09.196 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:23:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:23:09.112+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:23:09.196 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:23:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:23:09.152+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:09.152498+0000) 2024-02-20T09:23:09.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:23:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:23:09.193+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:23:09.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:23:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:23:09.193+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:23:09.886 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:23:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:23:09.660+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:23:09.886 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:23:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:23:09.660+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:23:09.886 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:23:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:23:09.660+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:23:10.156 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:23:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:23:09.944+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:09.944527+0000) 2024-02-20T09:23:10.156 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:23:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:23:10.059+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:10.059877+0000) 2024-02-20T09:23:10.460 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:23:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:23:10.189+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:23:10.460 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:23:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:23:10.189+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:23:10.460 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:23:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:23:10.189+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:23:10.461 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:23:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:23:10.154+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:10.154705+0000) 2024-02-20T09:23:10.461 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:23:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:23:10.161+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:23:10.461 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:23:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:23:10.161+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:23:10.461 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:23:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:23:10.161+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:23:10.461 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:23:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:23:10.161+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:23:10.742 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:23:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:23:10.679+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:23:10.742 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:23:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:23:10.679+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:23:10.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:23:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:23:10.679+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:23:11.202 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:23:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:23:10.944+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:10.944616+0000) 2024-02-20T09:23:11.203 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:23:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:23:11.157+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:11.158273+0000) 2024-02-20T09:23:11.203 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:23:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:23:11.059+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:11.059972+0000) 2024-02-20T09:23:11.203 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:23:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:23:11.147+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:23:11.203 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:23:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:23:11.147+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:23:11.203 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:23:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:23:11.147+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:23:11.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:23:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:23:11.201+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:23:11.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:23:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:23:11.201+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:23:11.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:23:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:23:11.201+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:23:11.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:23:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:23:11.201+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:23:11.851 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:23:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:23:11.730+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 68109 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:23:11.851 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:23:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:23:11.728+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:11.728657+0000) 2024-02-20T09:23:11.851 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:23:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:23:11.724+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:11.724761+0000) 2024-02-20T09:23:11.852 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:23:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:23:11.668+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:23:11.852 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:23:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:23:11.668+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:23:11.852 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:23:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:23:11.668+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:23:11.852 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:23:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:23:11.724+0000 7f8a5f86e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:11.724519+0000) 2024-02-20T09:23:12.157 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:23:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:23:11.944+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:11.944749+0000) 2024-02-20T09:23:12.157 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:23:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:23:12.059+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:12.060108+0000) 2024-02-20T09:23:12.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:23:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:23:12.172+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:23:12.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:23:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:23:12.172+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:23:12.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:23:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:23:12.172+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:23:12.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:23:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:23:12.155+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:23:12.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:23:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:23:12.155+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:23:12.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:23:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:23:12.155+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:23:12.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:23:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:23:12.155+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:23:12.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:23:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:23:12.158+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:12.158438+0000) 2024-02-20T09:23:12.934 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:23:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:23:12.699+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:23:12.935 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:23:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:23:12.699+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:23:12.935 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:23:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:23:12.699+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:23:13.234 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:23:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:23:12.944+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:12.944855+0000) 2024-02-20T09:23:13.235 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:23:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:23:13.159+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:13.160347+0000) 2024-02-20T09:23:13.235 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:23:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:23:13.166+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:23:13.235 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:23:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:23:13.166+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:23:13.235 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:23:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:23:13.166+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:23:13.235 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:23:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:23:13.166+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:23:13.235 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:23:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:23:13.059+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:13.060224+0000) 2024-02-20T09:23:13.235 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:23:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:23:13.147+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:23:13.236 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:23:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:23:13.147+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:23:13.236 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:23:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:23:13.147+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:23:13.935 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:23:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:23:13.689+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:23:13.935 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:23:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:23:13.689+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:23:13.935 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:23:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:23:13.689+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:23:14.128 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:23:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:23:13.944+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:13.944963+0000) 2024-02-20T09:23:14.128 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:23:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:23:14.059+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:14.060326+0000) 2024-02-20T09:23:14.128 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:23:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:23:14.125+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:23:14.129 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:23:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:23:14.125+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:23:14.129 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:23:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:23:14.125+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:23:14.218 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:23:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:23:14.161+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:14.162290+0000) 2024-02-20T09:23:14.218 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:23:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:23:14.216+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:23:14.218 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:23:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:23:14.216+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:23:14.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:23:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:23:14.216+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:23:14.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:23:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:23:14.216+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:23:14.935 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:23:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:23:14.738+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:23:14.935 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:23:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:23:14.738+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:23:14.935 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:23:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:23:14.738+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:23:15.204 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:23:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:23:14.944+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:14.945094+0000) 2024-02-20T09:23:15.204 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:23:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:23:15.060+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:15.060465+0000) 2024-02-20T09:23:15.204 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:23:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:23:15.168+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:23:15.205 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:23:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:23:15.168+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:23:15.205 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:23:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:23:15.168+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:23:15.205 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:23:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:23:15.163+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:15.164291+0000) 2024-02-20T09:23:15.205 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:23:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:23:15.202+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:23:15.205 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:23:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:23:15.202+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:23:15.205 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:23:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:23:15.202+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:23:15.206 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:23:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:23:15.202+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:23:15.935 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:23:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:23:15.755+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:23:15.935 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:23:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:23:15.755+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:23:15.935 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:23:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:23:15.755+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:23:16.209 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:23:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:23:16.060+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:16.060553+0000) 2024-02-20T09:23:16.210 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:23:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:23:15.944+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:15.945226+0000) 2024-02-20T09:23:16.210 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:23:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:23:16.165+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:16.166282+0000) 2024-02-20T09:23:16.210 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:23:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:23:16.167+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:23:16.210 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:23:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:23:16.167+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:23:16.210 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:23:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:23:16.167+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:23:16.211 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:23:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:23:16.167+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:23:16.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:23:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:23:16.208+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:23:16.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:23:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:23:16.208+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:23:16.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:23:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:23:16.208+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:23:16.857 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:23:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:23:16.738+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 68147 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:23:16.857 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:23:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:23:16.733+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:16.734340+0000) 2024-02-20T09:23:16.857 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:23:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:23:16.731+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:16.731602+0000) 2024-02-20T09:23:16.858 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:23:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:23:16.729+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:23:16.858 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:23:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:23:16.729+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:23:16.858 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:23:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:23:16.729+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:23:16.858 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:23:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:23:16.731+0000 7f8a5f86e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:16.731878+0000) 2024-02-20T09:23:17.135 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:23:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:23:16.944+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:16.945357+0000) 2024-02-20T09:23:17.136 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:23:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:23:17.060+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:17.060671+0000) 2024-02-20T09:23:17.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:23:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:23:17.210+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:23:17.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:23:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:23:17.210+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:23:17.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:23:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:23:17.210+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:23:17.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:23:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:23:17.133+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:23:17.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:23:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:23:17.133+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:23:17.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:23:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:23:17.133+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:23:17.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:23:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:23:17.134+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:23:17.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:23:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:23:17.166+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:17.166531+0000) 2024-02-20T09:23:17.935 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:23:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:23:17.703+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:23:17.935 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:23:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:23:17.703+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:23:17.936 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:23:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:23:17.703+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:23:18.238 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:23:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:23:18.060+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:18.060759+0000) 2024-02-20T09:23:18.239 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:23:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:23:17.945+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:17.945453+0000) 2024-02-20T09:23:18.239 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:23:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:23:18.133+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:23:18.239 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:23:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:23:18.133+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:23:18.239 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:23:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:23:18.133+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:23:18.240 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:23:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:23:18.134+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:23:18.240 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:23:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:23:18.168+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:18.168439+0000) 2024-02-20T09:23:18.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:23:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:23:18.259+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:23:18.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:23:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:23:18.259+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:23:18.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:23:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:23:18.259+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:23:18.935 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:23:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:23:18.733+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:23:18.935 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:23:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:23:18.733+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:23:18.936 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:23:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:23:18.733+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:23:19.240 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:23:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:23:18.945+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:18.945551+0000) 2024-02-20T09:23:19.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:23:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:23:19.060+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:19.060847+0000) 2024-02-20T09:23:19.241 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:23:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:23:19.156+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:23:19.241 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:23:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:23:19.156+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:23:19.241 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:23:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:23:19.156+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:23:19.241 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:23:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:23:19.156+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:23:19.241 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:23:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:23:19.170+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:19.170483+0000) 2024-02-20T09:23:19.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:23:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:23:19.283+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:23:19.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:23:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:23:19.283+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:23:19.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:23:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:23:19.284+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:23:19.885 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:23:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:23:19.718+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:23:19.885 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:23:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:23:19.718+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:23:19.885 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:23:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:23:19.718+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:23:20.174 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:23:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:23:19.945+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:19.945649+0000) 2024-02-20T09:23:20.175 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:23:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:23:20.060+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:20.060951+0000) 2024-02-20T09:23:20.460 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:23:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:23:20.299+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:23:20.461 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:23:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:23:20.299+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:23:20.461 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:23:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:23:20.299+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:23:20.461 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:23:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:23:20.172+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:20.173038+0000) 2024-02-20T09:23:20.461 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:23:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:23:20.178+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:23:20.462 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:23:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:23:20.178+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:23:20.462 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:23:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:23:20.178+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:23:20.462 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:23:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:23:20.178+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:23:20.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:23:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:23:20.727+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:23:20.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:23:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:23:20.727+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:23:20.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:23:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:23:20.727+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:23:20.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:23:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:23:20.945+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:20.945779+0000) 2024-02-20T09:23:21.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:23:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:23:21.060+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:21.061065+0000) 2024-02-20T09:23:21.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:23:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:23:21.298+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:23:21.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:23:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:23:21.298+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:23:21.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:23:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:23:21.298+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:23:21.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:23:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:23:21.138+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:23:21.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:23:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:23:21.138+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:23:21.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:23:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:23:21.138+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:23:21.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:23:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:23:21.138+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:23:21.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:23:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:23:21.176+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:21.176411+0000) 2024-02-20T09:23:21.865 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:23:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:23:21.744+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 68188 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:23:21.866 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:23:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:23:21.742+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:21.743753+0000) 2024-02-20T09:23:21.867 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:23:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:23:21.738+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:21.739863+0000) 2024-02-20T09:23:21.867 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:23:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:23:21.702+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:23:21.867 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:23:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:23:21.702+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:23:21.868 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:23:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:23:21.702+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:23:21.868 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:23:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:23:21.738+0000 7f8a5f86e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:21.740092+0000) 2024-02-20T09:23:22.150 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:23:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:23:21.944+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:21.945875+0000) 2024-02-20T09:23:22.151 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:23:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:23:22.059+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:22.061214+0000) 2024-02-20T09:23:22.489 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:23:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:23:22.270+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:23:22.489 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:23:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:23:22.270+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:23:22.489 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:23:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:23:22.270+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:23:22.489 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:23:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:23:22.148+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:23:22.489 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:23:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:23:22.148+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:23:22.490 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:23:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:23:22.148+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:23:22.490 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:23:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:23:22.148+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:23:22.490 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:23:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:23:22.175+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:22.176617+0000) 2024-02-20T09:23:22.742 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:23:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:23:22.697+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:23:22.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:23:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:23:22.697+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:23:22.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:23:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:23:22.697+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:23:23.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:23:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:23:22.944+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:22.945978+0000) 2024-02-20T09:23:23.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:23:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:23:23.059+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:23.061372+0000) 2024-02-20T09:23:23.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:23:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:23:23.135+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:23:23.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:23:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:23:23.135+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:23:23.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:23:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:23:23.135+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:23:23.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:23:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:23:23.135+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:23:23.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:23:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:23:23.177+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:23.179077+0000) 2024-02-20T09:23:23.591 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:23:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:23:23.315+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:23:23.592 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:23:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:23:23.315+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:23:23.592 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:23:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:23:23.315+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:23:23.936 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:23:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:23:23.721+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:23:23.937 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:23:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:23:23.721+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:23:23.937 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:23:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:23:23.721+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:23:24.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:23:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:23:23.944+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:23.946100+0000) 2024-02-20T09:23:24.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:23:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:23:24.060+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:24.061478+0000) 2024-02-20T09:23:24.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:23:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:23:24.141+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:23:24.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:23:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:23:24.141+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:23:24.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:23:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:23:24.141+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:23:24.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:23:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:23:24.141+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:23:24.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:23:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:23:24.180+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:24.181681+0000) 2024-02-20T09:23:24.591 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:23:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:23:24.316+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:23:24.591 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:23:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:23:24.316+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:23:24.592 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:23:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:23:24.316+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:23:24.936 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:23:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:23:24.694+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:23:24.936 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:23:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:23:24.694+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:23:24.937 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:23:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:23:24.694+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:23:25.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:23:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:23:24.944+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:24.946216+0000) 2024-02-20T09:23:25.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:23:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:23:25.060+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:25.061602+0000) 2024-02-20T09:23:25.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:23:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:23:25.098+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:23:25.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:23:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:23:25.098+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:23:25.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:23:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:23:25.098+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:23:25.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:23:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:23:25.098+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:23:25.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:23:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:23:25.182+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:25.184276+0000) 2024-02-20T09:23:25.591 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:23:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:23:25.305+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:23:25.592 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:23:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:23:25.305+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:23:25.592 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:23:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:23:25.305+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:23:25.936 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:23:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:23:25.740+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:23:25.937 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:23:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:23:25.740+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:23:25.937 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:23:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:23:25.740+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:23:26.187 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:23:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:23:26.060+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:26.061709+0000) 2024-02-20T09:23:26.188 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:23:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:23:25.944+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:25.946333+0000) 2024-02-20T09:23:26.188 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:23:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:23:26.134+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:23:26.191 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:23:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:23:26.134+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:23:26.191 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:23:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:23:26.134+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:23:26.191 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:23:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:23:26.134+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:23:26.191 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:23:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:23:26.185+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:26.186723+0000) 2024-02-20T09:23:26.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:23:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:23:26.309+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:23:26.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:23:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:23:26.309+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:23:26.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:23:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:23:26.310+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:23:26.873 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:23:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:23:26.749+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:26.750541+0000) 2024-02-20T09:23:26.873 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:23:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:23:26.746+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:26.748175+0000) 2024-02-20T09:23:26.874 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:23:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:23:26.716+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:23:26.874 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:23:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:23:26.716+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:23:26.874 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:23:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:23:26.716+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:23:26.874 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:23:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:23:26.746+0000 7f8a5f86e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:26.747626+0000) 2024-02-20T09:23:26.875 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:23:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:23:26.752+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 68226 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:23:27.127 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:23:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:23:26.945+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:26.946445+0000) 2024-02-20T09:23:27.128 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:23:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:23:27.060+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:27.061806+0000) 2024-02-20T09:23:27.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:23:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:23:27.347+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:23:27.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:23:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:23:27.347+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:23:27.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:23:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:23:27.347+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:23:27.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:23:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:23:27.125+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:23:27.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:23:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:23:27.125+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:23:27.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:23:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:23:27.125+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:23:27.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:23:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:23:27.125+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:23:27.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:23:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:23:27.185+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:27.187020+0000) 2024-02-20T09:23:27.936 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:23:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:23:27.698+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:23:27.937 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:23:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:23:27.698+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:23:27.937 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:23:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:23:27.698+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:23:28.190 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:23:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:23:27.945+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:27.946549+0000) 2024-02-20T09:23:28.191 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:23:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:23:28.060+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:28.061885+0000) 2024-02-20T09:23:28.191 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:23:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:23:28.138+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:23:28.191 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:23:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:23:28.138+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:23:28.191 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:23:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:23:28.138+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:23:28.191 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:23:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:23:28.138+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:23:28.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:23:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:23:28.187+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:28.189195+0000) 2024-02-20T09:23:28.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:23:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:23:28.366+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:23:28.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:23:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:23:28.366+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:23:28.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:23:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:23:28.366+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:23:28.936 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:23:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:23:28.677+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:23:28.937 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:23:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:23:28.677+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:23:28.937 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:23:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:23:28.677+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:23:29.192 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:23:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:23:29.060+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:29.062002+0000) 2024-02-20T09:23:29.193 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:23:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:23:28.945+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:28.946667+0000) 2024-02-20T09:23:29.193 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:23:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:23:29.100+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:23:29.193 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:23:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:23:29.100+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:23:29.193 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:23:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:23:29.100+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:23:29.193 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:23:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:23:29.100+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:23:29.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:23:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:23:29.189+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:29.191323+0000) 2024-02-20T09:23:29.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:23:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:23:29.316+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:23:29.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:23:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:23:29.316+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:23:29.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:23:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:23:29.316+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:23:29.886 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:23:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:23:29.697+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:23:29.886 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:23:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:23:29.697+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:23:29.886 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:23:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:23:29.697+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:23:30.196 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:23:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:23:29.945+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:29.946782+0000) 2024-02-20T09:23:30.196 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:23:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:23:30.060+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:30.062117+0000) 2024-02-20T09:23:30.196 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:23:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:23:30.081+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:23:30.197 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:23:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:23:30.081+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:23:30.197 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:23:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:23:30.081+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:23:30.197 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:23:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:23:30.081+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:23:30.461 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:23:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:23:30.193+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:30.194887+0000) 2024-02-20T09:23:30.461 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:23:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:23:30.328+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:23:30.461 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:23:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:23:30.328+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:23:30.461 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:23:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:23:30.328+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:23:30.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:23:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:23:30.737+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:23:30.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:23:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:23:30.737+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:23:30.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:23:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:23:30.737+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:23:30.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:23:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:23:30.945+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:30.946886+0000) 2024-02-20T09:23:31.325 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:23:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:23:31.060+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:31.062232+0000) 2024-02-20T09:23:31.326 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:23:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:23:31.323+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:23:31.326 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:23:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:23:31.100+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:23:31.326 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:23:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:23:31.100+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:23:31.326 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:23:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:23:31.100+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:23:31.326 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:23:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:23:31.100+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:23:31.326 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:23:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:23:31.196+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:31.197427+0000) 2024-02-20T09:23:31.592 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:23:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:23:31.323+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:23:31.592 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:23:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:23:31.323+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:23:31.879 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:23:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:23:31.760+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 68267 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:23:31.880 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:23:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:23:31.757+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:31.759102+0000) 2024-02-20T09:23:31.880 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:23:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:23:31.753+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:31.755370+0000) 2024-02-20T09:23:31.881 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:23:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:23:31.738+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:23:31.881 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:23:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:23:31.738+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:23:31.881 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:23:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:23:31.738+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:23:31.881 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:23:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:23:31.753+0000 7f8a5f86e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:31.755282+0000) 2024-02-20T09:23:32.198 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:23:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:23:31.945+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:31.946984+0000) 2024-02-20T09:23:32.199 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:23:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:23:32.061+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:32.062429+0000) 2024-02-20T09:23:32.199 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:23:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:23:32.064+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:23:32.200 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:23:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:23:32.064+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:23:32.200 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:23:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:23:32.064+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:23:32.200 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:23:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:23:32.064+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:23:32.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:23:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:23:32.196+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:32.197725+0000) 2024-02-20T09:23:32.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:23:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:23:32.339+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:23:32.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:23:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:23:32.339+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:23:32.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:23:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:23:32.339+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:23:32.937 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:23:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:23:32.727+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:23:32.937 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:23:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:23:32.727+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:23:32.937 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:23:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:23:32.727+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:23:33.202 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:23:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:23:32.945+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:32.947120+0000) 2024-02-20T09:23:33.202 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:23:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:23:33.061+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:33.062532+0000) 2024-02-20T09:23:33.202 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:23:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:23:33.023+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:23:33.203 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:23:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:23:33.023+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:23:33.203 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:23:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:23:33.023+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:23:33.203 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:23:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:23:33.023+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:23:33.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:23:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:23:33.199+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:33.200730+0000) 2024-02-20T09:23:33.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:23:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:23:33.341+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:23:33.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:23:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:23:33.341+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:23:33.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:23:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:23:33.341+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:23:33.937 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:23:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:23:33.737+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:23:33.937 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:23:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:23:33.737+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:23:33.938 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:23:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:23:33.737+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:23:34.205 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:23:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:23:33.945+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:33.947285+0000) 2024-02-20T09:23:34.206 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:23:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:23:34.061+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:34.062667+0000) 2024-02-20T09:23:34.206 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:23:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:23:34.070+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:23:34.206 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:23:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:23:34.070+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:23:34.206 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:23:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:23:34.070+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:23:34.206 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:23:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:23:34.070+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:23:34.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:23:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:23:34.202+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:34.204278+0000) 2024-02-20T09:23:34.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:23:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:23:34.322+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:23:34.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:23:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:23:34.322+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:23:34.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:23:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:23:34.322+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:23:34.937 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:23:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:23:34.744+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:23:34.937 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:23:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:23:34.744+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:23:34.938 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:23:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:23:34.744+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:23:35.209 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:23:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:23:34.946+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:34.947488+0000) 2024-02-20T09:23:35.209 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:23:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:23:35.061+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:35.062803+0000) 2024-02-20T09:23:35.210 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:23:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:23:35.037+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:23:35.210 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:23:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:23:35.037+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:23:35.210 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:23:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:23:35.037+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:23:35.210 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:23:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:23:35.037+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:23:35.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:23:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:23:35.206+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:35.207895+0000) 2024-02-20T09:23:35.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:23:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:23:35.319+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:23:35.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:23:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:23:35.319+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:23:35.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:23:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:23:35.319+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:23:35.937 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:23:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:23:35.767+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:23:35.938 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:23:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:23:35.767+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:23:35.938 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:23:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:23:35.767+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:23:36.212 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:23:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:23:35.946+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:35.947591+0000) 2024-02-20T09:23:36.212 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:23:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:23:36.061+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:36.062974+0000) 2024-02-20T09:23:36.213 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:23:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:23:36.024+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:23:36.213 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:23:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:23:36.024+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:23:36.213 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:23:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:23:36.024+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:23:36.213 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:23:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:23:36.024+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:23:36.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:23:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:23:36.208+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:36.210007+0000) 2024-02-20T09:23:36.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:23:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:23:36.278+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:23:36.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:23:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:23:36.278+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:23:36.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:23:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:23:36.278+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:23:36.888 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:23:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:23:36.768+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 68305 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:23:36.889 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:23:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:23:36.765+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:36.766733+0000) 2024-02-20T09:23:36.889 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:23:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:23:36.761+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:36.763139+0000) 2024-02-20T09:23:36.890 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:23:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:23:36.761+0000 7f8a5f86e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:36.763010+0000) 2024-02-20T09:23:36.890 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:23:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:23:36.763+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:23:36.890 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:23:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:23:36.763+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:23:36.890 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:23:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:23:36.763+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:23:37.211 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:23:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:23:37.061+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:37.063137+0000) 2024-02-20T09:23:37.212 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:23:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:23:36.946+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:36.947679+0000) 2024-02-20T09:23:37.212 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:23:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:23:37.036+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:23:37.212 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:23:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:23:37.036+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:23:37.212 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:23:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:23:37.036+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:23:37.212 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:23:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:23:37.036+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:23:37.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:23:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:23:37.208+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:37.210386+0000) 2024-02-20T09:23:37.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:23:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:23:37.306+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:23:37.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:23:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:23:37.306+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:23:37.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:23:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:23:37.306+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:23:37.938 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:23:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:23:37.778+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:23:37.938 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:23:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:23:37.778+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:23:37.938 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:23:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:23:37.778+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:23:38.215 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:23:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:23:37.946+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:37.947799+0000) 2024-02-20T09:23:38.215 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:23:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:23:38.061+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:38.063243+0000) 2024-02-20T09:23:38.216 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:23:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:23:38.060+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:23:38.216 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:23:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:23:38.060+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:23:38.216 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:23:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:23:38.060+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:23:38.216 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:23:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:23:38.060+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:23:38.216 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:23:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:23:38.212+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:38.213826+0000) 2024-02-20T09:23:38.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:23:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:23:38.262+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:23:38.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:23:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:23:38.263+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:23:38.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:23:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:23:38.263+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:23:38.938 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:23:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:23:38.772+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:23:38.938 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:23:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:23:38.772+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:23:38.938 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:23:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:23:38.772+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:23:39.217 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:23:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:23:38.946+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:38.947929+0000) 2024-02-20T09:23:39.218 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:23:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:23:39.061+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:39.063336+0000) 2024-02-20T09:23:39.218 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:23:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:23:39.045+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:23:39.218 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:23:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:23:39.045+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:23:39.218 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:23:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:23:39.045+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:23:39.218 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:23:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:23:39.045+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:23:39.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:23:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:23:39.214+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:39.216084+0000) 2024-02-20T09:23:39.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:23:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:23:39.302+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:23:39.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:23:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:23:39.302+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:23:39.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:23:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:23:39.302+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:23:39.885 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:23:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:23:39.757+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:23:39.886 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:23:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:23:39.757+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:23:39.886 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:23:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:23:39.757+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:23:40.220 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:23:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:23:39.946+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:39.948078+0000) 2024-02-20T09:23:40.220 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:23:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:23:40.062+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:40.063449+0000) 2024-02-20T09:23:40.221 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:23:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:23:40.011+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:23:40.221 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:23:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:23:40.011+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:23:40.221 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:23:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:23:40.011+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:23:40.221 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:23:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:23:40.011+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:23:40.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:23:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:23:40.217+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:40.218831+0000) 2024-02-20T09:23:40.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:23:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:23:40.270+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:23:40.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:23:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:23:40.270+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:23:40.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:23:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:23:40.270+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:23:40.921 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:23:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:23:40.796+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:23:40.921 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:23:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:23:40.796+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:23:40.921 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:23:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:23:40.796+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:23:41.222 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:23:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:23:41.062+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:41.063541+0000) 2024-02-20T09:23:41.223 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:23:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:23:40.946+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:40.948288+0000) 2024-02-20T09:23:41.223 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:23:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:23:41.022+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:23:41.223 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:23:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:23:41.022+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:23:41.223 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:23:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:23:41.022+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:23:41.223 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:23:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:23:41.022+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:23:41.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:23:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:23:41.219+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:41.221254+0000) 2024-02-20T09:23:41.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:23:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:23:41.286+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:23:41.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:23:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:23:41.286+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:23:41.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:23:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:23:41.286+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:23:41.897 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:23:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:23:41.776+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 68347 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:23:41.898 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:23:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:23:41.772+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:41.774321+0000) 2024-02-20T09:23:41.898 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:23:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:23:41.769+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:41.771218+0000) 2024-02-20T09:23:41.898 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:23:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:23:41.770+0000 7f8a5f86e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:41.771768+0000) 2024-02-20T09:23:41.898 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:23:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:23:41.791+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:23:41.898 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:23:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:23:41.791+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:23:41.899 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:23:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:23:41.791+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:23:42.222 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:23:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:23:41.947+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:41.948426+0000) 2024-02-20T09:23:42.223 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:23:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:23:42.062+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:42.063641+0000) 2024-02-20T09:23:42.223 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:23:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:23:41.982+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:23:42.223 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:23:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:23:41.982+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:23:42.223 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:23:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:23:41.982+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:23:42.223 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:23:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:23:41.982+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:23:42.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:23:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:23:42.220+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:42.221547+0000) 2024-02-20T09:23:42.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:23:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:23:42.239+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:23:42.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:23:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:23:42.239+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:23:42.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:23:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:23:42.239+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:23:42.938 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:23:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:23:42.745+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:23:42.938 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:23:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:23:42.745+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:23:42.939 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:23:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:23:42.745+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:23:43.225 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:23:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:23:43.062+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:43.063771+0000) 2024-02-20T09:23:43.225 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:23:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:23:42.947+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:42.948482+0000) 2024-02-20T09:23:43.226 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:23:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:23:42.948+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:23:43.226 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:23:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:23:42.948+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:23:43.226 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:23:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:23:42.948+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:23:43.226 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:23:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:23:42.948+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:23:43.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:23:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:23:43.222+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:43.224061+0000) 2024-02-20T09:23:43.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:23:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:23:43.257+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:23:43.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:23:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:23:43.257+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:23:43.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:23:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:23:43.257+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:23:43.909 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:23:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:23:43.718+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:23:43.910 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:23:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:23:43.718+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:23:43.910 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:23:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:23:43.718+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:23:44.227 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:23:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:23:44.062+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:44.063899+0000) 2024-02-20T09:23:44.228 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:23:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:23:43.947+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:43.948627+0000) 2024-02-20T09:23:44.228 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:23:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:23:43.907+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:23:44.228 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:23:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:23:43.907+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:23:44.228 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:23:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:23:43.907+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:23:44.229 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:23:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:23:43.907+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:23:44.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:23:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:23:44.225+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:44.226540+0000) 2024-02-20T09:23:44.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:23:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:23:44.262+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:23:44.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:23:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:23:44.262+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:23:44.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:23:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:23:44.262+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:23:44.938 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:23:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:23:44.759+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:23:44.939 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:23:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:23:44.759+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:23:44.939 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:23:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:23:44.759+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:23:45.231 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:23:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:23:44.947+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:44.948720+0000) 2024-02-20T09:23:45.232 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:23:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:23:44.943+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:23:45.232 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:23:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:23:44.943+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:23:45.232 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:23:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:23:44.943+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:23:45.232 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:23:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:23:44.943+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:23:45.233 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:23:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:23:45.062+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:45.064099+0000) 2024-02-20T09:23:45.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:23:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:23:45.228+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:45.230140+0000) 2024-02-20T09:23:45.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:23:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:23:45.272+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:23:45.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:23:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:23:45.272+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:23:45.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:23:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:23:45.272+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:23:45.918 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:23:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:23:45.805+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:23:45.918 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:23:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:23:45.805+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:23:45.918 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:23:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:23:45.805+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:23:46.234 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:23:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:23:46.062+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:46.064238+0000) 2024-02-20T09:23:46.234 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:23:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:23:45.915+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:23:46.234 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:23:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:23:45.915+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:23:46.235 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:23:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:23:45.915+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:23:46.235 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:23:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:23:45.915+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:23:46.235 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:23:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:23:45.947+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:45.948838+0000) 2024-02-20T09:23:46.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:23:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:23:46.231+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:46.233060+0000) 2024-02-20T09:23:46.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:23:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:23:46.292+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:23:46.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:23:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:23:46.292+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:23:46.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:23:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:23:46.292+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:23:46.888 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:23:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:23:46.784+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 68385 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:23:46.889 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:23:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:23:46.777+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:46.779121+0000) 2024-02-20T09:23:46.889 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:23:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:23:46.781+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:46.782894+0000) 2024-02-20T09:23:46.889 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:23:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:23:46.885+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:23:46.889 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:23:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:23:46.885+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:23:46.889 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:23:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:23:46.778+0000 7f8a5f86e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:46.779615+0000) 2024-02-20T09:23:46.890 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:23:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:23:46.828+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:23:46.890 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:23:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:23:46.828+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:23:46.890 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:23:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:23:46.828+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:23:47.234 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:23:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:23:46.947+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:46.948955+0000) 2024-02-20T09:23:47.235 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:23:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:23:46.885+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:23:47.235 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:23:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:23:46.886+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:23:47.235 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:23:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:23:47.062+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:47.064322+0000) 2024-02-20T09:23:47.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:23:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:23:47.232+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:47.233434+0000) 2024-02-20T09:23:47.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:23:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:23:47.256+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:23:47.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:23:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:23:47.256+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:23:47.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:23:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:23:47.256+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:23:48.237 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:23:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:23:48.063+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:48.064483+0000) 2024-02-20T09:23:48.237 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:23:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:23:47.867+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:23:48.237 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:23:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:23:47.867+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:23:48.237 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:23:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:23:47.867+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:23:48.237 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:23:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:23:47.947+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:47.949072+0000) 2024-02-20T09:23:48.238 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:23:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:23:47.898+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:23:48.238 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:23:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:23:47.898+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:23:48.238 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:23:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:23:47.898+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:23:48.238 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:23:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:23:47.898+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:23:48.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:23:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:23:48.234+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:48.235733+0000) 2024-02-20T09:23:48.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:23:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:23:48.298+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:23:48.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:23:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:23:48.298+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:23:48.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:23:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:23:48.298+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:23:49.239 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:23:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:23:49.063+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:49.064622+0000) 2024-02-20T09:23:49.240 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:23:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:23:48.892+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:23:49.240 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:23:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:23:48.892+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:23:49.240 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:23:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:23:48.892+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:23:49.240 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:23:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:23:48.947+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:48.949205+0000) 2024-02-20T09:23:49.240 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:23:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:23:48.890+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:23:49.240 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:23:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:23:48.890+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:23:49.241 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:23:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:23:48.890+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:23:49.241 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:23:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:23:48.891+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:23:49.241 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:23:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:23:49.236+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:49.238266+0000) 2024-02-20T09:23:49.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:23:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:23:49.252+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:23:49.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:23:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:23:49.252+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:23:49.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:23:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:23:49.252+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:23:50.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:23:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:23:50.063+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:50.064740+0000) 2024-02-20T09:23:50.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:23:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:23:49.885+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:23:50.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:23:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:23:49.885+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:23:50.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:23:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:23:49.885+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:23:50.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:23:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:23:49.885+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:23:50.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:23:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:23:49.928+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:23:50.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:23:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:23:49.928+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:23:50.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:23:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:23:49.928+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:23:50.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:23:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:23:49.947+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:49.949386+0000) 2024-02-20T09:23:50.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:23:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:23:50.239+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:50.240728+0000) 2024-02-20T09:23:50.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:23:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:23:50.299+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:23:50.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:23:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:23:50.299+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:23:50.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:23:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:23:50.299+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:23:51.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:23:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:23:51.064+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:51.064857+0000) 2024-02-20T09:23:51.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:23:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:23:50.873+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:23:51.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:23:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:23:50.873+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:23:51.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:23:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:23:50.873+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:23:51.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:23:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:23:50.873+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:23:51.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:23:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:23:50.892+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:23:51.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:23:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:23:50.892+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:23:51.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:23:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:23:50.892+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:23:51.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:23:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:23:50.948+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:50.949504+0000) 2024-02-20T09:23:51.594 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:23:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:23:51.241+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:51.242986+0000) 2024-02-20T09:23:51.595 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:23:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:23:51.273+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:23:51.595 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:23:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:23:51.273+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:23:51.595 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:23:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:23:51.273+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:23:51.902 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:23:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:23:51.791+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 68425 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:23:51.902 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:23:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:23:51.785+0000 7f8a5f86e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:51.787083+0000) 2024-02-20T09:23:51.902 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:23:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:23:51.785+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:51.787199+0000) 2024-02-20T09:23:51.903 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:23:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:23:51.788+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:51.789640+0000) 2024-02-20T09:23:51.903 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:23:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:23:51.899+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:23:52.239 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:23:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:23:51.899+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:23:52.239 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:23:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:23:51.899+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:23:52.239 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:23:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:23:51.899+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:23:52.240 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:23:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:23:52.064+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:52.065000+0000) 2024-02-20T09:23:52.240 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:23:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:23:52.237+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:23:52.240 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:23:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:23:52.237+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:23:52.240 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:23:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:23:51.927+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:23:52.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:23:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:23:51.927+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:23:52.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:23:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:23:51.927+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:23:52.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:23:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:23:51.949+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:51.949615+0000) 2024-02-20T09:23:52.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:23:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:23:52.242+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:52.243382+0000) 2024-02-20T09:23:52.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:23:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:23:52.237+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:23:53.208 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:23:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:23:53.063+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:53.065179+0000) 2024-02-20T09:23:53.209 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:23:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:23:52.903+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:23:53.209 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:23:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:23:52.903+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:23:53.209 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:23:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:23:52.903+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:23:53.209 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:23:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:23:52.948+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:52.949727+0000) 2024-02-20T09:23:53.210 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:23:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:23:52.896+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:23:53.210 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:23:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:23:52.896+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:23:53.210 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:23:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:23:52.896+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:23:53.210 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:23:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:23:52.896+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:23:53.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:23:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:23:53.244+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:53.245428+0000) 2024-02-20T09:23:53.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:23:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:23:53.218+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:23:53.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:23:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:23:53.218+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:23:53.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:23:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:23:53.218+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:23:54.211 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:23:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:23:53.949+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:53.949886+0000) 2024-02-20T09:23:54.211 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:23:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:23:53.950+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:23:54.211 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:23:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:23:53.950+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:23:54.211 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:23:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:23:53.950+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:23:54.212 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:23:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:23:53.882+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:23:54.212 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:23:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:23:53.882+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:23:54.212 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:23:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:23:53.882+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:23:54.212 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:23:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:23:53.882+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:23:54.212 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:23:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:23:54.064+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:54.065327+0000) 2024-02-20T09:23:54.213 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:23:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:23:54.209+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:23:54.213 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:23:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:23:54.209+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:23:54.213 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:23:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:23:54.209+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:23:54.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:23:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:23:54.247+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:54.247980+0000) 2024-02-20T09:23:55.172 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:23:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:23:54.909+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:23:55.172 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:23:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:23:54.909+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:23:55.172 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:23:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:23:54.909+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:23:55.172 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:23:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:23:54.949+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:54.950069+0000) 2024-02-20T09:23:55.173 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:23:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:23:54.876+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:23:55.173 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:23:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:23:54.876+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:23:55.173 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:23:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:23:54.876+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:23:55.173 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:23:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:23:54.876+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:23:55.173 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:23:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:23:55.065+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:55.065448+0000) 2024-02-20T09:23:55.173 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:23:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:23:55.170+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:23:55.173 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:23:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:23:55.170+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:23:55.174 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:23:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:23:55.170+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:23:55.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:23:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:23:55.250+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:55.250489+0000) 2024-02-20T09:23:56.153 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:23:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:23:56.065+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:56.065567+0000) 2024-02-20T09:23:56.153 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:23:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:23:56.152+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:23:56.154 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:23:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:23:55.896+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:23:56.154 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:23:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:23:55.896+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:23:56.154 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:23:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:23:55.896+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:23:56.154 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:23:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:23:55.949+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:55.950249+0000) 2024-02-20T09:23:56.154 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:23:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:23:55.904+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:23:56.154 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:23:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:23:55.904+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:23:56.155 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:23:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:23:55.904+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:23:56.155 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:23:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:23:55.904+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:23:56.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:23:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:23:56.252+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:56.252559+0000) 2024-02-20T09:23:56.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:23:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:23:56.152+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:23:56.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:23:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:23:56.152+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:23:56.856 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:23:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:23:56.800+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 68464 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:23:56.856 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:23:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:23:56.793+0000 7f8a5f86e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:56.794253+0000) 2024-02-20T09:23:56.857 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:23:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:23:56.794+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:56.794731+0000) 2024-02-20T09:23:56.857 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:23:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:23:56.797+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:56.798215+0000) 2024-02-20T09:23:56.857 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:23:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:23:56.854+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:23:56.857 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:23:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:23:56.854+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:23:57.191 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:23:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:23:57.065+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:57.065667+0000) 2024-02-20T09:23:57.191 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:23:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:23:57.189+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:23:57.191 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:23:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:23:56.854+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:23:57.191 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:23:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:23:56.854+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:23:57.192 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:23:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:23:56.863+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:23:57.192 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:23:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:23:56.863+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:23:57.192 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:23:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:23:56.863+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:23:57.192 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:23:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:23:56.950+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:56.950395+0000) 2024-02-20T09:23:57.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:23:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:23:57.252+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:57.252847+0000) 2024-02-20T09:23:57.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:23:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:23:57.189+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:23:57.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:23:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:23:57.189+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:23:57.857 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:23:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:23:57.819+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:23:57.857 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:23:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:23:57.819+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:23:57.858 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:23:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:23:57.819+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:23:57.858 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:23:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:23:57.819+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:23:58.193 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:23:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:23:57.856+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:23:58.193 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:23:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:23:57.856+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:23:58.193 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:23:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:23:57.856+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:23:58.193 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:23:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:23:57.950+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:57.950558+0000) 2024-02-20T09:23:58.193 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:23:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:23:58.065+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:58.065768+0000) 2024-02-20T09:23:58.193 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:23:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:23:58.191+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:23:58.194 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:23:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:23:58.191+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:23:58.194 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:23:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:23:58.191+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:23:58.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:23:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:23:58.254+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:58.255360+0000) 2024-02-20T09:23:58.874 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:23:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:23:58.831+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:23:58.875 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:23:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:23:58.831+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:23:58.875 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:23:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:23:58.831+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:23:58.875 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:23:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:23:58.832+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:23:59.221 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:23:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:23:58.872+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:23:59.222 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:23:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:23:58.872+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:23:59.222 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:23:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:23:58.872+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:23:59.222 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:23:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:23:58.950+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:58.950648+0000) 2024-02-20T09:23:59.222 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:23:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:23:59.065+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:59.065896+0000) 2024-02-20T09:23:59.222 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:23:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:23:59.220+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:23:59.222 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:23:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:23:59.220+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:23:59.223 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:23:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:23:59.220+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:23:59.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:23:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:23:59.258+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:59.259030+0000) 2024-02-20T09:23:59.876 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:23:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:23:59.811+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:23:59.876 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:23:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:23:59.811+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:23:59.876 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:23:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:23:59.811+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:23:59.876 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:23:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:23:59.811+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:24:00.206 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:24:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:24:00.065+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:00.066030+0000) 2024-02-20T09:24:00.207 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:23:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:23:59.874+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:24:00.207 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:23:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:23:59.874+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:24:00.207 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:23:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:23:59.874+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:24:00.207 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:23:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:23:59.950+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:23:59.950755+0000) 2024-02-20T09:24:00.462 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:24:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:24:00.262+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:00.262692+0000) 2024-02-20T09:24:00.463 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:24:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:24:00.204+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:24:00.463 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:24:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:24:00.205+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:24:00.463 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:24:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:24:00.205+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:24:01.190 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:24:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:24:01.065+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:01.066140+0000) 2024-02-20T09:24:01.190 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:24:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:24:01.188+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:24:01.190 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:24:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:24:00.860+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:24:01.191 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:24:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:24:00.860+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:24:01.191 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:24:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:24:00.860+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:24:01.191 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:24:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:24:00.950+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:00.950873+0000) 2024-02-20T09:24:01.191 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:24:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:24:00.828+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:24:01.191 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:24:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:24:00.828+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:24:01.191 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:24:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:24:00.828+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:24:01.191 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:24:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:24:00.828+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:24:01.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:24:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:24:01.266+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:01.266461+0000) 2024-02-20T09:24:01.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:24:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:24:01.188+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:24:01.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:24:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:24:01.188+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:24:01.929 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:24:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:24:01.802+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:01.802504+0000) 2024-02-20T09:24:01.929 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:24:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:24:01.781+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:24:01.929 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:24:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:24:01.781+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:24:01.929 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:24:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:24:01.781+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:24:01.930 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:24:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:24:01.781+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:24:01.930 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:24:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:24:01.805+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:01.805589+0000) 2024-02-20T09:24:01.930 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:24:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:24:01.802+0000 7f8a5f86e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:01.802725+0000) 2024-02-20T09:24:01.930 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:24:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:24:01.814+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:24:01.930 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:24:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:24:01.814+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:24:01.930 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:24:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:24:01.814+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:24:01.931 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:24:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:24:01.808+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 68504 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:24:02.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:24:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:24:01.950+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:01.951048+0000) 2024-02-20T09:24:02.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:24:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:24:02.065+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:02.066251+0000) 2024-02-20T09:24:02.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:24:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:24:02.142+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:24:02.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:24:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:24:02.142+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:24:02.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:24:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:24:02.142+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:24:02.596 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:24:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:24:02.266+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:02.266731+0000) 2024-02-20T09:24:02.941 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:24:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:24:02.776+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:24:02.941 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:24:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:24:02.776+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:24:02.941 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:24:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:24:02.776+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:24:02.941 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:24:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:24:02.827+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:24:02.941 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:24:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:24:02.827+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:24:02.942 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:24:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:24:02.827+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:24:02.942 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:24:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:24:02.827+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:24:03.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:24:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:24:02.950+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:02.951162+0000) 2024-02-20T09:24:03.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:24:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:24:03.065+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:03.066358+0000) 2024-02-20T09:24:03.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:24:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:24:03.141+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:24:03.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:24:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:24:03.141+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:24:03.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:24:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:24:03.141+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:24:03.596 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:24:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:24:03.268+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:03.269208+0000) 2024-02-20T09:24:03.941 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:24:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:24:03.732+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:24:03.941 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:24:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:24:03.732+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:24:03.941 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:24:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:24:03.732+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:24:03.941 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:24:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:24:03.819+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:24:03.941 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:24:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:24:03.819+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:24:03.942 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:24:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:24:03.819+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:24:03.942 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:24:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:24:03.819+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:24:04.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:24:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:24:03.950+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:03.951299+0000) 2024-02-20T09:24:04.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:24:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:24:04.066+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:04.066447+0000) 2024-02-20T09:24:04.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:24:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:24:04.168+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:24:04.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:24:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:24:04.168+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:24:04.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:24:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:24:04.168+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:24:04.596 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:24:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:24:04.270+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:04.271299+0000) 2024-02-20T09:24:04.864 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:24:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:24:04.756+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:24:04.865 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:24:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:24:04.756+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:24:04.865 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:24:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:24:04.756+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:24:05.217 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:24:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:24:04.951+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:04.951395+0000) 2024-02-20T09:24:05.217 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:24:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:24:05.066+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:05.066543+0000) 2024-02-20T09:24:05.217 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:24:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:24:04.863+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:24:05.217 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:24:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:24:04.863+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:24:05.218 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:24:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:24:04.863+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:24:05.218 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:24:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:24:04.863+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:24:05.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:24:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:24:05.272+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:05.273348+0000) 2024-02-20T09:24:05.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:24:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:24:05.215+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:24:05.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:24:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:24:05.215+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:24:05.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:24:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:24:05.215+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:24:05.941 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:24:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:24:05.718+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:24:05.941 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:24:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:24:05.718+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:24:05.941 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:24:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:24:05.718+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:24:05.942 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:24:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:24:05.833+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:24:05.942 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:24:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:24:05.833+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:24:05.942 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:24:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:24:05.833+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:24:05.942 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:24:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:24:05.833+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:24:06.228 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:24:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:24:05.951+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:05.951523+0000) 2024-02-20T09:24:06.229 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:24:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:24:06.066+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:06.066630+0000) 2024-02-20T09:24:06.229 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:24:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:24:06.227+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:24:06.229 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:24:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:24:06.227+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:24:06.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:24:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:24:06.227+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:24:06.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:24:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:24:06.275+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:06.275435+0000) 2024-02-20T09:24:06.936 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:24:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:24:06.816+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 68543 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:24:06.936 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:24:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:24:06.809+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:06.810023+0000) 2024-02-20T09:24:06.937 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:24:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:24:06.740+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:24:06.937 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:24:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:24:06.740+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:24:06.937 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:24:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:24:06.740+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:24:06.937 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:24:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:24:06.810+0000 7f8a5f86e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:06.810445+0000) 2024-02-20T09:24:06.937 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:24:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:24:06.813+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:06.813598+0000) 2024-02-20T09:24:06.937 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:24:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:24:06.821+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:24:06.938 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:24:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:24:06.821+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:24:06.938 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:24:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:24:06.821+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:24:06.938 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:24:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:24:06.821+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:24:07.229 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:24:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:24:06.951+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:06.951641+0000) 2024-02-20T09:24:07.229 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:24:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:24:07.066+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:07.066744+0000) 2024-02-20T09:24:07.229 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:24:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:24:07.227+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:24:07.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:24:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:24:07.275+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:07.275739+0000) 2024-02-20T09:24:07.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:24:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:24:07.227+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:24:07.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:24:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:24:07.227+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:24:07.860 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:24:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:24:07.760+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:24:07.860 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:24:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:24:07.760+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:24:07.860 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:24:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:24:07.760+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:24:08.228 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:24:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:24:07.951+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:07.951717+0000) 2024-02-20T09:24:08.228 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:24:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:24:07.858+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:24:08.229 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:24:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:24:07.858+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:24:08.229 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:24:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:24:07.858+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:24:08.229 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:24:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:24:07.858+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:24:08.229 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:24:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:24:08.066+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:08.066849+0000) 2024-02-20T09:24:08.229 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:24:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:24:08.226+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:24:08.229 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:24:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:24:08.226+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:24:08.229 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:24:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:24:08.226+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:24:08.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:24:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:24:08.277+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:08.277692+0000) 2024-02-20T09:24:08.858 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:24:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:24:08.722+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:24:08.859 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:24:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:24:08.722+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:24:08.859 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:24:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:24:08.722+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:24:09.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:24:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:24:08.951+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:08.951808+0000) 2024-02-20T09:24:09.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:24:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:24:09.066+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:09.066960+0000) 2024-02-20T09:24:09.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:24:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:24:08.857+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:24:09.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:24:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:24:08.857+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:24:09.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:24:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:24:08.857+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:24:09.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:24:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:24:08.857+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:24:09.597 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:24:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:24:09.279+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:09.279625+0000) 2024-02-20T09:24:09.597 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:24:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:24:09.253+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:24:09.597 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:24:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:24:09.253+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:24:09.597 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:24:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:24:09.253+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:24:09.881 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:24:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:24:09.704+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:24:09.881 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:24:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:24:09.704+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:24:09.881 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:24:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:24:09.704+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:24:10.238 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:24:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:24:09.951+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:09.951925+0000) 2024-02-20T09:24:10.239 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:24:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:24:09.879+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:24:10.239 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:24:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:24:09.879+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:24:10.239 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:24:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:24:09.879+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:24:10.239 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:24:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:24:09.879+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:24:10.239 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:24:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:24:10.066+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:10.067087+0000) 2024-02-20T09:24:10.240 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:24:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:24:10.237+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:24:10.240 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:24:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:24:10.237+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:24:10.240 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:24:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:24:10.237+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:24:10.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:24:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:24:10.282+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:10.283172+0000) 2024-02-20T09:24:10.921 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:24:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:24:10.658+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:24:10.922 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:24:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:24:10.658+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:24:10.922 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:24:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:24:10.658+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:24:11.201 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:24:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:24:10.951+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:10.952027+0000) 2024-02-20T09:24:11.202 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:24:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:24:10.919+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:24:11.202 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:24:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:24:10.919+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:24:11.202 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:24:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:24:10.919+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:24:11.202 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:24:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:24:10.920+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:24:11.202 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:24:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:24:11.066+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:11.067187+0000) 2024-02-20T09:24:11.203 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:24:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:24:11.199+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:24:11.203 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:24:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:24:11.199+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:24:11.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:24:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:24:11.284+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:11.285255+0000) 2024-02-20T09:24:11.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:24:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:24:11.199+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:24:11.922 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:24:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:24:11.823+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 68583 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:24:11.923 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:24:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:24:11.817+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:11.817583+0000) 2024-02-20T09:24:11.923 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:24:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:24:11.820+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:11.821294+0000) 2024-02-20T09:24:11.923 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:24:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:24:11.920+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:24:11.923 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:24:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:24:11.920+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:24:11.924 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:24:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:24:11.920+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:24:11.924 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:24:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:24:11.638+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:24:11.924 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:24:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:24:11.638+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:24:11.924 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:24:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:24:11.638+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:24:11.924 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:24:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:24:11.817+0000 7f8a5f86e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:11.817846+0000) 2024-02-20T09:24:12.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:24:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:24:11.951+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:11.952141+0000) 2024-02-20T09:24:12.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:24:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:24:11.921+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:24:12.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:24:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:24:12.066+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:12.067321+0000) 2024-02-20T09:24:12.597 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:24:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:24:12.285+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:12.285613+0000) 2024-02-20T09:24:12.598 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:24:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:24:12.241+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:24:12.598 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:24:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:24:12.241+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:24:12.598 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:24:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:24:12.241+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:24:12.905 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:24:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:24:12.641+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:24:12.905 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:24:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:24:12.641+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:24:12.905 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:24:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:24:12.641+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:24:13.219 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:24:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:24:12.951+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:12.952275+0000) 2024-02-20T09:24:13.220 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:24:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:24:12.903+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:24:13.220 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:24:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:24:12.903+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:24:13.220 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:24:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:24:12.903+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:24:13.220 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:24:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:24:12.903+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:24:13.221 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:24:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:24:13.067+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:13.067432+0000) 2024-02-20T09:24:13.221 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:24:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:24:13.217+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:24:13.221 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:24:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:24:13.217+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:24:13.221 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:24:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:24:13.217+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:24:13.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:24:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:24:13.288+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:13.288734+0000) 2024-02-20T09:24:13.926 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:24:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:24:13.601+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:24:13.926 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:24:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:24:13.601+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:24:13.927 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:24:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:24:13.601+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:24:14.177 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:24:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:24:13.952+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:13.952389+0000) 2024-02-20T09:24:14.177 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:24:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:24:14.067+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:14.067575+0000) 2024-02-20T09:24:14.177 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:24:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:24:14.175+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:24:14.178 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:24:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:24:14.175+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:24:14.178 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:24:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:24:13.924+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:24:14.178 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:24:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:24:13.924+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:24:14.178 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:24:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:24:13.924+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:24:14.178 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:24:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:24:13.925+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:24:14.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:24:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:24:14.291+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:14.291624+0000) 2024-02-20T09:24:14.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:24:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:24:14.175+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:24:14.942 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:24:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:24:14.650+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:24:14.943 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:24:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:24:14.650+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:24:14.943 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:24:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:24:14.650+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:24:15.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:24:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:24:14.952+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:14.952507+0000) 2024-02-20T09:24:15.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:24:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:24:14.962+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:24:15.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:24:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:24:14.962+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:24:15.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:24:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:24:14.962+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:24:15.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:24:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:24:14.962+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:24:15.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:24:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:24:15.067+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:15.067709+0000) 2024-02-20T09:24:15.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:24:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:24:15.127+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:24:15.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:24:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:24:15.127+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:24:15.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:24:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:24:15.127+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:24:15.597 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:24:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:24:15.294+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:15.295189+0000) 2024-02-20T09:24:15.925 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:24:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:24:15.635+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:24:15.925 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:24:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:24:15.635+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:24:15.925 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:24:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:24:15.635+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:24:16.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:24:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:24:15.951+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:15.952600+0000) 2024-02-20T09:24:16.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:24:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:24:15.922+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:24:16.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:24:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:24:15.922+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:24:16.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:24:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:24:15.922+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:24:16.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:24:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:24:15.922+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:24:16.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:24:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:24:16.066+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:16.067824+0000) 2024-02-20T09:24:16.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:24:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:24:16.131+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:24:16.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:24:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:24:16.131+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:24:16.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:24:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:24:16.131+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:24:16.598 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:24:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:24:16.295+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:16.297333+0000) 2024-02-20T09:24:16.848 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:24:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:24:16.827+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:16.828981+0000) 2024-02-20T09:24:16.849 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:24:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:24:16.823+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:16.824890+0000) 2024-02-20T09:24:16.849 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:24:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:24:16.638+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:24:16.849 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:24:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:24:16.638+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:24:16.849 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:24:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:24:16.638+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:24:16.849 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:24:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:24:16.823+0000 7f8a5f86e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:16.825196+0000) 2024-02-20T09:24:16.850 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:24:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:24:16.829+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 68622 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:24:17.111 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:24:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:24:16.951+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:16.952706+0000) 2024-02-20T09:24:17.116 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:24:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:24:16.874+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:24:17.116 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:24:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:24:16.874+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:24:17.116 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:24:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:24:16.874+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:24:17.117 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:24:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:24:16.874+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:24:17.117 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:24:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:24:17.066+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:17.067982+0000) 2024-02-20T09:24:17.117 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:24:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:24:17.108+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:24:17.117 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:24:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:24:17.108+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:24:17.117 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:24:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:24:17.108+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:24:17.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:24:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:24:17.296+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:17.297818+0000) 2024-02-20T09:24:17.889 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:24:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:24:17.632+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:24:17.890 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:24:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:24:17.632+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:24:17.890 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:24:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:24:17.632+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:24:18.142 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:24:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:24:17.951+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:17.952807+0000) 2024-02-20T09:24:18.142 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:24:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:24:18.066+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:18.068117+0000) 2024-02-20T09:24:18.142 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:24:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:24:17.886+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:24:18.142 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:24:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:24:17.886+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:24:18.143 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:24:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:24:17.886+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:24:18.143 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:24:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:24:17.887+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:24:18.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:24:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:24:18.299+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:18.300608+0000) 2024-02-20T09:24:18.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:24:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:24:18.139+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:24:18.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:24:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:24:18.139+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:24:18.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:24:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:24:18.139+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:24:18.935 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:24:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:24:18.588+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:24:18.935 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:24:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:24:18.588+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:24:18.936 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:24:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:24:18.588+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:24:19.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:24:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:24:18.951+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:18.952921+0000) 2024-02-20T09:24:19.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:24:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:24:18.932+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:24:19.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:24:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:24:18.932+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:24:19.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:24:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:24:18.932+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:24:19.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:24:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:24:18.932+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:24:19.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:24:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:24:19.066+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:19.068294+0000) 2024-02-20T09:24:19.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:24:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:24:19.113+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:24:19.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:24:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:24:19.113+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:24:19.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:24:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:24:19.113+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:24:19.598 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:24:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:24:19.301+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:19.302639+0000) 2024-02-20T09:24:19.886 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:24:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:24:19.596+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:24:19.886 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:24:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:24:19.596+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:24:19.886 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:24:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:24:19.596+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:24:20.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:24:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:24:19.951+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:19.953019+0000) 2024-02-20T09:24:20.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:24:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:24:20.067+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:20.068444+0000) 2024-02-20T09:24:20.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:24:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:24:20.090+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:24:20.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:24:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:24:20.090+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:24:20.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:24:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:24:20.090+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:24:20.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:24:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:24:19.909+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:24:20.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:24:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:24:19.909+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:24:20.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:24:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:24:19.909+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:24:20.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:24:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:24:19.909+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:24:20.584 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:24:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:24:20.304+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:20.306280+0000) 2024-02-20T09:24:20.856 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:24:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:24:20.582+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:24:20.857 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:24:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:24:20.582+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:24:20.857 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:24:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:24:20.582+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:24:21.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:24:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:24:20.951+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:20.953139+0000) 2024-02-20T09:24:21.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:24:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:24:20.958+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:24:21.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:24:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:24:20.958+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:24:21.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:24:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:24:20.958+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:24:21.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:24:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:24:20.958+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:24:21.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:24:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:24:21.067+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:21.068555+0000) 2024-02-20T09:24:21.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:24:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:24:21.086+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:24:21.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:24:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:24:21.086+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:24:21.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:24:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:24:21.086+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:24:21.593 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:24:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:24:21.307+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:21.308523+0000) 2024-02-20T09:24:21.845 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:24:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:24:21.837+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 68663 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:24:21.846 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:24:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:24:21.832+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:21.834282+0000) 2024-02-20T09:24:21.846 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:24:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:24:21.831+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:21.832515+0000) 2024-02-20T09:24:21.847 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:24:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:24:21.591+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:24:21.847 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:24:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:24:21.591+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:24:21.847 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:24:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:24:21.591+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:24:21.847 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:24:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:24:21.830+0000 7f8a5f86e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:21.832192+0000) 2024-02-20T09:24:22.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:24:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:24:21.951+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:21.953310+0000) 2024-02-20T09:24:22.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:24:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:24:21.970+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:24:22.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:24:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:24:21.970+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:24:22.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:24:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:24:21.970+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:24:22.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:24:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:24:21.970+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:24:22.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:24:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:24:22.067+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:22.068653+0000) 2024-02-20T09:24:22.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:24:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:24:22.086+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:24:22.245 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:24:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:24:22.086+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:24:22.245 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:24:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:24:22.086+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:24:22.598 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:24:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:24:22.307+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:22.308919+0000) 2024-02-20T09:24:22.943 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:24:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:24:22.598+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:24:22.943 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:24:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:24:22.598+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:24:22.943 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:24:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:24:22.598+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:24:23.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:24:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:24:22.952+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:22.953524+0000) 2024-02-20T09:24:23.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:24:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:24:22.949+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:24:23.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:24:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:24:22.949+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:24:23.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:24:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:24:22.949+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:24:23.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:24:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:24:22.949+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:24:23.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:24:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:24:23.067+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:23.068759+0000) 2024-02-20T09:24:23.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:24:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:24:23.101+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:24:23.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:24:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:24:23.101+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:24:23.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:24:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:24:23.101+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:24:23.589 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:24:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:24:23.309+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:23.310862+0000) 2024-02-20T09:24:23.924 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:24:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:24:23.586+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:24:23.924 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:24:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:24:23.586+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:24:23.925 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:24:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:24:23.586+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:24:24.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:24:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:24:23.952+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:23.953661+0000) 2024-02-20T09:24:24.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:24:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:24:24.067+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:24.068885+0000) 2024-02-20T09:24:24.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:24:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:24:24.150+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:24:24.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:24:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:24:24.150+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:24:24.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:24:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:24:24.150+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:24:24.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:24:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:24:23.922+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:24:24.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:24:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:24:23.922+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:24:24.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:24:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:24:23.922+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:24:24.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:24:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:24:23.922+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:24:24.599 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:24:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:24:24.311+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:24.313011+0000) 2024-02-20T09:24:24.935 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:24:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:24:24.630+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:24:24.935 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:24:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:24:24.630+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:24:24.935 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:24:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:24:24.630+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:24:25.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:24:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:24:24.952+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:24.953777+0000) 2024-02-20T09:24:25.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:24:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:24:25.067+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:25.069012+0000) 2024-02-20T09:24:25.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:24:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:24:25.167+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:24:25.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:24:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:24:25.167+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:24:25.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:24:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:24:25.167+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:24:25.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:24:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:24:24.932+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:24:25.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:24:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:24:24.932+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:24:25.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:24:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:24:24.932+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:24:25.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:24:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:24:24.932+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:24:25.599 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:24:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:24:25.313+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:25.315044+0000) 2024-02-20T09:24:25.944 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:24:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:24:25.616+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:24:25.944 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:24:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:24:25.616+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:24:25.944 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:24:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:24:25.616+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:24:26.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:24:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:24:25.952+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:25.953872+0000) 2024-02-20T09:24:26.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:24:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:24:26.067+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:26.069226+0000) 2024-02-20T09:24:26.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:24:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:24:26.127+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:24:26.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:24:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:24:26.127+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:24:26.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:24:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:24:26.127+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:24:26.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:24:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:24:25.940+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:24:26.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:24:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:24:25.940+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:24:26.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:24:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:24:25.940+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:24:26.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:24:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:24:25.940+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:24:26.599 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:24:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:24:26.315+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:26.317143+0000) 2024-02-20T09:24:26.852 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:24:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:24:26.845+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 68702 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:24:26.853 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:24:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:24:26.841+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:26.843264+0000) 2024-02-20T09:24:26.853 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:24:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:24:26.838+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:26.840079+0000) 2024-02-20T09:24:26.853 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:24:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:24:26.658+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:24:26.853 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:24:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:24:26.658+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:24:26.854 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:24:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:24:26.658+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:24:26.854 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:24:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:24:26.839+0000 7f8a5f86e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:26.840513+0000) 2024-02-20T09:24:27.180 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:24:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:24:26.952+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:26.953965+0000) 2024-02-20T09:24:27.180 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:24:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:24:26.965+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:24:27.181 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:24:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:24:26.965+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:24:27.181 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:24:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:24:26.965+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:24:27.181 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:24:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:24:26.965+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:24:27.181 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:24:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:24:27.067+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:27.069366+0000) 2024-02-20T09:24:27.181 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:24:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:24:27.177+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:24:27.181 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:24:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:24:27.177+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:24:27.182 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:24:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:24:27.177+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:24:27.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:24:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:24:27.316+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:27.317491+0000) 2024-02-20T09:24:27.943 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:24:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:24:27.636+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:24:27.944 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:24:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:24:27.636+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:24:27.944 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:24:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:24:27.636+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:24:28.207 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:24:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:24:27.952+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:27.954068+0000) 2024-02-20T09:24:28.207 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:24:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:24:28.068+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:28.069463+0000) 2024-02-20T09:24:28.208 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:24:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:24:27.992+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:24:28.208 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:24:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:24:27.992+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:24:28.208 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:24:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:24:27.992+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:24:28.208 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:24:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:24:27.992+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:24:28.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:24:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:24:28.318+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:28.319457+0000) 2024-02-20T09:24:28.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:24:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:24:28.204+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:24:28.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:24:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:24:28.204+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:24:28.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:24:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:24:28.204+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:24:28.943 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:24:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:24:28.678+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:24:28.944 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:24:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:24:28.678+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:24:28.944 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:24:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:24:28.678+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:24:29.240 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:24:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:24:28.952+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:28.954208+0000) 2024-02-20T09:24:29.240 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:24:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:24:28.976+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:24:29.240 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:24:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:24:28.976+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:24:29.241 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:24:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:24:28.976+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:24:29.241 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:24:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:24:28.976+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:24:29.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:24:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:24:29.068+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:29.069556+0000) 2024-02-20T09:24:29.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:24:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:24:29.237+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:24:29.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:24:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:24:29.237+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:24:29.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:24:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:24:29.237+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:24:29.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:24:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:24:29.320+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:29.321444+0000) 2024-02-20T09:24:29.886 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:24:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:24:29.685+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:24:29.886 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:24:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:24:29.685+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:24:29.886 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:24:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:24:29.685+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:24:30.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:24:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:24:29.952+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:29.954334+0000) 2024-02-20T09:24:30.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:24:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:24:30.068+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:30.069644+0000) 2024-02-20T09:24:30.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:24:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:24:29.969+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:24:30.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:24:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:24:29.969+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:24:30.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:24:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:24:29.969+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:24:30.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:24:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:24:29.969+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:24:30.599 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:24:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:24:30.322+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:30.324235+0000) 2024-02-20T09:24:30.600 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:24:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:24:30.261+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:24:30.600 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:24:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:24:30.261+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:24:30.600 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:24:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:24:30.261+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:24:30.944 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:24:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:24:30.693+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:24:30.944 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:24:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:24:30.693+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:24:30.944 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:24:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:24:30.693+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:24:31.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:24:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:24:30.953+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:30.954458+0000) 2024-02-20T09:24:31.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:24:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:24:31.068+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:31.069778+0000) 2024-02-20T09:24:31.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:24:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:24:30.966+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:24:31.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:24:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:24:30.966+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:24:31.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:24:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:24:30.966+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:24:31.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:24:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:24:30.966+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:24:31.599 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:24:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:24:31.324+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:31.326241+0000) 2024-02-20T09:24:31.600 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:24:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:24:31.299+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:24:31.600 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:24:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:24:31.299+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:24:31.600 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:24:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:24:31.299+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:24:31.850 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:24:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:24:31.847+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:31.848614+0000) 2024-02-20T09:24:31.851 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:24:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:24:31.712+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:24:31.851 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:24:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:24:31.712+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:24:31.851 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:24:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:24:31.712+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:24:31.851 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:24:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:24:31.846+0000 7f8a5f86e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:31.847877+0000) 2024-02-20T09:24:32.242 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:24:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:24:31.853+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 68742 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:24:32.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:24:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:24:31.953+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:31.954560+0000) 2024-02-20T09:24:32.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:24:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:24:32.068+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:32.069915+0000) 2024-02-20T09:24:32.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:24:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:24:31.849+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:31.850923+0000) 2024-02-20T09:24:32.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:24:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:24:31.971+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:24:32.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:24:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:24:31.971+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:24:32.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:24:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:24:31.971+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:24:32.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:24:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:24:31.971+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:24:32.600 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:24:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:24:32.325+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:32.326602+0000) 2024-02-20T09:24:32.600 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:24:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:24:32.267+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:24:32.600 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:24:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:24:32.267+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:24:32.600 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:24:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:24:32.267+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:24:32.944 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:24:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:24:32.732+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:24:32.944 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:24:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:24:32.732+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:24:32.945 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:24:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:24:32.732+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:24:33.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:24:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:24:32.953+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:32.954659+0000) 2024-02-20T09:24:33.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:24:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:24:33.068+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:33.069991+0000) 2024-02-20T09:24:33.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:24:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:24:32.971+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:24:33.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:24:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:24:32.971+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:24:33.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:24:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:24:32.971+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:24:33.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:24:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:24:32.971+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:24:33.600 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:24:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:24:33.327+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:33.328722+0000) 2024-02-20T09:24:33.600 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:24:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:24:33.301+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:24:33.600 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:24:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:24:33.301+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:24:33.600 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:24:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:24:33.301+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:24:33.930 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:24:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:24:33.723+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:24:33.930 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:24:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:24:33.723+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:24:33.930 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:24:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:24:33.723+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:24:34.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:24:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:24:33.953+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:33.954756+0000) 2024-02-20T09:24:34.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:24:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:24:34.068+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:34.070124+0000) 2024-02-20T09:24:34.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:24:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:24:33.927+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:24:34.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:24:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:24:33.927+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:24:34.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:24:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:24:33.927+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:24:34.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:24:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:24:33.927+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:24:34.600 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:24:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:24:34.329+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:34.330673+0000) 2024-02-20T09:24:34.600 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:24:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:24:34.255+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:24:34.600 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:24:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:24:34.255+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:24:34.601 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:24:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:24:34.255+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:24:34.928 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:24:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:24:34.772+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:24:34.928 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:24:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:24:34.772+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:24:34.928 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:24:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:24:34.772+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:24:35.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:24:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:24:34.953+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:34.954870+0000) 2024-02-20T09:24:35.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:24:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:24:35.068+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:35.070256+0000) 2024-02-20T09:24:35.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:24:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:24:34.925+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:24:35.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:24:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:24:34.925+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:24:35.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:24:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:24:34.925+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:24:35.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:24:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:24:34.925+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:24:35.600 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:24:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:24:35.331+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:35.332635+0000) 2024-02-20T09:24:35.600 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:24:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:24:35.265+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:24:35.600 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:24:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:24:35.265+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:24:35.601 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:24:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:24:35.265+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:24:35.904 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:24:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:24:35.794+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:24:35.904 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:24:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:24:35.794+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:24:35.907 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:24:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:24:35.794+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:24:36.219 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:24:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:24:35.953+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:35.954982+0000) 2024-02-20T09:24:36.219 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:24:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:24:36.068+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:36.070384+0000) 2024-02-20T09:24:36.220 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:24:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:24:36.216+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:24:36.220 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:24:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:24:35.901+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:24:36.220 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:24:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:24:35.901+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:24:36.220 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:24:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:24:35.901+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:24:36.220 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:24:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:24:35.901+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:24:36.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:24:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:24:36.333+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:36.334753+0000) 2024-02-20T09:24:36.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:24:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:24:36.216+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:24:36.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:24:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:24:36.216+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:24:36.856 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:24:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:24:36.843+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:24:36.856 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:24:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:24:36.843+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:24:36.857 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:24:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:24:36.843+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:24:37.242 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:24:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:24:36.861+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 68782 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:24:37.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:24:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:24:36.855+0000 7f8a5f86e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:36.856486+0000) 2024-02-20T09:24:37.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:24:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:24:36.953+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:36.955091+0000) 2024-02-20T09:24:37.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:24:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:24:36.854+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:36.856138+0000) 2024-02-20T09:24:37.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:24:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:24:37.069+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:37.070494+0000) 2024-02-20T09:24:37.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:24:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:24:36.853+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:24:37.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:24:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:24:36.853+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:24:37.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:24:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:24:36.853+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:24:37.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:24:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:24:36.853+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:24:37.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:24:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:24:36.858+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:36.859670+0000) 2024-02-20T09:24:37.600 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:24:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:24:37.333+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:37.335314+0000) 2024-02-20T09:24:37.731 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:24:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:24:37.251+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:24:37.731 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:24:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:24:37.251+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:24:37.731 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:24:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:24:37.251+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:24:38.216 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:24:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:24:38.069+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:38.070566+0000) 2024-02-20T09:24:38.217 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:24:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:24:37.860+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:24:38.217 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:24:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:24:37.860+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:24:38.217 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:24:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:24:37.860+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:24:38.217 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:24:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:24:37.860+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:24:38.217 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:24:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:24:37.871+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:24:38.217 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:24:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:24:37.871+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:24:38.218 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:24:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:24:37.871+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:24:38.218 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:24:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:24:37.953+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:37.955275+0000) 2024-02-20T09:24:38.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:24:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:24:38.336+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:38.337478+0000) 2024-02-20T09:24:38.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:24:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:24:38.214+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:24:38.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:24:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:24:38.214+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:24:38.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:24:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:24:38.214+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:24:39.238 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:24:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:24:39.069+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:39.070649+0000) 2024-02-20T09:24:39.238 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:24:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:24:39.235+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:24:39.238 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:24:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:24:39.235+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:24:39.238 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:24:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:24:38.901+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:24:39.238 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:24:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:24:38.901+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:24:39.239 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:24:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:24:38.901+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:24:39.239 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:24:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:24:38.954+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:38.955492+0000) 2024-02-20T09:24:39.239 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:24:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:24:38.883+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:24:39.239 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:24:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:24:38.883+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:24:39.239 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:24:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:24:38.883+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:24:39.239 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:24:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:24:38.883+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:24:39.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:24:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:24:39.338+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:39.339524+0000) 2024-02-20T09:24:39.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:24:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:24:39.235+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:24:40.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:24:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:24:40.069+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:40.070758+0000) 2024-02-20T09:24:40.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:24:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:24:39.854+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:24:40.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:24:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:24:39.854+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:24:40.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:24:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:24:39.854+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:24:40.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:24:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:24:39.954+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:39.955655+0000) 2024-02-20T09:24:40.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:24:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:24:39.887+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:24:40.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:24:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:24:39.887+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:24:40.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:24:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:24:39.887+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:24:40.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:24:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:24:39.887+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:24:40.601 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:24:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:24:40.340+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:40.341980+0000) 2024-02-20T09:24:40.601 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:24:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:24:40.256+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:24:40.601 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:24:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:24:40.256+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:24:40.601 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:24:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:24:40.256+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:24:40.854 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:24:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:24:40.815+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:24:40.855 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:24:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:24:40.815+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:24:40.855 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:24:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:24:40.815+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:24:41.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:24:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:24:40.954+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:40.955776+0000) 2024-02-20T09:24:41.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:24:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:24:41.069+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:41.070939+0000) 2024-02-20T09:24:41.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:24:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:24:40.852+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:24:41.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:24:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:24:40.852+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:24:41.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:24:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:24:40.852+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:24:41.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:24:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:24:40.852+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:24:41.601 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:24:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:24:41.342+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:41.344202+0000) 2024-02-20T09:24:41.601 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:24:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:24:41.248+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:24:41.601 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:24:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:24:41.248+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:24:41.601 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:24:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:24:41.248+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:24:41.853 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:24:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:24:41.812+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:24:41.853 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:24:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:24:41.812+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:24:41.853 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:24:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:24:41.812+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:24:41.854 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:24:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:24:41.812+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:24:42.213 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:24:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:24:41.869+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 68821 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:24:42.214 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:24:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:24:41.865+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:41.866487+0000) 2024-02-20T09:24:42.214 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:24:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:24:41.850+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:24:42.214 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:24:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:24:41.850+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:24:42.215 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:24:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:24:41.850+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:24:42.215 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:24:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:24:41.862+0000 7f8a5f86e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:41.864358+0000) 2024-02-20T09:24:42.215 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:24:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:24:41.954+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:41.955893+0000) 2024-02-20T09:24:42.215 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:24:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:24:41.862+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:41.863782+0000) 2024-02-20T09:24:42.215 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:24:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:24:42.069+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:42.071134+0000) 2024-02-20T09:24:42.215 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:24:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:24:42.211+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:24:42.215 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:24:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:24:42.211+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:24:42.216 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:24:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:24:42.211+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:24:42.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:24:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:24:42.343+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:42.344610+0000) 2024-02-20T09:24:42.946 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:24:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:24:42.837+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:24:42.946 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:24:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:24:42.837+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:24:42.946 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:24:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:24:42.837+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:24:42.947 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:24:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:24:42.790+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:24:42.948 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:24:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:24:42.790+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:24:42.949 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:24:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:24:42.790+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:24:42.949 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:24:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:24:42.790+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:24:43.215 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:24:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:24:42.954+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:42.956065+0000) 2024-02-20T09:24:43.216 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:24:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:24:43.069+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:43.071273+0000) 2024-02-20T09:24:43.216 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:24:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:24:43.212+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:24:43.216 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:24:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:24:43.212+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:24:43.216 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:24:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:24:43.212+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:24:43.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:24:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:24:43.345+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:43.347314+0000) 2024-02-20T09:24:43.945 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:24:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:24:43.845+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:24:43.946 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:24:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:24:43.845+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:24:43.946 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:24:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:24:43.845+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:24:43.946 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:24:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:24:43.772+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:24:43.946 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:24:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:24:43.772+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:24:43.946 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:24:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:24:43.772+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:24:43.946 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:24:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:24:43.772+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:24:44.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:24:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:24:43.954+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:43.956191+0000) 2024-02-20T09:24:44.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:24:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:24:44.070+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:44.071474+0000) 2024-02-20T09:24:44.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:24:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:24:44.186+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:24:44.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:24:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:24:44.186+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:24:44.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:24:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:24:44.186+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:24:44.601 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:24:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:24:44.348+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:44.349924+0000) 2024-02-20T09:24:44.945 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:24:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:24:44.806+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:24:44.946 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:24:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:24:44.806+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:24:44.946 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:24:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:24:44.806+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:24:44.946 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:24:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:24:44.791+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:24:44.946 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:24:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:24:44.791+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:24:44.947 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:24:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:24:44.791+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:24:44.947 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:24:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:24:44.791+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:24:45.215 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:24:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:24:44.954+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:44.956309+0000) 2024-02-20T09:24:45.215 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:24:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:24:45.070+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:45.071577+0000) 2024-02-20T09:24:45.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:24:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:24:45.350+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:45.352263+0000) 2024-02-20T09:24:45.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:24:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:24:45.212+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:24:45.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:24:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:24:45.212+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:24:45.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:24:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:24:45.212+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:24:45.945 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:24:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:24:45.810+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:24:45.946 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:24:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:24:45.810+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:24:45.946 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:24:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:24:45.810+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:24:45.946 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:24:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:24:45.814+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:24:45.946 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:24:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:24:45.814+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:24:45.946 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:24:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:24:45.814+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:24:45.947 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:24:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:24:45.815+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:24:46.215 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:24:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:24:45.955+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:45.956439+0000) 2024-02-20T09:24:46.215 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:24:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:24:46.071+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:46.071672+0000) 2024-02-20T09:24:46.216 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:24:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:24:46.212+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:24:46.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:24:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:24:46.352+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:46.354367+0000) 2024-02-20T09:24:46.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:24:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:24:46.212+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:24:46.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:24:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:24:46.212+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:24:46.862 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:24:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:24:46.810+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:24:46.862 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:24:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:24:46.810+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:24:46.862 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:24:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:24:46.810+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:24:47.202 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:24:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:24:46.877+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 68861 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:24:47.202 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:24:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:24:46.870+0000 7f8a5f86e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:46.871428+0000) 2024-02-20T09:24:47.202 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:24:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:24:46.955+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:46.956556+0000) 2024-02-20T09:24:47.203 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:24:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:24:46.860+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:24:47.203 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:24:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:24:46.860+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:24:47.203 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:24:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:24:46.860+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:24:47.203 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:24:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:24:46.860+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:24:47.203 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:24:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:24:46.872+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:46.873988+0000) 2024-02-20T09:24:47.204 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:24:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:24:46.870+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:46.871611+0000) 2024-02-20T09:24:47.204 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:24:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:24:47.070+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:47.071773+0000) 2024-02-20T09:24:47.204 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:24:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:24:47.199+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:24:47.204 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:24:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:24:47.199+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:24:47.204 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:24:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:24:47.199+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:24:47.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:24:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:24:47.353+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:47.354903+0000) 2024-02-20T09:24:47.946 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:24:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:24:47.781+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:24:47.946 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:24:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:24:47.781+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:24:47.946 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:24:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:24:47.781+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:24:47.946 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:24:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:24:47.817+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:24:47.946 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:24:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:24:47.817+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:24:47.946 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:24:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:24:47.817+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:24:47.947 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:24:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:24:47.817+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:24:48.231 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:24:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:24:47.955+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:47.956632+0000) 2024-02-20T09:24:48.231 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:24:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:24:48.070+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:48.071864+0000) 2024-02-20T09:24:48.231 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:24:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:24:48.228+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:24:48.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:24:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:24:48.356+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:48.356969+0000) 2024-02-20T09:24:48.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:24:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:24:48.228+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:24:48.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:24:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:24:48.228+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:24:48.946 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:24:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:24:48.812+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:24:48.946 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:24:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:24:48.812+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:24:48.946 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:24:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:24:48.812+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:24:48.946 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:24:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:24:48.812+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:24:48.947 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:24:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:24:48.818+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:24:48.947 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:24:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:24:48.818+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:24:48.947 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:24:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:24:48.818+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:24:49.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:24:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:24:48.955+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:48.956731+0000) 2024-02-20T09:24:49.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:24:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:24:49.070+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:49.072016+0000) 2024-02-20T09:24:49.602 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:24:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:24:49.358+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:49.359073+0000) 2024-02-20T09:24:49.602 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:24:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:24:49.272+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:24:49.602 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:24:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:24:49.272+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:24:49.602 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:24:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:24:49.272+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:24:49.886 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:24:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:24:49.830+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:24:49.886 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:24:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:24:49.830+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:24:49.886 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:24:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:24:49.830+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:24:49.886 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:24:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:24:49.777+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:24:49.886 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:24:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:24:49.777+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:24:49.887 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:24:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:24:49.777+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:24:49.887 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:24:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:24:49.777+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:24:50.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:24:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:24:49.956+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:49.956896+0000) 2024-02-20T09:24:50.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:24:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:24:50.071+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:50.072240+0000) 2024-02-20T09:24:50.602 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:24:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:24:50.361+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:50.361502+0000) 2024-02-20T09:24:50.602 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:24:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:24:50.288+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:24:50.602 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:24:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:24:50.288+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:24:50.603 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:24:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:24:50.288+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:24:50.947 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:24:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:24:50.839+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:24:50.947 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:24:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:24:50.839+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:24:50.947 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:24:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:24:50.839+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:24:50.948 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:24:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:24:50.802+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:24:50.948 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:24:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:24:50.802+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:24:50.948 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:24:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:24:50.802+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:24:50.948 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:24:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:24:50.802+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:24:51.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:24:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:24:50.956+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:50.957018+0000) 2024-02-20T09:24:51.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:24:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:24:51.071+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:51.072391+0000) 2024-02-20T09:24:51.576 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:24:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:24:51.363+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:51.363649+0000) 2024-02-20T09:24:51.577 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:24:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:24:51.264+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:24:51.577 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:24:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:24:51.264+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:24:51.577 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:24:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:24:51.264+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:24:52.239 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:24:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:24:51.885+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 68901 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:24:52.239 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:24:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:24:51.833+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:24:52.240 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:24:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:24:51.833+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:24:52.240 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:24:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:24:51.833+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:24:52.240 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:24:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:24:51.879+0000 7f8a5f86e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:51.879510+0000) 2024-02-20T09:24:52.240 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:24:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:24:51.956+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:51.957111+0000) 2024-02-20T09:24:52.240 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:24:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:24:51.844+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:24:52.241 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:24:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:24:51.844+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:24:52.241 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:24:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:24:51.844+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:24:52.241 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:24:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:24:51.844+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:24:52.241 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:24:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:24:51.882+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:51.882707+0000) 2024-02-20T09:24:52.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:24:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:24:51.879+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:51.879419+0000) 2024-02-20T09:24:52.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:24:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:24:52.072+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:52.072514+0000) 2024-02-20T09:24:52.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:24:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:24:52.237+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:24:52.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:24:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:24:52.237+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:24:52.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:24:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:24:52.237+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:24:52.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:24:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:24:52.363+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:52.364032+0000) 2024-02-20T09:24:52.884 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:24:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:24:52.833+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:24:52.885 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:24:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:24:52.833+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:24:52.885 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:24:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:24:52.833+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:24:53.194 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:24:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:24:52.956+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:52.957235+0000) 2024-02-20T09:24:53.194 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:24:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:24:53.072+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:53.072592+0000) 2024-02-20T09:24:53.194 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:24:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:24:53.192+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:24:53.195 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:24:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:24:53.192+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:24:53.195 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:24:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:24:53.192+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:24:53.195 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:24:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:24:52.883+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:24:53.195 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:24:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:24:52.883+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:24:53.195 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:24:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:24:52.883+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:24:53.195 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:24:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:24:52.883+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:24:53.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:24:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:24:53.365+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:53.366295+0000) 2024-02-20T09:24:54.229 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:24:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:24:53.852+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:24:54.229 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:24:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:24:53.852+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:24:54.229 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:24:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:24:53.852+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:24:54.229 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:24:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:24:53.956+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:53.957350+0000) 2024-02-20T09:24:54.230 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:24:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:24:54.072+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:54.072678+0000) 2024-02-20T09:24:54.230 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:24:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:24:54.227+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:24:54.230 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:24:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:24:54.227+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:24:54.230 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:24:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:24:53.850+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:24:54.231 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:24:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:24:53.850+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:24:54.231 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:24:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:24:53.850+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:24:54.231 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:24:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:24:53.851+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:24:54.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:24:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:24:54.227+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:24:54.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:24:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:24:54.369+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:54.369859+0000) 2024-02-20T09:24:54.946 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:24:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:24:54.842+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:24:54.947 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:24:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:24:54.842+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:24:54.947 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:24:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:24:54.842+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:24:54.947 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:24:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:24:54.842+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:24:54.947 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:24:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:24:54.848+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:24:54.947 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:24:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:24:54.848+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:24:54.947 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:24:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:24:54.848+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:24:55.215 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:24:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:24:54.957+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:54.957441+0000) 2024-02-20T09:24:55.215 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:24:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:24:55.072+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:55.072791+0000) 2024-02-20T09:24:55.215 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:24:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:24:55.213+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:24:55.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:24:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:24:55.371+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:55.371805+0000) 2024-02-20T09:24:55.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:24:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:24:55.213+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:24:55.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:24:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:24:55.213+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:24:56.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:24:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:24:56.072+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:56.072897+0000) 2024-02-20T09:24:56.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:24:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:24:55.873+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:24:56.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:24:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:24:55.873+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:24:56.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:24:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:24:55.873+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:24:56.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:24:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:24:55.957+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:55.957557+0000) 2024-02-20T09:24:56.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:24:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:24:55.853+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:24:56.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:24:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:24:55.853+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:24:56.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:24:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:24:55.853+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:24:56.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:24:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:24:55.853+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:24:56.602 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:24:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:24:56.375+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:56.375462+0000) 2024-02-20T09:24:56.603 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:24:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:24:56.246+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:24:56.603 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:24:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:24:56.246+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:24:56.603 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:24:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:24:56.246+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:24:56.863 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:24:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:24:56.845+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:24:56.863 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:24:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:24:56.845+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:24:56.863 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:24:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:24:56.845+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:24:56.863 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:24:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:24:56.845+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:24:57.203 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:24:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:24:56.893+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 68941 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:24:57.203 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:24:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:24:56.890+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:56.890807+0000) 2024-02-20T09:24:57.203 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:24:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:24:56.886+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:56.887234+0000) 2024-02-20T09:24:57.204 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:24:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:24:57.072+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:57.073016+0000) 2024-02-20T09:24:57.204 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:24:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:24:57.201+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:24:57.204 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:24:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:24:57.201+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:24:57.204 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:24:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:24:57.201+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:24:57.204 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:24:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:24:56.861+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:24:57.204 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:24:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:24:56.861+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:24:57.204 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:24:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:24:56.861+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:24:57.205 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:24:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:24:56.887+0000 7f8a5f86e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:56.887750+0000) 2024-02-20T09:24:57.205 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:24:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:24:56.957+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:56.957647+0000) 2024-02-20T09:24:57.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:24:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:24:57.375+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:57.375961+0000) 2024-02-20T09:24:58.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:24:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:24:58.072+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:58.073093+0000) 2024-02-20T09:24:58.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:24:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:24:57.894+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:24:58.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:24:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:24:57.894+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:24:58.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:24:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:24:57.894+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:24:58.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:24:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:24:57.894+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:24:58.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:24:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:24:57.852+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:24:58.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:24:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:24:57.852+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:24:58.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:24:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:24:57.852+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:24:58.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:24:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:24:57.957+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:57.957741+0000) 2024-02-20T09:24:58.603 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:24:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:24:58.377+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:58.378073+0000) 2024-02-20T09:24:58.603 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:24:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:24:58.247+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:24:58.603 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:24:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:24:58.247+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:24:58.604 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:24:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:24:58.247+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:24:58.929 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:24:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:24:58.848+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:24:58.930 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:24:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:24:58.848+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:24:58.930 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:24:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:24:58.848+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:24:59.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:24:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:24:58.957+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:58.957889+0000) 2024-02-20T09:24:59.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:24:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:24:59.072+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:59.073192+0000) 2024-02-20T09:24:59.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:24:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:24:58.928+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:24:59.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:24:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:24:58.928+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:24:59.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:24:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:24:58.928+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:24:59.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:24:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:24:58.928+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:24:59.603 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:24:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:24:59.379+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:59.380259+0000) 2024-02-20T09:24:59.603 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:24:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:24:59.254+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:24:59.603 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:24:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:24:59.254+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:24:59.603 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:24:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:24:59.254+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:25:00.210 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:25:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:25:00.072+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:00.073291+0000) 2024-02-20T09:25:00.210 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:24:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:24:59.878+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:25:00.210 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:24:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:24:59.878+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:25:00.211 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:24:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:24:59.878+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:25:00.211 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:24:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:24:59.957+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:24:59.958103+0000) 2024-02-20T09:25:00.211 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:24:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:24:59.947+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:25:00.211 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:24:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:24:59.947+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:25:00.211 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:24:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:24:59.947+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:25:00.211 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:24:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:24:59.947+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:25:00.467 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:25:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:25:00.382+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:00.382930+0000) 2024-02-20T09:25:00.467 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:25:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:25:00.208+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:25:00.467 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:25:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:25:00.208+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:25:00.467 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:25:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:25:00.208+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:25:01.074 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:25:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:25:00.928+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:25:01.075 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:25:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:25:00.928+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:25:01.075 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:25:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:25:00.928+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:25:01.075 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:25:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:25:00.957+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:00.958272+0000) 2024-02-20T09:25:01.075 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:25:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:25:00.974+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:25:01.075 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:25:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:25:00.974+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:25:01.076 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:25:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:25:00.974+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:25:01.076 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:25:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:25:00.974+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:25:01.336 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:25:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:25:01.073+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:01.073389+0000) 2024-02-20T09:25:01.336 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:25:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:25:01.185+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:25:01.336 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:25:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:25:01.185+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:25:01.336 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:25:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:25:01.185+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:25:01.603 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:25:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:25:01.386+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:01.386515+0000) 2024-02-20T09:25:02.232 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:25:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:25:01.901+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 68980 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:25:02.233 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:25:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:25:01.895+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:01.895639+0000) 2024-02-20T09:25:02.233 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:25:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:25:02.073+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:02.073501+0000) 2024-02-20T09:25:02.234 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:25:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:25:01.897+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:01.898160+0000) 2024-02-20T09:25:02.234 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:25:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:25:01.935+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:25:02.234 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:25:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:25:01.935+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:25:02.234 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:25:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:25:01.935+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:25:02.234 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:25:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:25:01.935+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:25:02.235 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:25:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:25:01.894+0000 7f8a5f86e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:01.895350+0000) 2024-02-20T09:25:02.235 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:25:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:25:01.958+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:01.958422+0000) 2024-02-20T09:25:02.235 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:25:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:25:01.969+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:25:02.235 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:25:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:25:01.969+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:25:02.235 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:25:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:25:01.969+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:25:02.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:25:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:25:02.386+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:02.386894+0000) 2024-02-20T09:25:02.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:25:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:25:02.231+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:25:02.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:25:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:25:02.231+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:25:02.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:25:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:25:02.231+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:25:03.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:25:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:25:03.073+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:03.073579+0000) 2024-02-20T09:25:03.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:25:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:25:02.958+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:02.958509+0000) 2024-02-20T09:25:03.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:25:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:25:03.017+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:25:03.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:25:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:25:03.017+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:25:03.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:25:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:25:03.018+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:25:03.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:25:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:25:02.905+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:25:03.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:25:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:25:02.905+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:25:03.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:25:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:25:02.905+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:25:03.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:25:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:25:02.906+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:25:03.603 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:25:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:25:03.388+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:03.389011+0000) 2024-02-20T09:25:03.603 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:25:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:25:03.253+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:25:03.604 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:25:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:25:03.253+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:25:03.604 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:25:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:25:03.253+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:25:04.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:25:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:25:04.073+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:04.073676+0000) 2024-02-20T09:25:04.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:25:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:25:03.922+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:25:04.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:25:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:25:03.922+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:25:04.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:25:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:25:03.922+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:25:04.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:25:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:25:03.922+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:25:04.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:25:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:25:03.958+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:03.958609+0000) 2024-02-20T09:25:04.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:25:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:25:04.025+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:25:04.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:25:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:25:04.025+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:25:04.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:25:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:25:04.025+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:25:04.604 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:25:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:25:04.390+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:04.391142+0000) 2024-02-20T09:25:04.604 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:25:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:25:04.274+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:25:04.604 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:25:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:25:04.274+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:25:04.604 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:25:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:25:04.274+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:25:05.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:25:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:25:05.073+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:05.073771+0000) 2024-02-20T09:25:05.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:25:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:25:04.958+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:04.958718+0000) 2024-02-20T09:25:05.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:25:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:25:05.047+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:25:05.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:25:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:25:05.047+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:25:05.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:25:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:25:05.047+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:25:05.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:25:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:25:04.965+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:25:05.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:25:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:25:04.965+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:25:05.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:25:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:25:04.965+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:25:05.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:25:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:25:04.966+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:25:05.603 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:25:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:25:05.392+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:05.393336+0000) 2024-02-20T09:25:05.604 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:25:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:25:05.312+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:25:05.604 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:25:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:25:05.312+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:25:05.604 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:25:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:25:05.312+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:25:06.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:25:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:25:06.073+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:06.073880+0000) 2024-02-20T09:25:06.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:25:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:25:05.998+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:25:06.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:25:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:25:05.998+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:25:06.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:25:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:25:05.998+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:25:06.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:25:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:25:05.998+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:25:06.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:25:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:25:05.958+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:05.958851+0000) 2024-02-20T09:25:06.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:25:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:25:06.006+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:25:06.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:25:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:25:06.006+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:25:06.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:25:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:25:06.006+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:25:06.603 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:25:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:25:06.395+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:06.395496+0000) 2024-02-20T09:25:06.604 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:25:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:25:06.316+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:25:06.604 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:25:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:25:06.316+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:25:06.604 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:25:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:25:06.316+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:25:07.242 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:25:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:25:06.909+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 69020 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:25:07.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:25:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:25:06.902+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:06.903055+0000) 2024-02-20T09:25:07.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:25:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:25:07.073+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:07.074057+0000) 2024-02-20T09:25:07.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:25:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:25:06.906+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:06.907090+0000) 2024-02-20T09:25:07.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:25:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:25:07.005+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:25:07.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:25:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:25:07.005+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:25:07.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:25:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:25:07.005+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:25:07.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:25:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:25:07.005+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:25:07.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:25:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:25:06.902+0000 7f8a5f86e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:06.903108+0000) 2024-02-20T09:25:07.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:25:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:25:06.958+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:06.958989+0000) 2024-02-20T09:25:07.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:25:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:25:06.965+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:25:07.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:25:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:25:06.965+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:25:07.245 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:25:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:25:06.965+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:25:07.604 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:25:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:25:07.395+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:07.396159+0000) 2024-02-20T09:25:07.604 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:25:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:25:07.351+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:25:07.604 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:25:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:25:07.351+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:25:07.604 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:25:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:25:07.351+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:25:08.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:25:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:25:08.073+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:08.074190+0000) 2024-02-20T09:25:08.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:25:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:25:07.925+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:25:08.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:25:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:25:07.925+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:25:08.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:25:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:25:07.925+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:25:08.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:25:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:25:07.958+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:07.959089+0000) 2024-02-20T09:25:08.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:25:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:25:07.976+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:25:08.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:25:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:25:07.976+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:25:08.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:25:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:25:07.976+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:25:08.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:25:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:25:07.976+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:25:08.604 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:25:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:25:08.397+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:08.398347+0000) 2024-02-20T09:25:08.604 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:25:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:25:08.305+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:25:08.604 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:25:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:25:08.305+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:25:08.604 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:25:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:25:08.305+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:25:09.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:25:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:25:09.074+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:09.074438+0000) 2024-02-20T09:25:09.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:25:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:25:08.944+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:25:09.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:25:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:25:08.944+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:25:09.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:25:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:25:08.944+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:25:09.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:25:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:25:08.958+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:08.959181+0000) 2024-02-20T09:25:09.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:25:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:25:08.988+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:25:09.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:25:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:25:08.988+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:25:09.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:25:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:25:08.988+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:25:09.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:25:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:25:08.988+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:25:09.604 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:25:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:25:09.399+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:09.400337+0000) 2024-02-20T09:25:09.604 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:25:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:25:09.344+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:25:09.604 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:25:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:25:09.344+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:25:09.605 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:25:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:25:09.344+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:25:10.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:25:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:25:10.074+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:10.074542+0000) 2024-02-20T09:25:10.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:25:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:25:09.958+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:09.959336+0000) 2024-02-20T09:25:10.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:25:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:25:09.960+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:25:10.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:25:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:25:09.960+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:25:10.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:25:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:25:09.960+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:25:10.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:25:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:25:10.005+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:25:10.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:25:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:25:10.005+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:25:10.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:25:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:25:10.005+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:25:10.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:25:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:25:10.005+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:25:10.604 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:25:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:25:10.402+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:10.402420+0000) 2024-02-20T09:25:10.605 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:25:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:25:10.296+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:25:10.605 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:25:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:25:10.296+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:25:10.605 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:25:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:25:10.296+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:25:11.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:25:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:25:11.074+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:11.074635+0000) 2024-02-20T09:25:11.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:25:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:25:11.004+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:25:11.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:25:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:25:11.004+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:25:11.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:25:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:25:11.004+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:25:11.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:25:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:25:11.004+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:25:11.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:25:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:25:10.959+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:10.959523+0000) 2024-02-20T09:25:11.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:25:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:25:10.984+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:25:11.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:25:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:25:10.984+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:25:11.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:25:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:25:10.984+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:25:11.604 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:25:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:25:11.404+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:11.404993+0000) 2024-02-20T09:25:11.605 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:25:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:25:11.272+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:25:11.605 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:25:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:25:11.272+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:25:11.605 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:25:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:25:11.272+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:25:12.242 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:25:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:25:11.917+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 69059 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:25:12.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:25:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:25:11.910+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:11.910457+0000) 2024-02-20T09:25:12.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:25:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:25:12.074+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:12.074749+0000) 2024-02-20T09:25:12.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:25:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:25:11.912+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:11.913074+0000) 2024-02-20T09:25:12.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:25:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:25:12.030+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:25:12.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:25:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:25:12.030+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:25:12.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:25:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:25:12.030+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:25:12.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:25:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:25:12.030+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:25:12.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:25:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:25:11.910+0000 7f8a5f86e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:11.910390+0000) 2024-02-20T09:25:12.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:25:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:25:11.959+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:11.959652+0000) 2024-02-20T09:25:12.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:25:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:25:11.966+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:25:12.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:25:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:25:11.966+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:25:12.245 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:25:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:25:11.966+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:25:12.604 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:25:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:25:12.404+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:12.405709+0000) 2024-02-20T09:25:12.605 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:25:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:25:12.251+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:25:12.605 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:25:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:25:12.251+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:25:12.605 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:25:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:25:12.251+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:25:13.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:25:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:25:13.073+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:13.074840+0000) 2024-02-20T09:25:13.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:25:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:25:12.958+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:12.959743+0000) 2024-02-20T09:25:13.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:25:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:25:13.008+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:25:13.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:25:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:25:13.008+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:25:13.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:25:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:25:13.008+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:25:13.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:25:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:25:13.015+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:25:13.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:25:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:25:13.015+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:25:13.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:25:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:25:13.015+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:25:13.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:25:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:25:13.015+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:25:13.604 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:25:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:25:13.406+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:13.407671+0000) 2024-02-20T09:25:13.605 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:25:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:25:13.265+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:25:13.605 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:25:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:25:13.265+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:25:13.605 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:25:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:25:13.265+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:25:14.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:25:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:25:14.073+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:14.074953+0000) 2024-02-20T09:25:14.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:25:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:25:13.958+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:13.959862+0000) 2024-02-20T09:25:14.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:25:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:25:14.003+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:25:14.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:25:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:25:14.003+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:25:14.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:25:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:25:14.003+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:25:14.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:25:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:25:14.027+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:25:14.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:25:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:25:14.027+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:25:14.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:25:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:25:14.027+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:25:14.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:25:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:25:14.027+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:25:14.604 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:25:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:25:14.408+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:14.409697+0000) 2024-02-20T09:25:14.605 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:25:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:25:14.292+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:25:14.605 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:25:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:25:14.292+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:25:14.605 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:25:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:25:14.292+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:25:15.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:25:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:25:15.073+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:15.075067+0000) 2024-02-20T09:25:15.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:25:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:25:14.958+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:14.959959+0000) 2024-02-20T09:25:15.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:25:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:25:15.001+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:25:15.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:25:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:25:15.001+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:25:15.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:25:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:25:15.001+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:25:15.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:25:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:25:14.979+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:25:15.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:25:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:25:14.979+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:25:15.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:25:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:25:14.979+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:25:15.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:25:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:25:14.979+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:25:15.605 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:25:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:25:15.411+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:15.413284+0000) 2024-02-20T09:25:15.605 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:25:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:25:15.340+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:25:15.605 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:25:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:25:15.340+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:25:15.605 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:25:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:25:15.340+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:25:16.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:25:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:25:16.073+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:16.075197+0000) 2024-02-20T09:25:16.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:25:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:25:15.969+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:25:16.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:25:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:25:15.969+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:25:16.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:25:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:25:15.969+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:25:16.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:25:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:25:15.969+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:25:16.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:25:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:25:15.958+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:15.960101+0000) 2024-02-20T09:25:16.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:25:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:25:16.041+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:25:16.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:25:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:25:16.041+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:25:16.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:25:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:25:16.041+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:25:16.605 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:25:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:25:16.415+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:16.416897+0000) 2024-02-20T09:25:16.605 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:25:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:25:16.345+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:25:16.605 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:25:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:25:16.345+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:25:16.606 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:25:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:25:16.345+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:25:17.242 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:25:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:25:16.923+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 69099 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:25:17.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:25:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:25:16.917+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:16.918446+0000) 2024-02-20T09:25:17.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:25:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:25:17.074+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:17.075384+0000) 2024-02-20T09:25:17.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:25:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:25:16.920+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:16.921740+0000) 2024-02-20T09:25:17.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:25:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:25:16.998+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:25:17.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:25:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:25:16.998+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:25:17.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:25:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:25:16.998+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:25:17.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:25:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:25:16.998+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:25:17.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:25:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:25:16.917+0000 7f8a5f86e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:16.918834+0000) 2024-02-20T09:25:17.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:25:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:25:16.958+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:16.960222+0000) 2024-02-20T09:25:17.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:25:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:25:17.086+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:25:17.245 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:25:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:25:17.086+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:25:17.245 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:25:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:25:17.086+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:25:17.605 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:25:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:25:17.415+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:17.417358+0000) 2024-02-20T09:25:17.605 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:25:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:25:17.342+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:25:17.606 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:25:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:25:17.342+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:25:17.606 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:25:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:25:17.342+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:25:18.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:25:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:25:18.074+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:18.075539+0000) 2024-02-20T09:25:18.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:25:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:25:18.017+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:25:18.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:25:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:25:18.017+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:25:18.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:25:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:25:18.017+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:25:18.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:25:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:25:18.017+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:25:18.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:25:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:25:17.958+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:17.960301+0000) 2024-02-20T09:25:18.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:25:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:25:18.102+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:25:18.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:25:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:25:18.102+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:25:18.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:25:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:25:18.102+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:25:18.605 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:25:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:25:18.418+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:18.419458+0000) 2024-02-20T09:25:18.605 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:25:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:25:18.391+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:25:18.606 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:25:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:25:18.391+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:25:18.606 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:25:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:25:18.391+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:25:19.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:25:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:25:19.074+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:19.075696+0000) 2024-02-20T09:25:19.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:25:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:25:19.016+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:25:19.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:25:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:25:19.016+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:25:19.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:25:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:25:19.016+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:25:19.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:25:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:25:19.017+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:25:19.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:25:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:25:18.959+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:18.960392+0000) 2024-02-20T09:25:19.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:25:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:25:19.080+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:25:19.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:25:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:25:19.080+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:25:19.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:25:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:25:19.080+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:25:19.605 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:25:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:25:19.420+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:19.422051+0000) 2024-02-20T09:25:19.605 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:25:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:25:19.357+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:25:19.606 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:25:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:25:19.357+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:25:19.606 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:25:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:25:19.357+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:25:20.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:25:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:25:20.074+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:20.075833+0000) 2024-02-20T09:25:20.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:25:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:25:20.020+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:25:20.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:25:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:25:20.020+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:25:20.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:25:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:25:20.020+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:25:20.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:25:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:25:20.020+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:25:20.245 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:25:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:25:19.959+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:19.960574+0000) 2024-02-20T09:25:20.245 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:25:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:25:20.068+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:25:20.245 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:25:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:25:20.068+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:25:20.246 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:25:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:25:20.068+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:25:20.605 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:25:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:25:20.423+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:20.424634+0000) 2024-02-20T09:25:20.606 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:25:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:25:20.332+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:25:20.606 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:25:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:25:20.332+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:25:20.606 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:25:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:25:20.332+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:25:21.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:25:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:25:21.074+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:21.075933+0000) 2024-02-20T09:25:21.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:25:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:25:20.959+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:20.960671+0000) 2024-02-20T09:25:21.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:25:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:25:21.028+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:25:21.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:25:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:25:21.028+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:25:21.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:25:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:25:21.028+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:25:21.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:25:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:25:21.006+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:25:21.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:25:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:25:21.006+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:25:21.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:25:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:25:21.006+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:25:21.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:25:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:25:21.006+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:25:21.605 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:25:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:25:21.425+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:21.426639+0000) 2024-02-20T09:25:21.606 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:25:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:25:21.333+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:25:21.606 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:25:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:25:21.333+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:25:21.606 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:25:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:25:21.333+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:25:22.242 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:25:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:25:21.931+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 69139 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:25:22.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:25:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:25:21.925+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:21.926855+0000) 2024-02-20T09:25:22.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:25:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:25:22.074+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:22.076073+0000) 2024-02-20T09:25:22.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:25:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:25:21.925+0000 7f8a5f86e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:21.926567+0000) 2024-02-20T09:25:22.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:25:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:25:21.959+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:21.960780+0000) 2024-02-20T09:25:22.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:25:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:25:21.986+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:25:22.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:25:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:25:21.986+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:25:22.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:25:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:25:21.986+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:25:22.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:25:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:25:21.928+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:21.930067+0000) 2024-02-20T09:25:22.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:25:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:25:22.017+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:25:22.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:25:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:25:22.017+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:25:22.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:25:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:25:22.017+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:25:22.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:25:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:25:22.017+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:25:22.605 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:25:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:25:22.425+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:22.427313+0000) 2024-02-20T09:25:22.606 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:25:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:25:22.367+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:25:22.606 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:25:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:25:22.367+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:25:22.606 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:25:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:25:22.367+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:25:23.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:25:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:25:23.074+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:23.076170+0000) 2024-02-20T09:25:23.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:25:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:25:22.959+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:25:23.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:25:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:25:22.959+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:25:23.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:25:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:25:22.959+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:25:23.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:25:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:25:22.959+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:22.960894+0000) 2024-02-20T09:25:23.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:25:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:25:23.044+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:25:23.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:25:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:25:23.044+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:25:23.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:25:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:25:23.044+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:25:23.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:25:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:25:23.044+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:25:23.605 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:25:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:25:23.427+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:23.429259+0000) 2024-02-20T09:25:23.606 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:25:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:25:23.332+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:25:23.606 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:25:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:25:23.332+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:25:23.606 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:25:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:25:23.332+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:25:24.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:25:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:25:24.074+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:24.076296+0000) 2024-02-20T09:25:24.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:25:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:25:23.995+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:25:24.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:25:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:25:23.995+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:25:24.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:25:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:25:23.995+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:25:24.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:25:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:25:23.995+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:25:24.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:25:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:25:23.959+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:23.961013+0000) 2024-02-20T09:25:24.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:25:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:25:24.006+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:25:24.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:25:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:25:24.006+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:25:24.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:25:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:25:24.006+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:25:24.606 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:25:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:25:24.429+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:24.431321+0000) 2024-02-20T09:25:24.606 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:25:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:25:24.325+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:25:24.606 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:25:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:25:24.325+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:25:24.606 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:25:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:25:24.325+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:25:25.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:25:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:25:25.075+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:25.076445+0000) 2024-02-20T09:25:25.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:25:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:25:24.980+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:25:25.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:25:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:25:24.980+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:25:25.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:25:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:25:24.980+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:25:25.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:25:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:25:24.980+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:25:25.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:25:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:25:24.959+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:24.961144+0000) 2024-02-20T09:25:25.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:25:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:25:24.962+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:25:25.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:25:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:25:24.962+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:25:25.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:25:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:25:24.962+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:25:25.606 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:25:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:25:25.432+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:25.433485+0000) 2024-02-20T09:25:25.606 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:25:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:25:25.297+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:25:25.606 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:25:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:25:25.297+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:25:25.606 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:25:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:25:25.297+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:25:26.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:25:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:25:26.075+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:26.076561+0000) 2024-02-20T09:25:26.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:25:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:25:25.997+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:25:26.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:25:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:25:25.997+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:25:26.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:25:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:25:25.997+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:25:26.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:25:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:25:25.997+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:25:26.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:25:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:25:25.958+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:25:26.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:25:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:25:25.958+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:25:26.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:25:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:25:25.958+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:25:26.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:25:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:25:25.959+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:25.961272+0000) 2024-02-20T09:25:26.606 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:25:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:25:26.435+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:26.437077+0000) 2024-02-20T09:25:26.607 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:25:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:25:26.328+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:25:26.607 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:25:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:25:26.328+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:25:26.607 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:25:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:25:26.328+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:25:27.242 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:25:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:25:26.939+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 69179 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:25:27.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:25:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:25:26.932+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:26.934052+0000) 2024-02-20T09:25:27.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:25:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:25:27.075+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:27.076672+0000) 2024-02-20T09:25:27.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:25:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:25:26.932+0000 7f8a5f86e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:26.934152+0000) 2024-02-20T09:25:27.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:25:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:25:26.938+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:25:27.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:25:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:25:26.938+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:25:27.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:25:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:25:26.938+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:25:27.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:25:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:25:26.959+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:26.961381+0000) 2024-02-20T09:25:27.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:25:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:25:26.936+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:26.938100+0000) 2024-02-20T09:25:27.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:25:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:25:27.034+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:25:27.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:25:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:25:27.034+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:25:27.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:25:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:25:27.034+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:25:27.245 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:25:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:25:27.034+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:25:27.606 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:25:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:25:27.436+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:27.437556+0000) 2024-02-20T09:25:27.607 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:25:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:25:27.329+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:25:27.607 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:25:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:25:27.329+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:25:27.607 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:25:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:25:27.329+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:25:28.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:25:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:25:28.075+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:28.076795+0000) 2024-02-20T09:25:28.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:25:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:25:27.917+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:25:28.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:25:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:25:27.917+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:25:28.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:25:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:25:27.917+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:25:28.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:25:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:25:27.960+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:27.961539+0000) 2024-02-20T09:25:28.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:25:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:25:28.018+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:25:28.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:25:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:25:28.018+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:25:28.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:25:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:25:28.018+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:25:28.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:25:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:25:28.018+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:25:28.707 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:25:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:25:28.438+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:28.440158+0000) 2024-02-20T09:25:28.708 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:25:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:25:28.379+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:25:28.708 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:25:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:25:28.379+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:25:28.708 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:25:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:25:28.379+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:25:28.963 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:25:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:25:28.880+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:25:28.963 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:25:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:25:28.880+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:25:28.963 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:25:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:25:28.880+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:25:29.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:25:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:25:28.960+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:28.961685+0000) 2024-02-20T09:25:29.245 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:25:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:25:29.075+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:29.076943+0000) 2024-02-20T09:25:29.245 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:25:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:25:29.049+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:25:29.245 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:25:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:25:29.049+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:25:29.246 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:25:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:25:29.049+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:25:29.246 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:25:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:25:29.050+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:25:29.707 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:25:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:25:29.440+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:29.442221+0000) 2024-02-20T09:25:29.708 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:25:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:25:29.421+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:25:29.708 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:25:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:25:29.421+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:25:29.708 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:25:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:25:29.421+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:25:29.963 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:25:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:25:29.866+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:25:29.963 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:25:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:25:29.866+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:25:29.964 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:25:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:25:29.866+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:25:30.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:25:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:25:29.960+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:29.961826+0000) 2024-02-20T09:25:30.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:25:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:25:30.075+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:30.077060+0000) 2024-02-20T09:25:30.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:25:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:25:30.004+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:25:30.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:25:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:25:30.004+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:25:30.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:25:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:25:30.004+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:25:30.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:25:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:25:30.004+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:25:30.708 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:25:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:25:30.442+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:30.444249+0000) 2024-02-20T09:25:30.708 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:25:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:25:30.468+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:25:30.708 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:25:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:25:30.468+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:25:30.709 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:25:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:25:30.468+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:25:30.963 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:25:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:25:30.854+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:25:30.963 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:25:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:25:30.854+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:25:30.963 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:25:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:25:30.854+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:25:31.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:25:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:25:30.960+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:30.961926+0000) 2024-02-20T09:25:31.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:25:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:25:31.075+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:31.077161+0000) 2024-02-20T09:25:31.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:25:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:25:30.992+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:25:31.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:25:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:25:30.992+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:25:31.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:25:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:25:30.992+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:25:31.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:25:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:25:30.993+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:25:31.708 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:25:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:25:31.444+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:31.446309+0000) 2024-02-20T09:25:31.708 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:25:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:25:31.470+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:25:31.709 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:25:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:25:31.470+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:25:31.709 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:25:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:25:31.470+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:25:31.963 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:25:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:25:31.947+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 69218 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:25:31.963 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:25:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:25:31.944+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:31.945635+0000) 2024-02-20T09:25:31.964 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:25:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:25:31.941+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:31.942552+0000) 2024-02-20T09:25:31.964 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:25:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:25:31.831+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:25:31.964 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:25:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:25:31.831+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:25:31.965 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:25:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:25:31.831+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:25:31.965 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:25:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:25:31.940+0000 7f8a5f86e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:31.942186+0000) 2024-02-20T09:25:32.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:25:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:25:32.075+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:32.077321+0000) 2024-02-20T09:25:32.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:25:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:25:31.960+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:31.962028+0000) 2024-02-20T09:25:32.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:25:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:25:32.005+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:25:32.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:25:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:25:32.005+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:25:32.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:25:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:25:32.005+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:25:32.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:25:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:25:32.005+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:25:32.708 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:25:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:25:32.445+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:32.446845+0000) 2024-02-20T09:25:32.709 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:25:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:25:32.460+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:25:32.709 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:25:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:25:32.460+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:25:32.709 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:25:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:25:32.460+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:25:32.963 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:25:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:25:32.882+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:25:32.963 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:25:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:25:32.882+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:25:32.963 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:25:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:25:32.882+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:25:33.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:25:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:25:32.960+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:32.962148+0000) 2024-02-20T09:25:33.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:25:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:25:33.076+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:33.077421+0000) 2024-02-20T09:25:33.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:25:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:25:32.981+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:25:33.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:25:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:25:32.981+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:25:33.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:25:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:25:32.981+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:25:33.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:25:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:25:32.981+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:25:33.708 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:25:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:25:33.447+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:33.448986+0000) 2024-02-20T09:25:33.709 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:25:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:25:33.490+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:25:33.709 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:25:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:25:33.490+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:25:33.709 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:25:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:25:33.490+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:25:33.963 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:25:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:25:33.857+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:25:33.963 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:25:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:25:33.857+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:25:33.963 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:25:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:25:33.857+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:25:34.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:25:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:25:33.960+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:33.962235+0000) 2024-02-20T09:25:34.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:25:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:25:34.076+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:34.077538+0000) 2024-02-20T09:25:34.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:25:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:25:34.003+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:25:34.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:25:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:25:34.003+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:25:34.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:25:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:25:34.003+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:25:34.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:25:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:25:34.003+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:25:34.708 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:25:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:25:34.450+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:34.451539+0000) 2024-02-20T09:25:34.709 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:25:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:25:34.495+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:25:34.709 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:25:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:25:34.495+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:25:34.709 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:25:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:25:34.495+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:25:34.963 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:25:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:25:34.816+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:25:34.964 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:25:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:25:34.816+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:25:34.964 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:25:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:25:34.816+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:25:35.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:25:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:25:34.960+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:34.962348+0000) 2024-02-20T09:25:35.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:25:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:25:35.076+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:35.077686+0000) 2024-02-20T09:25:35.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:25:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:25:35.053+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:25:35.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:25:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:25:35.053+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:25:35.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:25:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:25:35.053+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:25:35.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:25:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:25:35.053+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:25:35.708 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:25:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:25:35.452+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:35.453622+0000) 2024-02-20T09:25:35.709 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:25:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:25:35.465+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:25:35.709 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:25:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:25:35.465+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:25:35.709 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:25:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:25:35.465+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:25:35.963 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:25:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:25:35.855+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:25:35.963 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:25:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:25:35.855+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:25:35.963 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:25:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:25:35.855+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:25:36.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:25:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:25:35.961+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:35.962415+0000) 2024-02-20T09:25:36.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:25:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:25:36.076+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:36.077804+0000) 2024-02-20T09:25:36.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:25:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:25:36.077+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:25:36.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:25:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:25:36.077+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:25:36.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:25:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:25:36.077+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:25:36.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:25:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:25:36.077+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:25:36.607 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:25:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:25:36.454+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:36.455692+0000) 2024-02-20T09:25:36.607 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:25:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:25:36.437+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:25:36.608 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:25:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:25:36.437+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:25:36.608 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:25:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:25:36.437+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:25:36.951 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:25:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:25:36.844+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:25:36.951 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:25:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:25:36.844+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:25:36.951 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:25:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:25:36.844+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:25:36.951 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:25:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:25:36.948+0000 7f8a5f86e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:36.949849+0000) 2024-02-20T09:25:37.243 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:25:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:25:36.956+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 69258 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:25:37.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:25:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:25:36.961+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:36.962512+0000) 2024-02-20T09:25:37.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:25:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:25:36.948+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:36.949991+0000) 2024-02-20T09:25:37.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:25:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:25:37.076+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:37.077895+0000) 2024-02-20T09:25:37.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:25:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:25:36.951+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:36.953230+0000) 2024-02-20T09:25:37.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:25:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:25:37.124+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:25:37.245 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:25:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:25:37.124+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:25:37.245 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:25:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:25:37.124+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:25:37.245 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:25:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:25:37.124+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:25:37.709 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:25:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:25:37.454+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:37.456365+0000) 2024-02-20T09:25:37.709 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:25:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:25:37.463+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:25:37.709 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:25:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:25:37.463+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:25:37.709 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:25:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:25:37.463+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:25:37.963 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:25:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:25:37.854+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:25:37.964 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:25:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:25:37.854+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:25:37.964 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:25:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:25:37.854+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:25:38.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:25:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:25:37.961+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:37.962593+0000) 2024-02-20T09:25:38.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:25:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:25:38.076+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:38.077975+0000) 2024-02-20T09:25:38.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:25:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:25:38.127+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:25:38.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:25:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:25:38.127+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:25:38.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:25:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:25:38.127+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:25:38.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:25:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:25:38.127+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:25:38.709 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:25:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:25:38.458+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:38.459775+0000) 2024-02-20T09:25:38.709 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:25:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:25:38.427+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:25:38.709 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:25:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:25:38.427+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:25:38.709 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:25:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:25:38.427+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:25:38.964 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:25:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:25:38.837+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:25:38.964 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:25:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:25:38.837+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:25:38.964 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:25:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:25:38.837+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:25:39.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:25:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:25:38.961+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:38.962681+0000) 2024-02-20T09:25:39.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:25:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:25:39.076+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:39.078123+0000) 2024-02-20T09:25:39.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:25:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:25:39.173+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:25:39.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:25:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:25:39.173+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:25:39.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:25:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:25:39.173+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:25:39.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:25:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:25:39.174+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:25:39.709 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:25:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:25:39.461+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:39.463227+0000) 2024-02-20T09:25:39.709 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:25:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:25:39.477+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:25:39.709 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:25:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:25:39.477+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:25:39.709 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:25:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:25:39.477+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:25:39.964 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:25:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:25:39.799+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:25:39.964 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:25:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:25:39.799+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:25:39.964 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:25:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:25:39.799+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:25:40.214 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:25:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:25:39.961+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:39.962790+0000) 2024-02-20T09:25:40.214 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:25:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:25:40.076+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:40.078241+0000) 2024-02-20T09:25:40.214 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:25:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:25:40.211+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:25:40.215 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:25:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:25:40.211+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:25:40.215 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:25:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:25:40.211+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:25:40.215 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:25:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:25:40.211+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:25:40.468 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:25:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:25:40.457+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:25:40.468 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:25:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:25:40.457+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:25:40.468 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:25:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:25:40.457+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:25:40.728 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:25:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:25:40.465+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:40.466722+0000) 2024-02-20T09:25:40.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:25:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:25:40.840+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:25:40.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:25:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:25:40.840+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:25:40.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:25:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:25:40.840+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:25:40.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:25:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:25:40.961+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:40.962907+0000) 2024-02-20T09:25:41.371 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:25:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:25:41.076+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:41.078329+0000) 2024-02-20T09:25:41.372 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:25:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:25:41.201+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:25:41.372 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:25:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:25:41.201+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:25:41.372 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:25:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:25:41.201+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:25:41.372 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:25:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:25:41.201+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:25:41.709 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:25:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:25:41.468+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:41.470138+0000) 2024-02-20T09:25:41.710 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:25:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:25:41.417+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:25:41.710 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:25:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:25:41.417+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:25:41.710 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:25:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:25:41.418+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:25:41.959 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:25:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:25:41.841+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:25:41.960 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:25:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:25:41.841+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:25:41.960 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:25:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:25:41.841+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:25:42.242 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:25:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:25:41.963+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 69297 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:25:42.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:25:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:25:41.958+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:41.959685+0000) 2024-02-20T09:25:42.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:25:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:25:42.077+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:42.078494+0000) 2024-02-20T09:25:42.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:25:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:25:41.957+0000 7f8a5f86e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:41.958935+0000) 2024-02-20T09:25:42.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:25:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:25:41.961+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:41.963008+0000) 2024-02-20T09:25:42.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:25:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:25:41.961+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:41.962630+0000) 2024-02-20T09:25:42.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:25:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:25:42.173+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:25:42.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:25:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:25:42.173+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:25:42.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:25:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:25:42.173+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:25:42.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:25:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:25:42.173+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:25:42.709 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:25:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:25:42.469+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:42.470912+0000) 2024-02-20T09:25:42.710 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:25:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:25:42.389+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:25:42.710 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:25:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:25:42.389+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:25:42.710 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:25:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:25:42.389+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:25:42.964 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:25:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:25:42.843+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:25:42.964 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:25:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:25:42.843+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:25:42.965 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:25:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:25:42.843+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:25:43.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:25:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:25:42.962+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:42.963136+0000) 2024-02-20T09:25:43.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:25:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:25:43.078+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:43.078632+0000) 2024-02-20T09:25:43.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:25:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:25:43.187+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:25:43.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:25:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:25:43.187+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:25:43.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:25:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:25:43.187+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:25:43.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:25:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:25:43.187+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:25:43.608 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:25:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:25:43.472+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:43.472897+0000) 2024-02-20T09:25:43.608 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:25:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:25:43.352+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:25:43.609 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:25:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:25:43.353+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:25:43.609 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:25:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:25:43.353+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:25:44.151 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:25:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:25:44.078+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:44.078720+0000) 2024-02-20T09:25:44.152 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:25:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:25:43.886+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:25:44.152 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:25:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:25:43.886+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:25:44.152 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:25:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:25:43.886+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:25:44.152 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:25:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:25:43.962+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:43.963297+0000) 2024-02-20T09:25:44.476 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:25:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:25:44.150+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:25:44.476 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:25:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:25:44.150+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:25:44.476 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:25:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:25:44.150+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:25:44.477 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:25:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:25:44.150+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:25:44.477 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:25:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:25:44.309+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:25:44.477 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:25:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:25:44.309+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:25:44.477 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:25:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:25:44.309+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:25:44.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:25:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:25:44.474+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:44.474919+0000) 2024-02-20T09:25:45.196 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:25:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:25:45.078+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:45.078856+0000) 2024-02-20T09:25:45.197 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:25:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:25:44.905+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:25:45.197 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:25:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:25:44.905+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:25:45.197 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:25:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:25:44.905+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:25:45.197 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:25:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:25:44.963+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:44.963491+0000) 2024-02-20T09:25:45.478 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:25:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:25:45.266+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:25:45.479 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:25:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:25:45.266+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:25:45.479 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:25:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:25:45.266+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:25:45.479 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:25:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:25:45.194+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:25:45.479 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:25:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:25:45.194+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:25:45.479 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:25:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:25:45.194+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:25:45.479 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:25:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:25:45.194+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:25:45.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:25:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:25:45.475+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:45.477361+0000) 2024-02-20T09:25:46.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:25:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:25:46.077+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:46.078997+0000) 2024-02-20T09:25:46.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:25:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:25:45.919+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:25:46.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:25:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:25:45.919+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:25:46.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:25:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:25:45.919+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:25:46.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:25:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:25:45.963+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:45.963612+0000) 2024-02-20T09:25:46.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:25:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:25:46.150+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:25:46.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:25:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:25:46.150+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:25:46.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:25:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:25:46.150+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:25:46.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:25:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:25:46.150+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:25:46.608 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:25:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:25:46.479+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:46.479563+0000) 2024-02-20T09:25:46.609 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:25:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:25:46.317+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:25:46.609 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:25:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:25:46.317+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:25:46.609 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:25:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:25:46.317+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:25:47.181 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:25:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:25:46.972+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 69337 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:25:47.181 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:25:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:25:46.966+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:46.967141+0000) 2024-02-20T09:25:47.181 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:25:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:25:47.078+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:47.079109+0000) 2024-02-20T09:25:47.182 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:25:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:25:46.904+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:25:47.182 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:25:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:25:46.904+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:25:47.182 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:25:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:25:46.904+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:25:47.182 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:25:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:25:46.963+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:46.963703+0000) 2024-02-20T09:25:47.182 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:25:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:25:46.966+0000 7f8a5f86e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:46.967025+0000) 2024-02-20T09:25:47.183 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:25:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:25:46.970+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:46.970634+0000) 2024-02-20T09:25:47.183 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:25:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:25:47.179+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:25:47.183 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:25:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:25:47.179+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:25:47.183 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:25:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:25:47.179+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:25:47.183 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:25:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:25:47.179+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:25:47.481 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:25:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:25:47.345+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:25:47.481 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:25:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:25:47.345+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:25:47.481 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:25:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:25:47.345+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:25:47.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:25:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:25:47.479+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:47.480060+0000) 2024-02-20T09:25:48.218 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:25:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:25:48.078+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:48.079232+0000) 2024-02-20T09:25:48.219 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:25:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:25:47.903+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:25:48.219 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:25:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:25:47.903+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:25:48.219 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:25:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:25:47.903+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:25:48.219 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:25:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:25:47.963+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:47.963839+0000) 2024-02-20T09:25:48.484 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:25:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:25:48.346+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:25:48.484 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:25:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:25:48.346+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:25:48.484 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:25:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:25:48.346+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:25:48.484 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:25:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:25:48.217+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:25:48.485 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:25:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:25:48.217+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:25:48.485 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:25:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:25:48.217+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:25:48.485 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:25:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:25:48.217+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:25:48.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:25:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:25:48.481+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:48.482235+0000) 2024-02-20T09:25:49.234 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:25:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:25:49.078+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:49.079360+0000) 2024-02-20T09:25:49.235 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:25:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:25:48.942+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:25:49.235 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:25:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:25:48.942+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:25:49.235 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:25:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:25:48.942+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:25:49.235 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:25:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:25:48.963+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:48.963954+0000) 2024-02-20T09:25:49.485 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:25:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:25:49.379+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:25:49.485 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:25:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:25:49.379+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:25:49.485 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:25:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:25:49.379+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:25:49.486 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:25:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:25:49.232+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:25:49.486 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:25:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:25:49.232+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:25:49.486 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:25:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:25:49.232+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:25:49.486 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:25:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:25:49.232+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:25:49.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:25:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:25:49.483+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:49.484333+0000) 2024-02-20T09:25:50.204 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:25:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:25:50.079+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:50.079492+0000) 2024-02-20T09:25:50.205 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:25:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:25:49.962+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:25:50.205 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:25:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:25:49.962+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:25:50.205 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:25:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:25:49.962+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:25:50.205 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:25:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:25:49.963+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:49.964066+0000) 2024-02-20T09:25:50.472 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:25:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:25:50.382+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:25:50.472 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:25:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:25:50.382+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:25:50.472 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:25:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:25:50.382+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:25:50.472 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:25:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:25:50.203+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:25:50.472 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:25:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:25:50.203+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:25:50.473 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:25:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:25:50.203+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:25:50.473 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:25:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:25:50.203+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:25:50.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:25:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:25:50.486+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:50.486508+0000) 2024-02-20T09:25:51.220 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:25:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:25:51.079+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:51.079597+0000) 2024-02-20T09:25:51.221 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:25:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:25:50.946+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:25:51.221 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:25:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:25:50.946+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:25:51.221 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:25:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:25:50.946+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:25:51.221 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:25:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:25:50.963+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:50.964187+0000) 2024-02-20T09:25:51.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:25:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:25:51.365+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:25:51.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:25:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:25:51.365+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:25:51.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:25:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:25:51.365+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:25:51.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:25:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:25:51.219+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:25:51.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:25:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:25:51.219+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:25:51.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:25:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:25:51.219+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:25:51.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:25:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:25:51.219+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:25:51.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:25:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:25:51.489+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:51.490144+0000) 2024-02-20T09:25:52.227 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:25:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:25:51.980+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 69377 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:25:52.227 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:25:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:25:51.973+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:51.974366+0000) 2024-02-20T09:25:52.228 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:25:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:25:52.079+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:52.079694+0000) 2024-02-20T09:25:52.228 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:25:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:25:51.976+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:51.977166+0000) 2024-02-20T09:25:52.228 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:25:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:25:52.225+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:25:52.228 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:25:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:25:52.225+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:25:52.228 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:25:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:25:52.225+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:25:52.229 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:25:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:25:51.948+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:25:52.229 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:25:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:25:51.948+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:25:52.229 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:25:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:25:51.948+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:25:52.229 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:25:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:25:51.963+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:51.964302+0000) 2024-02-20T09:25:52.229 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:25:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:25:51.974+0000 7f8a5f86e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:51.974980+0000) 2024-02-20T09:25:52.491 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:25:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:25:52.225+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:25:52.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:25:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:25:52.368+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:25:52.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:25:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:25:52.368+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:25:52.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:25:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:25:52.368+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:25:52.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:25:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:25:52.490+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:52.490764+0000) 2024-02-20T09:25:53.203 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:25:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:25:53.079+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:53.079769+0000) 2024-02-20T09:25:53.203 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:25:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:25:52.933+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:25:53.203 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:25:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:25:52.933+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:25:53.203 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:25:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:25:52.933+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:25:53.204 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:25:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:25:52.964+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:52.964437+0000) 2024-02-20T09:25:53.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:25:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:25:53.201+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:25:53.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:25:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:25:53.201+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:25:53.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:25:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:25:53.201+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:25:53.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:25:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:25:53.201+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:25:53.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:25:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:25:53.349+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:25:53.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:25:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:25:53.349+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:25:53.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:25:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:25:53.349+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:25:53.954 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:25:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:25:53.492+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:53.492855+0000) 2024-02-20T09:25:54.240 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:25:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:25:54.079+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:54.079859+0000) 2024-02-20T09:25:54.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:25:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:25:53.957+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:25:54.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:25:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:25:53.957+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:25:54.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:25:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:25:53.957+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:25:54.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:25:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:25:53.964+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:53.964512+0000) 2024-02-20T09:25:54.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:25:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:25:54.326+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:25:54.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:25:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:25:54.326+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:25:54.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:25:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:25:54.326+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:25:54.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:25:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:25:54.238+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:25:54.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:25:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:25:54.238+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:25:54.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:25:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:25:54.238+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:25:54.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:25:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:25:54.238+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:25:54.954 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:25:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:25:54.494+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:54.494848+0000) 2024-02-20T09:25:55.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:25:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:25:55.079+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:55.079972+0000) 2024-02-20T09:25:55.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:25:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:25:54.964+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:54.964604+0000) 2024-02-20T09:25:55.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:25:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:25:54.996+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:25:55.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:25:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:25:54.996+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:25:55.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:25:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:25:54.996+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:25:55.609 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:25:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:25:55.328+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:25:55.610 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:25:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:25:55.328+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:25:55.610 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:25:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:25:55.328+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:25:55.610 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:25:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:25:55.282+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:25:55.610 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:25:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:25:55.282+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:25:55.610 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:25:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:25:55.282+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:25:55.611 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:25:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:25:55.282+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:25:55.611 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:25:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:25:55.496+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:55.496844+0000) 2024-02-20T09:25:56.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:25:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:25:56.079+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:56.080104+0000) 2024-02-20T09:25:56.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:25:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:25:55.964+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:55.964696+0000) 2024-02-20T09:25:56.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:25:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:25:56.028+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:25:56.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:25:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:25:56.028+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:25:56.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:25:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:25:56.028+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:25:56.610 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:25:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:25:56.288+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:25:56.610 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:25:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:25:56.288+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:25:56.610 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:25:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:25:56.288+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:25:56.610 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:25:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:25:56.313+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:25:56.610 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:25:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:25:56.313+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:25:56.611 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:25:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:25:56.313+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:25:56.611 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:25:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:25:56.313+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:25:56.611 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:25:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:25:56.498+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:56.498902+0000) 2024-02-20T09:25:57.242 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:25:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:25:56.987+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 69417 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:25:57.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:25:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:25:56.984+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:56.984940+0000) 2024-02-20T09:25:57.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:25:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:25:56.981+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:56.982079+0000) 2024-02-20T09:25:57.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:25:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:25:57.079+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:57.080207+0000) 2024-02-20T09:25:57.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:25:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:25:56.964+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:56.964790+0000) 2024-02-20T09:25:57.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:25:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:25:56.979+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:25:57.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:25:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:25:56.979+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:25:57.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:25:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:25:56.979+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:25:57.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:25:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:25:56.981+0000 7f8a5f86e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:56.982336+0000) 2024-02-20T09:25:57.610 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:25:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:25:57.256+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:25:57.610 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:25:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:25:57.256+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:25:57.610 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:25:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:25:57.256+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:25:57.610 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:25:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:25:57.310+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:25:57.611 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:25:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:25:57.310+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:25:57.611 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:25:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:25:57.310+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:25:57.611 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:25:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:25:57.310+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:25:57.611 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:25:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:25:57.499+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:57.499428+0000) 2024-02-20T09:25:58.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:25:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:25:58.079+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:58.080315+0000) 2024-02-20T09:25:58.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:25:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:25:57.940+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:25:58.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:25:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:25:57.940+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:25:58.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:25:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:25:57.940+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:25:58.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:25:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:25:57.964+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:57.964907+0000) 2024-02-20T09:25:58.609 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:25:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:25:58.260+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:25:58.610 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:25:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:25:58.260+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:25:58.610 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:25:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:25:58.260+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:25:58.610 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:25:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:25:58.304+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:25:58.610 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:25:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:25:58.304+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:25:58.610 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:25:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:25:58.304+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:25:58.611 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:25:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:25:58.304+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:25:58.611 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:25:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:25:58.502+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:58.502976+0000) 2024-02-20T09:25:59.233 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:25:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:25:58.960+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:25:59.233 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:25:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:25:58.960+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:25:59.233 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:25:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:25:58.960+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:25:59.233 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:25:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:25:58.964+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:58.965007+0000) 2024-02-20T09:25:59.233 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:25:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:25:59.080+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:59.080428+0000) 2024-02-20T09:25:59.234 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:25:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:25:59.231+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:25:59.234 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:25:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:25:59.231+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:25:59.234 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:25:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:25:59.231+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:25:59.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:25:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:25:59.286+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:25:59.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:25:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:25:59.286+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:25:59.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:25:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:25:59.286+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:25:59.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:25:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:25:59.286+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:25:59.885 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:25:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:25:59.504+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:59.505032+0000) 2024-02-20T09:26:00.205 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:25:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:25:59.918+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:26:00.205 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:25:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:25:59.918+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:26:00.205 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:25:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:25:59.918+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:26:00.205 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:25:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:25:59.964+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:25:59.965144+0000) 2024-02-20T09:26:00.206 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:26:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:26:00.080+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:00.080519+0000) 2024-02-20T09:26:00.206 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:26:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:26:00.203+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:26:00.206 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:26:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:26:00.203+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:26:00.206 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:26:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:26:00.203+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:26:00.472 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:26:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:26:00.267+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:26:00.472 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:26:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:26:00.267+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:26:00.473 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:26:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:26:00.267+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:26:00.473 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:26:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:26:00.267+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:26:00.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:26:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:26:00.506+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:00.507129+0000) 2024-02-20T09:26:01.220 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:26:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:26:01.080+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:01.080635+0000) 2024-02-20T09:26:01.220 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:26:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:26:01.218+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:26:01.221 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:26:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:26:00.911+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:26:01.221 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:26:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:26:00.911+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:26:01.221 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:26:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:26:00.911+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:26:01.221 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:26:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:26:00.964+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:00.965252+0000) 2024-02-20T09:26:01.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:26:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:26:01.218+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:26:01.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:26:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:26:01.218+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:26:01.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:26:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:26:01.279+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:26:01.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:26:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:26:01.279+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:26:01.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:26:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:26:01.279+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:26:01.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:26:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:26:01.279+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:26:01.907 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:26:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:26:01.508+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:01.509208+0000) 2024-02-20T09:26:02.194 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:26:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:26:01.995+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 69456 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:26:02.195 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:26:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:26:01.992+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:01.993063+0000) 2024-02-20T09:26:02.195 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:26:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:26:01.906+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:26:02.195 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:26:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:26:01.906+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:26:02.195 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:26:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:26:01.906+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:26:02.196 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:26:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:26:01.964+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:01.965344+0000) 2024-02-20T09:26:02.196 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:26:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:26:01.989+0000 7f8a5f86e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:01.989567+0000) 2024-02-20T09:26:02.196 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:26:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:26:01.989+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:01.989544+0000) 2024-02-20T09:26:02.196 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:26:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:26:02.080+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:02.080742+0000) 2024-02-20T09:26:02.196 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:26:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:26:02.193+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:26:02.196 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:26:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:26:02.193+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:26:02.196 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:26:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:26:02.193+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:26:02.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:26:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:26:02.323+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:26:02.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:26:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:26:02.323+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:26:02.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:26:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:26:02.323+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:26:02.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:26:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:26:02.323+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:26:02.908 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:26:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:26:02.509+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:02.509879+0000) 2024-02-20T09:26:03.212 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:26:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:26:02.906+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:26:03.212 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:26:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:26:02.906+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:26:03.212 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:26:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:26:02.906+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:26:03.212 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:26:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:26:02.965+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:02.965486+0000) 2024-02-20T09:26:03.212 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:26:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:26:03.080+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:03.080866+0000) 2024-02-20T09:26:03.213 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:26:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:26:03.210+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:26:03.213 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:26:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:26:03.210+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:26:03.213 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:26:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:26:03.210+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:26:03.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:26:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:26:03.307+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:26:03.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:26:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:26:03.307+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:26:03.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:26:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:26:03.307+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:26:03.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:26:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:26:03.307+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:26:03.938 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:26:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:26:03.511+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:03.511895+0000) 2024-02-20T09:26:04.236 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:26:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:26:04.080+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:04.080997+0000) 2024-02-20T09:26:04.236 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:26:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:26:04.234+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:26:04.237 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:26:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:26:03.936+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:26:04.237 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:26:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:26:03.936+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:26:04.237 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:26:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:26:03.936+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:26:04.237 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:26:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:26:03.965+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:03.965561+0000) 2024-02-20T09:26:04.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:26:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:26:04.234+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:26:04.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:26:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:26:04.234+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:26:04.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:26:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:26:04.331+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:26:04.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:26:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:26:04.331+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:26:04.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:26:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:26:04.331+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:26:04.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:26:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:26:04.331+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:26:04.955 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:26:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:26:04.513+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:04.514185+0000) 2024-02-20T09:26:05.206 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:26:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:26:05.080+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:05.081105+0000) 2024-02-20T09:26:05.206 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:26:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:26:05.204+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:26:05.206 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:26:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:26:04.965+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:04.965647+0000) 2024-02-20T09:26:05.206 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:26:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:26:04.968+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:26:05.207 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:26:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:26:04.968+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:26:05.207 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:26:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:26:04.968+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:26:05.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:26:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:26:05.204+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:26:05.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:26:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:26:05.204+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:26:05.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:26:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:26:05.366+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:26:05.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:26:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:26:05.366+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:26:05.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:26:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:26:05.366+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:26:05.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:26:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:26:05.366+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:26:05.955 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:26:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:26:05.515+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:05.516380+0000) 2024-02-20T09:26:06.206 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:26:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:26:06.080+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:06.081243+0000) 2024-02-20T09:26:06.206 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:26:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:26:05.965+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:05.965739+0000) 2024-02-20T09:26:06.206 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:26:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:26:06.013+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:26:06.206 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:26:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:26:06.013+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:26:06.206 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:26:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:26:06.013+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:26:06.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:26:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:26:06.204+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:26:06.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:26:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:26:06.204+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:26:06.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:26:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:26:06.204+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:26:06.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:26:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:26:06.347+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:26:06.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:26:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:26:06.347+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:26:06.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:26:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:26:06.347+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:26:06.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:26:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:26:06.347+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:26:06.955 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:26:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:26:06.518+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:06.518442+0000) 2024-02-20T09:26:07.237 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:26:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:26:06.999+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:07.000070+0000) 2024-02-20T09:26:07.238 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:26:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:26:07.003+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 69496 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:26:07.238 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:26:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:26:06.965+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:06.965829+0000) 2024-02-20T09:26:07.238 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:26:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:26:06.967+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:26:07.238 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:26:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:26:06.967+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:26:07.239 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:26:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:26:06.967+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:26:07.239 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:26:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:26:06.996+0000 7f8a5f86e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:06.997326+0000) 2024-02-20T09:26:07.239 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:26:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:26:06.997+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:06.997627+0000) 2024-02-20T09:26:07.239 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:26:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:26:07.080+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:07.081351+0000) 2024-02-20T09:26:07.239 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:26:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:26:07.236+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:26:07.239 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:26:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:26:07.236+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:26:07.239 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:26:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:26:07.236+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:26:07.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:26:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:26:07.375+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:26:07.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:26:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:26:07.375+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:26:07.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:26:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:26:07.375+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:26:07.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:26:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:26:07.375+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:26:07.955 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:26:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:26:07.518+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:07.519269+0000) 2024-02-20T09:26:08.234 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:26:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:26:07.965+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:07.965940+0000) 2024-02-20T09:26:08.234 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:26:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:26:07.979+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:26:08.234 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:26:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:26:07.979+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:26:08.234 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:26:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:26:07.979+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:26:08.235 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:26:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:26:08.081+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:08.081457+0000) 2024-02-20T09:26:08.235 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:26:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:26:08.232+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:26:08.235 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:26:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:26:08.232+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:26:08.235 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:26:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:26:08.232+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:26:08.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:26:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:26:08.392+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:26:08.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:26:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:26:08.392+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:26:08.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:26:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:26:08.392+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:26:08.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:26:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:26:08.393+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:26:08.943 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:26:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:26:08.521+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:08.521427+0000) 2024-02-20T09:26:09.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:26:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:26:09.080+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:09.081572+0000) 2024-02-20T09:26:09.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:26:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:26:08.941+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:26:09.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:26:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:26:08.941+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:26:09.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:26:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:26:08.941+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:26:09.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:26:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:26:08.964+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:08.966057+0000) 2024-02-20T09:26:09.524 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:26:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:26:09.252+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:26:09.525 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:26:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:26:09.252+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:26:09.525 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:26:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:26:09.252+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:26:09.525 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:26:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:26:09.378+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:26:09.525 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:26:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:26:09.378+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:26:09.525 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:26:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:26:09.378+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:26:09.525 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:26:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:26:09.378+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:26:09.885 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:26:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:26:09.522+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:09.523544+0000) 2024-02-20T09:26:10.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:26:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:26:10.080+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:10.081678+0000) 2024-02-20T09:26:10.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:26:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:26:09.919+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:26:10.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:26:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:26:09.919+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:26:10.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:26:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:26:09.919+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:26:10.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:26:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:26:09.964+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:09.966184+0000) 2024-02-20T09:26:10.611 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:26:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:26:10.279+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:26:10.611 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:26:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:26:10.279+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:26:10.611 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:26:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:26:10.279+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:26:10.612 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:26:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:26:10.350+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:26:10.612 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:26:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:26:10.350+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:26:10.612 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:26:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:26:10.350+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:26:10.612 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:26:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:26:10.350+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:26:10.612 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:26:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:26:10.525+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:10.527149+0000) 2024-02-20T09:26:11.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:26:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:26:11.080+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:11.081792+0000) 2024-02-20T09:26:11.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:26:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:26:10.942+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:26:11.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:26:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:26:10.942+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:26:11.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:26:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:26:10.942+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:26:11.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:26:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:26:10.964+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:10.966315+0000) 2024-02-20T09:26:11.611 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:26:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:26:11.325+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:26:11.611 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:26:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:26:11.325+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:26:11.611 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:26:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:26:11.325+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:26:11.612 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:26:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:26:11.334+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:26:11.612 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:26:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:26:11.334+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:26:11.612 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:26:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:26:11.334+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:26:11.612 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:26:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:26:11.334+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:26:11.612 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:26:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:26:11.527+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:11.529115+0000) 2024-02-20T09:26:12.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:26:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:26:12.006+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:12.007538+0000) 2024-02-20T09:26:12.243 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:26:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:26:12.009+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 69535 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:26:12.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:26:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:26:12.003+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:12.005094+0000) 2024-02-20T09:26:12.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:26:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:26:12.080+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:12.081935+0000) 2024-02-20T09:26:12.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:26:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:26:11.962+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:26:12.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:26:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:26:11.962+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:26:12.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:26:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:26:11.962+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:26:12.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:26:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:26:11.965+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:11.966444+0000) 2024-02-20T09:26:12.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:26:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:26:12.003+0000 7f8a5f86e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:12.004689+0000) 2024-02-20T09:26:12.611 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:26:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:26:12.285+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:26:12.612 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:26:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:26:12.285+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:26:12.612 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:26:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:26:12.285+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:26:12.612 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:26:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:26:12.365+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:26:12.612 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:26:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:26:12.365+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:26:12.613 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:26:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:26:12.365+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:26:12.613 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:26:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:26:12.365+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:26:12.613 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:26:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:26:12.528+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:12.529681+0000) 2024-02-20T09:26:13.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:26:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:26:13.080+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:13.082090+0000) 2024-02-20T09:26:13.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:26:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:26:12.964+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:26:13.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:26:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:26:12.964+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:26:13.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:26:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:26:12.964+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:26:13.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:26:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:26:12.965+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:12.966548+0000) 2024-02-20T09:26:13.611 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:26:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:26:13.281+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:26:13.612 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:26:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:26:13.281+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:26:13.612 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:26:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:26:13.281+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:26:13.612 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:26:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:26:13.369+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:26:13.612 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:26:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:26:13.369+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:26:13.613 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:26:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:26:13.369+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:26:13.613 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:26:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:26:13.369+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:26:13.613 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:26:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:26:13.530+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:13.531653+0000) 2024-02-20T09:26:14.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:26:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:26:14.080+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:14.082237+0000) 2024-02-20T09:26:14.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:26:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:26:13.965+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:13.966630+0000) 2024-02-20T09:26:14.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:26:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:26:13.984+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:26:14.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:26:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:26:13.984+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:26:14.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:26:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:26:13.984+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:26:14.611 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:26:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:26:14.383+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:26:14.612 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:26:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:26:14.383+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:26:14.612 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:26:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:26:14.383+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:26:14.612 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:26:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:26:14.384+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:26:14.612 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:26:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:26:14.532+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:14.534114+0000) 2024-02-20T09:26:14.612 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:26:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:26:14.327+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:26:14.613 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:26:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:26:14.327+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:26:14.613 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:26:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:26:14.327+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:26:15.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:26:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:26:15.080+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:15.082362+0000) 2024-02-20T09:26:15.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:26:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:26:14.965+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:14.966745+0000) 2024-02-20T09:26:15.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:26:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:26:15.003+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:26:15.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:26:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:26:15.003+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:26:15.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:26:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:26:15.003+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:26:15.612 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:26:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:26:15.333+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:26:15.612 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:26:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:26:15.333+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:26:15.612 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:26:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:26:15.333+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:26:15.612 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:26:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:26:15.347+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:26:15.613 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:26:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:26:15.347+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:26:15.613 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:26:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:26:15.347+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:26:15.613 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:26:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:26:15.347+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:26:15.613 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:26:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:26:15.534+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:15.536296+0000) 2024-02-20T09:26:16.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:26:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:26:16.081+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:16.082446+0000) 2024-02-20T09:26:16.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:26:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:26:15.965+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:15.966862+0000) 2024-02-20T09:26:16.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:26:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:26:16.022+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:26:16.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:26:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:26:16.022+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:26:16.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:26:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:26:16.022+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:26:16.612 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:26:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:26:16.335+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:26:16.612 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:26:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:26:16.335+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:26:16.612 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:26:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:26:16.335+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:26:16.612 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:26:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:26:16.363+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:26:16.612 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:26:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:26:16.363+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:26:16.613 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:26:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:26:16.363+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:26:16.613 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:26:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:26:16.363+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:26:16.613 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:26:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:26:16.537+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:16.539069+0000) 2024-02-20T09:26:17.242 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:26:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:26:17.017+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 69575 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:26:17.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:26:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:26:17.014+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:17.015884+0000) 2024-02-20T09:26:17.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:26:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:26:17.011+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:17.012433+0000) 2024-02-20T09:26:17.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:26:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:26:17.081+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:17.082534+0000) 2024-02-20T09:26:17.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:26:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:26:16.965+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:16.966975+0000) 2024-02-20T09:26:17.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:26:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:26:16.980+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:26:17.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:26:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:26:16.980+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:26:17.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:26:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:26:16.980+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:26:17.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:26:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:26:17.010+0000 7f8a5f86e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:17.012316+0000) 2024-02-20T09:26:17.612 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:26:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:26:17.354+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:26:17.612 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:26:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:26:17.354+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:26:17.612 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:26:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:26:17.354+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:26:17.613 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:26:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:26:17.384+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:26:17.613 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:26:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:26:17.384+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:26:17.613 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:26:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:26:17.384+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:26:17.613 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:26:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:26:17.384+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:26:17.613 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:26:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:26:17.538+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:17.539916+0000) 2024-02-20T09:26:18.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:26:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:26:18.081+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:18.082625+0000) 2024-02-20T09:26:18.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:26:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:26:17.962+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:26:18.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:26:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:26:17.962+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:26:18.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:26:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:26:17.962+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:26:18.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:26:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:26:17.965+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:17.967102+0000) 2024-02-20T09:26:18.612 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:26:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:26:18.325+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:26:18.613 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:26:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:26:18.325+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:26:18.613 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:26:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:26:18.325+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:26:18.613 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:26:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:26:18.369+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:26:18.613 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:26:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:26:18.369+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:26:18.613 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:26:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:26:18.369+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:26:18.614 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:26:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:26:18.369+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:26:18.614 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:26:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:26:18.540+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:18.542034+0000) 2024-02-20T09:26:19.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:26:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:26:19.081+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:19.082772+0000) 2024-02-20T09:26:19.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:26:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:26:18.965+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:18.967237+0000) 2024-02-20T09:26:19.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:26:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:26:18.996+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:26:19.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:26:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:26:18.996+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:26:19.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:26:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:26:18.996+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:26:19.612 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:26:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:26:19.338+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:26:19.613 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:26:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:26:19.338+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:26:19.613 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:26:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:26:19.338+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:26:19.613 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:26:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:26:19.348+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:26:19.614 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:26:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:26:19.348+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:26:19.614 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:26:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:26:19.348+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:26:19.614 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:26:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:26:19.348+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:26:19.614 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:26:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:26:19.543+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:19.544514+0000) 2024-02-20T09:26:20.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:26:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:26:20.081+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:20.082916+0000) 2024-02-20T09:26:20.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:26:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:26:19.954+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:26:20.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:26:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:26:19.954+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:26:20.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:26:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:26:19.954+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:26:20.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:26:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:26:19.965+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:19.967371+0000) 2024-02-20T09:26:20.612 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:26:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:26:20.323+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:26:20.613 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:26:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:26:20.323+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:26:20.613 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:26:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:26:20.323+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:26:20.613 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:26:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:26:20.316+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:26:20.613 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:26:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:26:20.316+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:26:20.613 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:26:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:26:20.316+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:26:20.613 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:26:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:26:20.316+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:26:20.613 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:26:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:26:20.545+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:20.546535+0000) 2024-02-20T09:26:21.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:26:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:26:21.081+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:21.083063+0000) 2024-02-20T09:26:21.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:26:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:26:20.966+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:20.967584+0000) 2024-02-20T09:26:21.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:26:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:26:20.970+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:26:21.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:26:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:26:20.970+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:26:21.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:26:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:26:20.970+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:26:21.549 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:26:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:26:21.317+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:26:21.550 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:26:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:26:21.317+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:26:21.550 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:26:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:26:21.317+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:26:21.550 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:26:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:26:21.288+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:26:21.550 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:26:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:26:21.288+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:26:21.550 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:26:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:26:21.288+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:26:21.550 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:26:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:26:21.289+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:26:21.940 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:26:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:26:21.547+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:21.548532+0000) 2024-02-20T09:26:22.242 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:26:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:26:22.025+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 69614 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:26:22.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:26:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:26:22.022+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:22.023968+0000) 2024-02-20T09:26:22.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:26:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:26:22.018+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:22.020131+0000) 2024-02-20T09:26:22.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:26:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:26:22.081+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:22.083262+0000) 2024-02-20T09:26:22.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:26:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:26:21.937+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:26:22.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:26:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:26:21.937+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:26:22.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:26:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:26:21.937+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:26:22.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:26:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:26:21.966+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:21.967719+0000) 2024-02-20T09:26:22.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:26:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:26:22.019+0000 7f8a5f86e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:22.020405+0000) 2024-02-20T09:26:22.550 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:26:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:26:22.306+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:26:22.550 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:26:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:26:22.306+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:26:22.551 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:26:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:26:22.306+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:26:22.551 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:26:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:26:22.286+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:26:22.551 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:26:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:26:22.286+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:26:22.551 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:26:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:26:22.286+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:26:22.551 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:26:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:26:22.286+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:26:22.949 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:26:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:26:22.547+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:22.549306+0000) 2024-02-20T09:26:23.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:26:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:26:23.082+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:23.083447+0000) 2024-02-20T09:26:23.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:26:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:26:22.946+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:26:23.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:26:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:26:22.946+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:26:23.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:26:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:26:22.946+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:26:23.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:26:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:26:22.966+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:22.967859+0000) 2024-02-20T09:26:23.552 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:26:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:26:23.273+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:26:23.553 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:26:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:26:23.273+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:26:23.553 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:26:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:26:23.273+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:26:23.553 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:26:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:26:23.273+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:26:23.553 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:26:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:26:23.304+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:26:23.553 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:26:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:26:23.304+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:26:23.553 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:26:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:26:23.304+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:26:23.957 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:26:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:26:23.550+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:23.551459+0000) 2024-02-20T09:26:24.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:26:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:26:24.082+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:24.083525+0000) 2024-02-20T09:26:24.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:26:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:26:23.961+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:26:24.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:26:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:26:23.961+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:26:24.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:26:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:26:23.961+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:26:24.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:26:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:26:23.966+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:23.967942+0000) 2024-02-20T09:26:24.555 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:26:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:26:24.258+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:26:24.555 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:26:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:26:24.258+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:26:24.555 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:26:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:26:24.258+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:26:24.556 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:26:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:26:24.323+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:26:24.556 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:26:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:26:24.323+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:26:24.556 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:26:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:26:24.323+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:26:24.556 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:26:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:26:24.323+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:26:24.957 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:26:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:26:24.552+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:24.553984+0000) 2024-02-20T09:26:25.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:26:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:26:25.082+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:25.083637+0000) 2024-02-20T09:26:25.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:26:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:26:24.966+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:24.968060+0000) 2024-02-20T09:26:25.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:26:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:26:25.009+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:26:25.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:26:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:26:25.009+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:26:25.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:26:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:26:25.009+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:26:25.557 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:26:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:26:25.285+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:26:25.557 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:26:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:26:25.285+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:26:25.558 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:26:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:26:25.285+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:26:25.558 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:26:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:26:25.294+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:26:25.558 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:26:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:26:25.294+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:26:25.558 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:26:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:26:25.294+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:26:25.558 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:26:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:26:25.295+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:26:25.957 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:26:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:26:25.555+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:25.556462+0000) 2024-02-20T09:26:26.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:26:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:26:26.082+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:26.083797+0000) 2024-02-20T09:26:26.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:26:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:26:25.966+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:25.968179+0000) 2024-02-20T09:26:26.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:26:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:26:25.970+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:26:26.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:26:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:26:25.970+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:26:26.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:26:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:26:25.970+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:26:26.613 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:26:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:26:26.308+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:26:26.613 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:26:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:26:26.308+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:26:26.613 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:26:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:26:26.308+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:26:26.614 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:26:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:26:26.312+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:26:26.614 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:26:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:26:26.312+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:26:26.614 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:26:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:26:26.312+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:26:26.614 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:26:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:26:26.312+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:26:26.614 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:26:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:26:26.557+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:26.558627+0000) 2024-02-20T09:26:27.242 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:26:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:26:27.032+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 69654 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:26:27.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:26:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:26:27.029+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:27.031180+0000) 2024-02-20T09:26:27.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:26:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:26:27.026+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:27.027863+0000) 2024-02-20T09:26:27.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:26:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:26:27.082+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:27.083912+0000) 2024-02-20T09:26:27.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:26:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:26:26.966+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:26.968309+0000) 2024-02-20T09:26:27.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:26:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:26:26.969+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:26:27.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:26:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:26:26.969+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:26:27.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:26:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:26:26.969+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:26:27.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:26:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:26:27.026+0000 7f8a5f86e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:27.027464+0000) 2024-02-20T09:26:27.613 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:26:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:26:27.314+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:26:27.613 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:26:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:26:27.314+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:26:27.614 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:26:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:26:27.314+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:26:27.614 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:26:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:26:27.315+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:26:27.614 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:26:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:26:27.557+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:27.559176+0000) 2024-02-20T09:26:27.614 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:26:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:26:27.354+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:26:27.614 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:26:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:26:27.354+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:26:27.614 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:26:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:26:27.354+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:26:28.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:26:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:26:28.082+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:28.084009+0000) 2024-02-20T09:26:28.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:26:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:26:27.967+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:27.968458+0000) 2024-02-20T09:26:28.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:26:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:26:27.971+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:26:28.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:26:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:26:27.971+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:26:28.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:26:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:26:27.971+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:26:28.613 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:26:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:26:28.385+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:26:28.614 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:26:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:26:28.385+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:26:28.614 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:26:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:26:28.385+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:26:28.614 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:26:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:26:28.354+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:26:28.614 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:26:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:26:28.354+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:26:28.614 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:26:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:26:28.354+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:26:28.615 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:26:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:26:28.354+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:26:28.615 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:26:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:26:28.559+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:28.561291+0000) 2024-02-20T09:26:29.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:26:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:26:29.082+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:29.084100+0000) 2024-02-20T09:26:29.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:26:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:26:28.967+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:28.968541+0000) 2024-02-20T09:26:29.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:26:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:26:29.018+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:26:29.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:26:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:26:29.018+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:26:29.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:26:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:26:29.018+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:26:29.707 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:26:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:26:29.392+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:26:29.708 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:26:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:26:29.392+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:26:29.708 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:26:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:26:29.392+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:26:29.708 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:26:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:26:29.379+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:26:29.708 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:26:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:26:29.379+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:26:29.708 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:26:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:26:29.379+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:26:29.709 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:26:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:26:29.379+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:26:29.709 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:26:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:26:29.562+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:29.563558+0000) 2024-02-20T09:26:30.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:26:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:26:30.082+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:30.084208+0000) 2024-02-20T09:26:30.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:26:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:26:29.967+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:29.968636+0000) 2024-02-20T09:26:30.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:26:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:26:30.010+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:26:30.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:26:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:26:30.010+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:26:30.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:26:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:26:30.010+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:26:30.614 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:26:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:26:30.415+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:26:30.615 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:26:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:26:30.416+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:26:30.615 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:26:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:26:30.416+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:26:30.615 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:26:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:26:30.360+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:26:30.615 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:26:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:26:30.360+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:26:30.615 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:26:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:26:30.360+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:26:30.615 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:26:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:26:30.360+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:26:30.616 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:26:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:26:30.565+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:30.567168+0000) 2024-02-20T09:26:31.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:26:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:26:31.082+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:31.084296+0000) 2024-02-20T09:26:31.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:26:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:26:30.967+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:30.968756+0000) 2024-02-20T09:26:31.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:26:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:26:30.991+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:26:31.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:26:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:26:30.991+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:26:31.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:26:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:26:30.991+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:26:31.708 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:26:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:26:31.401+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:26:31.708 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:26:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:26:31.401+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:26:31.708 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:26:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:26:31.401+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:26:31.709 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:26:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:26:31.396+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:26:31.709 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:26:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:26:31.396+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:26:31.709 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:26:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:26:31.396+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:26:31.709 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:26:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:26:31.396+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:26:31.709 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:26:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:26:31.567+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:31.569242+0000) 2024-02-20T09:26:32.242 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:26:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:26:32.040+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 69693 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:26:32.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:26:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:26:32.036+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:32.038295+0000) 2024-02-20T09:26:32.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:26:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:26:32.034+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:32.035408+0000) 2024-02-20T09:26:32.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:26:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:26:32.083+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:32.084443+0000) 2024-02-20T09:26:32.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:26:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:26:31.967+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:31.968847+0000) 2024-02-20T09:26:32.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:26:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:26:31.993+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:26:32.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:26:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:26:31.993+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:26:32.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:26:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:26:31.993+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:26:32.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:26:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:26:32.033+0000 7f8a5f86e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:32.035234+0000) 2024-02-20T09:26:32.708 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:26:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:26:32.444+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:26:32.708 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:26:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:26:32.444+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:26:32.708 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:26:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:26:32.444+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:26:32.708 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:26:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:26:32.364+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:26:32.709 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:26:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:26:32.364+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:26:32.709 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:26:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:26:32.364+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:26:32.709 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:26:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:26:32.364+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:26:32.709 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:26:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:26:32.568+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:32.569800+0000) 2024-02-20T09:26:33.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:26:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:26:33.083+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:33.084526+0000) 2024-02-20T09:26:33.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:26:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:26:32.967+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:32.969005+0000) 2024-02-20T09:26:33.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:26:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:26:33.015+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:26:33.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:26:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:26:33.015+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:26:33.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:26:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:26:33.015+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:26:33.614 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:26:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:26:33.492+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:26:33.614 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:26:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:26:33.492+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:26:33.614 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:26:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:26:33.492+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:26:33.615 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:26:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:26:33.354+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:26:33.615 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:26:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:26:33.354+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:26:33.615 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:26:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:26:33.354+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:26:33.615 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:26:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:26:33.354+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:26:33.615 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:26:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:26:33.570+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:33.571742+0000) 2024-02-20T09:26:34.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:26:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:26:34.083+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:34.084639+0000) 2024-02-20T09:26:34.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:26:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:26:33.967+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:33.969084+0000) 2024-02-20T09:26:34.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:26:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:26:34.057+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:26:34.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:26:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:26:34.057+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:26:34.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:26:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:26:34.057+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:26:34.575 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:26:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:26:34.494+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:26:34.575 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:26:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:26:34.494+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:26:34.575 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:26:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:26:34.494+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:26:34.575 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:26:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:26:34.305+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:26:34.575 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:26:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:26:34.305+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:26:34.576 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:26:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:26:34.305+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:26:34.576 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:26:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:26:34.305+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:26:34.958 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:26:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:26:34.572+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:34.573741+0000) 2024-02-20T09:26:35.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:26:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:26:35.083+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:35.084733+0000) 2024-02-20T09:26:35.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:26:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:26:34.967+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:34.969181+0000) 2024-02-20T09:26:35.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:26:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:26:35.030+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:26:35.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:26:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:26:35.030+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:26:35.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:26:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:26:35.030+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:26:35.577 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:26:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:26:35.533+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:26:35.577 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:26:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:26:35.533+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:26:35.577 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:26:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:26:35.533+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:26:35.577 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:26:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:26:35.306+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:26:35.578 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:26:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:26:35.306+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:26:35.578 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:26:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:26:35.306+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:26:35.578 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:26:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:26:35.306+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:26:35.958 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:26:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:26:35.574+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:35.575931+0000) 2024-02-20T09:26:36.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:26:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:26:36.083+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:36.084862+0000) 2024-02-20T09:26:36.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:26:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:26:35.967+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:35.969294+0000) 2024-02-20T09:26:36.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:26:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:26:36.058+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:26:36.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:26:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:26:36.058+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:26:36.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:26:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:26:36.058+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:26:36.579 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:26:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:26:36.291+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:26:36.579 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:26:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:26:36.291+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:26:36.580 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:26:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:26:36.291+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:26:36.580 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:26:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:26:36.291+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:26:36.959 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:26:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:26:36.576+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:36.578346+0000) 2024-02-20T09:26:36.959 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:26:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:26:36.579+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:26:36.959 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:26:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:26:36.579+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:26:36.959 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:26:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:26:36.579+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:26:37.242 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:26:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:26:37.048+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 69733 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:26:37.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:26:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:26:37.045+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:37.046963+0000) 2024-02-20T09:26:37.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:26:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:26:37.041+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:37.043153+0000) 2024-02-20T09:26:37.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:26:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:26:37.083+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:37.084983+0000) 2024-02-20T09:26:37.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:26:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:26:36.967+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:36.969376+0000) 2024-02-20T09:26:37.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:26:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:26:37.039+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:26:37.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:26:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:26:37.039+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:26:37.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:26:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:26:37.039+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:26:37.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:26:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:26:37.041+0000 7f8a5f86e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:37.043366+0000) 2024-02-20T09:26:37.580 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:26:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:26:37.261+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:26:37.580 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:26:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:26:37.261+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:26:37.580 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:26:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:26:37.261+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:26:37.580 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:26:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:26:37.261+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:26:37.959 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:26:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:26:37.577+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:37.579033+0000) 2024-02-20T09:26:37.959 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:26:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:26:37.623+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:26:37.959 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:26:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:26:37.623+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:26:37.959 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:26:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:26:37.623+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:26:38.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:26:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:26:38.083+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:38.085090+0000) 2024-02-20T09:26:38.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:26:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:26:37.968+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:37.969442+0000) 2024-02-20T09:26:38.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:26:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:26:38.082+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:26:38.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:26:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:26:38.082+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:26:38.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:26:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:26:38.082+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:26:38.582 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:26:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:26:38.296+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:26:38.583 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:26:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:26:38.296+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:26:38.583 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:26:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:26:38.296+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:26:38.583 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:26:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:26:38.296+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:26:38.959 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:26:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:26:38.580+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:38.581556+0000) 2024-02-20T09:26:38.959 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:26:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:26:38.652+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:26:38.959 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:26:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:26:38.652+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:26:38.959 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:26:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:26:38.652+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:26:39.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:26:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:26:39.083+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:39.085207+0000) 2024-02-20T09:26:39.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:26:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:26:38.968+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:38.969515+0000) 2024-02-20T09:26:39.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:26:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:26:39.097+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:26:39.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:26:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:26:39.097+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:26:39.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:26:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:26:39.097+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:26:39.585 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:26:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:26:39.292+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:26:39.585 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:26:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:26:39.292+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:26:39.585 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:26:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:26:39.292+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:26:39.585 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:26:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:26:39.292+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:26:39.885 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:26:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:26:39.582+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:39.583571+0000) 2024-02-20T09:26:39.886 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:26:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:26:39.698+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:26:39.886 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:26:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:26:39.698+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:26:39.886 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:26:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:26:39.698+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:26:40.138 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:26:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:26:39.969+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:39.969606+0000) 2024-02-20T09:26:40.138 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:26:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:26:40.084+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:40.085300+0000) 2024-02-20T09:26:40.429 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:26:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:26:40.135+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:26:40.429 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:26:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:26:40.135+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:26:40.429 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:26:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:26:40.135+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:26:40.430 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:26:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:26:40.276+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:26:40.430 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:26:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:26:40.276+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:26:40.430 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:26:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:26:40.276+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:26:40.430 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:26:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:26:40.276+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:26:40.700 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:26:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:26:40.584+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:40.585950+0000) 2024-02-20T09:26:40.959 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:26:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:26:40.698+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:26:40.960 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:26:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:26:40.698+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:26:40.960 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:26:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:26:40.698+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:26:41.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:26:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:26:41.084+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:41.085420+0000) 2024-02-20T09:26:41.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:26:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:26:40.968+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:40.969716+0000) 2024-02-20T09:26:41.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:26:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:26:41.180+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:26:41.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:26:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:26:41.180+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:26:41.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:26:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:26:41.180+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:26:41.590 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:26:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:26:41.253+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:26:41.590 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:26:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:26:41.253+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:26:41.590 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:26:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:26:41.253+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:26:41.590 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:26:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:26:41.253+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:26:41.959 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:26:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:26:41.586+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:41.588312+0000) 2024-02-20T09:26:41.960 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:26:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:26:41.689+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:26:41.960 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:26:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:26:41.689+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:26:41.960 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:26:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:26:41.689+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:26:42.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:26:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:26:42.053+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:42.055090+0000) 2024-02-20T09:26:42.243 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:26:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:26:42.057+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 69772 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:26:42.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:26:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:26:42.049+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:42.051011+0000) 2024-02-20T09:26:42.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:26:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:26:42.084+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:42.085543+0000) 2024-02-20T09:26:42.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:26:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:26:41.969+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:41.969843+0000) 2024-02-20T09:26:42.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:26:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:26:42.049+0000 7f8a5f86e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:42.051362+0000) 2024-02-20T09:26:42.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:26:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:26:42.140+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:26:42.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:26:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:26:42.140+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:26:42.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:26:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:26:42.140+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:26:42.590 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:26:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:26:42.272+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:26:42.590 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:26:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:26:42.272+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:26:42.590 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:26:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:26:42.272+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:26:42.590 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:26:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:26:42.272+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:26:42.960 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:26:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:26:42.588+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:42.588970+0000) 2024-02-20T09:26:42.960 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:26:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:26:42.694+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:26:42.960 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:26:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:26:42.694+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:26:42.960 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:26:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:26:42.694+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:26:43.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:26:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:26:43.085+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:43.085654+0000) 2024-02-20T09:26:43.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:26:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:26:42.969+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:42.969985+0000) 2024-02-20T09:26:43.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:26:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:26:43.129+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:26:43.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:26:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:26:43.129+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:26:43.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:26:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:26:43.129+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:26:43.592 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:26:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:26:43.244+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:26:43.592 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:26:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:26:43.244+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:26:43.592 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:26:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:26:43.244+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:26:43.593 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:26:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:26:43.244+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:26:43.959 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:26:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:26:43.590+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:43.591201+0000) 2024-02-20T09:26:43.960 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:26:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:26:43.728+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:26:43.960 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:26:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:26:43.728+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:26:43.960 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:26:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:26:43.728+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:26:44.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:26:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:26:44.085+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:44.085786+0000) 2024-02-20T09:26:44.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:26:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:26:43.969+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:43.970107+0000) 2024-02-20T09:26:44.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:26:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:26:44.114+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:26:44.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:26:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:26:44.114+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:26:44.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:26:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:26:44.114+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:26:44.595 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:26:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:26:44.283+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:26:44.595 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:26:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:26:44.283+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:26:44.595 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:26:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:26:44.283+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:26:44.595 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:26:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:26:44.283+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:26:44.960 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:26:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:26:44.593+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:44.593939+0000) 2024-02-20T09:26:44.960 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:26:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:26:44.746+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:26:44.960 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:26:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:26:44.746+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:26:44.960 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:26:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:26:44.746+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:26:45.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:26:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:26:45.085+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:45.085876+0000) 2024-02-20T09:26:45.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:26:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:26:44.969+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:44.970221+0000) 2024-02-20T09:26:45.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:26:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:26:45.077+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:26:45.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:26:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:26:45.077+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:26:45.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:26:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:26:45.077+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:26:45.597 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:26:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:26:45.268+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:26:45.598 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:26:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:26:45.268+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:26:45.598 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:26:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:26:45.268+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:26:45.598 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:26:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:26:45.268+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:26:45.960 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:26:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:26:45.596+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:45.596506+0000) 2024-02-20T09:26:45.960 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:26:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:26:45.783+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:26:45.960 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:26:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:26:45.783+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:26:45.960 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:26:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:26:45.783+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:26:46.228 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:26:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:26:46.085+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:46.085963+0000) 2024-02-20T09:26:46.228 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:26:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:26:45.969+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:45.970334+0000) 2024-02-20T09:26:46.228 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:26:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:26:46.052+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:26:46.228 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:26:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:26:46.052+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:26:46.228 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:26:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:26:46.052+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:26:46.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:26:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:26:46.226+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:26:46.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:26:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:26:46.226+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:26:46.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:26:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:26:46.226+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:26:46.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:26:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:26:46.226+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:26:46.960 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:26:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:26:46.598+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:46.599040+0000) 2024-02-20T09:26:46.960 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:26:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:26:46.736+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:26:46.960 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:26:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:26:46.736+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:26:46.961 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:26:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:26:46.736+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:26:47.242 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:26:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:26:47.066+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 69812 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:26:47.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:26:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:26:47.061+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:47.061703+0000) 2024-02-20T09:26:47.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:26:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:26:47.058+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:47.059077+0000) 2024-02-20T09:26:47.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:26:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:26:47.085+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:47.086059+0000) 2024-02-20T09:26:47.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:26:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:26:46.970+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:46.970436+0000) 2024-02-20T09:26:47.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:26:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:26:47.037+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:26:47.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:26:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:26:47.037+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:26:47.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:26:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:26:47.037+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:26:47.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:26:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:26:47.059+0000 7f8a5f86e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:47.059664+0000) 2024-02-20T09:26:47.601 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:26:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:26:47.258+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:26:47.601 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:26:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:26:47.258+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:26:47.601 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:26:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:26:47.258+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:26:47.601 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:26:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:26:47.258+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:26:47.960 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:26:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:26:47.599+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:47.599783+0000) 2024-02-20T09:26:47.960 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:26:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:26:47.744+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:26:47.961 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:26:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:26:47.744+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:26:47.961 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:26:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:26:47.744+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:26:48.240 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:26:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:26:48.085+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:48.086164+0000) 2024-02-20T09:26:48.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:26:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:26:47.970+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:47.970496+0000) 2024-02-20T09:26:48.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:26:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:26:48.001+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:26:48.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:26:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:26:48.001+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:26:48.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:26:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:26:48.001+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:26:48.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:26:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:26:48.239+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:26:48.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:26:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:26:48.239+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:26:48.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:26:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:26:48.239+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:26:48.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:26:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:26:48.239+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:26:48.960 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:26:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:26:48.601+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:48.601842+0000) 2024-02-20T09:26:48.960 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:26:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:26:48.773+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:26:48.961 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:26:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:26:48.773+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:26:48.961 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:26:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:26:48.773+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:26:49.227 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:26:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:26:49.085+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:49.086296+0000) 2024-02-20T09:26:49.228 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:26:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:26:48.970+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:48.970578+0000) 2024-02-20T09:26:49.228 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:26:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:26:49.039+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:26:49.228 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:26:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:26:49.039+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:26:49.228 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:26:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:26:49.039+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:26:49.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:26:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:26:49.226+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:26:49.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:26:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:26:49.226+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:26:49.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:26:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:26:49.226+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:26:49.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:26:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:26:49.226+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:26:49.885 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:26:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:26:49.603+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:49.603830+0000) 2024-02-20T09:26:49.885 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:26:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:26:49.760+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:26:49.885 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:26:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:26:49.760+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:26:49.885 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:26:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:26:49.760+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:26:50.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:26:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:26:50.086+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:50.086449+0000) 2024-02-20T09:26:50.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:26:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:26:49.970+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:49.970674+0000) 2024-02-20T09:26:50.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:26:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:26:50.006+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:26:50.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:26:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:26:50.006+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:26:50.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:26:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:26:50.006+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:26:50.607 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:26:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:26:50.253+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:26:50.607 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:26:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:26:50.253+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:26:50.607 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:26:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:26:50.253+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:26:50.608 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:26:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:26:50.253+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:26:50.608 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:26:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:26:50.605+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:50.606035+0000) 2024-02-20T09:26:50.960 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:26:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:26:50.735+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:26:50.961 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:26:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:26:50.735+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:26:50.961 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:26:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:26:50.735+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:26:51.233 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:26:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:26:51.086+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:51.086540+0000) 2024-02-20T09:26:51.233 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:26:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:26:50.970+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:50.970797+0000) 2024-02-20T09:26:51.234 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:26:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:26:51.006+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:26:51.234 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:26:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:26:51.006+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:26:51.234 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:26:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:26:51.006+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:26:51.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:26:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:26:51.231+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:26:51.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:26:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:26:51.231+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:26:51.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:26:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:26:51.231+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:26:51.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:26:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:26:51.231+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:26:51.960 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:26:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:26:51.607+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:51.608276+0000) 2024-02-20T09:26:51.961 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:26:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:26:51.718+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:26:51.961 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:26:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:26:51.718+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:26:51.961 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:26:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:26:51.718+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:26:52.243 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:26:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:26:52.074+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 69851 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:26:52.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:26:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:26:52.070+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:52.070805+0000) 2024-02-20T09:26:52.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:26:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:26:52.067+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:52.068331+0000) 2024-02-20T09:26:52.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:26:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:26:52.086+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:52.086637+0000) 2024-02-20T09:26:52.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:26:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:26:51.970+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:51.970905+0000) 2024-02-20T09:26:52.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:26:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:26:52.003+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:26:52.245 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:26:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:26:52.003+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:26:52.245 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:26:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:26:52.003+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:26:52.245 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:26:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:26:52.067+0000 7f8a5f86e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:52.068169+0000) 2024-02-20T09:26:52.610 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:26:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:26:52.256+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:26:52.610 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:26:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:26:52.256+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:26:52.610 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:26:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:26:52.256+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:26:52.610 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:26:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:26:52.256+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:26:52.611 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:26:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:26:52.608+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:52.609061+0000) 2024-02-20T09:26:52.961 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:26:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:26:52.713+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:26:52.961 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:26:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:26:52.713+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:26:52.961 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:26:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:26:52.713+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:26:53.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:26:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:26:53.086+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:53.086718+0000) 2024-02-20T09:26:53.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:26:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:26:52.970+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:52.971024+0000) 2024-02-20T09:26:53.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:26:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:26:53.031+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:26:53.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:26:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:26:53.031+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:26:53.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:26:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:26:53.031+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:26:53.612 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:26:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:26:53.248+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:26:53.612 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:26:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:26:53.248+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:26:53.612 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:26:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:26:53.248+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:26:53.613 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:26:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:26:53.249+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:26:53.961 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:26:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:26:53.610+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:53.611353+0000) 2024-02-20T09:26:53.961 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:26:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:26:53.718+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:26:53.961 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:26:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:26:53.718+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:26:53.961 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:26:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:26:53.718+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:26:54.219 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:26:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:26:54.086+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:54.086843+0000) 2024-02-20T09:26:54.220 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:26:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:26:53.970+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:53.971139+0000) 2024-02-20T09:26:54.220 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:26:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:26:54.012+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:26:54.220 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:26:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:26:54.012+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:26:54.220 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:26:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:26:54.012+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:26:54.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:26:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:26:54.218+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:26:54.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:26:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:26:54.218+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:26:54.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:26:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:26:54.218+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:26:54.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:26:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:26:54.218+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:26:54.961 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:26:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:26:54.613+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:54.613763+0000) 2024-02-20T09:26:54.961 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:26:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:26:54.764+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:26:54.961 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:26:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:26:54.764+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:26:54.961 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:26:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:26:54.764+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:26:55.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:26:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:26:55.086+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:55.086973+0000) 2024-02-20T09:26:55.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:26:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:26:54.970+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:54.971272+0000) 2024-02-20T09:26:55.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:26:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:26:54.975+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:26:55.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:26:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:26:54.975+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:26:55.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:26:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:26:54.975+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:26:55.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:26:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:26:55.172+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:26:55.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:26:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:26:55.172+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:26:55.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:26:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:26:55.172+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:26:55.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:26:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:26:55.172+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:26:55.710 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:26:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:26:55.615+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:55.615815+0000) 2024-02-20T09:26:55.961 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:26:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:26:55.810+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:26:55.961 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:26:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:26:55.810+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:26:55.961 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:26:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:26:55.810+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:26:56.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:26:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:26:56.086+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:56.087107+0000) 2024-02-20T09:26:56.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:26:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:26:55.971+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:55.971425+0000) 2024-02-20T09:26:56.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:26:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:26:56.024+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:26:56.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:26:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:26:56.024+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:26:56.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:26:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:26:56.024+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:26:56.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:26:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:26:56.203+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:26:56.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:26:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:26:56.203+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:26:56.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:26:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:26:56.203+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:26:56.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:26:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:26:56.203+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:26:56.710 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:26:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:26:56.617+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:56.617922+0000) 2024-02-20T09:26:56.961 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:26:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:26:56.794+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:26:56.961 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:26:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:26:56.794+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:26:56.961 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:26:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:26:56.794+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:26:57.242 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:26:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:26:57.082+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 69891 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:26:57.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:26:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:26:57.075+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:57.075816+0000) 2024-02-20T09:26:57.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:26:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:26:57.086+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:57.087223+0000) 2024-02-20T09:26:57.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:26:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:26:57.077+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:57.077973+0000) 2024-02-20T09:26:57.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:26:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:26:57.179+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:26:57.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:26:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:26:57.179+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:26:57.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:26:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:26:57.179+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:26:57.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:26:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:26:57.179+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:26:57.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:26:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:26:56.971+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:56.971517+0000) 2024-02-20T09:26:57.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:26:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:26:57.053+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:26:57.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:26:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:26:57.053+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:26:57.245 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:26:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:26:57.053+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:26:57.245 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:26:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:26:57.075+0000 7f8a5f86e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:57.076091+0000) 2024-02-20T09:26:57.710 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:26:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:26:57.618+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:57.618529+0000) 2024-02-20T09:26:57.961 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:26:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:26:57.750+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:26:57.961 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:26:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:26:57.750+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:26:57.961 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:26:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:26:57.750+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:26:58.211 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:26:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:26:58.086+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:58.087308+0000) 2024-02-20T09:26:58.212 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:26:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:26:57.971+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:57.971617+0000) 2024-02-20T09:26:58.212 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:26:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:26:58.037+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:26:58.212 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:26:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:26:58.037+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:26:58.212 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:26:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:26:58.037+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:26:58.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:26:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:26:58.210+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:26:58.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:26:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:26:58.210+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:26:58.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:26:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:26:58.210+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:26:58.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:26:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:26:58.210+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:26:58.962 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:26:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:26:58.620+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:58.620575+0000) 2024-02-20T09:26:58.962 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:26:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:26:58.781+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:26:58.962 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:26:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:26:58.781+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:26:58.963 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:26:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:26:58.781+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:26:59.235 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:26:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:26:59.087+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:59.087470+0000) 2024-02-20T09:26:59.236 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:26:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:26:58.971+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:58.971725+0000) 2024-02-20T09:26:59.236 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:26:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:26:59.073+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:26:59.236 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:26:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:26:59.073+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:26:59.236 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:26:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:26:59.073+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:26:59.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:26:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:26:59.234+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:26:59.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:26:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:26:59.234+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:26:59.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:26:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:26:59.234+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:26:59.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:26:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:26:59.234+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:26:59.885 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:26:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:26:59.622+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:59.622673+0000) 2024-02-20T09:26:59.886 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:26:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:26:59.803+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:26:59.886 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:26:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:26:59.803+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:26:59.886 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:26:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:26:59.803+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:27:00.214 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:27:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:27:00.087+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:00.087608+0000) 2024-02-20T09:27:00.214 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:26:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:26:59.971+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:26:59.971841+0000) 2024-02-20T09:27:00.214 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:27:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:27:00.048+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:27:00.214 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:27:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:27:00.048+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:27:00.214 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:27:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:27:00.048+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:27:00.476 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:27:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:27:00.212+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:27:00.476 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:27:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:27:00.212+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:27:00.476 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:27:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:27:00.212+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:27:00.476 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:27:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:27:00.212+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:27:00.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:27:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:27:00.624+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:00.625187+0000) 2024-02-20T09:27:01.089 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:27:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:27:00.795+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:27:01.089 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:27:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:27:00.795+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:27:01.089 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:27:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:27:00.795+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:27:01.089 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:27:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:27:00.971+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:00.971986+0000) 2024-02-20T09:27:01.089 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:27:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:27:01.029+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:27:01.089 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:27:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:27:01.029+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:27:01.090 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:27:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:27:01.029+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:27:01.450 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:27:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:27:01.087+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:01.087783+0000) 2024-02-20T09:27:01.525 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:27:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:27:01.256+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:27:01.525 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:27:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:27:01.256+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:27:01.525 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:27:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:27:01.256+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:27:01.525 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:27:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:27:01.256+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:27:01.711 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:27:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:27:01.626+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:01.627351+0000) 2024-02-20T09:27:01.961 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:27:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:27:01.809+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:27:01.962 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:27:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:27:01.809+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:27:01.962 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:27:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:27:01.809+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:27:02.242 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:27:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:27:02.090+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 69930 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:27:02.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:27:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:27:02.085+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:02.086118+0000) 2024-02-20T09:27:02.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:27:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:27:02.084+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:02.084605+0000) 2024-02-20T09:27:02.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:27:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:27:02.087+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:02.087880+0000) 2024-02-20T09:27:02.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:27:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:27:01.971+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:01.972077+0000) 2024-02-20T09:27:02.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:27:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:27:01.991+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:27:02.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:27:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:27:01.991+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:27:02.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:27:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:27:01.991+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:27:02.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:27:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:27:02.083+0000 7f8a5f86e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:02.083452+0000) 2024-02-20T09:27:02.617 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:27:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:27:02.305+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:27:02.617 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:27:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:27:02.305+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:27:02.617 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:27:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:27:02.305+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:27:02.617 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:27:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:27:02.305+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:27:02.962 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:27:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:27:02.627+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:02.628342+0000) 2024-02-20T09:27:02.962 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:27:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:27:02.779+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:27:02.962 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:27:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:27:02.779+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:27:02.962 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:27:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:27:02.779+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:27:03.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:27:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:27:03.087+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:03.087958+0000) 2024-02-20T09:27:03.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:27:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:27:02.971+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:02.972224+0000) 2024-02-20T09:27:03.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:27:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:27:02.994+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:27:03.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:27:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:27:02.994+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:27:03.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:27:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:27:02.994+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:27:03.617 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:27:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:27:03.260+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:27:03.617 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:27:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:27:03.260+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:27:03.617 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:27:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:27:03.260+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:27:03.617 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:27:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:27:03.260+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:27:03.962 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:27:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:27:03.630+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:03.630430+0000) 2024-02-20T09:27:03.962 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:27:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:27:03.807+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:27:03.962 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:27:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:27:03.807+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:27:03.962 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:27:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:27:03.807+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:27:04.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:27:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:27:04.087+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:04.088077+0000) 2024-02-20T09:27:04.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:27:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:27:03.971+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:03.972335+0000) 2024-02-20T09:27:04.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:27:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:27:04.004+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:27:04.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:27:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:27:04.004+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:27:04.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:27:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:27:04.004+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:27:04.617 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:27:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:27:04.255+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:27:04.617 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:27:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:27:04.255+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:27:04.618 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:27:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:27:04.255+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:27:04.618 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:27:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:27:04.255+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:27:04.962 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:27:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:27:04.632+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:04.632838+0000) 2024-02-20T09:27:04.962 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:27:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:27:04.801+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:27:04.962 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:27:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:27:04.801+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:27:04.962 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:27:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:27:04.801+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:27:05.231 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:27:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:27:05.086+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:05.088192+0000) 2024-02-20T09:27:05.232 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:27:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:27:04.971+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:04.972455+0000) 2024-02-20T09:27:05.232 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:27:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:27:05.022+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:27:05.232 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:27:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:27:05.022+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:27:05.233 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:27:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:27:05.022+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:27:05.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:27:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:27:05.228+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:27:05.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:27:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:27:05.228+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:27:05.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:27:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:27:05.228+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:27:05.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:27:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:27:05.228+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:27:05.962 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:27:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:27:05.633+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:05.634831+0000) 2024-02-20T09:27:05.962 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:27:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:27:05.812+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:27:05.962 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:27:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:27:05.812+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:27:05.962 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:27:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:27:05.812+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:27:06.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:27:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:27:06.086+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:06.088305+0000) 2024-02-20T09:27:06.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:27:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:27:05.971+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:05.972544+0000) 2024-02-20T09:27:06.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:27:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:27:06.034+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:27:06.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:27:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:27:06.034+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:27:06.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:27:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:27:06.034+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:27:06.617 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:27:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:27:06.258+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:27:06.618 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:27:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:27:06.258+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:27:06.618 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:27:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:27:06.258+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:27:06.618 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:27:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:27:06.259+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:27:06.962 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:27:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:27:06.635+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:06.637318+0000) 2024-02-20T09:27:06.963 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:27:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:27:06.774+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:27:06.963 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:27:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:27:06.774+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:27:06.963 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:27:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:27:06.774+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:27:07.217 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:27:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:27:07.097+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 69970 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:27:07.218 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:27:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:27:07.094+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:07.096137+0000) 2024-02-20T09:27:07.218 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:27:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:27:07.087+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:07.088395+0000) 2024-02-20T09:27:07.218 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:27:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:27:07.091+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:07.092806+0000) 2024-02-20T09:27:07.219 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:27:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:27:06.971+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:06.972636+0000) 2024-02-20T09:27:07.219 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:27:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:27:07.004+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:27:07.219 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:27:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:27:07.004+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:27:07.219 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:27:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:27:07.004+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:27:07.219 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:27:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:27:07.091+0000 7f8a5f86e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:07.092513+0000) 2024-02-20T09:27:07.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:27:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:27:07.278+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:27:07.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:27:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:27:07.278+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:27:07.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:27:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:27:07.278+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:27:07.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:27:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:27:07.278+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:27:07.962 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:27:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:27:07.636+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:07.637970+0000) 2024-02-20T09:27:07.963 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:27:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:27:07.804+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:27:07.963 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:27:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:27:07.804+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:27:07.963 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:27:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:27:07.804+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:27:08.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:27:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:27:08.087+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:08.088522+0000) 2024-02-20T09:27:08.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:27:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:27:07.971+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:07.972725+0000) 2024-02-20T09:27:08.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:27:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:27:08.011+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:27:08.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:27:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:27:08.011+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:27:08.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:27:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:27:08.011+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:27:08.617 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:27:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:27:08.283+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:27:08.618 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:27:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:27:08.283+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:27:08.618 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:27:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:27:08.283+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:27:08.618 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:27:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:27:08.283+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:27:08.962 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:27:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:27:08.638+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:08.640014+0000) 2024-02-20T09:27:08.963 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:27:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:27:08.806+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:27:08.963 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:27:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:27:08.806+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:27:08.963 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:27:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:27:08.806+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:27:09.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:27:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:27:09.087+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:09.088620+0000) 2024-02-20T09:27:09.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:27:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:27:08.971+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:08.972877+0000) 2024-02-20T09:27:09.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:27:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:27:08.976+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:27:09.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:27:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:27:08.976+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:27:09.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:27:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:27:08.976+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:27:09.618 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:27:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:27:09.325+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:27:09.618 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:27:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:27:09.325+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:27:09.618 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:27:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:27:09.325+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:27:09.618 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:27:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:27:09.325+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:27:09.886 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:27:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:27:09.640+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:09.641955+0000) 2024-02-20T09:27:09.887 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:27:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:27:09.780+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:27:09.887 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:27:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:27:09.780+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:27:09.887 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:27:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:27:09.780+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:27:10.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:27:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:27:10.087+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:10.088712+0000) 2024-02-20T09:27:10.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:27:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:27:09.971+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:09.973018+0000) 2024-02-20T09:27:10.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:27:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:27:10.020+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:27:10.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:27:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:27:10.020+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:27:10.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:27:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:27:10.020+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:27:10.618 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:27:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:27:10.296+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:27:10.618 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:27:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:27:10.296+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:27:10.618 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:27:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:27:10.296+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:27:10.618 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:27:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:27:10.296+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:27:10.962 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:27:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:27:10.642+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:10.644179+0000) 2024-02-20T09:27:10.963 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:27:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:27:10.811+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:27:10.963 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:27:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:27:10.811+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:27:10.963 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:27:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:27:10.811+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:27:11.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:27:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:27:11.087+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:11.088871+0000) 2024-02-20T09:27:11.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:27:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:27:10.971+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:10.973131+0000) 2024-02-20T09:27:11.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:27:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:27:11.056+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:27:11.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:27:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:27:11.056+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:27:11.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:27:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:27:11.056+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:27:11.618 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:27:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:27:11.326+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:27:11.618 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:27:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:27:11.326+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:27:11.618 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:27:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:27:11.326+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:27:11.618 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:27:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:27:11.326+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:27:11.963 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:27:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:27:11.646+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:11.647882+0000) 2024-02-20T09:27:11.963 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:27:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:27:11.770+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:27:11.963 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:27:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:27:11.770+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:27:11.963 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:27:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:27:11.770+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:27:12.224 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:27:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:27:12.101+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:12.102804+0000) 2024-02-20T09:27:12.225 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:27:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:27:12.087+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:12.089004+0000) 2024-02-20T09:27:12.225 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:27:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:27:12.099+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:12.100471+0000) 2024-02-20T09:27:12.225 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:27:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:27:11.971+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:11.973266+0000) 2024-02-20T09:27:12.225 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:27:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:27:12.066+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:27:12.225 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:27:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:27:12.066+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:27:12.226 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:27:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:27:12.066+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:27:12.226 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:27:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:27:12.099+0000 7f8a5f86e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:12.100851+0000) 2024-02-20T09:27:12.226 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:27:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:27:12.105+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 70009 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:27:12.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:27:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:27:12.361+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:27:12.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:27:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:27:12.361+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:27:12.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:27:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:27:12.361+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:27:12.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:27:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:27:12.361+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:27:12.888 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:27:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:27:12.647+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:12.648898+0000) 2024-02-20T09:27:12.889 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:27:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:27:12.781+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:27:12.889 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:27:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:27:12.781+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:27:12.889 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:27:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:27:12.781+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:27:13.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:27:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:27:13.087+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:13.089132+0000) 2024-02-20T09:27:13.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:27:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:27:12.971+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:12.973385+0000) 2024-02-20T09:27:13.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:27:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:27:13.049+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:27:13.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:27:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:27:13.049+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:27:13.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:27:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:27:13.049+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:27:13.712 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:27:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:27:13.400+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:27:13.712 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:27:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:27:13.400+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:27:13.712 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:27:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:27:13.400+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:27:13.713 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:27:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:27:13.400+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:27:13.713 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:27:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:27:13.649+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:13.650881+0000) 2024-02-20T09:27:13.963 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:27:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:27:13.773+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:27:13.963 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:27:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:27:13.773+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:27:13.963 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:27:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:27:13.773+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:27:14.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:27:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:27:14.087+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:14.089229+0000) 2024-02-20T09:27:14.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:27:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:27:13.972+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:13.973497+0000) 2024-02-20T09:27:14.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:27:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:27:14.054+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:27:14.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:27:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:27:14.054+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:27:14.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:27:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:27:14.054+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:27:14.712 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:27:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:27:14.402+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:27:14.712 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:27:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:27:14.402+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:27:14.713 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:27:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:27:14.402+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:27:14.713 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:27:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:27:14.402+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:27:14.713 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:27:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:27:14.651+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:14.653281+0000) 2024-02-20T09:27:14.963 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:27:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:27:14.803+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:27:14.963 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:27:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:27:14.803+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:27:14.963 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:27:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:27:14.803+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:27:15.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:27:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:27:15.087+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:15.089348+0000) 2024-02-20T09:27:15.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:27:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:27:14.972+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:14.973596+0000) 2024-02-20T09:27:15.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:27:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:27:15.031+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:27:15.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:27:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:27:15.031+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:27:15.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:27:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:27:15.031+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:27:15.656 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:27:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:27:15.380+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:27:15.656 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:27:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:27:15.380+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:27:15.656 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:27:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:27:15.380+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:27:15.657 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:27:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:27:15.380+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:27:15.963 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:27:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:27:15.653+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:15.655234+0000) 2024-02-20T09:27:15.963 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:27:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:27:15.782+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:27:15.964 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:27:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:27:15.782+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:27:15.964 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:27:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:27:15.782+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:27:16.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:27:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:27:16.088+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:16.089464+0000) 2024-02-20T09:27:16.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:27:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:27:15.972+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:15.973686+0000) 2024-02-20T09:27:16.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:27:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:27:16.022+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:27:16.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:27:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:27:16.022+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:27:16.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:27:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:27:16.022+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:27:16.618 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:27:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:27:16.334+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:27:16.619 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:27:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:27:16.334+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:27:16.619 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:27:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:27:16.334+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:27:16.619 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:27:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:27:16.334+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:27:16.963 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:27:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:27:16.656+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:16.657623+0000) 2024-02-20T09:27:16.963 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:27:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:27:16.790+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:27:16.963 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:27:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:27:16.790+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:27:16.964 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:27:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:27:16.790+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:27:17.236 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:27:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:27:17.114+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 70049 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:27:17.237 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:27:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:27:17.109+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:17.111089+0000) 2024-02-20T09:27:17.237 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:27:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:27:17.088+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:17.089556+0000) 2024-02-20T09:27:17.237 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:27:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:27:17.106+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:17.107925+0000) 2024-02-20T09:27:17.237 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:27:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:27:16.972+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:16.973776+0000) 2024-02-20T09:27:17.237 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:27:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:27:16.976+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:27:17.238 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:27:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:27:16.976+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:27:17.238 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:27:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:27:16.976+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:27:17.238 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:27:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:27:17.106+0000 7f8a5f86e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:17.108061+0000) 2024-02-20T09:27:17.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:27:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:27:17.313+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:27:17.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:27:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:27:17.313+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:27:17.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:27:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:27:17.313+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:27:17.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:27:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:27:17.313+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:27:17.947 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:27:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:27:17.656+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:17.658260+0000) 2024-02-20T09:27:17.947 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:27:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:27:17.831+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:27:17.948 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:27:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:27:17.831+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:27:17.948 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:27:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:27:17.831+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:27:18.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:27:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:27:18.088+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:18.089653+0000) 2024-02-20T09:27:18.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:27:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:27:17.945+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:27:18.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:27:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:27:17.945+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:27:18.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:27:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:27:17.945+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:27:18.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:27:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:27:17.972+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:17.973865+0000) 2024-02-20T09:27:18.619 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:27:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:27:18.286+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:27:18.619 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:27:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:27:18.286+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:27:18.620 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:27:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:27:18.286+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:27:18.620 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:27:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:27:18.286+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:27:18.956 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:27:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:27:18.658+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:18.660244+0000) 2024-02-20T09:27:18.957 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:27:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:27:18.831+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:27:18.957 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:27:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:27:18.831+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:27:18.957 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:27:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:27:18.831+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:27:19.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:27:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:27:19.088+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:19.089768+0000) 2024-02-20T09:27:19.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:27:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:27:18.954+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:27:19.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:27:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:27:18.954+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:27:19.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:27:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:27:18.954+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:27:19.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:27:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:27:18.972+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:18.973961+0000) 2024-02-20T09:27:19.619 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:27:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:27:19.282+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:27:19.619 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:27:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:27:19.282+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:27:19.619 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:27:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:27:19.282+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:27:19.619 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:27:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:27:19.282+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:27:19.885 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:27:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:27:19.661+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:19.662516+0000) 2024-02-20T09:27:19.886 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:27:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:27:19.796+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:27:19.886 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:27:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:27:19.796+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:27:19.886 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:27:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:27:19.796+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:27:20.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:27:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:27:20.088+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:20.089884+0000) 2024-02-20T09:27:20.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:27:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:27:19.972+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:19.974074+0000) 2024-02-20T09:27:20.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:27:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:27:19.985+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:27:20.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:27:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:27:19.985+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:27:20.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:27:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:27:19.985+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:27:20.619 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:27:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:27:20.304+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:27:20.619 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:27:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:27:20.304+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:27:20.619 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:27:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:27:20.304+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:27:20.619 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:27:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:27:20.304+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:27:20.963 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:27:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:27:20.663+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:20.664556+0000) 2024-02-20T09:27:20.964 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:27:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:27:20.777+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:27:20.964 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:27:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:27:20.777+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:27:20.964 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:27:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:27:20.777+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:27:21.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:27:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:27:21.088+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:21.089976+0000) 2024-02-20T09:27:21.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:27:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:27:20.972+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:20.974192+0000) 2024-02-20T09:27:21.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:27:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:27:21.033+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:27:21.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:27:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:27:21.033+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:27:21.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:27:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:27:21.033+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:27:21.619 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:27:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:27:21.340+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:27:21.619 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:27:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:27:21.340+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:27:21.619 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:27:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:27:21.340+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:27:21.619 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:27:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:27:21.340+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:27:21.963 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:27:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:27:21.665+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:21.666734+0000) 2024-02-20T09:27:21.964 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:27:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:27:21.759+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:27:21.965 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:27:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:27:21.759+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:27:21.965 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:27:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:27:21.759+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:27:22.242 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:27:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:27:22.122+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 70089 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:27:22.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:27:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:27:22.119+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:22.120591+0000) 2024-02-20T09:27:22.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:27:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:27:22.088+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:22.090064+0000) 2024-02-20T09:27:22.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:27:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:27:22.115+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:22.116554+0000) 2024-02-20T09:27:22.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:27:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:27:21.972+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:21.974329+0000) 2024-02-20T09:27:22.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:27:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:27:22.074+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:27:22.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:27:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:27:22.074+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:27:22.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:27:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:27:22.074+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:27:22.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:27:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:27:22.115+0000 7f8a5f86e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:22.116682+0000) 2024-02-20T09:27:22.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:27:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:27:22.341+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:27:22.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:27:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:27:22.341+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:27:22.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:27:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:27:22.341+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:27:22.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:27:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:27:22.341+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:27:22.963 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:27:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:27:22.666+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:22.667514+0000) 2024-02-20T09:27:22.964 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:27:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:27:22.760+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:27:22.964 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:27:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:27:22.760+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:27:22.964 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:27:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:27:22.760+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:27:23.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:27:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:27:23.088+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:23.090203+0000) 2024-02-20T09:27:23.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:27:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:27:22.973+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:22.974443+0000) 2024-02-20T09:27:23.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:27:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:27:23.087+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:27:23.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:27:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:27:23.087+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:27:23.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:27:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:27:23.087+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:27:23.619 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:27:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:27:23.317+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:27:23.620 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:27:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:27:23.317+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:27:23.620 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:27:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:27:23.317+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:27:23.620 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:27:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:27:23.317+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:27:23.964 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:27:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:27:23.668+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:23.669447+0000) 2024-02-20T09:27:23.965 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:27:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:27:23.757+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:27:23.965 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:27:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:27:23.757+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:27:23.965 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:27:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:27:23.757+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:27:24.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:27:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:27:24.088+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:24.090277+0000) 2024-02-20T09:27:24.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:27:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:27:23.973+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:23.974521+0000) 2024-02-20T09:27:24.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:27:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:27:24.067+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:27:24.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:27:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:27:24.067+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:27:24.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:27:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:27:24.067+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:27:24.619 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:27:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:27:24.295+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:27:24.619 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:27:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:27:24.295+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:27:24.620 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:27:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:27:24.295+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:27:24.620 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:27:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:27:24.295+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:27:24.964 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:27:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:27:24.670+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:24.671544+0000) 2024-02-20T09:27:24.964 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:27:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:27:24.798+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:27:24.965 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:27:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:27:24.798+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:27:24.965 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:27:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:27:24.798+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:27:25.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:27:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:27:25.089+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:25.090388+0000) 2024-02-20T09:27:25.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:27:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:27:24.973+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:24.974622+0000) 2024-02-20T09:27:25.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:27:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:27:25.082+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:27:25.245 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:27:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:27:25.082+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:27:25.245 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:27:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:27:25.082+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:27:25.619 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:27:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:27:25.336+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:27:25.620 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:27:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:27:25.336+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:27:25.620 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:27:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:27:25.336+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:27:25.620 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:27:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:27:25.336+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:27:25.964 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:27:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:27:25.672+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:25.673526+0000) 2024-02-20T09:27:25.964 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:27:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:27:25.829+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:27:25.965 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:27:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:27:25.829+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:27:25.965 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:27:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:27:25.829+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:27:26.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:27:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:27:26.089+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:26.090502+0000) 2024-02-20T09:27:26.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:27:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:27:25.973+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:25.974718+0000) 2024-02-20T09:27:26.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:27:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:27:26.068+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:27:26.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:27:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:27:26.068+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:27:26.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:27:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:27:26.068+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:27:26.678 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:27:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:27:26.374+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:27:26.678 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:27:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:27:26.374+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:27:26.678 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:27:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:27:26.374+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:27:26.678 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:27:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:27:26.374+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:27:26.964 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:27:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:27:26.675+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:26.677081+0000) 2024-02-20T09:27:26.965 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:27:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:27:26.862+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:27:26.965 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:27:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:27:26.862+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:27:26.965 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:27:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:27:26.862+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:27:27.242 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:27:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:27:27.129+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 70129 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:27:27.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:27:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:27:27.125+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:27.127226+0000) 2024-02-20T09:27:27.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:27:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:27:27.089+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:27.090594+0000) 2024-02-20T09:27:27.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:27:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:27:27.123+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:27.124467+0000) 2024-02-20T09:27:27.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:27:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:27:26.973+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:26.974807+0000) 2024-02-20T09:27:27.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:27:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:27:27.047+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:27:27.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:27:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:27:27.047+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:27:27.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:27:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:27:27.047+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:27:27.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:27:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:27:27.123+0000 7f8a5f86e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:27.124962+0000) 2024-02-20T09:27:27.620 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:27:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:27:27.356+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:27:27.620 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:27:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:27:27.356+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:27:27.620 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:27:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:27:27.356+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:27:27.620 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:27:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:27:27.356+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:27:27.964 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:27:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:27:27.676+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:27.677931+0000) 2024-02-20T09:27:27.965 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:27:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:27:27.858+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:27:27.965 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:27:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:27:27.858+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:27:27.965 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:27:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:27:27.858+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:27:28.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:27:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:27:28.089+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:28.090704+0000) 2024-02-20T09:27:28.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:27:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:27:27.973+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:27.974963+0000) 2024-02-20T09:27:28.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:27:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:27:28.043+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:27:28.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:27:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:27:28.043+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:27:28.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:27:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:27:28.043+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:27:28.620 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:27:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:27:28.347+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:27:28.620 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:27:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:27:28.347+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:27:28.620 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:27:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:27:28.347+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:27:28.620 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:27:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:27:28.347+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:27:28.964 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:27:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:27:28.680+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:28.681471+0000) 2024-02-20T09:27:28.965 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:27:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:27:28.861+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:27:28.965 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:27:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:27:28.861+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:27:28.965 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:27:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:27:28.861+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:27:29.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:27:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:27:29.089+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:29.090826+0000) 2024-02-20T09:27:29.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:27:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:27:28.973+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:28.975052+0000) 2024-02-20T09:27:29.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:27:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:27:29.038+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:27:29.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:27:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:27:29.038+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:27:29.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:27:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:27:29.038+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:27:29.620 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:27:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:27:29.312+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:27:29.620 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:27:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:27:29.312+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:27:29.621 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:27:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:27:29.312+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:27:29.621 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:27:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:27:29.312+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:27:29.885 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:27:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:27:29.682+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:29.683552+0000) 2024-02-20T09:27:29.885 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:27:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:27:29.835+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:27:29.885 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:27:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:27:29.835+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:27:29.886 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:27:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:27:29.835+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:27:30.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:27:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:27:30.089+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:30.091001+0000) 2024-02-20T09:27:30.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:27:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:27:29.973+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:29.975140+0000) 2024-02-20T09:27:30.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:27:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:27:30.036+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:27:30.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:27:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:27:30.036+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:27:30.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:27:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:27:30.036+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:27:30.620 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:27:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:27:30.334+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:27:30.620 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:27:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:27:30.334+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:27:30.621 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:27:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:27:30.334+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:27:30.621 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:27:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:27:30.334+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:27:30.965 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:27:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:27:30.684+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:30.685583+0000) 2024-02-20T09:27:30.965 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:27:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:27:30.789+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:27:30.965 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:27:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:27:30.789+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:27:30.965 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:27:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:27:30.789+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:27:31.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:27:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:27:31.089+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:31.091148+0000) 2024-02-20T09:27:31.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:27:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:27:30.973+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:30.975279+0000) 2024-02-20T09:27:31.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:27:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:27:31.028+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:27:31.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:27:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:27:31.028+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:27:31.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:27:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:27:31.028+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:27:31.620 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:27:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:27:31.301+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:27:31.620 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:27:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:27:31.301+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:27:31.621 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:27:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:27:31.301+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:27:31.621 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:27:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:27:31.302+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:27:31.965 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:27:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:27:31.687+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:31.689215+0000) 2024-02-20T09:27:31.965 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:27:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:27:31.818+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:27:31.965 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:27:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:27:31.818+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:27:31.965 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:27:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:27:31.818+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:27:32.242 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:27:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:27:32.138+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 70168 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:27:32.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:27:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:27:32.133+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:32.134930+0000) 2024-02-20T09:27:32.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:27:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:27:32.089+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:32.091285+0000) 2024-02-20T09:27:32.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:27:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:27:32.131+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:32.132635+0000) 2024-02-20T09:27:32.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:27:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:27:31.974+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:31.975504+0000) 2024-02-20T09:27:32.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:27:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:27:32.026+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:27:32.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:27:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:27:32.026+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:27:32.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:27:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:27:32.026+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:27:32.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:27:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:27:32.130+0000 7f8a5f86e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:32.132343+0000) 2024-02-20T09:27:32.620 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:27:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:27:32.313+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:27:32.621 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:27:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:27:32.313+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:27:32.621 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:27:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:27:32.313+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:27:32.621 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:27:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:27:32.313+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:27:32.965 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:27:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:27:32.688+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:32.689941+0000) 2024-02-20T09:27:32.965 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:27:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:27:32.836+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:27:32.965 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:27:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:27:32.836+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:27:32.965 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:27:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:27:32.836+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:27:33.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:27:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:27:33.090+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:33.091421+0000) 2024-02-20T09:27:33.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:27:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:27:32.974+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:32.975654+0000) 2024-02-20T09:27:33.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:27:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:27:33.046+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:27:33.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:27:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:27:33.046+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:27:33.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:27:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:27:33.046+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:27:33.621 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:27:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:27:33.270+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:27:33.621 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:27:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:27:33.270+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:27:33.621 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:27:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:27:33.270+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:27:33.621 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:27:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:27:33.270+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:27:33.965 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:27:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:27:33.691+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:33.692534+0000) 2024-02-20T09:27:33.965 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:27:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:27:33.822+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:27:33.965 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:27:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:27:33.822+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:27:33.966 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:27:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:27:33.822+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:27:34.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:27:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:27:34.090+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:34.091499+0000) 2024-02-20T09:27:34.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:27:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:27:33.974+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:33.975729+0000) 2024-02-20T09:27:34.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:27:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:27:34.023+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:27:34.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:27:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:27:34.023+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:27:34.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:27:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:27:34.023+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:27:34.621 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:27:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:27:34.301+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:27:34.621 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:27:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:27:34.301+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:27:34.621 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:27:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:27:34.301+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:27:34.621 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:27:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:27:34.301+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:27:34.965 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:27:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:27:34.693+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:34.694632+0000) 2024-02-20T09:27:34.965 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:27:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:27:34.861+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:27:34.966 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:27:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:27:34.861+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:27:34.966 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:27:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:27:34.861+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:27:35.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:27:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:27:35.090+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:35.091590+0000) 2024-02-20T09:27:35.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:27:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:27:34.974+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:34.975838+0000) 2024-02-20T09:27:35.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:27:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:27:34.989+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:27:35.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:27:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:27:34.989+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:27:35.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:27:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:27:34.989+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:27:35.621 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:27:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:27:35.274+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:27:35.621 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:27:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:27:35.274+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:27:35.621 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:27:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:27:35.274+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:27:35.621 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:27:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:27:35.274+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:27:35.945 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:27:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:27:35.695+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:35.696729+0000) 2024-02-20T09:27:35.945 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:27:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:27:35.864+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:27:35.946 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:27:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:27:35.864+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:27:35.946 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:27:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:27:35.864+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:27:36.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:27:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:27:36.090+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:36.091717+0000) 2024-02-20T09:27:36.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:27:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:27:35.942+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:27:36.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:27:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:27:35.942+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:27:36.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:27:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:27:35.942+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:27:36.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:27:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:27:35.974+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:35.975952+0000) 2024-02-20T09:27:36.621 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:27:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:27:36.251+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:27:36.621 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:27:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:27:36.251+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:27:36.621 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:27:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:27:36.251+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:27:36.621 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:27:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:27:36.251+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:27:36.878 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:27:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:27:36.698+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:36.700342+0000) 2024-02-20T09:27:37.142 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:27:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:27:36.876+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:27:37.142 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:27:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:27:36.876+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:27:37.142 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:27:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:27:36.876+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:27:37.142 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:27:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:27:37.090+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:37.091829+0000) 2024-02-20T09:27:37.142 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:27:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:27:36.971+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:27:37.143 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:27:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:27:36.971+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:27:37.143 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:27:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:27:36.971+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:27:37.143 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:27:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:27:36.974+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:36.976059+0000) 2024-02-20T09:27:37.143 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:27:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:27:37.139+0000 7f8a5f86e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:37.140776+0000) 2024-02-20T09:27:37.487 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:27:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:27:37.147+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 70208 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:27:37.487 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:27:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:27:37.139+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:37.141173+0000) 2024-02-20T09:27:37.487 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:27:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:27:37.143+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:37.144503+0000) 2024-02-20T09:27:37.487 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:27:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:27:37.251+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:27:37.488 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:27:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:27:37.251+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:27:37.488 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:27:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:27:37.251+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:27:37.488 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:27:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:27:37.251+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:27:37.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:27:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:27:37.700+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:37.701417+0000) 2024-02-20T09:27:38.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:27:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:27:37.847+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:27:38.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:27:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:27:37.847+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:27:38.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:27:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:27:37.847+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:27:38.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:27:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:27:38.091+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:38.092008+0000) 2024-02-20T09:27:38.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:27:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:27:37.954+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:27:38.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:27:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:27:37.954+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:27:38.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:27:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:27:37.954+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:27:38.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:27:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:27:37.974+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:37.976230+0000) 2024-02-20T09:27:38.621 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:27:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:27:38.282+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:27:38.621 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:27:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:27:38.282+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:27:38.622 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:27:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:27:38.282+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:27:38.622 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:27:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:27:38.282+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:27:38.953 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:27:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:27:38.703+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:38.703441+0000) 2024-02-20T09:27:38.954 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:27:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:27:38.848+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:27:38.954 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:27:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:27:38.848+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:27:38.954 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:27:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:27:38.848+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:27:39.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:27:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:27:39.091+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:39.092166+0000) 2024-02-20T09:27:39.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:27:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:27:38.952+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:27:39.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:27:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:27:38.952+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:27:39.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:27:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:27:38.952+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:27:39.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:27:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:27:38.975+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:38.976359+0000) 2024-02-20T09:27:39.621 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:27:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:27:39.294+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:27:39.622 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:27:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:27:39.294+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:27:39.622 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:27:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:27:39.294+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:27:39.622 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:27:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:27:39.294+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:27:39.885 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:27:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:27:39.705+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:39.705439+0000) 2024-02-20T09:27:39.885 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:27:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:27:39.835+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:27:39.886 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:27:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:27:39.835+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:27:39.886 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:27:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:27:39.835+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:27:40.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:27:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:27:40.091+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:40.092342+0000) 2024-02-20T09:27:40.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:27:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:27:39.911+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:27:40.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:27:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:27:39.911+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:27:40.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:27:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:27:39.911+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:27:40.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:27:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:27:39.976+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:39.976502+0000) 2024-02-20T09:27:40.622 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:27:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:27:40.305+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:27:40.622 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:27:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:27:40.305+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:27:40.622 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:27:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:27:40.305+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:27:40.622 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:27:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:27:40.305+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:27:40.950 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:27:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:27:40.707+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:40.707442+0000) 2024-02-20T09:27:40.950 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:27:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:27:40.809+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:27:40.950 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:27:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:27:40.809+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:27:40.950 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:27:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:27:40.809+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:27:41.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:27:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:27:41.092+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:41.092436+0000) 2024-02-20T09:27:41.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:27:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:27:40.948+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:27:41.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:27:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:27:40.948+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:27:41.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:27:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:27:40.948+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:27:41.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:27:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:27:40.976+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:40.976672+0000) 2024-02-20T09:27:41.622 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:27:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:27:41.273+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:27:41.622 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:27:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:27:41.273+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:27:41.622 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:27:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:27:41.273+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:27:41.622 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:27:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:27:41.273+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:27:41.966 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:27:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:27:41.709+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:41.709869+0000) 2024-02-20T09:27:41.966 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:27:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:27:41.801+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:27:41.967 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:27:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:27:41.801+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:27:41.967 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:27:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:27:41.801+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:27:42.242 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:27:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:27:42.156+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 70244 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:27:42.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:27:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:27:42.152+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:42.153091+0000) 2024-02-20T09:27:42.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:27:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:27:42.092+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:42.092564+0000) 2024-02-20T09:27:42.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:27:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:27:42.149+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:42.150319+0000) 2024-02-20T09:27:42.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:27:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:27:41.976+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:41.976812+0000) 2024-02-20T09:27:42.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:27:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:27:41.982+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:27:42.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:27:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:27:41.982+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:27:42.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:27:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:27:41.982+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:27:42.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:27:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:27:42.149+0000 7f8a5f86e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:42.150306+0000) 2024-02-20T09:27:42.622 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:27:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:27:42.316+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:27:42.622 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:27:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:27:42.316+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:27:42.622 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:27:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:27:42.316+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:27:42.622 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:27:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:27:42.316+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:27:42.966 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:27:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:27:42.710+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:42.710569+0000) 2024-02-20T09:27:42.968 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:27:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:27:42.787+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:27:42.968 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:27:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:27:42.787+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:27:42.968 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:27:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:27:42.787+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:27:43.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:27:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:27:43.092+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:43.092699+0000) 2024-02-20T09:27:43.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:27:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:27:42.976+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:42.976946+0000) 2024-02-20T09:27:43.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:27:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:27:43.007+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:27:43.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:27:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:27:43.007+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:27:43.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:27:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:27:43.007+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:27:43.608 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:27:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:27:43.303+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:27:43.609 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:27:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:27:43.303+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:27:43.609 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:27:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:27:43.303+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:27:43.609 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:27:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:27:43.303+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:27:43.966 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:27:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:27:43.712+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:43.713048+0000) 2024-02-20T09:27:43.967 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:27:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:27:43.757+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:27:43.967 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:27:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:27:43.757+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:27:43.967 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:27:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:27:43.757+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:27:44.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:27:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:27:44.092+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:44.092788+0000) 2024-02-20T09:27:44.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:27:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:27:43.976+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:43.977025+0000) 2024-02-20T09:27:44.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:27:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:27:44.004+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:27:44.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:27:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:27:44.005+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:27:44.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:27:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:27:44.005+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:27:44.622 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:27:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:27:44.268+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:27:44.622 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:27:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:27:44.268+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:27:44.622 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:27:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:27:44.268+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:27:44.622 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:27:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:27:44.268+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:27:44.966 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:27:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:27:44.715+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:44.716081+0000) 2024-02-20T09:27:44.967 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:27:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:27:44.802+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:27:44.967 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:27:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:27:44.802+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:27:44.967 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:27:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:27:44.802+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:27:45.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:27:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:27:45.092+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:45.092879+0000) 2024-02-20T09:27:45.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:27:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:27:44.976+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:44.977132+0000) 2024-02-20T09:27:45.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:27:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:27:45.042+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:27:45.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:27:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:27:45.042+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:27:45.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:27:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:27:45.042+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:27:45.622 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:27:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:27:45.304+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:27:45.622 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:27:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:27:45.304+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:27:45.622 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:27:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:27:45.304+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:27:45.623 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:27:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:27:45.304+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:27:45.966 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:27:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:27:45.718+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:45.718591+0000) 2024-02-20T09:27:45.967 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:27:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:27:45.833+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:27:45.967 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:27:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:27:45.833+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:27:45.967 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:27:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:27:45.833+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:27:46.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:27:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:27:46.092+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:46.092968+0000) 2024-02-20T09:27:46.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:27:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:27:45.976+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:45.977227+0000) 2024-02-20T09:27:46.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:27:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:27:46.004+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:27:46.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:27:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:27:46.004+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:27:46.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:27:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:27:46.004+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:27:46.622 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:27:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:27:46.340+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:27:46.622 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:27:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:27:46.340+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:27:46.622 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:27:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:27:46.340+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:27:46.623 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:27:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:27:46.340+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:27:46.966 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:27:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:27:46.720+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:46.721180+0000) 2024-02-20T09:27:46.967 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:27:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:27:46.811+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:27:46.967 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:27:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:27:46.811+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:27:46.967 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:27:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:27:46.811+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:27:47.242 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:27:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:27:47.164+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 70284 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:27:47.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:27:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:27:47.161+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:47.161529+0000) 2024-02-20T09:27:47.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:27:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:27:47.092+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:47.093102+0000) 2024-02-20T09:27:47.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:27:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:27:47.157+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:47.158324+0000) 2024-02-20T09:27:47.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:27:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:27:46.976+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:46.977343+0000) 2024-02-20T09:27:47.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:27:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:27:47.011+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:27:47.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:27:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:27:47.011+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:27:47.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:27:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:27:47.011+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:27:47.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:27:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:27:47.158+0000 7f8a5f86e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:47.158497+0000) 2024-02-20T09:27:47.623 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:27:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:27:47.323+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:27:47.624 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:27:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:27:47.323+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:27:47.624 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:27:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:27:47.323+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:27:47.624 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:27:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:27:47.323+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:27:47.967 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:27:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:27:47.721+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:47.721930+0000) 2024-02-20T09:27:47.967 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:27:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:27:47.827+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:27:47.967 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:27:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:27:47.827+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:27:47.967 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:27:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:27:47.827+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:27:48.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:27:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:27:48.092+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:48.093225+0000) 2024-02-20T09:27:48.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:27:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:27:47.977+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:47.977457+0000) 2024-02-20T09:27:48.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:27:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:27:48.021+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:27:48.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:27:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:27:48.021+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:27:48.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:27:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:27:48.021+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:27:48.622 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:27:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:27:48.350+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:27:48.622 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:27:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:27:48.350+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:27:48.623 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:27:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:27:48.350+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:27:48.623 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:27:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:27:48.351+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:27:48.967 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:27:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:27:48.725+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:48.725461+0000) 2024-02-20T09:27:48.967 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:27:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:27:48.812+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:27:48.967 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:27:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:27:48.812+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:27:48.967 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:27:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:27:48.813+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:27:49.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:27:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:27:49.092+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:49.093326+0000) 2024-02-20T09:27:49.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:27:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:27:48.977+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:48.977535+0000) 2024-02-20T09:27:49.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:27:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:27:49.052+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:27:49.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:27:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:27:49.052+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:27:49.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:27:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:27:49.052+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:27:49.622 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:27:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:27:49.309+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:27:49.623 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:27:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:27:49.309+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:27:49.623 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:27:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:27:49.309+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:27:49.623 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:27:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:27:49.309+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:27:49.885 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:27:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:27:49.728+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:49.729040+0000) 2024-02-20T09:27:49.885 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:27:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:27:49.823+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:27:49.886 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:27:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:27:49.823+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:27:49.886 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:27:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:27:49.823+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:27:50.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:27:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:27:50.093+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:50.093458+0000) 2024-02-20T09:27:50.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:27:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:27:49.977+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:49.977626+0000) 2024-02-20T09:27:50.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:27:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:27:50.009+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:27:50.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:27:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:27:50.009+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:27:50.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:27:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:27:50.009+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:27:50.622 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:27:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:27:50.340+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:27:50.623 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:27:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:27:50.340+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:27:50.623 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:27:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:27:50.340+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:27:50.623 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:27:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:27:50.341+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:27:50.967 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:27:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:27:50.731+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:50.731445+0000) 2024-02-20T09:27:50.968 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:27:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:27:50.816+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:27:50.968 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:27:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:27:50.816+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:27:50.968 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:27:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:27:50.816+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:27:51.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:27:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:27:51.093+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:51.093546+0000) 2024-02-20T09:27:51.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:27:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:27:50.977+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:50.977736+0000) 2024-02-20T09:27:51.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:27:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:27:50.981+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:27:51.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:27:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:27:50.981+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:27:51.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:27:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:27:50.981+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:27:51.622 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:27:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:27:51.305+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:27:51.623 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:27:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:27:51.305+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:27:51.623 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:27:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:27:51.305+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:27:51.623 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:27:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:27:51.305+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:27:51.967 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:27:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:27:51.733+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:51.734052+0000) 2024-02-20T09:27:51.967 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:27:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:27:51.860+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:27:51.968 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:27:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:27:51.860+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:27:51.968 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:27:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:27:51.860+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:27:52.242 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:27:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:27:52.172+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 70323 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:27:52.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:27:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:27:52.169+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:52.170257+0000) 2024-02-20T09:27:52.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:27:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:27:52.093+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:52.093636+0000) 2024-02-20T09:27:52.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:27:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:27:52.166+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:52.167051+0000) 2024-02-20T09:27:52.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:27:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:27:51.977+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:51.977880+0000) 2024-02-20T09:27:52.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:27:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:27:51.989+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:27:52.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:27:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:27:51.989+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:27:52.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:27:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:27:51.989+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:27:52.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:27:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:27:52.166+0000 7f8a5f86e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:52.166979+0000) 2024-02-20T09:27:52.623 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:27:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:27:52.265+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:27:52.623 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:27:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:27:52.265+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:27:52.623 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:27:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:27:52.265+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:27:52.623 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:27:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:27:52.266+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:27:52.964 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:27:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:27:52.734+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:52.734895+0000) 2024-02-20T09:27:52.964 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:27:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:27:52.848+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:27:52.964 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:27:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:27:52.848+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:27:52.964 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:27:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:27:52.848+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:27:53.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:27:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:27:53.093+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:53.093776+0000) 2024-02-20T09:27:53.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:27:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:27:52.962+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:27:53.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:27:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:27:52.962+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:27:53.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:27:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:27:52.962+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:27:53.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:27:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:27:52.977+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:52.977999+0000) 2024-02-20T09:27:53.623 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:27:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:27:53.309+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:27:53.623 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:27:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:27:53.309+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:27:53.623 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:27:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:27:53.309+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:27:53.623 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:27:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:27:53.309+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:27:53.931 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:27:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:27:53.737+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:53.738355+0000) 2024-02-20T09:27:53.931 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:27:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:27:53.871+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:27:53.932 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:27:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:27:53.871+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:27:53.932 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:27:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:27:53.871+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:27:54.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:27:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:27:54.093+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:54.093874+0000) 2024-02-20T09:27:54.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:27:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:27:53.929+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:27:54.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:27:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:27:53.929+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:27:54.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:27:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:27:53.929+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:27:54.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:27:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:27:53.977+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:53.978112+0000) 2024-02-20T09:27:54.623 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:27:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:27:54.327+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:27:54.623 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:27:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:27:54.327+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:27:54.623 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:27:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:27:54.327+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:27:54.623 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:27:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:27:54.327+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:27:54.954 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:27:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:27:54.741+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:54.741837+0000) 2024-02-20T09:27:54.954 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:27:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:27:54.836+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:27:54.954 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:27:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:27:54.836+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:27:54.954 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:27:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:27:54.836+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:27:55.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:27:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:27:55.093+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:55.093999+0000) 2024-02-20T09:27:55.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:27:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:27:54.952+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:27:55.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:27:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:27:54.952+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:27:55.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:27:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:27:54.952+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:27:55.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:27:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:27:54.977+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:54.978240+0000) 2024-02-20T09:27:55.623 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:27:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:27:55.319+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:27:55.623 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:27:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:27:55.319+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:27:55.624 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:27:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:27:55.319+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:27:55.624 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:27:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:27:55.319+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:27:55.914 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:27:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:27:55.744+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:55.745300+0000) 2024-02-20T09:27:55.914 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:27:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:27:55.855+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:27:55.915 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:27:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:27:55.855+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:27:55.915 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:27:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:27:55.855+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:27:56.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:27:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:27:56.093+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:56.094111+0000) 2024-02-20T09:27:56.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:27:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:27:55.912+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:27:56.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:27:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:27:55.912+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:27:56.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:27:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:27:55.912+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:27:56.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:27:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:27:55.977+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:55.978367+0000) 2024-02-20T09:27:56.623 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:27:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:27:56.315+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:27:56.623 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:27:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:27:56.315+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:27:56.623 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:27:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:27:56.315+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:27:56.624 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:27:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:27:56.315+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:27:56.899 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:27:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:27:56.748+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:56.748786+0000) 2024-02-20T09:27:56.899 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:27:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:27:56.851+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:27:56.899 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:27:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:27:56.851+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:27:56.900 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:27:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:27:56.851+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:27:57.175 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:27:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:27:57.093+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:57.094219+0000) 2024-02-20T09:27:57.175 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:27:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:27:56.897+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:27:57.176 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:27:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:27:56.897+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:27:57.176 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:27:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:27:56.897+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:27:57.176 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:27:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:27:56.978+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:56.978479+0000) 2024-02-20T09:27:57.492 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:27:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:27:57.174+0000 7f8a5f86e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:57.174665+0000) 2024-02-20T09:27:57.493 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:27:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:27:57.180+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 70362 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:27:57.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:27:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:27:57.174+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:57.174840+0000) 2024-02-20T09:27:57.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:27:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:27:57.176+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:57.176755+0000) 2024-02-20T09:27:57.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:27:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:27:57.313+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:27:57.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:27:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:27:57.313+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:27:57.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:27:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:27:57.313+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:27:57.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:27:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:27:57.313+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:27:57.900 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:27:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:27:57.749+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:57.749966+0000) 2024-02-20T09:27:58.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:27:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:27:57.940+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:27:58.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:27:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:27:57.940+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:27:58.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:27:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:27:57.940+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:27:58.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:27:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:27:57.978+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:57.978583+0000) 2024-02-20T09:27:58.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:27:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:27:57.898+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:27:58.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:27:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:27:57.898+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:27:58.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:27:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:27:57.898+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:27:58.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:27:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:27:58.093+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:58.094352+0000) 2024-02-20T09:27:58.623 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:27:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:27:58.310+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:27:58.624 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:27:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:27:58.310+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:27:58.624 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:27:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:27:58.310+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:27:58.624 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:27:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:27:58.310+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:27:58.935 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:27:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:27:58.751+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:58.752010+0000) 2024-02-20T09:27:59.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:27:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:27:58.946+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:27:59.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:27:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:27:58.946+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:27:59.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:27:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:27:58.946+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:27:59.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:27:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:27:59.094+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:59.094485+0000) 2024-02-20T09:27:59.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:27:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:27:58.933+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:27:59.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:27:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:27:58.933+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:27:59.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:27:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:27:58.933+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:27:59.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:27:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:27:58.978+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:58.978660+0000) 2024-02-20T09:27:59.623 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:27:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:27:59.297+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:27:59.624 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:27:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:27:59.297+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:27:59.624 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:27:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:27:59.297+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:27:59.624 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:27:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:27:59.297+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:27:59.885 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:27:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:27:59.753+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:59.754313+0000) 2024-02-20T09:28:00.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:27:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:27:59.905+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:28:00.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:27:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:27:59.905+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:28:00.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:27:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:27:59.905+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:28:00.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:28:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:28:00.094+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:00.094599+0000) 2024-02-20T09:28:00.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:27:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:27:59.976+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:28:00.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:27:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:27:59.976+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:28:00.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:27:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:27:59.976+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:28:00.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:27:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:27:59.978+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:27:59.978777+0000) 2024-02-20T09:28:00.623 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:28:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:28:00.316+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:28:00.624 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:28:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:28:00.316+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:28:00.624 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:28:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:28:00.316+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:28:00.624 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:28:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:28:00.316+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:28:00.898 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:28:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:28:00.756+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:00.756416+0000) 2024-02-20T09:28:01.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:28:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:28:00.971+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:28:01.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:28:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:28:00.971+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:28:01.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:28:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:28:00.971+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:28:01.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:28:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:28:00.977+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:00.978912+0000) 2024-02-20T09:28:01.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:28:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:28:00.897+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:28:01.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:28:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:28:00.897+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:28:01.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:28:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:28:00.897+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:28:01.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:28:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:28:01.093+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:01.094711+0000) 2024-02-20T09:28:01.623 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:28:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:28:01.344+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:28:01.624 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:28:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:28:01.344+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:28:01.624 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:28:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:28:01.344+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:28:01.624 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:28:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:28:01.344+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:28:01.894 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:28:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:28:01.757+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:01.758488+0000) 2024-02-20T09:28:02.183 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:28:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:28:01.891+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:28:02.183 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:28:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:28:01.891+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:28:02.183 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:28:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:28:01.891+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:28:02.183 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:28:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:28:02.093+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:02.094818+0000) 2024-02-20T09:28:02.184 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:28:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:28:01.977+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:01.978999+0000) 2024-02-20T09:28:02.184 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:28:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:28:01.978+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:28:02.184 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:28:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:28:01.978+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:28:02.184 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:28:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:28:01.978+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:28:02.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:28:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:28:02.180+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:02.182343+0000) 2024-02-20T09:28:02.493 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:28:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:28:02.187+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 70402 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:28:02.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:28:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:28:02.181+0000 7f8a5f86e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:02.182948+0000) 2024-02-20T09:28:02.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:28:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:28:02.183+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:02.185181+0000) 2024-02-20T09:28:02.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:28:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:28:02.305+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:28:02.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:28:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:28:02.305+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:28:02.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:28:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:28:02.305+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:28:02.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:28:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:28:02.305+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:28:02.854 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:28:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:28:02.757+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:02.759181+0000) 2024-02-20T09:28:03.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:28:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:28:02.851+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:28:03.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:28:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:28:02.851+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:28:03.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:28:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:28:02.851+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:28:03.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:28:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:28:03.093+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:03.094944+0000) 2024-02-20T09:28:03.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:28:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:28:02.977+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:02.979124+0000) 2024-02-20T09:28:03.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:28:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:28:03.028+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:28:03.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:28:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:28:03.028+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:28:03.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:28:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:28:03.028+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:28:03.624 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:28:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:28:03.286+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:28:03.624 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:28:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:28:03.286+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:28:03.624 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:28:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:28:03.286+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:28:03.624 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:28:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:28:03.286+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:28:03.968 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:28:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:28:03.760+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:03.761665+0000) 2024-02-20T09:28:03.969 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:28:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:28:03.870+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:28:03.969 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:28:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:28:03.870+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:28:03.969 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:28:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:28:03.870+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:28:04.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:28:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:28:04.093+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:04.095033+0000) 2024-02-20T09:28:04.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:28:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:28:03.977+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:03.979229+0000) 2024-02-20T09:28:04.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:28:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:28:04.025+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:28:04.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:28:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:28:04.025+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:28:04.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:28:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:28:04.025+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:28:04.624 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:28:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:28:04.308+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:28:04.625 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:28:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:28:04.308+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:28:04.625 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:28:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:28:04.308+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:28:04.625 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:28:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:28:04.308+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:28:04.908 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:28:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:28:04.762+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:04.764078+0000) 2024-02-20T09:28:05.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:28:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:28:04.978+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:04.979378+0000) 2024-02-20T09:28:05.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:28:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:28:04.982+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:28:05.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:28:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:28:04.982+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:28:05.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:28:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:28:04.982+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:28:05.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:28:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:28:04.906+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:28:05.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:28:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:28:04.906+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:28:05.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:28:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:28:04.906+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:28:05.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:28:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:28:05.093+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:05.095158+0000) 2024-02-20T09:28:05.624 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:28:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:28:05.289+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:28:05.624 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:28:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:28:05.289+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:28:05.624 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:28:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:28:05.289+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:28:05.625 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:28:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:28:05.289+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:28:05.913 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:28:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:28:05.764+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:05.766066+0000) 2024-02-20T09:28:06.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:28:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:28:05.978+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:05.979511+0000) 2024-02-20T09:28:06.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:28:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:28:06.027+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:28:06.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:28:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:28:06.027+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:28:06.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:28:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:28:06.027+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:28:06.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:28:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:28:05.911+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:28:06.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:28:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:28:05.911+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:28:06.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:28:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:28:05.911+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:28:06.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:28:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:28:06.093+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:06.095276+0000) 2024-02-20T09:28:06.519 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:28:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:28:06.247+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:28:06.519 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:28:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:28:06.247+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:28:06.519 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:28:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:28:06.247+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:28:06.519 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:28:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:28:06.247+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:28:07.068 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:28:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:28:06.766+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:06.768206+0000) 2024-02-20T09:28:07.069 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:28:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:28:06.978+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:06.979601+0000) 2024-02-20T09:28:07.069 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:28:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:28:07.066+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:28:07.069 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:28:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:28:06.901+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:28:07.070 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:28:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:28:06.901+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:28:07.070 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:28:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:28:06.901+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:28:07.492 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:28:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:28:07.195+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 70441 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:28:07.495 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:28:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:28:07.094+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:07.095424+0000) 2024-02-20T09:28:07.495 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:28:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:28:07.189+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:07.190440+0000) 2024-02-20T09:28:07.495 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:28:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:28:07.066+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:28:07.495 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:28:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:28:07.066+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:28:07.495 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:28:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:28:07.189+0000 7f8a5f86e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:07.190820+0000) 2024-02-20T09:28:07.496 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:28:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:28:07.192+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:07.193707+0000) 2024-02-20T09:28:07.496 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:28:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:28:07.202+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:28:07.496 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:28:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:28:07.202+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:28:07.496 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:28:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:28:07.202+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:28:07.496 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:28:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:28:07.202+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:28:08.020 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:28:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:28:07.767+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:07.769043+0000) 2024-02-20T09:28:08.020 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:28:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:28:07.978+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:07.979735+0000) 2024-02-20T09:28:08.021 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:28:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:28:08.017+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:28:08.021 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:28:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:28:07.869+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:28:08.021 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:28:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:28:07.869+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:28:08.021 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:28:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:28:07.869+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:28:08.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:28:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:28:08.094+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:08.095476+0000) 2024-02-20T09:28:08.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:28:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:28:08.154+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:28:08.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:28:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:28:08.154+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:28:08.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:28:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:28:08.154+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:28:08.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:28:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:28:08.154+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:28:08.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:28:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:28:08.017+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:28:08.494 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:28:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:28:08.017+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:28:09.022 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:28:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:28:08.770+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:08.771434+0000) 2024-02-20T09:28:09.023 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:28:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:28:08.901+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:28:09.023 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:28:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:28:08.901+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:28:09.023 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:28:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:28:08.901+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:28:09.023 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:28:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:28:08.978+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:08.979838+0000) 2024-02-20T09:28:09.023 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:28:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:28:09.020+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:28:09.024 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:28:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:28:09.020+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:28:09.024 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:28:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:28:09.020+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:28:09.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:28:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:28:09.094+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:09.095617+0000) 2024-02-20T09:28:09.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:28:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:28:09.121+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:28:09.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:28:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:28:09.121+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:28:09.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:28:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:28:09.121+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:28:09.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:28:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:28:09.121+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:28:10.096 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:28:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:28:09.772+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:09.773584+0000) 2024-02-20T09:28:10.097 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:28:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:28:09.858+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:28:10.097 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:28:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:28:09.858+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:28:10.097 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:28:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:28:09.858+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:28:10.097 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:28:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:28:09.978+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:09.979971+0000) 2024-02-20T09:28:10.098 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:28:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:28:09.999+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:28:10.098 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:28:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:28:09.999+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:28:10.098 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:28:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:28:09.999+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:28:10.479 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:28:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:28:10.094+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:10.095740+0000) 2024-02-20T09:28:10.480 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:28:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:28:10.104+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:28:10.480 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:28:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:28:10.104+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:28:10.480 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:28:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:28:10.104+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:28:10.480 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:28:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:28:10.104+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:28:11.074 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:28:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:28:10.978+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:10.980105+0000) 2024-02-20T09:28:11.075 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:28:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:28:11.023+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:28:11.075 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:28:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:28:11.023+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:28:11.075 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:28:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:28:11.023+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:28:11.075 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:28:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:28:10.814+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:28:11.075 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:28:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:28:10.814+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:28:11.076 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:28:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:28:10.814+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:28:11.076 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:28:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:28:10.774+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:10.775770+0000) 2024-02-20T09:28:11.076 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:28:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:28:11.072+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:28:11.076 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:28:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:28:11.072+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:28:11.076 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:28:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:28:11.072+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:28:11.076 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:28:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:28:11.072+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:28:11.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:28:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:28:11.094+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:11.095835+0000) 2024-02-20T09:28:12.073 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:28:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:28:11.853+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:28:12.073 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:28:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:28:11.853+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:28:12.073 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:28:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:28:11.853+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:28:12.074 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:28:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:28:11.978+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:11.980214+0000) 2024-02-20T09:28:12.074 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:28:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:28:12.020+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:28:12.074 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:28:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:28:12.020+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:28:12.074 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:28:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:28:12.020+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:28:12.074 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:28:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:28:11.776+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:11.777826+0000) 2024-02-20T09:28:12.075 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:28:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:28:12.070+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:28:12.075 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:28:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:28:12.070+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:28:12.075 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:28:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:28:12.070+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:28:12.492 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:28:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:28:12.202+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 70481 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:28:12.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:28:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:28:12.196+0000 7f8a5f86e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:12.197883+0000) 2024-02-20T09:28:12.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:28:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:28:12.070+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:28:12.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:28:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:28:12.200+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:12.201485+0000) 2024-02-20T09:28:12.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:28:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:28:12.094+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:12.095942+0000) 2024-02-20T09:28:12.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:28:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:28:12.196+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:12.198078+0000) 2024-02-20T09:28:13.044 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:28:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:28:12.978+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:12.980362+0000) 2024-02-20T09:28:13.045 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:28:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:28:12.902+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:28:13.045 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:28:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:28:12.902+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:28:13.045 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:28:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:28:12.902+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:28:13.045 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:28:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:28:12.777+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:12.779074+0000) 2024-02-20T09:28:13.045 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:28:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:28:13.041+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:28:13.045 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:28:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:28:13.041+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:28:13.045 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:28:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:28:13.041+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:28:13.046 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:28:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:28:13.041+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:28:13.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:28:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:28:13.094+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:13.096074+0000) 2024-02-20T09:28:13.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:28:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:28:13.064+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:28:13.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:28:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:28:13.064+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:28:13.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:28:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:28:13.064+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:28:14.051 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:28:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:28:13.979+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:13.980444+0000) 2024-02-20T09:28:14.051 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:28:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:28:13.779+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:13.781058+0000) 2024-02-20T09:28:14.052 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:28:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:28:13.889+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:28:14.052 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:28:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:28:13.889+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:28:14.052 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:28:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:28:13.889+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:28:14.328 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:28:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:28:14.094+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:14.096148+0000) 2024-02-20T09:28:14.329 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:28:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:28:14.098+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:28:14.329 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:28:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:28:14.098+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:28:14.329 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:28:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:28:14.098+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:28:14.329 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:28:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:28:14.047+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:28:14.330 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:28:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:28:14.047+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:28:14.330 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:28:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:28:14.047+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:28:14.330 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:28:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:28:14.048+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:28:14.939 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:28:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:28:14.781+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:14.783269+0000) 2024-02-20T09:28:15.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:28:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:28:15.095+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:28:15.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:28:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:28:15.095+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:28:15.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:28:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:28:15.095+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:28:15.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:28:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:28:15.095+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:28:15.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:28:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:28:14.937+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:28:15.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:28:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:28:14.937+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:28:15.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:28:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:28:14.937+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:28:15.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:28:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:28:15.094+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:15.096224+0000) 2024-02-20T09:28:15.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:28:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:28:14.979+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:14.980562+0000) 2024-02-20T09:28:15.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:28:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:28:15.103+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:28:15.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:28:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:28:15.103+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:28:15.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:28:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:28:15.103+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:28:16.068 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:28:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:28:15.784+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:15.785847+0000) 2024-02-20T09:28:16.069 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:28:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:28:15.985+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:28:16.069 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:28:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:28:15.985+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:28:16.069 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:28:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:28:15.985+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:28:16.069 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:28:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:28:15.979+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:15.980655+0000) 2024-02-20T09:28:16.069 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:28:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:28:16.066+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:28:16.070 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:28:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:28:16.066+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:28:16.070 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:28:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:28:16.066+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:28:16.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:28:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:28:16.094+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:16.096353+0000) 2024-02-20T09:28:16.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:28:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:28:16.138+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:28:16.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:28:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:28:16.138+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:28:16.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:28:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:28:16.138+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:28:16.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:28:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:28:16.138+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:28:17.070 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:28:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:28:16.786+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:16.788001+0000) 2024-02-20T09:28:17.070 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:28:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:28:16.993+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:28:17.070 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:28:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:28:16.993+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:28:17.071 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:28:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:28:16.993+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:28:17.071 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:28:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:28:16.979+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:16.980744+0000) 2024-02-20T09:28:17.071 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:28:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:28:17.067+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:28:17.071 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:28:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:28:17.067+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:28:17.071 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:28:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:28:17.067+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:28:17.331 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:28:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:28:17.210+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 70520 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:28:17.331 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:28:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:28:17.203+0000 7f8a5f86e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:17.205337+0000) 2024-02-20T09:28:17.332 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:28:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:28:17.095+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:17.096464+0000) 2024-02-20T09:28:17.332 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:28:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:28:17.204+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:17.205556+0000) 2024-02-20T09:28:17.332 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:28:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:28:17.112+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:28:17.332 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:28:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:28:17.112+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:28:17.333 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:28:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:28:17.112+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:28:17.333 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:28:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:28:17.112+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:28:17.333 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:28:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:28:17.207+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:17.209093+0000) 2024-02-20T09:28:17.970 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:28:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:28:17.787+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:17.788919+0000) 2024-02-20T09:28:18.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:28:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:28:17.979+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:17.980834+0000) 2024-02-20T09:28:18.287 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:28:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:28:18.065+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:28:18.287 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:28:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:28:18.065+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:28:18.288 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:28:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:28:18.065+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:28:18.288 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:28:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:28:18.082+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:28:18.288 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:28:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:28:18.082+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:28:18.288 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:28:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:28:18.082+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:28:18.289 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:28:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:28:18.083+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:28:18.289 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:28:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:28:17.976+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:28:18.289 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:28:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:28:17.976+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:28:18.289 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:28:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:28:17.976+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:28:18.289 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:28:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:28:18.095+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:18.096578+0000) 2024-02-20T09:28:19.049 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:28:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:28:19.012+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:28:19.050 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:28:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:28:19.012+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:28:19.050 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:28:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:28:19.012+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:28:19.050 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:28:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:28:18.979+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:18.980915+0000) 2024-02-20T09:28:19.051 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:28:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:28:19.030+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:28:19.051 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:28:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:28:19.030+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:28:19.051 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:28:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:28:19.030+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:28:19.051 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:28:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:28:18.789+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:18.790958+0000) 2024-02-20T09:28:19.051 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:28:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:28:19.046+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:28:19.051 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:28:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:28:19.046+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:28:19.052 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:28:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:28:19.046+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:28:19.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:28:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:28:19.047+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:28:19.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:28:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:28:19.095+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:19.096718+0000) 2024-02-20T09:28:20.098 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:28:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:28:20.029+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:28:20.098 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:28:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:28:20.029+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:28:20.098 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:28:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:28:20.029+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:28:20.098 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:28:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:28:19.979+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:19.981047+0000) 2024-02-20T09:28:20.099 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:28:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:28:20.005+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:28:20.099 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:28:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:28:20.005+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:28:20.099 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:28:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:28:20.005+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:28:20.099 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:28:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:28:19.791+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:19.792958+0000) 2024-02-20T09:28:20.099 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:28:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:28:20.037+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:28:20.099 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:28:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:28:20.037+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:28:20.100 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:28:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:28:20.037+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:28:20.100 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:28:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:28:20.037+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:28:20.480 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:28:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:28:20.095+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:20.096850+0000) 2024-02-20T09:28:21.047 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:28:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:28:21.000+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:28:21.048 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:28:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:28:21.000+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:28:21.048 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:28:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:28:21.000+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:28:21.049 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:28:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:28:20.793+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:20.795058+0000) 2024-02-20T09:28:21.049 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:28:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:28:21.044+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:28:21.049 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:28:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:28:21.044+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:28:21.050 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:28:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:28:20.979+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:20.981163+0000) 2024-02-20T09:28:21.050 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:28:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:28:20.991+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:28:21.050 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:28:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:28:20.991+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:28:21.050 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:28:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:28:20.991+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:28:21.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:28:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:28:21.095+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:21.096972+0000) 2024-02-20T09:28:21.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:28:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:28:21.044+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:28:21.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:28:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:28:21.044+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:28:22.057 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:28:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:28:21.795+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:21.797052+0000) 2024-02-20T09:28:22.057 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:28:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:28:22.054+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:28:22.058 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:28:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:28:21.956+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:28:22.058 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:28:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:28:21.956+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:28:22.058 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:28:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:28:21.956+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:28:22.058 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:28:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:28:21.963+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:28:22.058 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:28:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:28:21.963+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:28:22.059 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:28:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:28:21.963+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:28:22.059 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:28:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:28:21.979+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:21.981292+0000) 2024-02-20T09:28:22.338 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:28:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:28:22.212+0000 7f8a5f86e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:22.213630+0000) 2024-02-20T09:28:22.338 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:28:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:28:22.095+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:22.097090+0000) 2024-02-20T09:28:22.338 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:28:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:28:22.211+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:22.212867+0000) 2024-02-20T09:28:22.339 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:28:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:28:22.054+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:28:22.339 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:28:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:28:22.054+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:28:22.339 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:28:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:28:22.054+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:28:22.339 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:28:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:28:22.213+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:22.215356+0000) 2024-02-20T09:28:22.339 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:28:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:28:22.217+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 70560 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:28:22.910 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:28:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:28:22.796+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:22.798020+0000) 2024-02-20T09:28:23.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:28:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:28:23.032+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:28:23.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:28:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:28:23.032+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:28:23.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:28:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:28:23.032+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:28:23.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:28:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:28:23.033+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:28:23.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:28:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:28:22.907+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:28:23.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:28:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:28:22.907+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:28:23.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:28:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:28:22.907+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:28:23.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:28:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:28:23.095+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:23.097170+0000) 2024-02-20T09:28:23.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:28:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:28:22.977+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:28:23.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:28:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:28:22.977+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:28:23.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:28:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:28:22.977+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:28:23.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:28:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:28:22.980+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:22.981426+0000) 2024-02-20T09:28:24.098 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:28:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:28:23.903+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:28:24.098 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:28:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:28:23.903+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:28:24.098 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:28:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:28:23.903+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:28:24.099 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:28:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:28:23.941+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:28:24.099 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:28:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:28:23.941+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:28:24.100 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:28:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:28:23.941+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:28:24.100 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:28:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:28:23.980+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:23.981511+0000) 2024-02-20T09:28:24.100 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:28:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:28:23.799+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:23.800516+0000) 2024-02-20T09:28:24.100 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:28:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:28:24.011+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:28:24.100 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:28:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:28:24.011+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:28:24.101 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:28:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:28:24.012+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:28:24.101 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:28:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:28:24.012+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:28:24.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:28:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:28:24.095+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:24.097244+0000) 2024-02-20T09:28:25.098 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:28:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:28:24.879+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:28:25.098 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:28:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:28:24.879+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:28:25.098 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:28:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:28:24.879+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:28:25.099 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:28:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:28:24.980+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:28:25.099 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:28:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:28:24.980+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:28:25.099 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:28:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:28:24.980+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:28:25.099 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:28:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:28:24.980+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:24.981619+0000) 2024-02-20T09:28:25.100 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:28:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:28:24.801+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:24.802838+0000) 2024-02-20T09:28:25.100 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:28:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:28:25.043+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:28:25.100 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:28:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:28:25.043+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:28:25.100 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:28:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:28:25.043+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:28:25.100 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:28:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:28:25.043+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:28:25.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:28:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:28:25.095+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:25.097352+0000) 2024-02-20T09:28:26.098 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:28:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:28:25.913+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:28:26.098 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:28:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:28:25.913+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:28:26.099 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:28:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:28:25.913+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:28:26.099 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:28:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:28:25.803+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:25.804926+0000) 2024-02-20T09:28:26.099 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:28:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:28:26.049+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:28:26.099 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:28:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:28:26.049+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:28:26.099 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:28:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:28:26.049+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:28:26.099 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:28:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:28:26.049+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:28:26.100 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:28:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:28:25.973+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:28:26.100 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:28:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:28:25.973+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:28:26.100 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:28:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:28:25.973+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:28:26.100 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:28:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:28:25.980+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:25.981722+0000) 2024-02-20T09:28:26.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:28:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:28:26.096+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:26.097468+0000) 2024-02-20T09:28:27.079 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:28:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:28:26.883+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:28:27.079 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:28:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:28:26.883+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:28:27.079 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:28:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:28:26.883+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:28:27.080 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:28:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:28:26.805+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:26.807045+0000) 2024-02-20T09:28:27.080 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:28:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:28:27.076+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:28:27.080 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:28:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:28:27.076+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:28:27.081 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:28:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:28:27.076+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:28:27.081 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:28:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:28:27.076+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:28:27.081 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:28:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:28:26.980+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:26.981897+0000) 2024-02-20T09:28:27.082 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:28:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:28:26.992+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:28:27.082 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:28:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:28:26.992+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:28:27.082 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:28:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:28:26.992+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:28:27.346 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:28:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:28:27.225+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 70599 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:28:27.347 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:28:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:28:27.219+0000 7f8a5f86e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:27.220780+0000) 2024-02-20T09:28:27.347 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:28:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:28:27.222+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:27.223517+0000) 2024-02-20T09:28:27.347 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:28:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:28:27.096+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:27.097582+0000) 2024-02-20T09:28:27.347 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:28:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:28:27.219+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:27.220639+0000) 2024-02-20T09:28:27.916 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:28:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:28:27.806+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:27.807927+0000) 2024-02-20T09:28:28.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:28:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:28:28.035+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:28:28.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:28:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:28:28.035+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:28:28.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:28:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:28:28.035+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:28:28.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:28:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:28:28.036+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:28:28.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:28:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:28:27.914+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:28:28.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:28:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:28:27.914+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:28:28.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:28:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:28:27.914+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:28:28.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:28:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:28:28.096+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:28.097681+0000) 2024-02-20T09:28:28.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:28:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:28:27.980+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:27.982042+0000) 2024-02-20T09:28:28.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:28:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:28:28.010+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:28:28.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:28:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:28:28.010+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:28:28.245 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:28:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:28:28.010+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:28:29.099 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:28:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:28:28.866+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:28:29.099 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:28:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:28:28.866+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:28:29.099 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:28:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:28:28.866+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:28:29.099 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:28:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:28:28.980+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:28.982147+0000) 2024-02-20T09:28:29.099 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:28:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:28:29.012+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:28:29.100 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:28:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:28:29.012+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:28:29.100 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:28:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:28:29.012+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:28:29.100 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:28:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:28:28.808+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:28.809923+0000) 2024-02-20T09:28:29.100 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:28:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:28:29.054+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:28:29.100 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:28:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:28:29.054+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:28:29.100 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:28:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:28:29.054+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:28:29.100 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:28:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:28:29.054+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:28:29.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:28:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:28:29.096+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:29.097759+0000) 2024-02-20T09:28:30.073 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:28:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:28:29.980+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:29.982275+0000) 2024-02-20T09:28:30.073 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:28:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:28:29.998+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:28:30.073 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:28:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:28:29.998+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:28:30.074 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:28:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:28:29.998+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:28:30.074 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:28:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:28:29.811+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:29.812067+0000) 2024-02-20T09:28:30.074 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:28:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:28:30.070+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:28:30.074 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:28:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:28:30.070+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:28:30.074 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:28:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:28:30.070+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:28:30.074 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:28:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:28:29.859+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:28:30.075 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:28:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:28:29.859+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:28:30.075 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:28:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:28:29.859+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:28:30.480 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:28:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:28:30.070+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:28:30.481 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:28:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:28:30.096+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:30.097855+0000) 2024-02-20T09:28:31.077 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:28:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:28:30.837+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:28:31.078 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:28:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:28:30.837+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:28:31.078 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:28:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:28:30.837+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:28:31.078 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:28:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:28:30.974+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:28:31.078 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:28:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:28:30.974+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:28:31.078 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:28:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:28:30.974+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:28:31.078 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:28:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:28:30.980+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:30.982369+0000) 2024-02-20T09:28:31.079 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:28:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:28:30.815+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:30.815851+0000) 2024-02-20T09:28:31.079 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:28:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:28:31.075+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:28:31.079 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:28:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:28:31.075+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:28:31.079 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:28:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:28:31.075+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:28:31.079 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:28:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:28:31.075+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:28:31.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:28:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:28:31.096+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:31.097976+0000) 2024-02-20T09:28:32.062 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:28:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:28:31.817+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:31.817862+0000) 2024-02-20T09:28:32.062 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:28:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:28:32.059+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:28:32.063 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:28:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:28:31.806+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:28:32.063 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:28:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:28:31.806+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:28:32.063 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:28:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:28:31.806+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:28:32.063 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:28:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:28:31.970+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:28:32.063 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:28:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:28:31.970+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:28:32.063 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:28:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:28:31.970+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:28:32.064 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:28:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:28:31.982+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:31.982521+0000) 2024-02-20T09:28:32.352 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:28:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:28:32.233+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 70639 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:28:32.353 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:28:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:28:32.227+0000 7f8a5f86e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:32.229076+0000) 2024-02-20T09:28:32.353 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:28:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:28:32.096+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:32.098070+0000) 2024-02-20T09:28:32.353 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:28:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:28:32.226+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:32.228356+0000) 2024-02-20T09:28:32.354 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:28:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:28:32.059+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:28:32.354 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:28:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:28:32.059+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:28:32.354 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:28:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:28:32.059+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:28:32.354 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:28:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:28:32.230+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:32.231857+0000) 2024-02-20T09:28:32.972 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:28:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:28:32.817+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:32.818676+0000) 2024-02-20T09:28:32.972 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:28:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:28:32.785+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:28:32.972 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:28:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:28:32.785+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:28:32.972 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:28:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:28:32.785+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:28:33.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:28:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:28:33.096+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:33.098191+0000) 2024-02-20T09:28:33.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:28:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:28:32.969+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:28:33.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:28:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:28:32.969+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:28:33.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:28:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:28:32.969+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:28:33.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:28:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:28:32.981+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:32.982618+0000) 2024-02-20T09:28:33.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:28:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:28:33.055+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:28:33.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:28:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:28:33.055+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:28:33.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:28:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:28:33.055+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:28:33.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:28:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:28:33.055+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:28:34.099 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:28:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:28:33.820+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:33.822185+0000) 2024-02-20T09:28:34.100 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:28:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:28:34.029+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:28:34.100 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:28:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:28:34.029+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:28:34.100 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:28:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:28:34.029+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:28:34.100 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:28:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:28:34.029+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:28:34.100 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:28:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:28:33.819+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:28:34.101 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:28:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:28:33.819+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:28:34.101 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:28:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:28:33.819+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:28:34.101 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:28:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:28:33.955+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:28:34.101 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:28:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:28:33.955+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:28:34.101 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:28:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:28:33.955+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:28:34.101 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:28:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:28:33.981+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:33.982701+0000) 2024-02-20T09:28:34.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:28:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:28:34.096+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:34.098337+0000) 2024-02-20T09:28:35.099 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:28:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:28:34.919+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:28:35.100 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:28:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:28:34.919+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:28:35.100 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:28:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:28:34.919+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:28:35.100 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:28:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:28:34.982+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:34.982879+0000) 2024-02-20T09:28:35.101 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:28:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:28:34.861+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:28:35.101 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:28:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:28:34.861+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:28:35.101 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:28:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:28:34.861+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:28:35.101 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:28:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:28:35.098+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:35.098439+0000) 2024-02-20T09:28:35.101 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:28:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:28:34.823+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:34.824126+0000) 2024-02-20T09:28:35.101 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:28:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:28:35.049+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:28:35.101 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:28:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:28:35.049+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:28:35.102 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:28:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:28:35.049+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:28:35.102 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:28:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:28:35.049+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:28:36.099 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:28:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:28:35.870+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:28:36.100 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:28:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:28:35.870+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:28:36.100 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:28:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:28:35.870+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:28:36.100 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:28:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:28:36.098+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:36.098550+0000) 2024-02-20T09:28:36.100 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:28:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:28:35.934+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:28:36.100 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:28:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:28:35.934+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:28:36.101 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:28:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:28:35.934+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:28:36.101 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:28:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:28:35.982+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:35.982988+0000) 2024-02-20T09:28:36.101 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:28:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:28:35.825+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:35.826343+0000) 2024-02-20T09:28:36.101 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:28:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:28:36.031+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:28:36.101 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:28:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:28:36.031+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:28:36.101 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:28:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:28:36.031+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:28:36.101 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:28:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:28:36.031+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:28:37.100 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:28:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:28:36.912+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:28:37.100 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:28:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:28:36.912+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:28:37.100 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:28:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:28:36.912+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:28:37.100 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:28:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:28:37.098+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:37.098664+0000) 2024-02-20T09:28:37.101 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:28:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:28:36.922+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:28:37.101 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:28:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:28:36.922+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:28:37.101 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:28:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:28:36.922+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:28:37.101 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:28:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:28:36.982+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:36.983107+0000) 2024-02-20T09:28:37.101 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:28:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:28:36.828+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:36.828511+0000) 2024-02-20T09:28:37.101 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:28:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:28:36.991+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:28:37.102 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:28:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:28:36.991+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:28:37.102 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:28:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:28:36.991+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:28:37.102 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:28:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:28:36.991+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:28:37.362 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:28:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:28:37.235+0000 7f8a5f86e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:37.236358+0000) 2024-02-20T09:28:37.363 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:28:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:28:37.239+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:37.240098+0000) 2024-02-20T09:28:37.363 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:28:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:28:37.235+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:37.236280+0000) 2024-02-20T09:28:37.364 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:28:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:28:37.243+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 70678 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:28:37.892 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:28:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:28:37.829+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:37.829855+0000) 2024-02-20T09:28:38.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:28:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:28:37.890+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:28:38.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:28:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:28:37.890+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:28:38.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:28:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:28:37.890+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:28:38.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:28:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:28:37.982+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:37.983302+0000) 2024-02-20T09:28:38.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:28:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:28:37.985+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:28:38.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:28:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:28:37.985+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:28:38.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:28:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:28:37.985+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:28:38.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:28:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:28:37.985+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:28:38.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:28:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:28:37.944+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:28:38.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:28:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:28:37.944+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:28:38.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:28:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:28:37.944+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:28:38.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:28:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:28:38.098+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:38.098774+0000) 2024-02-20T09:28:39.100 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:28:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:28:38.873+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:28:39.100 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:28:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:28:38.873+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:28:39.100 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:28:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:28:38.873+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:28:39.100 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:28:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:28:38.983+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:38.983484+0000) 2024-02-20T09:28:39.101 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:28:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:28:38.832+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:38.833409+0000) 2024-02-20T09:28:39.101 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:28:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:28:38.999+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:28:39.101 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:28:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:28:38.999+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:28:39.101 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:28:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:28:38.999+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:28:39.101 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:28:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:28:38.999+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:28:39.102 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:28:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:28:38.979+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:28:39.102 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:28:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:28:38.979+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:28:39.102 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:28:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:28:38.979+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:28:39.102 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:28:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:28:39.098+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:39.098864+0000) 2024-02-20T09:28:40.100 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:28:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:28:39.986+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:28:40.100 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:28:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:28:39.986+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:28:40.100 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:28:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:28:39.986+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:28:40.100 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:28:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:28:40.098+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:40.098950+0000) 2024-02-20T09:28:40.101 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:28:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:28:39.835+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:39.836102+0000) 2024-02-20T09:28:40.101 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:28:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:28:39.984+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:28:40.101 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:28:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:28:39.984+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:28:40.101 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:28:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:28:39.984+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:28:40.101 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:28:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:28:39.984+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:28:40.102 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:28:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:28:39.923+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:28:40.102 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:28:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:28:39.923+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:28:40.102 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:28:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:28:39.923+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:28:40.102 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:28:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:28:39.983+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:39.983641+0000) 2024-02-20T09:28:41.103 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:28:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:28:40.941+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:28:41.103 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:28:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:28:40.941+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:28:41.103 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:28:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:28:40.941+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:28:41.104 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:28:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:28:40.915+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:28:41.104 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:28:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:28:40.915+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:28:41.104 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:28:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:28:40.915+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:28:41.104 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:28:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:28:40.983+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:40.983770+0000) 2024-02-20T09:28:41.104 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:28:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:28:40.838+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:40.838688+0000) 2024-02-20T09:28:41.104 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:28:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:28:41.022+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:28:41.105 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:28:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:28:41.022+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:28:41.105 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:28:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:28:41.022+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:28:41.105 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:28:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:28:41.022+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:28:41.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:28:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:28:41.098+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:41.099083+0000) 2024-02-20T09:28:42.100 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:28:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:28:41.895+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:28:42.101 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:28:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:28:41.895+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:28:42.101 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:28:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:28:41.895+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:28:42.101 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:28:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:28:41.922+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:28:42.101 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:28:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:28:41.922+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:28:42.102 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:28:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:28:41.922+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:28:42.102 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:28:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:28:41.983+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:41.983859+0000) 2024-02-20T09:28:42.102 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:28:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:28:41.840+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:41.841205+0000) 2024-02-20T09:28:42.102 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:28:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:28:42.036+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:28:42.102 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:28:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:28:42.036+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:28:42.102 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:28:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:28:42.036+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:28:42.102 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:28:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:28:42.036+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:28:42.370 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:28:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:28:42.250+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 70718 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:28:42.371 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:28:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:28:42.244+0000 7f8a5f86e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:42.245021+0000) 2024-02-20T09:28:42.371 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:28:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:28:42.247+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:42.248155+0000) 2024-02-20T09:28:42.371 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:28:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:28:42.098+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:42.099239+0000) 2024-02-20T09:28:42.371 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:28:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:28:42.244+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:42.244603+0000) 2024-02-20T09:28:42.940 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:28:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:28:42.842+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:42.842547+0000) 2024-02-20T09:28:43.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:28:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:28:42.987+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:28:43.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:28:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:28:42.987+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:28:43.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:28:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:28:42.987+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:28:43.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:28:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:28:42.987+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:28:43.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:28:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:28:42.939+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:28:43.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:28:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:28:42.939+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:28:43.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:28:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:28:42.939+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:28:43.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:28:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:28:43.099+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:43.099418+0000) 2024-02-20T09:28:43.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:28:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:28:42.951+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:28:43.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:28:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:28:42.951+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:28:43.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:28:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:28:42.951+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:28:43.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:28:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:28:42.983+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:42.983976+0000) 2024-02-20T09:28:44.100 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:28:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:28:43.899+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:28:44.101 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:28:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:28:43.899+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:28:44.101 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:28:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:28:43.899+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:28:44.101 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:28:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:28:43.913+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:28:44.101 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:28:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:28:43.913+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:28:44.101 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:28:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:28:43.913+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:28:44.102 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:28:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:28:43.983+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:43.984084+0000) 2024-02-20T09:28:44.102 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:28:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:28:43.844+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:43.844456+0000) 2024-02-20T09:28:44.102 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:28:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:28:43.986+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:28:44.102 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:28:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:28:43.986+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:28:44.102 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:28:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:28:43.986+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:28:44.102 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:28:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:28:43.986+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:28:44.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:28:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:28:44.099+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:44.099497+0000) 2024-02-20T09:28:45.100 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:28:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:28:44.912+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:28:45.101 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:28:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:28:44.912+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:28:45.101 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:28:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:28:44.912+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:28:45.102 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:28:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:28:44.866+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:28:45.102 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:28:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:28:44.866+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:28:45.102 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:28:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:28:44.866+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:28:45.102 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:28:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:28:44.983+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:44.984206+0000) 2024-02-20T09:28:45.103 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:28:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:28:44.846+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:44.846497+0000) 2024-02-20T09:28:45.103 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:28:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:28:44.969+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:28:45.103 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:28:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:28:44.969+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:28:45.103 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:28:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:28:44.969+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:28:45.103 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:28:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:28:44.969+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:28:45.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:28:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:28:45.099+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:45.099582+0000) 2024-02-20T09:28:46.100 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:28:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:28:45.829+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:28:46.101 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:28:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:28:45.829+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:28:46.101 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:28:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:28:45.829+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:28:46.101 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:28:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:28:45.983+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:45.984314+0000) 2024-02-20T09:28:46.101 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:28:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:28:45.867+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:28:46.102 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:28:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:28:45.867+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:28:46.102 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:28:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:28:45.867+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:28:46.102 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:28:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:28:46.099+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:46.099667+0000) 2024-02-20T09:28:46.102 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:28:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:28:45.849+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:45.850116+0000) 2024-02-20T09:28:46.102 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:28:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:28:45.994+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:28:46.102 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:28:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:28:45.994+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:28:46.102 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:28:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:28:45.994+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:28:46.103 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:28:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:28:45.994+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:28:47.100 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:28:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:28:46.839+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:28:47.101 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:28:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:28:46.839+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:28:47.101 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:28:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:28:46.839+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:28:47.101 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:28:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:28:46.984+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:46.984470+0000) 2024-02-20T09:28:47.101 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:28:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:28:46.915+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:28:47.102 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:28:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:28:46.915+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:28:47.102 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:28:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:28:46.915+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:28:47.102 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:28:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:28:46.851+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:46.852283+0000) 2024-02-20T09:28:47.102 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:28:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:28:47.010+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:28:47.102 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:28:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:28:47.010+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:28:47.102 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:28:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:28:47.010+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:28:47.103 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:28:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:28:47.010+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:28:47.379 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:28:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:28:47.258+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 70757 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:28:47.379 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:28:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:28:47.252+0000 7f8a5f86e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:47.252411+0000) 2024-02-20T09:28:47.380 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:28:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:28:47.254+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:47.254429+0000) 2024-02-20T09:28:47.380 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:28:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:28:47.099+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:47.099752+0000) 2024-02-20T09:28:47.380 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:28:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:28:47.251+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:47.251902+0000) 2024-02-20T09:28:47.974 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:28:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:28:47.853+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:47.853542+0000) 2024-02-20T09:28:47.974 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:28:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:28:47.862+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:28:47.974 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:28:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:28:47.862+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:28:47.975 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:28:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:28:47.862+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:28:47.975 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:28:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:28:47.933+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:28:47.975 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:28:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:28:47.933+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:28:47.975 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:28:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:28:47.933+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:28:48.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:28:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:28:47.984+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:47.984564+0000) 2024-02-20T09:28:48.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:28:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:28:48.099+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:48.099883+0000) 2024-02-20T09:28:48.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:28:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:28:47.985+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:28:48.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:28:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:28:47.985+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:28:48.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:28:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:28:47.985+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:28:48.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:28:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:28:47.985+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:28:49.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:28:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:28:48.907+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:28:49.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:28:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:28:48.907+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:28:49.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:28:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:28:48.907+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:28:49.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:28:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:28:48.984+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:48.984640+0000) 2024-02-20T09:28:49.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:28:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:28:48.886+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:28:49.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:28:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:28:48.886+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:28:49.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:28:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:28:48.886+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:28:49.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:28:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:28:49.099+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:49.099961+0000) 2024-02-20T09:28:49.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:28:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:28:48.855+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:48.856274+0000) 2024-02-20T09:28:49.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:28:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:28:49.007+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:28:49.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:28:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:28:49.007+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:28:49.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:28:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:28:49.007+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:28:49.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:28:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:28:49.007+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:28:50.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:28:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:28:49.882+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:28:50.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:28:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:28:49.882+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:28:50.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:28:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:28:49.882+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:28:50.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:28:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:28:49.984+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:49.984744+0000) 2024-02-20T09:28:50.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:28:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:28:49.870+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:28:50.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:28:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:28:49.870+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:28:50.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:28:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:28:49.870+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:28:50.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:28:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:28:50.099+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:50.100051+0000) 2024-02-20T09:28:50.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:28:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:28:49.858+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:49.858545+0000) 2024-02-20T09:28:50.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:28:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:28:50.036+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:28:50.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:28:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:28:50.036+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:28:50.245 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:28:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:28:50.036+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:28:50.245 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:28:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:28:50.036+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:28:51.192 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:28:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:28:50.852+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:28:51.193 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:28:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:28:50.852+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:28:51.193 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:28:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:28:50.852+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:28:51.193 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:28:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:28:51.099+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:51.100169+0000) 2024-02-20T09:28:51.193 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:28:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:28:50.860+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:50.860664+0000) 2024-02-20T09:28:51.194 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:28:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:28:51.052+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:28:51.194 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:28:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:28:51.052+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:28:51.194 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:28:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:28:51.052+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:28:51.194 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:28:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:28:51.052+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:28:51.194 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:28:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:28:50.850+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:28:51.194 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:28:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:28:50.850+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:28:51.194 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:28:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:28:50.850+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:28:51.195 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:28:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:28:50.984+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:50.984950+0000) 2024-02-20T09:28:51.829 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:28:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:28:51.809+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:28:51.830 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:28:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:28:51.809+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:28:51.830 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:28:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:28:51.809+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:28:52.101 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:28:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:28:51.984+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:51.985124+0000) 2024-02-20T09:28:52.101 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:28:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:28:51.828+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:28:52.102 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:28:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:28:51.828+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:28:52.102 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:28:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:28:51.828+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:28:52.102 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:28:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:28:51.862+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:51.862744+0000) 2024-02-20T09:28:52.102 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:28:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:28:52.073+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:28:52.102 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:28:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:28:52.073+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:28:52.102 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:28:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:28:52.073+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:28:52.102 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:28:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:28:52.073+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:28:52.386 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:28:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:28:52.260+0000 7f8a5f86e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:52.260511+0000) 2024-02-20T09:28:52.387 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:28:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:28:52.263+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:52.264330+0000) 2024-02-20T09:28:52.387 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:28:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:28:52.099+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:52.100291+0000) 2024-02-20T09:28:52.387 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:28:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:28:52.259+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:52.260235+0000) 2024-02-20T09:28:52.388 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:28:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:28:52.266+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 70797 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:28:52.974 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:28:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:28:52.863+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:52.863737+0000) 2024-02-20T09:28:52.975 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:28:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:28:52.878+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:28:52.975 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:28:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:28:52.878+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:28:52.975 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:28:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:28:52.878+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:28:52.975 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:28:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:28:52.796+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:28:52.975 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:28:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:28:52.796+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:28:52.976 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:28:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:28:52.796+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:28:53.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:28:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:28:52.984+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:52.985232+0000) 2024-02-20T09:28:53.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:28:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:28:53.100+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:53.100408+0000) 2024-02-20T09:28:53.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:28:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:28:53.084+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:28:53.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:28:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:28:53.084+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:28:53.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:28:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:28:53.084+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:28:53.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:28:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:28:53.084+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:28:53.846 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:28:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:28:53.781+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:28:53.846 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:28:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:28:53.781+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:28:53.846 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:28:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:28:53.781+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:28:54.101 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:28:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:28:53.984+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:53.985311+0000) 2024-02-20T09:28:54.102 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:28:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:28:53.844+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:28:54.102 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:28:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:28:53.844+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:28:54.102 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:28:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:28:53.844+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:28:54.102 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:28:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:28:53.865+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:53.865732+0000) 2024-02-20T09:28:54.102 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:28:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:28:54.091+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:28:54.103 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:28:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:28:54.091+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:28:54.103 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:28:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:28:54.091+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:28:54.103 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:28:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:28:54.092+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:28:54.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:28:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:28:54.100+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:54.100499+0000) 2024-02-20T09:28:54.832 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:28:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:28:54.807+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:28:54.833 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:28:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:28:54.807+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:28:54.833 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:28:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:28:54.807+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:28:55.086 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:28:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:28:54.831+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:28:55.086 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:28:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:28:54.831+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:28:55.086 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:28:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:28:54.831+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:28:55.086 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:28:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:28:54.985+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:54.985405+0000) 2024-02-20T09:28:55.087 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:28:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:28:54.867+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:54.868377+0000) 2024-02-20T09:28:55.087 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:28:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:28:55.084+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:28:55.087 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:28:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:28:55.084+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:28:55.087 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:28:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:28:55.084+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:28:55.087 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:28:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:28:55.084+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:28:55.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:28:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:28:55.100+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:55.100590+0000) 2024-02-20T09:28:55.871 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:28:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:28:55.795+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:28:55.872 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:28:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:28:55.795+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:28:55.872 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:28:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:28:55.795+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:28:55.872 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:28:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:28:55.812+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:28:55.872 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:28:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:28:55.812+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:28:55.872 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:28:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:28:55.812+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:28:56.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:28:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:28:55.985+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:55.985516+0000) 2024-02-20T09:28:56.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:28:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:28:56.100+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:56.100697+0000) 2024-02-20T09:28:56.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:28:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:28:55.869+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:55.870355+0000) 2024-02-20T09:28:56.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:28:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:28:56.101+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:28:56.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:28:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:28:56.101+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:28:56.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:28:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:28:56.101+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:28:56.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:28:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:28:56.101+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:28:56.831 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:28:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:28:56.809+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:28:56.831 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:28:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:28:56.809+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:28:56.831 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:28:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:28:56.809+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:28:57.102 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:28:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:28:56.872+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:56.872460+0000) 2024-02-20T09:28:57.102 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:28:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:28:56.829+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:28:57.102 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:28:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:28:56.829+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:28:57.102 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:28:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:28:56.829+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:28:57.103 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:28:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:28:56.984+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:56.985629+0000) 2024-02-20T09:28:57.393 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:28:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:28:57.273+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 70836 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:28:57.394 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:28:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:28:57.267+0000 7f8a5f86e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:57.268469+0000) 2024-02-20T09:28:57.394 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:28:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:28:57.099+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:57.100814+0000) 2024-02-20T09:28:57.395 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:28:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:28:57.267+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:57.268609+0000) 2024-02-20T09:28:57.395 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:28:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:28:57.136+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:28:57.395 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:28:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:28:57.136+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:28:57.395 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:28:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:28:57.136+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:28:57.395 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:28:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:28:57.137+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:28:57.395 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:28:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:28:57.270+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:57.271860+0000) 2024-02-20T09:28:57.975 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:28:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:28:57.871+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:57.873300+0000) 2024-02-20T09:28:57.975 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:28:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:28:57.856+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:28:57.975 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:28:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:28:57.856+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:28:57.975 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:28:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:28:57.856+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:28:57.976 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:28:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:28:57.857+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:28:57.976 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:28:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:28:57.857+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:28:57.976 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:28:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:28:57.857+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:28:58.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:28:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:28:57.984+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:57.985751+0000) 2024-02-20T09:28:58.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:28:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:28:58.099+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:58.100920+0000) 2024-02-20T09:28:58.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:28:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:28:58.151+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:28:58.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:28:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:28:58.151+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:28:58.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:28:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:28:58.151+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:28:58.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:28:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:28:58.151+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:28:58.874 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:28:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:28:58.811+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:28:58.875 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:28:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:28:58.811+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:28:58.875 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:28:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:28:58.811+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:28:59.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:28:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:28:58.984+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:58.985848+0000) 2024-02-20T09:28:59.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:28:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:28:58.875+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:58.876858+0000) 2024-02-20T09:28:59.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:28:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:28:59.117+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:28:59.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:28:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:28:59.117+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:28:59.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:28:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:28:59.117+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:28:59.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:28:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:28:59.117+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:28:59.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:28:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:28:58.872+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:28:59.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:28:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:28:58.872+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:28:59.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:28:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:28:58.872+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:28:59.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:28:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:28:59.099+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:59.101023+0000) 2024-02-20T09:28:59.876 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:28:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:28:59.761+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:28:59.876 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:28:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:28:59.761+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:28:59.877 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:28:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:28:59.761+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:29:00.141 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:28:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:28:59.878+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:59.879585+0000) 2024-02-20T09:29:00.142 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:28:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:28:59.984+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:28:59.985984+0000) 2024-02-20T09:29:00.142 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:28:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:28:59.874+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:29:00.142 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:28:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:28:59.874+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:29:00.143 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:28:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:28:59.874+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:29:00.143 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:29:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:29:00.099+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:00.101139+0000) 2024-02-20T09:29:00.482 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:29:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:29:00.139+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:29:00.482 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:29:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:29:00.139+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:29:00.483 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:29:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:29:00.139+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:29:00.483 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:29:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:29:00.139+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:29:01.102 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:29:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:29:00.880+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:00.881549+0000) 2024-02-20T09:29:01.103 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:29:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:29:00.828+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:29:01.103 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:29:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:29:00.828+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:29:01.103 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:29:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:29:00.828+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:29:01.103 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:29:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:29:00.782+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:29:01.103 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:29:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:29:00.782+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:29:01.103 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:29:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:29:00.782+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:29:01.103 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:29:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:29:00.984+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:00.986098+0000) 2024-02-20T09:29:01.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:29:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:29:01.099+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:01.101263+0000) 2024-02-20T09:29:01.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:29:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:29:01.151+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:29:01.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:29:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:29:01.151+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:29:01.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:29:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:29:01.151+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:29:01.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:29:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:29:01.151+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:29:01.849 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:29:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:29:01.770+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:29:01.849 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:29:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:29:01.770+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:29:01.849 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:29:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:29:01.770+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:29:02.102 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:29:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:29:01.882+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:01.883519+0000) 2024-02-20T09:29:02.103 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:29:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:29:01.984+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:01.986207+0000) 2024-02-20T09:29:02.104 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:29:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:29:01.847+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:29:02.104 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:29:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:29:01.847+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:29:02.104 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:29:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:29:01.847+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:29:02.411 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:29:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:29:02.281+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 70876 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:29:02.411 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:29:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:29:02.274+0000 7f8a5f86e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:02.275559+0000) 2024-02-20T09:29:02.412 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:29:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:29:02.100+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:02.101395+0000) 2024-02-20T09:29:02.412 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:29:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:29:02.274+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:02.275919+0000) 2024-02-20T09:29:02.412 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:29:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:29:02.149+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:29:02.412 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:29:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:29:02.149+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:29:02.412 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:29:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:29:02.149+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:29:02.412 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:29:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:29:02.150+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:29:02.413 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:29:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:29:02.277+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:02.279193+0000) 2024-02-20T09:29:02.975 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:29:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:29:02.882+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:02.884343+0000) 2024-02-20T09:29:02.976 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:29:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:29:02.742+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:29:02.976 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:29:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:29:02.742+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:29:02.976 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:29:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:29:02.742+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:29:02.976 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:29:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:29:02.896+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:29:02.976 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:29:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:29:02.896+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:29:02.977 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:29:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:29:02.896+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:29:03.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:29:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:29:02.984+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:02.986340+0000) 2024-02-20T09:29:03.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:29:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:29:03.100+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:03.101505+0000) 2024-02-20T09:29:03.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:29:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:29:03.140+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:29:03.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:29:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:29:03.140+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:29:03.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:29:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:29:03.140+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:29:03.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:29:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:29:03.140+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:29:03.732 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:29:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:29:03.706+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:29:03.732 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:29:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:29:03.706+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:29:03.732 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:29:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:29:03.706+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:29:03.987 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:29:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:29:03.885+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:03.886429+0000) 2024-02-20T09:29:03.988 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:29:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:29:03.921+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:29:03.988 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:29:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:29:03.921+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:29:03.988 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:29:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:29:03.921+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:29:04.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:29:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:29:03.985+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:03.986455+0000) 2024-02-20T09:29:04.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:29:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:29:04.100+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:04.101578+0000) 2024-02-20T09:29:04.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:29:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:29:04.132+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:29:04.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:29:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:29:04.132+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:29:04.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:29:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:29:04.132+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:29:04.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:29:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:29:04.132+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:29:04.885 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:29:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:29:04.712+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:29:04.885 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:29:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:29:04.712+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:29:04.886 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:29:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:29:04.712+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:29:05.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:29:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:29:04.985+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:04.986568+0000) 2024-02-20T09:29:05.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:29:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:29:04.883+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:29:05.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:29:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:29:04.883+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:29:05.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:29:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:29:04.883+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:29:05.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:29:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:29:05.100+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:05.101668+0000) 2024-02-20T09:29:05.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:29:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:29:04.887+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:04.888393+0000) 2024-02-20T09:29:05.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:29:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:29:05.100+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:29:05.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:29:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:29:05.100+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:29:05.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:29:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:29:05.100+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:29:05.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:29:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:29:05.100+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:29:05.837 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:29:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:29:05.734+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:29:05.837 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:29:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:29:05.734+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:29:05.838 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:29:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:29:05.734+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:29:06.098 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:29:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:29:05.985+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:05.986679+0000) 2024-02-20T09:29:06.098 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:29:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:29:05.834+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:29:06.098 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:29:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:29:05.835+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:29:06.098 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:29:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:29:05.835+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:29:06.099 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:29:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:29:05.889+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:05.890440+0000) 2024-02-20T09:29:06.099 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:29:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:29:06.095+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:29:06.099 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:29:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:29:06.095+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:29:06.099 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:29:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:29:06.095+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:29:06.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:29:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:29:06.095+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:29:06.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:29:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:29:06.100+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:06.101753+0000) 2024-02-20T09:29:06.893 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:29:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:29:06.728+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:29:06.893 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:29:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:29:06.728+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:29:06.894 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:29:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:29:06.728+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:29:06.894 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:29:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:29:06.811+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:29:06.894 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:29:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:29:06.811+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:29:06.894 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:29:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:29:06.811+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:29:07.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:29:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:29:06.985+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:06.986772+0000) 2024-02-20T09:29:07.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:29:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:29:07.100+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:07.101854+0000) 2024-02-20T09:29:07.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:29:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:29:06.891+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:06.892429+0000) 2024-02-20T09:29:07.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:29:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:29:07.112+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:29:07.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:29:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:29:07.113+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:29:07.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:29:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:29:07.113+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:29:07.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:29:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:29:07.113+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:29:07.583 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:29:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:29:07.282+0000 7f8a5f86e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:07.284033+0000) 2024-02-20T09:29:07.583 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:29:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:29:07.284+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:07.286182+0000) 2024-02-20T09:29:07.583 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:29:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:29:07.282+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:07.283858+0000) 2024-02-20T09:29:07.584 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:29:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:29:07.288+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 70915 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:29:07.894 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:29:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:29:07.820+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:29:07.895 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:29:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:29:07.820+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:29:07.895 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:29:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:29:07.820+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:29:07.895 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:29:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:29:07.708+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:29:07.895 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:29:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:29:07.708+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:29:07.895 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:29:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:29:07.708+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:29:08.163 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:29:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:29:07.985+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:07.986908+0000) 2024-02-20T09:29:08.163 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:29:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:29:08.100+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:08.101973+0000) 2024-02-20T09:29:08.163 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:29:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:29:07.892+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:07.893504+0000) 2024-02-20T09:29:08.164 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:29:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:29:08.160+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:29:08.164 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:29:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:29:08.160+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:29:08.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:29:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:29:08.160+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:29:08.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:29:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:29:08.160+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:29:08.897 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:29:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:29:08.797+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:29:08.897 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:29:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:29:08.797+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:29:08.897 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:29:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:29:08.797+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:29:08.898 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:29:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:29:08.686+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:29:08.898 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:29:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:29:08.686+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:29:08.898 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:29:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:29:08.686+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:29:09.162 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:29:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:29:08.985+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:08.987000+0000) 2024-02-20T09:29:09.162 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:29:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:29:09.100+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:09.102051+0000) 2024-02-20T09:29:09.162 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:29:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:29:08.894+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:08.896021+0000) 2024-02-20T09:29:09.163 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:29:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:29:09.159+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:29:09.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:29:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:29:09.159+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:29:09.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:29:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:29:09.159+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:29:09.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:29:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:29:09.159+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:29:09.886 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:29:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:29:09.775+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:29:09.886 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:29:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:29:09.775+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:29:09.886 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:29:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:29:09.775+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:29:09.886 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:29:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:29:09.711+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:29:09.886 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:29:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:29:09.711+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:29:09.887 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:29:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:29:09.711+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:29:10.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:29:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:29:09.985+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:09.987125+0000) 2024-02-20T09:29:10.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:29:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:29:10.100+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:10.102155+0000) 2024-02-20T09:29:10.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:29:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:29:09.897+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:09.898583+0000) 2024-02-20T09:29:10.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:29:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:29:10.127+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:29:10.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:29:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:29:10.127+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:29:10.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:29:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:29:10.127+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:29:10.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:29:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:29:10.127+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:29:10.742 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:29:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:29:10.666+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:29:10.742 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:29:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:29:10.666+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:29:10.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:29:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:29:10.666+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:29:10.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:29:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:29:10.728+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:29:10.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:29:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:29:10.728+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:29:10.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:29:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:29:10.728+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:29:11.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:29:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:29:10.985+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:10.987250+0000) 2024-02-20T09:29:11.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:29:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:29:11.100+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:11.102250+0000) 2024-02-20T09:29:11.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:29:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:29:10.899+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:10.900616+0000) 2024-02-20T09:29:11.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:29:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:29:11.133+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:29:11.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:29:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:29:11.133+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:29:11.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:29:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:29:11.133+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:29:11.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:29:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:29:11.134+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:29:11.904 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:29:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:29:11.625+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:29:11.904 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:29:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:29:11.625+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:29:11.904 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:29:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:29:11.625+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:29:11.904 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:29:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:29:11.737+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:29:11.904 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:29:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:29:11.737+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:29:11.905 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:29:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:29:11.737+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:29:12.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:29:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:29:12.101+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:12.102393+0000) 2024-02-20T09:29:12.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:29:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:29:11.985+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:11.987381+0000) 2024-02-20T09:29:12.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:29:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:29:11.901+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:11.902674+0000) 2024-02-20T09:29:12.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:29:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:29:12.126+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:29:12.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:29:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:29:12.126+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:29:12.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:29:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:29:12.126+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:29:12.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:29:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:29:12.126+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:29:12.589 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:29:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:29:12.296+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 70955 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:29:12.590 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:29:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:29:12.292+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:12.294305+0000) 2024-02-20T09:29:12.590 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:29:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:29:12.289+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:12.291285+0000) 2024-02-20T09:29:12.590 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:29:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:29:12.290+0000 7f8a5f86e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:12.291564+0000) 2024-02-20T09:29:12.905 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:29:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:29:12.600+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:29:12.905 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:29:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:29:12.600+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:29:12.905 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:29:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:29:12.600+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:29:12.905 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:29:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:29:12.770+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:29:12.906 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:29:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:29:12.770+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:29:12.906 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:29:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:29:12.770+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:29:13.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:29:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:29:12.986+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:12.987506+0000) 2024-02-20T09:29:13.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:29:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:29:13.101+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:13.102512+0000) 2024-02-20T09:29:13.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:29:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:29:12.902+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:12.904048+0000) 2024-02-20T09:29:13.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:29:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:29:13.090+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:29:13.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:29:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:29:13.090+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:29:13.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:29:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:29:13.090+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:29:13.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:29:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:29:13.091+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:29:13.907 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:29:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:29:13.595+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:29:13.907 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:29:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:29:13.595+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:29:13.907 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:29:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:29:13.595+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:29:13.908 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:29:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:29:13.790+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:29:13.908 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:29:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:29:13.790+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:29:13.908 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:29:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:29:13.790+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:29:14.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:29:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:29:13.986+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:13.987594+0000) 2024-02-20T09:29:14.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:29:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:29:14.101+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:14.102612+0000) 2024-02-20T09:29:14.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:29:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:29:13.904+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:13.906248+0000) 2024-02-20T09:29:14.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:29:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:29:14.106+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:29:14.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:29:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:29:14.106+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:29:14.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:29:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:29:14.106+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:29:14.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:29:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:29:14.106+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:29:14.909 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:29:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:29:14.626+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:29:14.910 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:29:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:29:14.626+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:29:14.910 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:29:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:29:14.626+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:29:14.910 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:29:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:29:14.822+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:29:14.910 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:29:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:29:14.822+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:29:14.910 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:29:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:29:14.822+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:29:15.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:29:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:29:14.986+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:14.987713+0000) 2024-02-20T09:29:15.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:29:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:29:15.101+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:15.102704+0000) 2024-02-20T09:29:15.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:29:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:29:14.907+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:14.908415+0000) 2024-02-20T09:29:15.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:29:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:29:15.145+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:29:15.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:29:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:29:15.145+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:29:15.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:29:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:29:15.145+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:29:15.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:29:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:29:15.145+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:29:15.912 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:29:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:29:15.607+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:29:15.912 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:29:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:29:15.607+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:29:15.912 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:29:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:29:15.607+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:29:15.913 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:29:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:29:15.792+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:29:15.913 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:29:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:29:15.792+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:29:15.913 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:29:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:29:15.792+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:29:16.172 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:29:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:29:15.986+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:15.987799+0000) 2024-02-20T09:29:16.173 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:29:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:29:16.101+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:16.102855+0000) 2024-02-20T09:29:16.173 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:29:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:29:15.909+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:15.910949+0000) 2024-02-20T09:29:16.173 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:29:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:29:16.170+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:29:16.173 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:29:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:29:16.170+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:29:16.173 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:29:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:29:16.170+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:29:16.174 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:29:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:29:16.170+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:29:16.914 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:29:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:29:16.651+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:29:16.914 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:29:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:29:16.651+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:29:16.915 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:29:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:29:16.651+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:29:16.915 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:29:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:29:16.764+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:29:16.915 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:29:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:29:16.764+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:29:16.915 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:29:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:29:16.764+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:29:17.210 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:29:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:29:16.986+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:16.987931+0000) 2024-02-20T09:29:17.210 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:29:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:29:17.101+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:17.102980+0000) 2024-02-20T09:29:17.211 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:29:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:29:16.912+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:16.913410+0000) 2024-02-20T09:29:17.211 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:29:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:29:17.208+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:29:17.211 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:29:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:29:17.208+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:29:17.211 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:29:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:29:17.208+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:29:17.211 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:29:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:29:17.208+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:29:17.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:29:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:29:17.300+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:17.301813+0000) 2024-02-20T09:29:17.493 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:29:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:29:17.304+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 70994 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:29:17.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:29:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:29:17.297+0000 7f8a5f86e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:17.299300+0000) 2024-02-20T09:29:17.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:29:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:29:17.298+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:17.299770+0000) 2024-02-20T09:29:17.916 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:29:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:29:17.661+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:29:17.916 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:29:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:29:17.662+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:29:17.916 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:29:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:29:17.662+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:29:17.916 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:29:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:29:17.790+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:29:17.917 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:29:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:29:17.790+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:29:17.917 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:29:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:29:17.790+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:29:18.173 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:29:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:29:17.986+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:17.988060+0000) 2024-02-20T09:29:18.173 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:29:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:29:18.101+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:18.103112+0000) 2024-02-20T09:29:18.173 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:29:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:29:17.913+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:17.914959+0000) 2024-02-20T09:29:18.174 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:29:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:29:18.170+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:29:18.174 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:29:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:29:18.170+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:29:18.174 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:29:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:29:18.170+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:29:18.174 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:29:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:29:18.170+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:29:18.919 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:29:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:29:18.757+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:29:18.919 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:29:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:29:18.757+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:29:18.919 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:29:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:29:18.757+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:29:18.919 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:29:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:29:18.647+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:29:18.919 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:29:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:29:18.647+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:29:18.919 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:29:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:29:18.647+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:29:19.169 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:29:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:29:18.986+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:18.988178+0000) 2024-02-20T09:29:19.170 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:29:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:29:18.916+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:18.917761+0000) 2024-02-20T09:29:19.170 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:29:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:29:19.101+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:19.103188+0000) 2024-02-20T09:29:19.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:29:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:29:19.167+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:29:19.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:29:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:29:19.167+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:29:19.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:29:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:29:19.167+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:29:19.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:29:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:29:19.167+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:29:19.886 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:29:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:29:19.620+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:29:19.886 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:29:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:29:19.620+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:29:19.886 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:29:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:29:19.620+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:29:19.886 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:29:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:29:19.743+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:29:19.887 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:29:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:29:19.743+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:29:19.887 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:29:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:29:19.743+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:29:20.175 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:29:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:29:19.986+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:19.988312+0000) 2024-02-20T09:29:20.176 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:29:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:29:20.101+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:20.103278+0000) 2024-02-20T09:29:20.177 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:29:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:29:19.918+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:19.920311+0000) 2024-02-20T09:29:20.177 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:29:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:29:20.172+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:29:20.177 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:29:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:29:20.172+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:29:20.177 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:29:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:29:20.172+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:29:20.177 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:29:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:29:20.172+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:29:20.742 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:29:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:29:20.629+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:29:20.742 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:29:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:29:20.629+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:29:20.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:29:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:29:20.629+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:29:20.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:29:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:29:20.699+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:29:20.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:29:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:29:20.699+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:29:20.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:29:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:29:20.699+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:29:21.179 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:29:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:29:20.987+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:20.988446+0000) 2024-02-20T09:29:21.179 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:29:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:29:21.101+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:21.103373+0000) 2024-02-20T09:29:21.180 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:29:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:29:20.921+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:20.922462+0000) 2024-02-20T09:29:21.180 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:29:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:29:21.176+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:29:21.180 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:29:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:29:21.176+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:29:21.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:29:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:29:21.176+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:29:21.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:29:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:29:21.176+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:29:21.912 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:29:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:29:21.675+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:29:21.913 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:29:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:29:21.675+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:29:21.913 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:29:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:29:21.675+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:29:21.913 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:29:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:29:21.668+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:29:21.913 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:29:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:29:21.668+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:29:21.913 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:29:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:29:21.668+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:29:22.214 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:29:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:29:21.987+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:21.988541+0000) 2024-02-20T09:29:22.215 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:29:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:29:22.102+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:22.103442+0000) 2024-02-20T09:29:22.215 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:29:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:29:21.923+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:21.924502+0000) 2024-02-20T09:29:22.215 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:29:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:29:22.212+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:29:22.216 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:29:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:29:22.212+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:29:22.216 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:29:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:29:22.212+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:29:22.492 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:29:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:29:22.312+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 71034 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:29:22.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:29:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:29:22.305+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:22.307236+0000) 2024-02-20T09:29:22.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:29:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:29:22.305+0000 7f8a5f86e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:22.306962+0000) 2024-02-20T09:29:22.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:29:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:29:22.212+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:29:22.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:29:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:29:22.308+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:22.310356+0000) 2024-02-20T09:29:22.926 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:29:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:29:22.688+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:29:22.926 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:29:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:29:22.688+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:29:22.927 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:29:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:29:22.688+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:29:22.927 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:29:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:29:22.676+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:29:22.927 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:29:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:29:22.676+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:29:22.927 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:29:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:29:22.676+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:29:23.220 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:29:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:29:22.987+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:22.988641+0000) 2024-02-20T09:29:23.220 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:29:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:29:23.102+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:23.103533+0000) 2024-02-20T09:29:23.221 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:29:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:29:22.924+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:22.925392+0000) 2024-02-20T09:29:23.221 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:29:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:29:23.217+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:29:23.221 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:29:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:29:23.217+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:29:23.221 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:29:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:29:23.217+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:29:23.221 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:29:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:29:23.218+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:29:23.930 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:29:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:29:23.649+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:29:23.930 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:29:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:29:23.649+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:29:23.930 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:29:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:29:23.649+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:29:23.930 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:29:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:29:23.679+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:29:23.930 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:29:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:29:23.679+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:29:23.931 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:29:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:29:23.679+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:29:24.180 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:29:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:29:23.987+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:23.988727+0000) 2024-02-20T09:29:24.180 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:29:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:29:24.102+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:24.103612+0000) 2024-02-20T09:29:24.180 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:29:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:29:23.927+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:23.928817+0000) 2024-02-20T09:29:24.180 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:29:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:29:24.177+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:29:24.181 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:29:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:29:24.177+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:29:24.181 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:29:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:29:24.177+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:29:24.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:29:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:29:24.177+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:29:24.933 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:29:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:29:24.704+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:29:24.933 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:29:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:29:24.704+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:29:24.933 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:29:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:29:24.704+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:29:24.934 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:29:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:29:24.691+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:29:24.934 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:29:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:29:24.691+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:29:24.934 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:29:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:29:24.691+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:29:25.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:29:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:29:24.987+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:24.988823+0000) 2024-02-20T09:29:25.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:29:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:29:25.102+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:25.103755+0000) 2024-02-20T09:29:25.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:29:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:29:24.930+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:24.931585+0000) 2024-02-20T09:29:25.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:29:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:29:25.169+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:29:25.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:29:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:29:25.169+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:29:25.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:29:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:29:25.169+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:29:25.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:29:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:29:25.169+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:29:25.935 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:29:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:29:25.707+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:29:25.935 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:29:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:29:25.707+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:29:25.935 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:29:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:29:25.707+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:29:25.935 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:29:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:29:25.662+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:29:25.935 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:29:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:29:25.662+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:29:25.936 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:29:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:29:25.662+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:29:26.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:29:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:29:25.987+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:25.988912+0000) 2024-02-20T09:29:26.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:29:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:29:26.102+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:26.103849+0000) 2024-02-20T09:29:26.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:29:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:29:25.932+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:25.933732+0000) 2024-02-20T09:29:26.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:29:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:29:26.136+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:29:26.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:29:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:29:26.136+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:29:26.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:29:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:29:26.136+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:29:26.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:29:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:29:26.136+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:29:26.937 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:29:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:29:26.695+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:29:26.938 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:29:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:29:26.695+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:29:26.938 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:29:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:29:26.695+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:29:26.938 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:29:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:29:26.636+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:29:26.938 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:29:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:29:26.636+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:29:26.938 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:29:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:29:26.636+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:29:27.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:29:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:29:26.987+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:26.989023+0000) 2024-02-20T09:29:27.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:29:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:29:27.102+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:27.103957+0000) 2024-02-20T09:29:27.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:29:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:29:26.934+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:26.935951+0000) 2024-02-20T09:29:27.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:29:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:29:27.098+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:29:27.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:29:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:29:27.098+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:29:27.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:29:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:29:27.098+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:29:27.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:29:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:29:27.098+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:29:27.595 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:29:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:29:27.314+0000 7f8a5f86e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:27.315466+0000) 2024-02-20T09:29:27.595 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:29:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:29:27.316+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:27.317765+0000) 2024-02-20T09:29:27.596 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:29:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:29:27.314+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:27.315730+0000) 2024-02-20T09:29:27.596 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:29:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:29:27.320+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 71073 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:29:27.938 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:29:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:29:27.714+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:29:27.938 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:29:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:29:27.714+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:29:27.938 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:29:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:29:27.714+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:29:27.938 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:29:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:29:27.592+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:29:27.939 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:29:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:29:27.592+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:29:27.939 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:29:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:29:27.592+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:29:28.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:29:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:29:27.987+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:27.989133+0000) 2024-02-20T09:29:28.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:29:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:29:28.102+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:28.104061+0000) 2024-02-20T09:29:28.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:29:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:29:27.935+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:27.936939+0000) 2024-02-20T09:29:28.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:29:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:29:28.113+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:29:28.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:29:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:29:28.113+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:29:28.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:29:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:29:28.113+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:29:28.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:29:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:29:28.113+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:29:28.940 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:29:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:29:28.747+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:29:28.941 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:29:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:29:28.747+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:29:28.941 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:29:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:29:28.747+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:29:28.941 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:29:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:29:28.643+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:29:28.941 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:29:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:29:28.643+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:29:28.941 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:29:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:29:28.643+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:29:29.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:29:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:29:28.988+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:28.989215+0000) 2024-02-20T09:29:29.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:29:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:29:29.102+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:29.104162+0000) 2024-02-20T09:29:29.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:29:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:29:28.938+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:28.939563+0000) 2024-02-20T09:29:29.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:29:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:29:29.156+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:29:29.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:29:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:29:29.156+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:29:29.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:29:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:29:29.156+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:29:29.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:29:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:29:29.156+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:29:29.885 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:29:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:29:29.789+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:29:29.886 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:29:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:29:29.789+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:29:29.886 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:29:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:29:29.789+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:29:29.886 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:29:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:29:29.684+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:29:29.886 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:29:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:29:29.684+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:29:29.886 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:29:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:29:29.684+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:29:30.185 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:29:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:29:29.987+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:29.989331+0000) 2024-02-20T09:29:30.186 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:29:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:29:29.940+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:29.942264+0000) 2024-02-20T09:29:30.186 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:29:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:29:30.102+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:30.104297+0000) 2024-02-20T09:29:30.485 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:29:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:29:30.183+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:29:30.485 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:29:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:29:30.183+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:29:30.485 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:29:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:29:30.183+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:29:30.485 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:29:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:29:30.183+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:29:30.735 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:29:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:29:30.687+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:29:30.735 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:29:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:29:30.687+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:29:30.735 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:29:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:29:30.687+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:29:30.990 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:29:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:29:30.944+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:30.944888+0000) 2024-02-20T09:29:30.990 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:29:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:29:30.765+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:29:30.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:29:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:29:30.765+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:29:30.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:29:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:29:30.765+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:29:31.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:29:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:29:30.989+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:30.989446+0000) 2024-02-20T09:29:31.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:29:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:29:31.104+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:31.104444+0000) 2024-02-20T09:29:31.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:29:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:29:31.192+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:29:31.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:29:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:29:31.192+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:29:31.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:29:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:29:31.192+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:29:31.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:29:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:29:31.192+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:29:31.948 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:29:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:29:31.754+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:29:31.949 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:29:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:29:31.754+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:29:31.949 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:29:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:29:31.754+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:29:31.949 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:29:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:29:31.724+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:29:31.949 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:29:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:29:31.724+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:29:31.949 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:29:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:29:31.724+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:29:32.226 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:29:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:29:31.989+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:31.989534+0000) 2024-02-20T09:29:32.227 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:29:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:29:32.104+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:32.104557+0000) 2024-02-20T09:29:32.227 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:29:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:29:31.947+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:31.947521+0000) 2024-02-20T09:29:32.227 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:29:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:29:32.225+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:29:32.227 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:29:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:29:32.225+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:29:32.228 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:29:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:29:32.225+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:29:32.492 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:29:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:29:32.323+0000 7f8a5f86e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:32.323654+0000) 2024-02-20T09:29:32.493 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:29:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:29:32.330+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 71113 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:29:32.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:29:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:29:32.323+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:32.324183+0000) 2024-02-20T09:29:32.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:29:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:29:32.225+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:29:32.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:29:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:29:32.326+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:32.326879+0000) 2024-02-20T09:29:32.949 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:29:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:29:32.679+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:29:32.950 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:29:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:29:32.679+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:29:32.950 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:29:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:29:32.679+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:29:32.950 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:29:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:29:32.775+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:29:32.950 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:29:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:29:32.775+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:29:32.950 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:29:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:29:32.775+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:29:33.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:29:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:29:32.948+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:32.948524+0000) 2024-02-20T09:29:33.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:29:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:29:33.104+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:33.104643+0000) 2024-02-20T09:29:33.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:29:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:29:32.989+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:32.989630+0000) 2024-02-20T09:29:33.613 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:29:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:29:33.264+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:29:33.613 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:29:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:29:33.264+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:29:33.613 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:29:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:29:33.264+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:29:33.613 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:29:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:29:33.264+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:29:33.951 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:29:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:29:33.677+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:29:33.952 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:29:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:29:33.677+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:29:33.952 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:29:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:29:33.677+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:29:33.952 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:29:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:29:33.734+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:29:33.952 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:29:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:29:33.734+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:29:33.953 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:29:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:29:33.734+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:29:34.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:29:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:29:33.989+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:33.989710+0000) 2024-02-20T09:29:34.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:29:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:29:33.950+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:33.950619+0000) 2024-02-20T09:29:34.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:29:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:29:34.104+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:34.104716+0000) 2024-02-20T09:29:34.615 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:29:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:29:34.249+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:29:34.616 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:29:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:29:34.249+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:29:34.616 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:29:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:29:34.249+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:29:34.616 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:29:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:29:34.249+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:29:34.954 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:29:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:29:34.686+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:29:34.954 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:29:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:29:34.686+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:29:34.954 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:29:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:29:34.686+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:29:34.955 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:29:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:29:34.640+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:29:34.955 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:29:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:29:34.640+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:29:34.955 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:29:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:29:34.640+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:29:35.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:29:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:29:34.989+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:34.989814+0000) 2024-02-20T09:29:35.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:29:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:29:34.952+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:34.953148+0000) 2024-02-20T09:29:35.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:29:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:29:35.104+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:35.104833+0000) 2024-02-20T09:29:35.615 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:29:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:29:35.259+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:29:35.615 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:29:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:29:35.259+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:29:35.616 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:29:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:29:35.259+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:29:35.616 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:29:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:29:35.259+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:29:35.956 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:29:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:29:35.725+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:29:35.956 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:29:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:29:35.725+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:29:35.956 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:29:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:29:35.725+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:29:35.957 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:29:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:29:35.683+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:29:35.957 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:29:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:29:35.683+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:29:35.957 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:29:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:29:35.683+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:29:36.238 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:29:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:29:35.989+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:35.989928+0000) 2024-02-20T09:29:36.238 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:29:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:29:36.104+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:36.104926+0000) 2024-02-20T09:29:36.238 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:29:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:29:35.954+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:35.955229+0000) 2024-02-20T09:29:36.239 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:29:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:29:36.236+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:29:36.239 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:29:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:29:36.236+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:29:36.239 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:29:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:29:36.236+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:29:36.239 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:29:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:29:36.236+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:29:36.960 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:29:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:29:36.678+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:29:36.960 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:29:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:29:36.678+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:29:36.961 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:29:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:29:36.678+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:29:36.961 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:29:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:29:36.774+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:29:36.961 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:29:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:29:36.774+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:29:36.961 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:29:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:29:36.774+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:29:37.239 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:29:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:29:36.989+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:36.990055+0000) 2024-02-20T09:29:37.240 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:29:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:29:37.104+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:37.105034+0000) 2024-02-20T09:29:37.240 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:29:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:29:36.958+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:36.958761+0000) 2024-02-20T09:29:37.240 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:29:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:29:37.238+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:29:37.240 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:29:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:29:37.238+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:29:37.240 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:29:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:29:37.238+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:29:37.492 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:29:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:29:37.338+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 71152 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:29:37.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:29:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:29:37.332+0000 7f8a5f86e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:37.332884+0000) 2024-02-20T09:29:37.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:29:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:29:37.332+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:37.332392+0000) 2024-02-20T09:29:37.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:29:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:29:37.238+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:29:37.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:29:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:29:37.334+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:37.334748+0000) 2024-02-20T09:29:37.961 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:29:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:29:37.812+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:29:37.961 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:29:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:29:37.812+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:29:37.961 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:29:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:29:37.812+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:29:37.962 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:29:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:29:37.646+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:29:37.962 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:29:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:29:37.646+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:29:37.962 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:29:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:29:37.646+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:29:38.213 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:29:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:29:37.989+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:37.990146+0000) 2024-02-20T09:29:38.213 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:29:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:29:37.959+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:37.960171+0000) 2024-02-20T09:29:38.214 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:29:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:29:38.104+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:38.105168+0000) 2024-02-20T09:29:38.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:29:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:29:38.211+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:29:38.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:29:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:29:38.211+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:29:38.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:29:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:29:38.211+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:29:38.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:29:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:29:38.211+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:29:38.964 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:29:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:29:38.617+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:29:38.964 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:29:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:29:38.617+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:29:38.964 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:29:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:29:38.617+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:29:38.964 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:29:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:29:38.829+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:29:38.964 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:29:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:29:38.829+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:29:38.965 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:29:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:29:38.829+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:29:39.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:29:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:29:38.989+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:38.990242+0000) 2024-02-20T09:29:39.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:29:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:29:39.104+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:39.105269+0000) 2024-02-20T09:29:39.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:29:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:29:38.962+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:38.962776+0000) 2024-02-20T09:29:39.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:29:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:29:39.203+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:29:39.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:29:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:29:39.203+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:29:39.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:29:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:29:39.203+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:29:39.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:29:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:29:39.203+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:29:39.886 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:29:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:29:39.818+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:29:39.886 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:29:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:29:39.818+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:29:39.886 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:29:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:29:39.818+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:29:39.886 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:29:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:29:39.664+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:29:39.887 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:29:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:29:39.664+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:29:39.887 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:29:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:29:39.664+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:29:40.168 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:29:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:29:39.990+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:39.990378+0000) 2024-02-20T09:29:40.169 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:29:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:29:39.964+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:39.965276+0000) 2024-02-20T09:29:40.169 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:29:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:29:40.104+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:40.105377+0000) 2024-02-20T09:29:40.486 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:29:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:29:40.167+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:29:40.486 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:29:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:29:40.167+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:29:40.486 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:29:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:29:40.167+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:29:40.486 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:29:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:29:40.167+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:29:40.736 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:29:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:29:40.682+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:29:40.736 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:29:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:29:40.682+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:29:40.737 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:29:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:29:40.682+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:29:40.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:29:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:29:40.967+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:40.967821+0000) 2024-02-20T09:29:40.992 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:29:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:29:40.858+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:29:40.992 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:29:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:29:40.858+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:29:40.992 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:29:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:29:40.858+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:29:41.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:29:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:29:41.105+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:41.105493+0000) 2024-02-20T09:29:41.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:29:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:29:40.990+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:40.990498+0000) 2024-02-20T09:29:41.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:29:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:29:41.163+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:29:41.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:29:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:29:41.163+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:29:41.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:29:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:29:41.163+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:29:41.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:29:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:29:41.163+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:29:41.900 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:29:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:29:41.719+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:29:41.900 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:29:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:29:41.719+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:29:41.900 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:29:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:29:41.719+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:29:42.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:29:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:29:41.990+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:41.990583+0000) 2024-02-20T09:29:42.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:29:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:29:41.969+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:41.969791+0000) 2024-02-20T09:29:42.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:29:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:29:42.123+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:29:42.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:29:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:29:42.123+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:29:42.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:29:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:29:42.123+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:29:42.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:29:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:29:42.123+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:29:42.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:29:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:29:41.898+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:29:42.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:29:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:29:41.898+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:29:42.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:29:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:29:41.898+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:29:42.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:29:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:29:42.105+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:42.105606+0000) 2024-02-20T09:29:42.621 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:29:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:29:42.346+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 71192 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:29:42.622 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:29:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:29:42.339+0000 7f8a5f86e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:42.340174+0000) 2024-02-20T09:29:42.623 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:29:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:29:42.343+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:42.343599+0000) 2024-02-20T09:29:42.623 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:29:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:29:42.339+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:42.340298+0000) 2024-02-20T09:29:42.943 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:29:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:29:42.738+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:29:42.943 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:29:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:29:42.738+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:29:42.943 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:29:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:29:42.738+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:29:43.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:29:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:29:42.990+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:42.990715+0000) 2024-02-20T09:29:43.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:29:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:29:42.941+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:29:43.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:29:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:29:42.941+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:29:43.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:29:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:29:42.941+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:29:43.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:29:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:29:43.105+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:43.105722+0000) 2024-02-20T09:29:43.245 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:29:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:29:42.970+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:42.970707+0000) 2024-02-20T09:29:43.245 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:29:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:29:43.117+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:29:43.245 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:29:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:29:43.117+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:29:43.245 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:29:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:29:43.117+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:29:43.245 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:29:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:29:43.117+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:29:43.973 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:29:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:29:43.689+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:29:43.973 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:29:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:29:43.689+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:29:43.973 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:29:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:29:43.689+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:29:44.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:29:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:29:43.990+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:43.990853+0000) 2024-02-20T09:29:44.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:29:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:29:43.971+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:29:44.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:29:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:29:43.971+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:29:44.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:29:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:29:43.971+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:29:44.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:29:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:29:44.105+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:44.105826+0000) 2024-02-20T09:29:44.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:29:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:29:43.972+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:43.972686+0000) 2024-02-20T09:29:44.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:29:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:29:44.111+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:29:44.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:29:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:29:44.111+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:29:44.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:29:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:29:44.111+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:29:44.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:29:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:29:44.111+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:29:44.976 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:29:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:29:44.739+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:29:44.976 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:29:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:29:44.739+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:29:44.976 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:29:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:29:44.739+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:29:45.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:29:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:29:44.990+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:44.991027+0000) 2024-02-20T09:29:45.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:29:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:29:44.993+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:29:45.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:29:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:29:44.993+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:29:45.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:29:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:29:44.993+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:29:45.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:29:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:29:45.105+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:45.105941+0000) 2024-02-20T09:29:45.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:29:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:29:44.974+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:44.974685+0000) 2024-02-20T09:29:45.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:29:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:29:45.121+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:29:45.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:29:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:29:45.121+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:29:45.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:29:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:29:45.121+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:29:45.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:29:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:29:45.121+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:29:45.960 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:29:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:29:45.776+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:29:45.961 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:29:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:29:45.776+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:29:45.961 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:29:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:29:45.776+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:29:46.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:29:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:29:45.990+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:45.991214+0000) 2024-02-20T09:29:46.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:29:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:29:45.959+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:29:46.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:29:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:29:45.959+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:29:46.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:29:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:29:45.959+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:29:46.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:29:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:29:46.105+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:46.106062+0000) 2024-02-20T09:29:46.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:29:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:29:45.976+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:45.977350+0000) 2024-02-20T09:29:46.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:29:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:29:46.131+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:29:46.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:29:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:29:46.131+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:29:46.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:29:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:29:46.131+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:29:46.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:29:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:29:46.131+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:29:46.980 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:29:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:29:46.735+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:29:46.981 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:29:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:29:46.735+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:29:46.981 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:29:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:29:46.735+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:29:47.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:29:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:29:46.990+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:46.991372+0000) 2024-02-20T09:29:47.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:29:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:29:47.003+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:29:47.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:29:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:29:47.003+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:29:47.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:29:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:29:47.003+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:29:47.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:29:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:29:47.105+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:47.106175+0000) 2024-02-20T09:29:47.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:29:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:29:46.978+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:46.979387+0000) 2024-02-20T09:29:47.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:29:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:29:47.165+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:29:47.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:29:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:29:47.165+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:29:47.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:29:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:29:47.165+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:29:47.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:29:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:29:47.165+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:29:47.626 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:29:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:29:47.354+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 71231 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:29:47.626 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:29:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:29:47.347+0000 7f8a5f86e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:47.348196+0000) 2024-02-20T09:29:47.627 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:29:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:29:47.350+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:47.351375+0000) 2024-02-20T09:29:47.627 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:29:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:29:47.348+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:47.348436+0000) 2024-02-20T09:29:47.980 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:29:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:29:47.765+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:29:47.981 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:29:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:29:47.765+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:29:47.981 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:29:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:29:47.765+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:29:48.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:29:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:29:47.991+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:47.991470+0000) 2024-02-20T09:29:48.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:29:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:29:48.006+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:29:48.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:29:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:29:48.006+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:29:48.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:29:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:29:48.006+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:29:48.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:29:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:29:48.105+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:48.106289+0000) 2024-02-20T09:29:48.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:29:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:29:47.980+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:47.980906+0000) 2024-02-20T09:29:48.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:29:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:29:48.196+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:29:48.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:29:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:29:48.196+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:29:48.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:29:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:29:48.196+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:29:48.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:29:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:29:48.196+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:29:48.980 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:29:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:29:48.810+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:29:48.981 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:29:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:29:48.810+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:29:48.981 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:29:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:29:48.810+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:29:49.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:29:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:29:48.991+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:48.991550+0000) 2024-02-20T09:29:49.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:29:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:29:48.982+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:48.982968+0000) 2024-02-20T09:29:49.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:29:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:29:49.181+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:29:49.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:29:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:29:49.181+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:29:49.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:29:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:29:49.181+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:29:49.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:29:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:29:49.181+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:29:49.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:29:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:29:49.042+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:29:49.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:29:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:29:49.042+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:29:49.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:29:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:29:49.042+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:29:49.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:29:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:29:49.106+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:49.106427+0000) 2024-02-20T09:29:49.885 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:29:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:29:49.812+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:29:49.885 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:29:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:29:49.812+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:29:49.885 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:29:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:29:49.812+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:29:50.149 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:29:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:29:49.991+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:49.991635+0000) 2024-02-20T09:29:50.150 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:29:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:29:49.984+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:49.984903+0000) 2024-02-20T09:29:50.150 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:29:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:29:50.147+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:29:50.150 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:29:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:29:50.005+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:29:50.150 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:29:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:29:50.005+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:29:50.150 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:29:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:29:50.005+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:29:50.150 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:29:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:29:50.106+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:50.106571+0000) 2024-02-20T09:29:50.486 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:29:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:29:50.147+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:29:50.486 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:29:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:29:50.148+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:29:50.486 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:29:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:29:50.148+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:29:50.988 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:29:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:29:50.977+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:29:50.988 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:29:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:29:50.977+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:29:50.988 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:29:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:29:50.977+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:29:50.989 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:29:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:29:50.811+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:29:50.989 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:29:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:29:50.811+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:29:50.989 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:29:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:29:50.811+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:29:51.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:29:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:29:50.991+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:50.991751+0000) 2024-02-20T09:29:51.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:29:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:29:51.106+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:51.106668+0000) 2024-02-20T09:29:51.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:29:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:29:50.986+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:50.987111+0000) 2024-02-20T09:29:51.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:29:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:29:51.183+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:29:51.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:29:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:29:51.183+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:29:51.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:29:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:29:51.183+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:29:51.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:29:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:29:51.183+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:29:51.944 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:29:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:29:51.800+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:29:51.944 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:29:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:29:51.800+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:29:51.944 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:29:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:29:51.800+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:29:52.220 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:29:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:29:51.991+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:51.991855+0000) 2024-02-20T09:29:52.220 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:29:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:29:51.942+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:29:52.221 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:29:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:29:51.942+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:29:52.221 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:29:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:29:51.942+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:29:52.221 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:29:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:29:52.106+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:52.106764+0000) 2024-02-20T09:29:52.221 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:29:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:29:51.988+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:51.989243+0000) 2024-02-20T09:29:52.221 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:29:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:29:52.218+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:29:52.221 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:29:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:29:52.218+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:29:52.222 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:29:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:29:52.218+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:29:52.222 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:29:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:29:52.218+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:29:52.482 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:29:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:29:52.362+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 71271 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:29:52.483 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:29:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:29:52.356+0000 7f8a5f86e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:52.356473+0000) 2024-02-20T09:29:52.483 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:29:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:29:52.359+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:52.360205+0000) 2024-02-20T09:29:52.484 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:29:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:29:52.355+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:52.356070+0000) 2024-02-20T09:29:52.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:29:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:29:52.756+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:29:52.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:29:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:29:52.756+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:29:52.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:29:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:29:52.756+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:29:52.992 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:29:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:29:52.944+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:29:52.992 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:29:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:29:52.944+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:29:52.992 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:29:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:29:52.944+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:29:53.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:29:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:29:53.106+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:53.106947+0000) 2024-02-20T09:29:53.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:29:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:29:52.991+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:52.992011+0000) 2024-02-20T09:29:53.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:29:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:29:52.989+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:52.990309+0000) 2024-02-20T09:29:53.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:29:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:29:53.213+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:29:53.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:29:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:29:53.213+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:29:53.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:29:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:29:53.213+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:29:53.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:29:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:29:53.213+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:29:53.925 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:29:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:29:53.722+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:29:53.925 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:29:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:29:53.722+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:29:53.925 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:29:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:29:53.722+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:29:54.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:29:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:29:53.990+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:53.992099+0000) 2024-02-20T09:29:54.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:29:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:29:53.922+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:29:54.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:29:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:29:53.922+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:29:54.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:29:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:29:53.922+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:29:54.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:29:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:29:54.105+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:54.107099+0000) 2024-02-20T09:29:54.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:29:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:29:53.991+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:53.992934+0000) 2024-02-20T09:29:54.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:29:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:29:54.168+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:29:54.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:29:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:29:54.168+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:29:54.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:29:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:29:54.168+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:29:54.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:29:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:29:54.168+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:29:54.932 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:29:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:29:54.768+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:29:54.932 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:29:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:29:54.768+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:29:54.932 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:29:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:29:54.768+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:29:55.214 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:29:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:29:54.990+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:54.992196+0000) 2024-02-20T09:29:55.215 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:29:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:29:54.929+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:29:55.215 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:29:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:29:54.929+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:29:55.215 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:29:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:29:54.929+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:29:55.215 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:29:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:29:55.105+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:55.107247+0000) 2024-02-20T09:29:55.216 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:29:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:29:54.993+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:54.994944+0000) 2024-02-20T09:29:55.216 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:29:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:29:55.211+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:29:55.216 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:29:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:29:55.211+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:29:55.216 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:29:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:29:55.211+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:29:55.216 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:29:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:29:55.211+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:29:55.919 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:29:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:29:55.778+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:29:55.920 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:29:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:29:55.778+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:29:55.920 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:29:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:29:55.778+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:29:56.178 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:29:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:29:55.990+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:55.992317+0000) 2024-02-20T09:29:56.179 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:29:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:29:55.917+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:29:56.179 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:29:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:29:55.917+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:29:56.180 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:29:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:29:55.917+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:29:56.180 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:29:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:29:56.105+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:56.107346+0000) 2024-02-20T09:29:56.180 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:29:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:29:55.995+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:55.997352+0000) 2024-02-20T09:29:56.181 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:29:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:29:56.175+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:29:56.181 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:29:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:29:56.175+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:29:56.181 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:29:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:29:56.175+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:29:56.181 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:29:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:29:56.176+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:29:56.982 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:29:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:29:56.878+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:29:56.982 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:29:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:29:56.878+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:29:56.982 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:29:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:29:56.878+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:29:56.982 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:29:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:29:56.826+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:29:56.982 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:29:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:29:56.826+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:29:56.983 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:29:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:29:56.826+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:29:57.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:29:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:29:56.991+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:56.992428+0000) 2024-02-20T09:29:57.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:29:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:29:57.106+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:57.107414+0000) 2024-02-20T09:29:57.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:29:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:29:56.998+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:56.999557+0000) 2024-02-20T09:29:57.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:29:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:29:57.130+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:29:57.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:29:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:29:57.130+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:29:57.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:29:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:29:57.130+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:29:57.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:29:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:29:57.130+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:29:57.636 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:29:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:29:57.369+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 71310 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:29:57.637 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:29:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:29:57.363+0000 7f8a5f86e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:57.364800+0000) 2024-02-20T09:29:57.637 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:29:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:29:57.366+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:57.367608+0000) 2024-02-20T09:29:57.637 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:29:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:29:57.363+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:57.364975+0000) 2024-02-20T09:29:57.916 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:29:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:29:57.779+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:29:57.916 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:29:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:29:57.779+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:29:57.916 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:29:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:29:57.779+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:29:58.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:29:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:29:57.991+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:57.992547+0000) 2024-02-20T09:29:58.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:29:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:29:57.999+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:58.000498+0000) 2024-02-20T09:29:58.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:29:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:29:58.110+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:29:58.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:29:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:29:58.110+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:29:58.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:29:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:29:58.110+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:29:58.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:29:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:29:58.110+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:29:58.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:29:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:29:57.913+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:29:58.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:29:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:29:57.913+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:29:58.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:29:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:29:57.913+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:29:58.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:29:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:29:58.106+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:58.107507+0000) 2024-02-20T09:29:58.901 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:29:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:29:58.792+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:29:58.901 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:29:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:29:58.792+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:29:58.902 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:29:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:29:58.792+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:29:59.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:29:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:29:58.991+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:58.992629+0000) 2024-02-20T09:29:59.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:29:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:29:58.898+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:29:59.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:29:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:29:58.898+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:29:59.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:29:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:29:58.898+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:29:59.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:29:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:29:59.106+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:59.107590+0000) 2024-02-20T09:29:59.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:29:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:29:59.001+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:59.002987+0000) 2024-02-20T09:29:59.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:29:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:29:59.090+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:29:59.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:29:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:29:59.090+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:29:59.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:29:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:29:59.090+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:29:59.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:29:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:29:59.090+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:29:59.982 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:29:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:29:59.771+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:29:59.982 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:29:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:29:59.771+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:29:59.982 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:29:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:29:59.771+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:29:59.983 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:29:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:29:59.855+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:29:59.983 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:29:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:29:59.855+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:29:59.983 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:29:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:29:59.855+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:30:00.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:29:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:29:59.991+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:29:59.992720+0000) 2024-02-20T09:30:00.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:30:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:30:00.106+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:00.107687+0000) 2024-02-20T09:30:00.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:30:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:30:00.004+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:00.005662+0000) 2024-02-20T09:30:00.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:30:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:30:00.116+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:30:00.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:30:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:30:00.116+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:30:00.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:30:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:30:00.116+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:30:00.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:30:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:30:00.117+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:30:00.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:30:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:30:00.819+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:30:00.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:30:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:30:00.819+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:30:00.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:30:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:30:00.819+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:30:00.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:30:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:30:00.811+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:30:00.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:30:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:30:00.811+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:30:00.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:30:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:30:00.811+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:30:01.492 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:30:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:30:00.991+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:00.992810+0000) 2024-02-20T09:30:01.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:30:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:30:01.106+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:01.107807+0000) 2024-02-20T09:30:01.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:30:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:30:01.007+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:01.008410+0000) 2024-02-20T09:30:01.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:30:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:30:01.104+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:30:01.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:30:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:30:01.104+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:30:01.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:30:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:30:01.104+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:30:01.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:30:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:30:01.104+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:30:01.982 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:30:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:30:01.860+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:30:01.983 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:30:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:30:01.860+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:30:01.983 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:30:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:30:01.860+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:30:01.983 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:30:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:30:01.833+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:30:01.984 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:30:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:30:01.833+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:30:01.984 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:30:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:30:01.833+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:30:02.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:30:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:30:01.991+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:01.992932+0000) 2024-02-20T09:30:02.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:30:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:30:02.106+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:02.107908+0000) 2024-02-20T09:30:02.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:30:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:30:02.009+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:02.010512+0000) 2024-02-20T09:30:02.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:30:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:30:02.121+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:30:02.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:30:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:30:02.121+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:30:02.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:30:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:30:02.121+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:30:02.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:30:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:30:02.121+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:30:02.637 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:30:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:30:02.377+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 71350 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:30:02.638 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:30:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:30:02.371+0000 7f8a5f86e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:02.372900+0000) 2024-02-20T09:30:02.638 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:30:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:30:02.373+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:02.375019+0000) 2024-02-20T09:30:02.639 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:30:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:30:02.370+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:02.372275+0000) 2024-02-20T09:30:02.982 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:30:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:30:02.873+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:30:02.983 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:30:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:30:02.873+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:30:02.983 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:30:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:30:02.873+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:30:02.983 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:30:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:30:02.819+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:30:02.983 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:30:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:30:02.819+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:30:02.983 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:30:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:30:02.819+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:30:03.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:30:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:30:02.991+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:02.993072+0000) 2024-02-20T09:30:03.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:30:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:30:03.106+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:03.108027+0000) 2024-02-20T09:30:03.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:30:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:30:03.010+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:03.011463+0000) 2024-02-20T09:30:03.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:30:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:30:03.083+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:30:03.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:30:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:30:03.083+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:30:03.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:30:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:30:03.083+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:30:03.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:30:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:30:03.083+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:30:03.982 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:30:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:30:03.859+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:30:03.983 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:30:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:30:03.859+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:30:03.983 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:30:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:30:03.859+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:30:03.983 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:30:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:30:03.874+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:30:03.983 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:30:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:30:03.874+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:30:03.983 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:30:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:30:03.874+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:30:04.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:30:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:30:03.991+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:03.993167+0000) 2024-02-20T09:30:04.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:30:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:30:04.106+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:04.108102+0000) 2024-02-20T09:30:04.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:30:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:30:04.012+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:04.013445+0000) 2024-02-20T09:30:04.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:30:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:30:04.117+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:30:04.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:30:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:30:04.117+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:30:04.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:30:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:30:04.117+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:30:04.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:30:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:30:04.117+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:30:04.898 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:30:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:30:04.875+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:30:04.898 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:30:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:30:04.875+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:30:04.899 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:30:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:30:04.875+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:30:05.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:30:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:30:05.106+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:05.108234+0000) 2024-02-20T09:30:05.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:30:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:30:04.895+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:30:05.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:30:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:30:04.895+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:30:05.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:30:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:30:04.895+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:30:05.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:30:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:30:04.991+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:04.993282+0000) 2024-02-20T09:30:05.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:30:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:30:05.014+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:05.015594+0000) 2024-02-20T09:30:05.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:30:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:30:05.079+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:30:05.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:30:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:30:05.079+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:30:05.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:30:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:30:05.079+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:30:05.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:30:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:30:05.079+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:30:05.890 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:30:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:30:05.874+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:30:05.890 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:30:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:30:05.874+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:30:05.890 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:30:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:30:05.874+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:30:06.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:30:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:30:06.106+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:06.108388+0000) 2024-02-20T09:30:06.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:30:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:30:05.887+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:30:06.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:30:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:30:05.887+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:30:06.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:30:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:30:05.887+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:30:06.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:30:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:30:05.992+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:05.993391+0000) 2024-02-20T09:30:06.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:30:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:30:06.016+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:06.017599+0000) 2024-02-20T09:30:06.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:30:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:30:06.121+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:30:06.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:30:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:30:06.121+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:30:06.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:30:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:30:06.121+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:30:06.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:30:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:30:06.121+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:30:06.918 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:30:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:30:06.879+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:30:06.918 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:30:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:30:06.879+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:30:06.918 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:30:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:30:06.879+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:30:07.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:30:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:30:07.107+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:07.108509+0000) 2024-02-20T09:30:07.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:30:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:30:06.915+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:30:07.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:30:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:30:06.915+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:30:07.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:30:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:30:06.915+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:30:07.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:30:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:30:06.992+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:06.993503+0000) 2024-02-20T09:30:07.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:30:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:30:07.018+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:07.019611+0000) 2024-02-20T09:30:07.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:30:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:30:07.082+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:30:07.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:30:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:30:07.082+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:30:07.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:30:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:30:07.082+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:30:07.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:30:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:30:07.082+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:30:07.638 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:30:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:30:07.385+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 71389 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:30:07.638 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:30:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:30:07.378+0000 7f8a5f86e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:07.380059+0000) 2024-02-20T09:30:07.639 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:30:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:30:07.381+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:07.382558+0000) 2024-02-20T09:30:07.639 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:30:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:30:07.378+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:07.380270+0000) 2024-02-20T09:30:07.915 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:30:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:30:07.869+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:30:07.916 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:30:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:30:07.869+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:30:07.916 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:30:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:30:07.869+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:30:08.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:30:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:30:08.107+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:08.108597+0000) 2024-02-20T09:30:08.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:30:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:30:07.913+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:30:08.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:30:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:30:07.913+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:30:08.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:30:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:30:07.913+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:30:08.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:30:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:30:07.992+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:07.993599+0000) 2024-02-20T09:30:08.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:30:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:30:08.019+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:08.020583+0000) 2024-02-20T09:30:08.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:30:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:30:08.044+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:30:08.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:30:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:30:08.044+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:30:08.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:30:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:30:08.044+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:30:08.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:30:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:30:08.044+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:30:08.983 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:30:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:30:08.822+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:30:08.983 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:30:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:30:08.822+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:30:08.983 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:30:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:30:08.822+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:30:08.984 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:30:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:30:08.873+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:30:08.984 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:30:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:30:08.873+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:30:08.984 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:30:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:30:08.873+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:30:09.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:30:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:30:08.992+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:08.993705+0000) 2024-02-20T09:30:09.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:30:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:30:09.107+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:09.108697+0000) 2024-02-20T09:30:09.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:30:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:30:09.021+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:09.023086+0000) 2024-02-20T09:30:09.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:30:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:30:09.075+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:30:09.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:30:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:30:09.075+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:30:09.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:30:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:30:09.075+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:30:09.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:30:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:30:09.075+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:30:09.921 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:30:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:30:09.791+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:30:09.921 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:30:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:30:09.791+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:30:09.921 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:30:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:30:09.791+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:30:10.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:30:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:30:10.107+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:10.108829+0000) 2024-02-20T09:30:10.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:30:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:30:09.918+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:30:10.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:30:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:30:09.918+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:30:10.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:30:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:30:09.918+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:30:10.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:30:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:30:09.992+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:09.993807+0000) 2024-02-20T09:30:10.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:30:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:30:10.024+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:10.025576+0000) 2024-02-20T09:30:10.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:30:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:30:10.058+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:30:10.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:30:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:30:10.058+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:30:10.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:30:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:30:10.058+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:30:10.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:30:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:30:10.058+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:30:10.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:30:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:30:10.953+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:30:10.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:30:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:30:10.953+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:30:10.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:30:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:30:10.953+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:30:10.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:30:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:30:10.786+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:30:10.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:30:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:30:10.786+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:30:10.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:30:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:30:10.786+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:30:11.492 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:30:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:30:10.992+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:10.993908+0000) 2024-02-20T09:30:11.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:30:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:30:11.107+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:11.109039+0000) 2024-02-20T09:30:11.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:30:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:30:11.026+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:11.027686+0000) 2024-02-20T09:30:11.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:30:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:30:11.063+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:30:11.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:30:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:30:11.063+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:30:11.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:30:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:30:11.063+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:30:11.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:30:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:30:11.063+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:30:11.983 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:30:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:30:11.746+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:30:11.983 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:30:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:30:11.746+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:30:11.984 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:30:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:30:11.746+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:30:12.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:30:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:30:12.107+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:12.109236+0000) 2024-02-20T09:30:12.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:30:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:30:11.981+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:30:12.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:30:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:30:11.981+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:30:12.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:30:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:30:11.981+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:30:12.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:30:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:30:11.992+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:11.994028+0000) 2024-02-20T09:30:12.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:30:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:30:12.028+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:12.029780+0000) 2024-02-20T09:30:12.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:30:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:30:12.041+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:30:12.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:30:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:30:12.041+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:30:12.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:30:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:30:12.041+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:30:12.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:30:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:30:12.041+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:30:12.652 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:30:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:30:12.393+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 71429 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:30:12.653 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:30:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:30:12.387+0000 7f8a5f86e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:12.388774+0000) 2024-02-20T09:30:12.653 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:30:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:30:12.389+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:12.390804+0000) 2024-02-20T09:30:12.653 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:30:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:30:12.386+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:12.388324+0000) 2024-02-20T09:30:12.973 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:30:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:30:12.792+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:30:12.973 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:30:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:30:12.792+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:30:12.973 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:30:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:30:12.792+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:30:13.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:30:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:30:13.108+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:13.109420+0000) 2024-02-20T09:30:13.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:30:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:30:12.970+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:30:13.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:30:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:30:12.970+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:30:13.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:30:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:30:12.970+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:30:13.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:30:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:30:12.992+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:12.994155+0000) 2024-02-20T09:30:13.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:30:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:30:13.009+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:30:13.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:30:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:30:13.009+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:30:13.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:30:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:30:13.009+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:30:13.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:30:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:30:13.009+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:30:13.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:30:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:30:13.030+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:13.031541+0000) 2024-02-20T09:30:13.932 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:30:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:30:13.783+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:30:13.933 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:30:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:30:13.784+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:30:13.933 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:30:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:30:13.784+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:30:14.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:30:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:30:14.108+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:14.109558+0000) 2024-02-20T09:30:14.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:30:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:30:13.930+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:30:14.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:30:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:30:13.930+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:30:14.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:30:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:30:13.930+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:30:14.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:30:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:30:13.992+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:13.994260+0000) 2024-02-20T09:30:14.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:30:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:30:14.014+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:30:14.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:30:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:30:14.014+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:30:14.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:30:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:30:14.014+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:30:14.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:30:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:30:14.014+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:30:14.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:30:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:30:14.032+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:14.033749+0000) 2024-02-20T09:30:14.950 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:30:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:30:14.748+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:30:14.950 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:30:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:30:14.748+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:30:14.950 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:30:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:30:14.748+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:30:15.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:30:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:30:15.108+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:15.109729+0000) 2024-02-20T09:30:15.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:30:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:30:14.947+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:30:15.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:30:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:30:14.947+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:30:15.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:30:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:30:14.947+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:30:15.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:30:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:30:14.992+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:14.994384+0000) 2024-02-20T09:30:15.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:30:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:30:14.977+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:30:15.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:30:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:30:14.977+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:30:15.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:30:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:30:14.977+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:30:15.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:30:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:30:14.977+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:30:15.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:30:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:30:15.034+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:15.035931+0000) 2024-02-20T09:30:15.968 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:30:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:30:15.784+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:30:15.968 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:30:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:30:15.784+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:30:15.968 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:30:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:30:15.784+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:30:16.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:30:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:30:16.108+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:16.109885+0000) 2024-02-20T09:30:16.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:30:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:30:15.965+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:30:16.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:30:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:30:15.965+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:30:16.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:30:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:30:15.965+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:30:16.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:30:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:30:15.993+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:15.994486+0000) 2024-02-20T09:30:16.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:30:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:30:15.969+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:30:16.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:30:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:30:15.969+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:30:16.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:30:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:30:15.969+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:30:16.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:30:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:30:15.969+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:30:16.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:30:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:30:16.036+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:16.037990+0000) 2024-02-20T09:30:16.932 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:30:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:30:16.824+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:30:16.932 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:30:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:30:16.824+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:30:16.932 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:30:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:30:16.824+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:30:17.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:30:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:30:17.108+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:17.110007+0000) 2024-02-20T09:30:17.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:30:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:30:16.929+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:30:17.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:30:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:30:16.929+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:30:17.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:30:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:30:16.929+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:30:17.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:30:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:30:16.930+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:30:17.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:30:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:30:17.038+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:17.040035+0000) 2024-02-20T09:30:17.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:30:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:30:16.963+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:30:17.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:30:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:30:16.963+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:30:17.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:30:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:30:16.963+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:30:17.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:30:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:30:16.993+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:16.994575+0000) 2024-02-20T09:30:17.658 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:30:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:30:17.395+0000 7f8a5f86e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:17.396551+0000) 2024-02-20T09:30:17.659 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:30:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:30:17.398+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:17.400344+0000) 2024-02-20T09:30:17.659 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:30:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:30:17.395+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:17.396919+0000) 2024-02-20T09:30:17.660 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:30:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:30:17.401+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 71468 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:30:17.984 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:30:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:30:17.836+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:30:17.985 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:30:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:30:17.836+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:30:17.985 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:30:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:30:17.836+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:30:17.985 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:30:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:30:17.894+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:30:17.985 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:30:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:30:17.894+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:30:17.985 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:30:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:30:17.894+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:30:17.986 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:30:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:30:17.894+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:30:18.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:30:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:30:18.039+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:18.041043+0000) 2024-02-20T09:30:18.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:30:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:30:18.108+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:18.110148+0000) 2024-02-20T09:30:18.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:30:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:30:17.992+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:30:18.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:30:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:30:17.992+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:30:18.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:30:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:30:17.992+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:30:18.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:30:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:30:17.993+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:17.994661+0000) 2024-02-20T09:30:18.908 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:30:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:30:18.821+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:30:18.908 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:30:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:30:18.821+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:30:18.908 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:30:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:30:18.821+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:30:19.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:30:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:30:19.108+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:19.110249+0000) 2024-02-20T09:30:19.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:30:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:30:18.993+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:18.994752+0000) 2024-02-20T09:30:19.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:30:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:30:19.034+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:30:19.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:30:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:30:19.034+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:30:19.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:30:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:30:19.034+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:30:19.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:30:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:30:18.905+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:30:19.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:30:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:30:18.905+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:30:19.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:30:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:30:18.905+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:30:19.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:30:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:30:18.905+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:30:19.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:30:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:30:19.042+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:19.043430+0000) 2024-02-20T09:30:19.899 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:30:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:30:19.823+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:30:19.899 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:30:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:30:19.823+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:30:19.899 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:30:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:30:19.823+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:30:20.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:30:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:30:20.108+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:20.110344+0000) 2024-02-20T09:30:20.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:30:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:30:19.896+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:30:20.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:30:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:30:19.896+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:30:20.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:30:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:30:19.896+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:30:20.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:30:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:30:19.896+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:30:20.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:30:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:30:20.044+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:20.046344+0000) 2024-02-20T09:30:20.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:30:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:30:19.993+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:19.994865+0000) 2024-02-20T09:30:20.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:30:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:30:20.045+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:30:20.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:30:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:30:20.045+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:30:20.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:30:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:30:20.045+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:30:20.992 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:30:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:30:20.778+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:30:20.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:30:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:30:20.778+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:30:20.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:30:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:30:20.778+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:30:20.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:30:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:30:20.932+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:30:20.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:30:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:30:20.932+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:30:20.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:30:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:30:20.932+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:30:20.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:30:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:30:20.932+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:30:21.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:30:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:30:21.046+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:21.048353+0000) 2024-02-20T09:30:21.494 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:30:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:30:21.109+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:21.110494+0000) 2024-02-20T09:30:21.494 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:30:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:30:20.993+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:20.994970+0000) 2024-02-20T09:30:21.495 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:30:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:30:21.037+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:30:21.495 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:30:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:30:21.037+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:30:21.495 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:30:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:30:21.037+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:30:21.974 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:30:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:30:21.753+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:30:21.974 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:30:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:30:21.753+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:30:21.974 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:30:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:30:21.753+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:30:22.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:30:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:30:22.109+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:22.110595+0000) 2024-02-20T09:30:22.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:30:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:30:21.993+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:21.995082+0000) 2024-02-20T09:30:22.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:30:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:30:22.014+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:30:22.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:30:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:30:22.014+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:30:22.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:30:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:30:22.014+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:30:22.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:30:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:30:21.971+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:30:22.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:30:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:30:21.971+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:30:22.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:30:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:30:21.971+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:30:22.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:30:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:30:21.971+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:30:22.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:30:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:30:22.049+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:22.050388+0000) 2024-02-20T09:30:22.663 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:30:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:30:22.409+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 71509 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:30:22.664 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:30:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:30:22.403+0000 7f8a5f86e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:22.404917+0000) 2024-02-20T09:30:22.665 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:30:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:30:22.405+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:22.407144+0000) 2024-02-20T09:30:22.665 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:30:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:30:22.403+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:22.404555+0000) 2024-02-20T09:30:22.984 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:30:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:30:22.738+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:30:22.985 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:30:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:30:22.738+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:30:22.985 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:30:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:30:22.738+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:30:23.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:30:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:30:23.109+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:23.110681+0000) 2024-02-20T09:30:23.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:30:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:30:23.017+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:30:23.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:30:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:30:23.017+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:30:23.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:30:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:30:23.017+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:30:23.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:30:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:30:23.017+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:30:23.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:30:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:30:23.050+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:23.051425+0000) 2024-02-20T09:30:23.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:30:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:30:22.993+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:22.995174+0000) 2024-02-20T09:30:23.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:30:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:30:23.052+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:30:23.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:30:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:30:23.052+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:30:23.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:30:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:30:23.052+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:30:23.973 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:30:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:30:23.761+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:30:23.974 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:30:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:30:23.761+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:30:23.974 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:30:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:30:23.761+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:30:24.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:30:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:30:24.109+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:24.110760+0000) 2024-02-20T09:30:24.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:30:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:30:23.971+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:30:24.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:30:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:30:23.971+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:30:24.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:30:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:30:23.971+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:30:24.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:30:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:30:23.971+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:30:24.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:30:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:30:24.052+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:24.053658+0000) 2024-02-20T09:30:24.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:30:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:30:23.993+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:23.995271+0000) 2024-02-20T09:30:24.245 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:30:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:30:24.096+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:30:24.245 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:30:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:30:24.096+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:30:24.245 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:30:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:30:24.096+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:30:24.985 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:30:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:30:24.719+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:30:24.985 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:30:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:30:24.719+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:30:24.985 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:30:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:30:24.719+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:30:25.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:30:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:30:25.109+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:25.110881+0000) 2024-02-20T09:30:25.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:30:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:30:25.019+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:30:25.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:30:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:30:25.019+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:30:25.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:30:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:30:25.019+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:30:25.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:30:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:30:25.019+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:30:25.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:30:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:30:25.054+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:25.055924+0000) 2024-02-20T09:30:25.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:30:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:30:24.994+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:24.995389+0000) 2024-02-20T09:30:25.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:30:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:30:25.069+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:30:25.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:30:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:30:25.069+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:30:25.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:30:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:30:25.069+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:30:25.985 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:30:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:30:25.683+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:30:25.985 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:30:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:30:25.683+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:30:25.986 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:30:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:30:25.683+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:30:26.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:30:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:30:26.109+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:26.111004+0000) 2024-02-20T09:30:26.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:30:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:30:25.994+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:25.995512+0000) 2024-02-20T09:30:26.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:30:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:30:26.113+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:30:26.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:30:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:30:26.113+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:30:26.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:30:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:30:26.113+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:30:26.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:30:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:30:26.058+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:26.059525+0000) 2024-02-20T09:30:26.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:30:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:30:26.058+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:30:26.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:30:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:30:26.058+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:30:26.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:30:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:30:26.058+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:30:26.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:30:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:30:26.058+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:30:26.985 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:30:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:30:26.707+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:30:26.985 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:30:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:30:26.707+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:30:26.986 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:30:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:30:26.707+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:30:27.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:30:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:30:27.109+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:27.111118+0000) 2024-02-20T09:30:27.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:30:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:30:26.994+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:26.995603+0000) 2024-02-20T09:30:27.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:30:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:30:27.119+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:30:27.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:30:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:30:27.119+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:30:27.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:30:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:30:27.119+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:30:27.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:30:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:30:27.060+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:27.061585+0000) 2024-02-20T09:30:27.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:30:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:30:27.096+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:30:27.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:30:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:30:27.096+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:30:27.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:30:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:30:27.096+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:30:27.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:30:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:30:27.096+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:30:27.671 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:30:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:30:27.413+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:27.414667+0000) 2024-02-20T09:30:27.671 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:30:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:30:27.410+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:27.412231+0000) 2024-02-20T09:30:27.672 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:30:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:30:27.410+0000 7f8a5f86e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:27.412300+0000) 2024-02-20T09:30:27.672 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:30:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:30:27.417+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 71548 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:30:27.985 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:30:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:30:27.733+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:30:27.985 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:30:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:30:27.733+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:30:27.986 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:30:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:30:27.733+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:30:28.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:30:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:30:28.110+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:28.111294+0000) 2024-02-20T09:30:28.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:30:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:30:28.060+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:30:28.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:30:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:30:28.060+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:30:28.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:30:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:30:28.060+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:30:28.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:30:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:30:28.060+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:30:28.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:30:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:30:28.061+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:28.062564+0000) 2024-02-20T09:30:28.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:30:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:30:27.995+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:27.995729+0000) 2024-02-20T09:30:28.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:30:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:30:28.135+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:30:28.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:30:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:30:28.135+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:30:28.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:30:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:30:28.135+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:30:28.985 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:30:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:30:28.735+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:30:28.986 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:30:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:30:28.735+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:30:28.986 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:30:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:30:28.735+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:30:29.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:30:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:30:29.110+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:29.111383+0000) 2024-02-20T09:30:29.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:30:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:30:29.040+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:30:29.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:30:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:30:29.040+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:30:29.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:30:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:30:29.040+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:30:29.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:30:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:30:29.040+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:30:29.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:30:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:30:29.064+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:29.064704+0000) 2024-02-20T09:30:29.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:30:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:30:28.995+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:28.995816+0000) 2024-02-20T09:30:29.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:30:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:30:29.089+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:30:29.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:30:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:30:29.089+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:30:29.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:30:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:30:29.089+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:30:29.985 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:30:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:30:29.685+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:30:29.986 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:30:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:30:29.685+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:30:29.986 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:30:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:30:29.685+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:30:30.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:30:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:30:30.111+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:30.111486+0000) 2024-02-20T09:30:30.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:30:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:30:30.031+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:30:30.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:30:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:30:30.031+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:30:30.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:30:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:30:30.031+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:30:30.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:30:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:30:30.031+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:30:30.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:30:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:30:30.066+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:30.066716+0000) 2024-02-20T09:30:30.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:30:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:30:29.995+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:29.995922+0000) 2024-02-20T09:30:30.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:30:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:30:30.057+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:30:30.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:30:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:30:30.057+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:30:30.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:30:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:30:30.057+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:30:30.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:30:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:30:30.664+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:30:30.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:30:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:30:30.664+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:30:30.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:30:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:30:30.664+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:30:31.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:30:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:30:31.111+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:31.111599+0000) 2024-02-20T09:30:31.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:30:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:30:31.014+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:30:31.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:30:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:30:31.014+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:30:31.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:30:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:30:31.014+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:30:31.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:30:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:30:31.014+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:30:31.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:30:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:30:31.069+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:31.069414+0000) 2024-02-20T09:30:31.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:30:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:30:30.995+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:30.996099+0000) 2024-02-20T09:30:31.494 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:30:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:30:31.013+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:30:31.494 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:30:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:30:31.013+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:30:31.494 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:30:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:30:31.013+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:30:31.986 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:30:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:30:31.709+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:30:31.986 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:30:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:30:31.709+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:30:31.986 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:30:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:30:31.709+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:30:32.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:30:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:30:32.111+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:32.111706+0000) 2024-02-20T09:30:32.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:30:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:30:31.995+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:31.996257+0000) 2024-02-20T09:30:32.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:30:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:30:32.040+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:30:32.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:30:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:30:32.040+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:30:32.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:30:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:30:32.040+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:30:32.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:30:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:30:32.040+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:30:32.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:30:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:30:32.040+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:30:32.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:30:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:30:32.040+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:30:32.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:30:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:30:32.040+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:30:32.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:30:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:30:32.071+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:32.071869+0000) 2024-02-20T09:30:32.677 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:30:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:30:32.427+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 71588 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:30:32.678 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:30:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:30:32.419+0000 7f8a5f86e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:32.420361+0000) 2024-02-20T09:30:32.678 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:30:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:30:32.423+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:32.423944+0000) 2024-02-20T09:30:32.678 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:30:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:30:32.419+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:32.420072+0000) 2024-02-20T09:30:32.986 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:30:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:30:32.693+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:30:32.986 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:30:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:30:32.693+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:30:32.986 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:30:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:30:32.693+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:30:33.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:30:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:30:33.111+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:33.111851+0000) 2024-02-20T09:30:33.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:30:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:30:32.996+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:32.996400+0000) 2024-02-20T09:30:33.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:30:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:30:33.075+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:30:33.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:30:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:30:33.075+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:30:33.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:30:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:30:33.075+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:30:33.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:30:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:30:33.055+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:30:33.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:30:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:30:33.055+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:30:33.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:30:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:30:33.055+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:30:33.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:30:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:30:33.055+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:30:33.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:30:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:30:33.072+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:33.072938+0000) 2024-02-20T09:30:33.986 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:30:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:30:33.739+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:30:33.986 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:30:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:30:33.739+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:30:33.986 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:30:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:30:33.739+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:30:34.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:30:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:30:34.111+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:34.111959+0000) 2024-02-20T09:30:34.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:30:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:30:33.996+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:33.996473+0000) 2024-02-20T09:30:34.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:30:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:30:34.078+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:30:34.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:30:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:30:34.078+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:30:34.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:30:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:30:34.078+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:30:34.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:30:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:30:34.020+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:30:34.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:30:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:30:34.020+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:30:34.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:30:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:30:34.020+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:30:34.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:30:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:30:34.020+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:30:34.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:30:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:30:34.074+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:34.075079+0000) 2024-02-20T09:30:34.986 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:30:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:30:34.715+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:30:34.986 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:30:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:30:34.715+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:30:34.986 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:30:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:30:34.715+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:30:35.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:30:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:30:35.111+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:35.112083+0000) 2024-02-20T09:30:35.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:30:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:30:34.996+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:34.996540+0000) 2024-02-20T09:30:35.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:30:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:30:35.084+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:30:35.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:30:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:30:35.084+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:30:35.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:30:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:30:35.084+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:30:35.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:30:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:30:35.050+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:30:35.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:30:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:30:35.050+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:30:35.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:30:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:30:35.050+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:30:35.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:30:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:30:35.050+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:30:35.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:30:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:30:35.076+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:35.077217+0000) 2024-02-20T09:30:35.986 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:30:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:30:35.730+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:30:35.986 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:30:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:30:35.730+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:30:35.987 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:30:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:30:35.730+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:30:36.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:30:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:30:36.111+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:36.112175+0000) 2024-02-20T09:30:36.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:30:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:30:35.996+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:35.996632+0000) 2024-02-20T09:30:36.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:30:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:30:36.090+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:30:36.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:30:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:30:36.090+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:30:36.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:30:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:30:36.090+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:30:36.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:30:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:30:36.079+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:30:36.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:30:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:30:36.079+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:30:36.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:30:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:30:36.079+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:30:36.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:30:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:30:36.079+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:30:36.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:30:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:30:36.080+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:36.080645+0000) 2024-02-20T09:30:36.986 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:30:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:30:36.752+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:30:36.986 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:30:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:30:36.752+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:30:36.986 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:30:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:30:36.752+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:30:37.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:30:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:30:37.111+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:37.112313+0000) 2024-02-20T09:30:37.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:30:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:30:36.996+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:36.996723+0000) 2024-02-20T09:30:37.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:30:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:30:37.048+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:30:37.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:30:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:30:37.048+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:30:37.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:30:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:30:37.048+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:30:37.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:30:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:30:37.080+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:30:37.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:30:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:30:37.080+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:30:37.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:30:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:30:37.080+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:30:37.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:30:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:30:37.080+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:30:37.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:30:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:30:37.082+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:37.082779+0000) 2024-02-20T09:30:37.682 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:30:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:30:37.434+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 71627 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:30:37.683 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:30:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:30:37.428+0000 7f8a5f86e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:37.429223+0000) 2024-02-20T09:30:37.683 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:30:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:30:37.430+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:37.430968+0000) 2024-02-20T09:30:37.683 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:30:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:30:37.428+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:37.428611+0000) 2024-02-20T09:30:37.986 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:30:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:30:37.790+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:30:37.986 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:30:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:30:37.790+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:30:37.987 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:30:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:30:37.790+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:30:38.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:30:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:30:38.112+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:38.112464+0000) 2024-02-20T09:30:38.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:30:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:30:37.996+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:37.996835+0000) 2024-02-20T09:30:38.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:30:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:30:38.046+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:30:38.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:30:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:30:38.046+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:30:38.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:30:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:30:38.046+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:30:38.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:30:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:30:38.071+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:30:38.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:30:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:30:38.071+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:30:38.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:30:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:30:38.071+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:30:38.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:30:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:30:38.071+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:30:38.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:30:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:30:38.083+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:38.084082+0000) 2024-02-20T09:30:38.987 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:30:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:30:38.835+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:30:38.987 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:30:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:30:38.835+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:30:38.987 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:30:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:30:38.835+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:30:39.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:30:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:30:39.112+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:39.112557+0000) 2024-02-20T09:30:39.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:30:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:30:38.996+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:38.996934+0000) 2024-02-20T09:30:39.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:30:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:30:39.058+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:30:39.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:30:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:30:39.058+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:30:39.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:30:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:30:39.058+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:30:39.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:30:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:30:39.086+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:39.086566+0000) 2024-02-20T09:30:39.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:30:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:30:39.087+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:30:39.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:30:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:30:39.087+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:30:39.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:30:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:30:39.087+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:30:39.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:30:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:30:39.087+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:30:39.987 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:30:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:30:39.869+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:30:39.987 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:30:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:30:39.869+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:30:39.987 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:30:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:30:39.869+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:30:40.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:30:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:30:40.112+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:40.112674+0000) 2024-02-20T09:30:40.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:30:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:30:39.996+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:39.997038+0000) 2024-02-20T09:30:40.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:30:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:30:40.094+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:30:40.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:30:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:30:40.094+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:30:40.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:30:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:30:40.094+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:30:40.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:30:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:30:40.088+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:40.088805+0000) 2024-02-20T09:30:40.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:30:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:30:40.125+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:30:40.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:30:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:30:40.125+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:30:40.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:30:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:30:40.125+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:30:40.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:30:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:30:40.126+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:30:40.998 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:30:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:30:40.892+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:30:40.998 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:30:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:30:40.892+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:30:40.999 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:30:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:30:40.892+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:30:41.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:30:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:30:41.112+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:41.112794+0000) 2024-02-20T09:30:41.492 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:30:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:30:40.996+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:40.997160+0000) 2024-02-20T09:30:41.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:30:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:30:41.114+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:30:41.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:30:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:30:41.114+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:30:41.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:30:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:30:41.114+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:30:41.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:30:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:30:41.092+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:41.092508+0000) 2024-02-20T09:30:41.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:30:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:30:41.130+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:30:41.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:30:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:30:41.130+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:30:41.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:30:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:30:41.130+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:30:41.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:30:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:30:41.130+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:30:42.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:30:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:30:41.938+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:30:42.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:30:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:30:41.938+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:30:42.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:30:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:30:41.938+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:30:42.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:30:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:30:42.112+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:42.112916+0000) 2024-02-20T09:30:42.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:30:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:30:41.996+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:41.997302+0000) 2024-02-20T09:30:42.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:30:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:30:42.070+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:30:42.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:30:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:30:42.070+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:30:42.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:30:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:30:42.070+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:30:42.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:30:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:30:42.094+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:42.094670+0000) 2024-02-20T09:30:42.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:30:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:30:42.126+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:30:42.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:30:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:30:42.126+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:30:42.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:30:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:30:42.126+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:30:42.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:30:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:30:42.126+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:30:42.689 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:30:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:30:42.442+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 71667 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:30:42.689 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:30:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:30:42.436+0000 7f8a5f86e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:42.436589+0000) 2024-02-20T09:30:42.689 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:30:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:30:42.439+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:42.439681+0000) 2024-02-20T09:30:42.690 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:30:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:30:42.436+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:42.437104+0000) 2024-02-20T09:30:42.987 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:30:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:30:42.925+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:30:42.987 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:30:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:30:42.925+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:30:42.987 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:30:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:30:42.925+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:30:43.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:30:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:30:43.112+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:43.113032+0000) 2024-02-20T09:30:43.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:30:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:30:42.997+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:42.997448+0000) 2024-02-20T09:30:43.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:30:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:30:43.079+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:30:43.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:30:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:30:43.079+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:30:43.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:30:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:30:43.079+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:30:43.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:30:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:30:43.095+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:43.096382+0000) 2024-02-20T09:30:43.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:30:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:30:43.150+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:30:43.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:30:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:30:43.150+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:30:43.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:30:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:30:43.150+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:30:43.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:30:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:30:43.150+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:30:44.147 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:30:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:30:44.098+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:44.098560+0000) 2024-02-20T09:30:44.147 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:30:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:30:44.145+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:30:44.147 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:30:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:30:44.145+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:30:44.148 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:30:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:30:43.997+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:43.997528+0000) 2024-02-20T09:30:44.148 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:30:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:30:44.063+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:30:44.148 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:30:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:30:44.063+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:30:44.148 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:30:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:30:44.063+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:30:44.148 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:30:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:30:43.893+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:30:44.149 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:30:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:30:43.893+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:30:44.149 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:30:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:30:43.893+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:30:44.149 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:30:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:30:44.112+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:44.113131+0000) 2024-02-20T09:30:44.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:30:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:30:44.145+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:30:44.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:30:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:30:44.146+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:30:44.987 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:30:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:30:44.864+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:30:44.987 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:30:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:30:44.864+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:30:44.988 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:30:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:30:44.864+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:30:45.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:30:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:30:45.112+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:45.113260+0000) 2024-02-20T09:30:45.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:30:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:30:44.997+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:44.997624+0000) 2024-02-20T09:30:45.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:30:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:30:45.102+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:30:45.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:30:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:30:45.102+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:30:45.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:30:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:30:45.102+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:30:45.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:30:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:30:45.101+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:45.102081+0000) 2024-02-20T09:30:45.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:30:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:30:45.122+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:30:45.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:30:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:30:45.122+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:30:45.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:30:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:30:45.122+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:30:45.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:30:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:30:45.123+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:30:46.172 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:30:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:30:45.997+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:45.997721+0000) 2024-02-20T09:30:46.172 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:30:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:30:46.104+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:30:46.173 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:30:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:30:46.104+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:30:46.173 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:30:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:30:46.104+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:30:46.173 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:30:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:30:45.893+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:30:46.173 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:30:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:30:45.893+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:30:46.173 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:30:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:30:45.893+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:30:46.173 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:30:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:30:46.112+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:46.113360+0000) 2024-02-20T09:30:46.174 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:30:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:30:46.103+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:46.104227+0000) 2024-02-20T09:30:46.174 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:30:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:30:46.171+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:30:46.174 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:30:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:30:46.171+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:30:46.174 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:30:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:30:46.171+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:30:46.174 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:30:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:30:46.171+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:30:47.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:30:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:30:46.916+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:30:47.420 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:30:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:30:46.916+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:30:47.420 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:30:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:30:46.916+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:30:47.420 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:30:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:30:47.113+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:47.113470+0000) 2024-02-20T09:30:47.421 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:30:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:30:46.997+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:46.997829+0000) 2024-02-20T09:30:47.421 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:30:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:30:47.119+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:30:47.421 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:30:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:30:47.119+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:30:47.421 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:30:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:30:47.119+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:30:47.421 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:30:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:30:47.106+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:47.106491+0000) 2024-02-20T09:30:47.422 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:30:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:30:47.146+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:30:47.422 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:30:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:30:47.146+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:30:47.422 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:30:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:30:47.146+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:30:47.422 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:30:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:30:47.146+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:30:47.736 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:30:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:30:47.450+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 71706 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:30:47.736 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:30:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:30:47.446+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:47.446954+0000) 2024-02-20T09:30:47.737 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:30:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:30:47.444+0000 7f8a5f86e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:47.444651+0000) 2024-02-20T09:30:47.737 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:30:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:30:47.444+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:47.444519+0000) 2024-02-20T09:30:47.987 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:30:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:30:47.920+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:30:47.988 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:30:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:30:47.920+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:30:47.988 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:30:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:30:47.920+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:30:48.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:30:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:30:48.113+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:48.113567+0000) 2024-02-20T09:30:48.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:30:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:30:47.997+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:47.997941+0000) 2024-02-20T09:30:48.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:30:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:30:48.137+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:30:48.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:30:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:30:48.137+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:30:48.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:30:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:30:48.137+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:30:48.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:30:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:30:48.107+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:48.107498+0000) 2024-02-20T09:30:48.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:30:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:30:48.147+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:30:48.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:30:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:30:48.147+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:30:48.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:30:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:30:48.147+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:30:48.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:30:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:30:48.147+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:30:49.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:30:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:30:48.936+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:30:49.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:30:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:30:48.936+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:30:49.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:30:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:30:48.936+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:30:49.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:30:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:30:49.113+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:49.113645+0000) 2024-02-20T09:30:49.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:30:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:30:48.997+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:48.998039+0000) 2024-02-20T09:30:49.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:30:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:30:49.101+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:30:49.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:30:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:30:49.101+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:30:49.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:30:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:30:49.101+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:30:49.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:30:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:30:49.109+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:49.109429+0000) 2024-02-20T09:30:49.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:30:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:30:49.136+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:30:49.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:30:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:30:49.136+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:30:49.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:30:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:30:49.136+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:30:49.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:30:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:30:49.136+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:30:49.988 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:30:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:30:49.887+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:30:49.988 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:30:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:30:49.887+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:30:49.988 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:30:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:30:49.887+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:30:50.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:30:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:30:50.113+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:50.113774+0000) 2024-02-20T09:30:50.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:30:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:30:50.097+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:30:50.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:30:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:30:50.097+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:30:50.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:30:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:30:50.097+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:30:50.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:30:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:30:50.097+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:30:50.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:30:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:30:50.112+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:50.113007+0000) 2024-02-20T09:30:50.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:30:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:30:49.997+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:49.998150+0000) 2024-02-20T09:30:50.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:30:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:30:50.092+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:30:50.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:30:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:30:50.092+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:30:50.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:30:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:30:50.092+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:30:50.999 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:30:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:30:50.924+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:30:50.999 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:30:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:30:50.924+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:30:50.999 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:30:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:30:50.924+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:30:51.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:30:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:30:51.113+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:51.113865+0000) 2024-02-20T09:30:51.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:30:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:30:51.097+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:30:51.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:30:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:30:51.097+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:30:51.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:30:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:30:51.097+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:30:51.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:30:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:30:51.097+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:30:51.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:30:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:30:51.114+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:51.115039+0000) 2024-02-20T09:30:51.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:30:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:30:50.997+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:50.998280+0000) 2024-02-20T09:30:51.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:30:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:30:51.124+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:30:51.494 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:30:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:30:51.124+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:30:51.494 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:30:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:30:51.124+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:30:52.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:30:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:30:51.923+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:30:52.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:30:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:30:51.923+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:30:52.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:30:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:30:51.923+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:30:52.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:30:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:30:52.112+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:52.113997+0000) 2024-02-20T09:30:52.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:30:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:30:51.997+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:51.998391+0000) 2024-02-20T09:30:52.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:30:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:30:52.125+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:30:52.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:30:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:30:52.125+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:30:52.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:30:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:30:52.125+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:30:52.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:30:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:30:52.115+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:52.117037+0000) 2024-02-20T09:30:52.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:30:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:30:52.116+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:30:52.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:30:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:30:52.116+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:30:52.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:30:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:30:52.116+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:30:52.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:30:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:30:52.116+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:30:52.736 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:30:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:30:52.457+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 71746 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:30:52.737 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:30:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:30:52.450+0000 7f8a5f86e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:52.452074+0000) 2024-02-20T09:30:52.737 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:30:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:30:52.454+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:52.455948+0000) 2024-02-20T09:30:52.737 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:30:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:30:52.450+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:52.452116+0000) 2024-02-20T09:30:52.988 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:30:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:30:52.939+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:30:52.988 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:30:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:30:52.939+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:30:52.989 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:30:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:30:52.939+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:30:53.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:30:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:30:53.112+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:53.114128+0000) 2024-02-20T09:30:53.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:30:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:30:52.997+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:52.998508+0000) 2024-02-20T09:30:53.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:30:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:30:53.128+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:30:53.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:30:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:30:53.128+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:30:53.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:30:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:30:53.128+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:30:53.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:30:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:30:53.079+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:30:53.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:30:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:30:53.079+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:30:53.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:30:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:30:53.079+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:30:53.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:30:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:30:53.079+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:30:53.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:30:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:30:53.116+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:53.118101+0000) 2024-02-20T09:30:54.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:30:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:30:53.997+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:53.998585+0000) 2024-02-20T09:30:54.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:30:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:30:54.155+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:30:54.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:30:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:30:54.155+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:30:54.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:30:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:30:54.155+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:30:54.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:30:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:30:53.939+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:30:54.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:30:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:30:53.939+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:30:54.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:30:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:30:53.939+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:30:54.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:30:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:30:54.112+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:54.114232+0000) 2024-02-20T09:30:54.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:30:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:30:54.069+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:30:54.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:30:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:30:54.069+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:30:54.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:30:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:30:54.069+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:30:54.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:30:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:30:54.069+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:30:54.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:30:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:30:54.118+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:54.120135+0000) 2024-02-20T09:30:55.202 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:30:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:30:54.997+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:54.998698+0000) 2024-02-20T09:30:55.203 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:30:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:30:55.200+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:30:55.203 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:30:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:30:55.200+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:30:55.203 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:30:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:30:55.200+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:30:55.203 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:30:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:30:54.933+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:30:55.203 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:30:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:30:54.933+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:30:55.203 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:30:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:30:54.933+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:30:55.203 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:30:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:30:55.112+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:55.114355+0000) 2024-02-20T09:30:55.204 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:30:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:30:55.088+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:30:55.204 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:30:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:30:55.088+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:30:55.204 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:30:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:30:55.088+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:30:55.204 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:30:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:30:55.088+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:30:55.204 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:30:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:30:55.120+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:55.122211+0000) 2024-02-20T09:30:56.241 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:30:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:30:55.997+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:55.998819+0000) 2024-02-20T09:30:56.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:30:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:30:56.239+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:30:56.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:30:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:30:56.073+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:30:56.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:30:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:30:56.073+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:30:56.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:30:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:30:56.073+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:30:56.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:30:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:30:56.073+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:30:56.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:30:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:30:56.122+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:56.124228+0000) 2024-02-20T09:30:56.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:30:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:30:55.938+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:30:56.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:30:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:30:55.938+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:30:56.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:30:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:30:55.938+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:30:56.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:30:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:30:56.113+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:56.114445+0000) 2024-02-20T09:30:56.492 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:30:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:30:56.239+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:30:56.492 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:30:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:30:56.239+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:30:57.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:30:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:30:56.997+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:56.998998+0000) 2024-02-20T09:30:57.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:30:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:30:57.113+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:30:57.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:30:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:30:57.113+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:30:57.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:30:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:30:57.113+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:30:57.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:30:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:30:57.113+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:30:57.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:30:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:30:57.124+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:57.126242+0000) 2024-02-20T09:30:57.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:30:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:30:56.960+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:30:57.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:30:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:30:56.960+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:30:57.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:30:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:30:56.960+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:30:57.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:30:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:30:57.113+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:57.114555+0000) 2024-02-20T09:30:57.586 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:30:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:30:57.461+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:57.463006+0000) 2024-02-20T09:30:57.586 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:30:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:30:57.459+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:57.460693+0000) 2024-02-20T09:30:57.586 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:30:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:30:57.279+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:30:57.587 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:30:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:30:57.279+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:30:57.587 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:30:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:30:57.279+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:30:57.587 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:30:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:30:57.459+0000 7f8a5f86e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:57.460481+0000) 2024-02-20T09:30:57.587 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:30:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:30:57.466+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 71785 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:30:58.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:30:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:30:57.997+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:57.999086+0000) 2024-02-20T09:30:58.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:30:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:30:57.926+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:30:58.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:30:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:30:57.926+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:30:58.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:30:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:30:57.926+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:30:58.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:30:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:30:58.113+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:58.114646+0000) 2024-02-20T09:30:58.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:30:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:30:58.125+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:58.127295+0000) 2024-02-20T09:30:58.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:30:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:30:58.127+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:30:58.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:30:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:30:58.127+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:30:58.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:30:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:30:58.127+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:30:58.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:30:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:30:58.127+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:30:58.643 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:30:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:30:58.272+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:30:58.644 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:30:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:30:58.272+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:30:58.644 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:30:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:30:58.272+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:30:59.165 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:30:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:30:58.997+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:58.999186+0000) 2024-02-20T09:30:59.166 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:30:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:30:59.127+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:59.129325+0000) 2024-02-20T09:30:59.166 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:30:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:30:59.163+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:30:59.166 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:30:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:30:59.163+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:30:59.166 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:30:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:30:59.163+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:30:59.167 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:30:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:30:58.902+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:30:59.167 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:30:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:30:58.902+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:30:59.167 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:30:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:30:58.902+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:30:59.167 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:30:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:30:59.113+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:59.114725+0000) 2024-02-20T09:30:59.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:30:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:30:59.163+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:30:59.492 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:30:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:30:59.231+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:30:59.492 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:30:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:30:59.231+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:30:59.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:30:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:30:59.231+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:31:00.194 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:30:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:30:59.997+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:30:59.999322+0000) 2024-02-20T09:31:00.194 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:31:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:31:00.131+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:00.132816+0000) 2024-02-20T09:31:00.194 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:30:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:30:59.893+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:31:00.195 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:30:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:30:59.893+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:31:00.195 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:30:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:30:59.893+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:31:00.195 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:31:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:31:00.113+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:00.114834+0000) 2024-02-20T09:31:00.489 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:31:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:31:00.273+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:31:00.490 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:31:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:31:00.273+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:31:00.490 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:31:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:31:00.273+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:31:00.490 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:31:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:31:00.191+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:31:00.490 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:31:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:31:00.191+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:31:00.490 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:31:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:31:00.191+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:31:00.490 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:31:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:31:00.191+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:31:01.000 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:31:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:31:00.928+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:31:01.000 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:31:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:31:00.928+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:31:01.000 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:31:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:31:00.928+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:31:01.305 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:31:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:31:01.113+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:01.114953+0000) 2024-02-20T09:31:01.306 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:31:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:31:00.998+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:00.999457+0000) 2024-02-20T09:31:01.306 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:31:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:31:01.303+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:31:01.306 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:31:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:31:01.303+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:31:01.307 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:31:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:31:01.133+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:01.134844+0000) 2024-02-20T09:31:01.307 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:31:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:31:01.175+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:31:01.307 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:31:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:31:01.175+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:31:01.307 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:31:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:31:01.175+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:31:01.307 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:31:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:31:01.175+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:31:01.940 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:31:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:31:01.303+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:31:02.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:31:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:31:01.998+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:01.999553+0000) 2024-02-20T09:31:02.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:31:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:31:02.135+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:02.136969+0000) 2024-02-20T09:31:02.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:31:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:31:02.153+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:31:02.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:31:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:31:02.153+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:31:02.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:31:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:31:02.153+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:31:02.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:31:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:31:02.153+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:31:02.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:31:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:31:01.938+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:31:02.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:31:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:31:01.938+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:31:02.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:31:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:31:01.938+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:31:02.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:31:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:31:02.113+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:02.115087+0000) 2024-02-20T09:31:02.644 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:31:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:31:02.471+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:02.472724+0000) 2024-02-20T09:31:02.644 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:31:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:31:02.467+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:02.468901+0000) 2024-02-20T09:31:02.645 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:31:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:31:02.346+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:31:02.645 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:31:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:31:02.346+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:31:02.645 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:31:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:31:02.346+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:31:02.645 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:31:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:31:02.467+0000 7f8a5f86e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:02.468669+0000) 2024-02-20T09:31:02.645 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:31:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:31:02.473+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 71825 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:31:03.194 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:31:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:31:02.998+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:02.999654+0000) 2024-02-20T09:31:03.195 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:31:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:31:03.136+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:03.138117+0000) 2024-02-20T09:31:03.195 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:31:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:31:03.192+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:31:03.195 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:31:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:31:02.922+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:31:03.196 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:31:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:31:02.922+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:31:03.196 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:31:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:31:02.922+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:31:03.196 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:31:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:31:03.113+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:03.115203+0000) 2024-02-20T09:31:03.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:31:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:31:03.192+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:31:03.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:31:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:31:03.192+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:31:03.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:31:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:31:03.192+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:31:03.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:31:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:31:03.395+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:31:03.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:31:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:31:03.395+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:31:03.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:31:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:31:03.395+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:31:03.989 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:31:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:31:03.885+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:31:03.990 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:31:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:31:03.885+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:31:03.990 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:31:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:31:03.885+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:31:04.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:31:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:31:04.113+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:04.115284+0000) 2024-02-20T09:31:04.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:31:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:31:03.998+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:03.999753+0000) 2024-02-20T09:31:04.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:31:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:31:04.139+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:04.140452+0000) 2024-02-20T09:31:04.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:31:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:31:04.208+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:31:04.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:31:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:31:04.208+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:31:04.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:31:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:31:04.208+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:31:04.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:31:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:31:04.208+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:31:04.644 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:31:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:31:04.350+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:31:04.644 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:31:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:31:04.350+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:31:04.644 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:31:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:31:04.350+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:31:05.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:31:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:31:04.998+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:04.999864+0000) 2024-02-20T09:31:05.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:31:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:31:05.141+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:05.142431+0000) 2024-02-20T09:31:05.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:31:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:31:04.902+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:31:05.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:31:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:31:04.902+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:31:05.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:31:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:31:04.902+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:31:05.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:31:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:31:05.114+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:05.115395+0000) 2024-02-20T09:31:05.644 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:31:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:31:05.350+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:31:05.644 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:31:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:31:05.350+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:31:05.645 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:31:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:31:05.350+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:31:05.645 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:31:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:31:05.255+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:31:05.645 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:31:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:31:05.255+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:31:05.645 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:31:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:31:05.255+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:31:05.645 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:31:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:31:05.255+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:31:05.990 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:31:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:31:05.882+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:31:05.990 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:31:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:31:05.882+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:31:05.990 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:31:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:31:05.882+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:31:06.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:31:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:31:06.143+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:06.144443+0000) 2024-02-20T09:31:06.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:31:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:31:06.114+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:06.115498+0000) 2024-02-20T09:31:06.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:31:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:31:05.998+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:05.999971+0000) 2024-02-20T09:31:06.644 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:31:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:31:06.346+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:31:06.644 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:31:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:31:06.346+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:31:06.645 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:31:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:31:06.346+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:31:06.645 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:31:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:31:06.248+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:31:06.645 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:31:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:31:06.248+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:31:06.645 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:31:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:31:06.248+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:31:06.645 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:31:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:31:06.248+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:31:06.990 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:31:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:31:06.886+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:31:06.990 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:31:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:31:06.886+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:31:06.990 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:31:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:31:06.886+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:31:07.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:31:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:31:06.998+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:07.000086+0000) 2024-02-20T09:31:07.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:31:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:31:07.114+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:07.115588+0000) 2024-02-20T09:31:07.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:31:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:31:07.145+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:07.146457+0000) 2024-02-20T09:31:07.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:31:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:31:07.223+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:31:07.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:31:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:31:07.223+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:31:07.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:31:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:31:07.223+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:31:07.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:31:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:31:07.223+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:31:07.644 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:31:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:31:07.481+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 71864 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:31:07.645 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:31:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:31:07.477+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:07.478823+0000) 2024-02-20T09:31:07.645 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:31:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:31:07.474+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:07.476125+0000) 2024-02-20T09:31:07.645 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:31:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:31:07.357+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:31:07.645 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:31:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:31:07.357+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:31:07.646 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:31:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:31:07.357+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:31:07.646 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:31:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:31:07.474+0000 7f8a5f86e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:07.476217+0000) 2024-02-20T09:31:08.226 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:31:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:31:07.998+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:08.000218+0000) 2024-02-20T09:31:08.227 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:31:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:31:07.897+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:31:08.227 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:31:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:31:07.897+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:31:08.227 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:31:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:31:07.897+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:31:08.227 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:31:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:31:08.114+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:08.115679+0000) 2024-02-20T09:31:08.227 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:31:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:31:08.146+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:08.148295+0000) 2024-02-20T09:31:08.228 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:31:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:31:08.224+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:31:08.228 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:31:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:31:08.224+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:31:08.228 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:31:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:31:08.224+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:31:08.228 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:31:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:31:08.224+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:31:08.492 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:31:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:31:08.343+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:31:08.492 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:31:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:31:08.343+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:31:08.492 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:31:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:31:08.343+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:31:08.990 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:31:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:31:08.870+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:31:08.990 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:31:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:31:08.870+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:31:08.990 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:31:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:31:08.870+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:31:09.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:31:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:31:08.998+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:09.000335+0000) 2024-02-20T09:31:09.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:31:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:31:09.114+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:09.115753+0000) 2024-02-20T09:31:09.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:31:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:31:09.150+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:09.151811+0000) 2024-02-20T09:31:09.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:31:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:31:09.184+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:31:09.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:31:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:31:09.184+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:31:09.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:31:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:31:09.184+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:31:09.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:31:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:31:09.184+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:31:09.644 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:31:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:31:09.378+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:31:09.645 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:31:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:31:09.378+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:31:09.645 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:31:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:31:09.378+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:31:09.990 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:31:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:31:09.826+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:31:09.990 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:31:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:31:09.826+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:31:09.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:31:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:31:09.826+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:31:10.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:31:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:31:09.999+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:10.000443+0000) 2024-02-20T09:31:10.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:31:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:31:10.114+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:10.115860+0000) 2024-02-20T09:31:10.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:31:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:31:10.152+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:10.153949+0000) 2024-02-20T09:31:10.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:31:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:31:10.156+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:31:10.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:31:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:31:10.156+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:31:10.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:31:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:31:10.156+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:31:10.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:31:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:31:10.156+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:31:10.718 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:31:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:31:10.409+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:31:10.719 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:31:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:31:10.409+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:31:10.719 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:31:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:31:10.409+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:31:10.990 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:31:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:31:10.857+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:31:10.990 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:31:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:31:10.857+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:31:10.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:31:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:31:10.857+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:31:11.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:31:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:31:10.999+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:11.000533+0000) 2024-02-20T09:31:11.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:31:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:31:11.114+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:11.115994+0000) 2024-02-20T09:31:11.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:31:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:31:11.146+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:31:11.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:31:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:31:11.146+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:31:11.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:31:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:31:11.146+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:31:11.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:31:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:31:11.146+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:31:11.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:31:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:31:11.156+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:11.157493+0000) 2024-02-20T09:31:11.719 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:31:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:31:11.454+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:31:11.719 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:31:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:31:11.454+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:31:11.719 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:31:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:31:11.454+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:31:11.990 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:31:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:31:11.861+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:31:11.990 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:31:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:31:11.861+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:31:11.990 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:31:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:31:11.861+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:31:12.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:31:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:31:11.999+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:12.000624+0000) 2024-02-20T09:31:12.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:31:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:31:12.114+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:12.116126+0000) 2024-02-20T09:31:12.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:31:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:31:12.131+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:31:12.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:31:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:31:12.131+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:31:12.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:31:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:31:12.131+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:31:12.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:31:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:31:12.131+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:31:12.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:31:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:31:12.159+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:12.160950+0000) 2024-02-20T09:31:12.738 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:31:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:31:12.490+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 71904 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:31:12.739 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:31:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:31:12.482+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:12.484155+0000) 2024-02-20T09:31:12.739 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:31:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:31:12.486+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:12.487413+0000) 2024-02-20T09:31:12.739 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:31:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:31:12.474+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:31:12.740 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:31:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:31:12.474+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:31:12.740 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:31:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:31:12.474+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:31:12.740 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:31:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:31:12.482+0000 7f8a5f86e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:12.483982+0000) 2024-02-20T09:31:12.990 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:31:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:31:12.906+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:31:12.990 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:31:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:31:12.906+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:31:12.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:31:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:31:12.906+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:31:13.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:31:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:31:13.114+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:13.116268+0000) 2024-02-20T09:31:13.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:31:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:31:12.999+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:13.000729+0000) 2024-02-20T09:31:13.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:31:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:31:13.157+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:31:13.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:31:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:31:13.157+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:31:13.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:31:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:31:13.157+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:31:13.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:31:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:31:13.157+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:31:13.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:31:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:31:13.161+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:13.162774+0000) 2024-02-20T09:31:13.721 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:31:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:31:13.440+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:31:13.721 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:31:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:31:13.440+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:31:13.721 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:31:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:31:13.440+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:31:13.990 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:31:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:31:13.884+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:31:13.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:31:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:31:13.884+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:31:13.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:31:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:31:13.884+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:31:14.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:31:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:31:13.999+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:14.000821+0000) 2024-02-20T09:31:14.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:31:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:31:14.114+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:14.116387+0000) 2024-02-20T09:31:14.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:31:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:31:14.155+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:31:14.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:31:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:31:14.155+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:31:14.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:31:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:31:14.155+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:31:14.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:31:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:31:14.155+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:31:14.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:31:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:31:14.164+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:14.166179+0000) 2024-02-20T09:31:14.723 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:31:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:31:14.394+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:31:14.723 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:31:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:31:14.394+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:31:14.723 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:31:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:31:14.394+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:31:14.990 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:31:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:31:14.838+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:31:14.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:31:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:31:14.838+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:31:14.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:31:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:31:14.838+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:31:15.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:31:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:31:14.999+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:15.000937+0000) 2024-02-20T09:31:15.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:31:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:31:15.115+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:15.116481+0000) 2024-02-20T09:31:15.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:31:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:31:15.168+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:15.169665+0000) 2024-02-20T09:31:15.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:31:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:31:15.188+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:31:15.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:31:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:31:15.188+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:31:15.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:31:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:31:15.188+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:31:15.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:31:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:31:15.188+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:31:15.723 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:31:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:31:15.427+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:31:15.723 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:31:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:31:15.427+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:31:15.723 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:31:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:31:15.427+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:31:15.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:31:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:31:15.840+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:31:15.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:31:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:31:15.840+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:31:15.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:31:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:31:15.840+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:31:16.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:31:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:31:15.999+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:16.001051+0000) 2024-02-20T09:31:16.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:31:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:31:16.115+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:16.116577+0000) 2024-02-20T09:31:16.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:31:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:31:16.170+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:16.171731+0000) 2024-02-20T09:31:16.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:31:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:31:16.237+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:31:16.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:31:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:31:16.237+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:31:16.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:31:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:31:16.237+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:31:16.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:31:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:31:16.237+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:31:16.739 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:31:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:31:16.472+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:31:16.739 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:31:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:31:16.472+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:31:16.739 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:31:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:31:16.472+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:31:16.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:31:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:31:16.803+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:31:16.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:31:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:31:16.803+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:31:16.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:31:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:31:16.803+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:31:17.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:31:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:31:17.115+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:17.116672+0000) 2024-02-20T09:31:17.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:31:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:31:16.999+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:17.001166+0000) 2024-02-20T09:31:17.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:31:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:31:17.172+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:17.174170+0000) 2024-02-20T09:31:17.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:31:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:31:17.203+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:31:17.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:31:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:31:17.203+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:31:17.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:31:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:31:17.203+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:31:17.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:31:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:31:17.203+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:31:17.723 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:31:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:31:17.498+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 71943 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:31:17.724 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:31:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:31:17.494+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:17.495845+0000) 2024-02-20T09:31:17.724 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:31:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:31:17.491+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:17.493255+0000) 2024-02-20T09:31:17.724 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:31:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:31:17.465+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:31:17.725 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:31:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:31:17.465+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:31:17.725 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:31:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:31:17.465+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:31:17.725 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:31:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:31:17.492+0000 7f8a5f86e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:17.493857+0000) 2024-02-20T09:31:17.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:31:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:31:17.780+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:31:18.033 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:31:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:31:17.780+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:31:18.033 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:31:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:31:17.780+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:31:18.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:31:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:31:18.115+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:18.116790+0000) 2024-02-20T09:31:18.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:31:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:31:17.999+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:18.001295+0000) 2024-02-20T09:31:18.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:31:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:31:18.174+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:18.175393+0000) 2024-02-20T09:31:18.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:31:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:31:18.189+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:31:18.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:31:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:31:18.189+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:31:18.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:31:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:31:18.189+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:31:18.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:31:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:31:18.189+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:31:18.726 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:31:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:31:18.469+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:31:18.726 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:31:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:31:18.469+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:31:18.726 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:31:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:31:18.469+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:31:18.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:31:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:31:18.814+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:31:18.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:31:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:31:18.814+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:31:18.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:31:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:31:18.815+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:31:19.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:31:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:31:19.115+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:19.116903+0000) 2024-02-20T09:31:19.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:31:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:31:19.000+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:19.001435+0000) 2024-02-20T09:31:19.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:31:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:31:19.176+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:19.177488+0000) 2024-02-20T09:31:19.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:31:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:31:19.204+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:31:19.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:31:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:31:19.204+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:31:19.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:31:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:31:19.204+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:31:19.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:31:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:31:19.204+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:31:19.747 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:31:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:31:19.493+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:31:19.747 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:31:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:31:19.493+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:31:19.747 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:31:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:31:19.493+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:31:20.002 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:31:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:31:19.804+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:31:20.003 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:31:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:31:19.804+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:31:20.003 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:31:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:31:19.804+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:31:20.480 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:31:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:31:20.115+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:20.117038+0000) 2024-02-20T09:31:20.481 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:31:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:31:20.000+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:20.001537+0000) 2024-02-20T09:31:20.481 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:31:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:31:20.476+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:31:20.481 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:31:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:31:20.476+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:31:20.481 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:31:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:31:20.476+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:31:20.481 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:31:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:31:20.178+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:20.179795+0000) 2024-02-20T09:31:20.482 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:31:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:31:20.195+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:31:20.482 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:31:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:31:20.195+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:31:20.482 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:31:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:31:20.195+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:31:20.482 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:31:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:31:20.196+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:31:20.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:31:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:31:20.778+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:31:20.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:31:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:31:20.778+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:31:20.992 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:31:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:31:20.778+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:31:21.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:31:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:31:21.000+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:21.001656+0000) 2024-02-20T09:31:21.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:31:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:31:21.115+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:21.117155+0000) 2024-02-20T09:31:21.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:31:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:31:21.180+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:21.182126+0000) 2024-02-20T09:31:21.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:31:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:31:21.227+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:31:21.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:31:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:31:21.227+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:31:21.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:31:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:31:21.227+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:31:21.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:31:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:31:21.227+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:31:21.747 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:31:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:31:21.491+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:31:21.747 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:31:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:31:21.491+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:31:21.747 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:31:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:31:21.491+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:31:22.002 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:31:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:31:21.787+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:31:22.003 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:31:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:31:21.787+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:31:22.003 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:31:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:31:21.787+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:31:22.482 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:31:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:31:22.000+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:22.001745+0000) 2024-02-20T09:31:22.482 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:31:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:31:22.480+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:31:22.483 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:31:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:31:22.480+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:31:22.483 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:31:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:31:22.480+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:31:22.483 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:31:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:31:22.115+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:22.117284+0000) 2024-02-20T09:31:22.483 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:31:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:31:22.183+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:22.184634+0000) 2024-02-20T09:31:22.483 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:31:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:31:22.218+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:31:22.484 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:31:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:31:22.218+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:31:22.484 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:31:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:31:22.218+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:31:22.484 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:31:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:31:22.218+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:31:22.740 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:31:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:31:22.500+0000 7f8a5f86e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:22.501501+0000) 2024-02-20T09:31:22.740 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:31:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:31:22.502+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:22.503443+0000) 2024-02-20T09:31:22.741 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:31:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:31:22.500+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:22.501391+0000) 2024-02-20T09:31:22.742 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:31:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:31:22.506+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 71983 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:31:22.991 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:31:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:31:22.814+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:31:22.992 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:31:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:31:22.814+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:31:22.992 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:31:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:31:22.814+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:31:23.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:31:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:31:23.000+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:23.001886+0000) 2024-02-20T09:31:23.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:31:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:31:23.116+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:23.117483+0000) 2024-02-20T09:31:23.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:31:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:31:23.184+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:23.185877+0000) 2024-02-20T09:31:23.496 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:31:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:31:23.241+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:31:23.496 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:31:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:31:23.241+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:31:23.496 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:31:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:31:23.241+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:31:23.496 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:31:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:31:23.241+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:31:23.748 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:31:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:31:23.493+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:31:23.748 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:31:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:31:23.493+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:31:23.748 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:31:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:31:23.493+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:31:24.003 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:31:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:31:23.836+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:31:24.003 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:31:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:31:23.836+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:31:24.003 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:31:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:31:23.836+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:31:24.492 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:31:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:31:24.000+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:24.002028+0000) 2024-02-20T09:31:24.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:31:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:31:24.116+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:24.117571+0000) 2024-02-20T09:31:24.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:31:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:31:24.186+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:24.188004+0000) 2024-02-20T09:31:24.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:31:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:31:24.208+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:31:24.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:31:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:31:24.208+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:31:24.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:31:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:31:24.208+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:31:24.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:31:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:31:24.208+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:31:24.747 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:31:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:31:24.493+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:31:24.748 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:31:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:31:24.493+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:31:24.748 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:31:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:31:24.493+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:31:25.003 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:31:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:31:24.876+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:31:25.003 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:31:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:31:24.876+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:31:25.003 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:31:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:31:24.876+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:31:25.492 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:31:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:31:25.001+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:25.002230+0000) 2024-02-20T09:31:25.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:31:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:31:25.116+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:25.117659+0000) 2024-02-20T09:31:25.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:31:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:31:25.189+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:25.190237+0000) 2024-02-20T09:31:25.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:31:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:31:25.196+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:31:25.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:31:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:31:25.196+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:31:25.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:31:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:31:25.196+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:31:25.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:31:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:31:25.196+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:31:25.836 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:31:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:31:25.508+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:31:25.836 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:31:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:31:25.508+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:31:25.836 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:31:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:31:25.508+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:31:26.118 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:31:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:31:26.002+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:26.002443+0000) 2024-02-20T09:31:26.119 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:31:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:31:25.833+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:31:26.119 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:31:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:31:25.833+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:31:26.119 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:31:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:31:25.833+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:31:26.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:31:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:31:26.117+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:26.117747+0000) 2024-02-20T09:31:26.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:31:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:31:26.155+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:31:26.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:31:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:31:26.156+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:31:26.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:31:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:31:26.156+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:31:26.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:31:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:31:26.156+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:31:26.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:31:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:31:26.191+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:26.192819+0000) 2024-02-20T09:31:26.849 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:31:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:31:26.521+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:31:26.849 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:31:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:31:26.521+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:31:26.849 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:31:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:31:26.521+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:31:27.118 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:31:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:31:27.002+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:27.002530+0000) 2024-02-20T09:31:27.119 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:31:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:31:26.847+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:31:27.119 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:31:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:31:26.847+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:31:27.119 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:31:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:31:26.847+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:31:27.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:31:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:31:27.117+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:27.117844+0000) 2024-02-20T09:31:27.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:31:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:31:27.126+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:31:27.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:31:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:31:27.126+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:31:27.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:31:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:31:27.126+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:31:27.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:31:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:31:27.126+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:31:27.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:31:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:31:27.194+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:27.195343+0000) 2024-02-20T09:31:27.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:31:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:31:27.491+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:31:27.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:31:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:31:27.491+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:31:27.494 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:31:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:31:27.491+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:31:27.748 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:31:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:31:27.509+0000 7f8a5f86e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:27.510244+0000) 2024-02-20T09:31:27.748 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:31:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:31:27.516+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 72022 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:31:27.749 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:31:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:31:27.511+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:27.511878+0000) 2024-02-20T09:31:27.749 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:31:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:31:27.509+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:27.509889+0000) 2024-02-20T09:31:28.003 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:31:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:31:27.865+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:31:28.004 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:31:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:31:27.865+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:31:28.004 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:31:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:31:27.865+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:31:28.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:31:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:31:28.117+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:28.117961+0000) 2024-02-20T09:31:28.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:31:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:31:28.002+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:28.002623+0000) 2024-02-20T09:31:28.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:31:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:31:28.142+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:31:28.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:31:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:31:28.142+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:31:28.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:31:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:31:28.142+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:31:28.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:31:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:31:28.142+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:31:28.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:31:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:31:28.196+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:28.197000+0000) 2024-02-20T09:31:28.870 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:31:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:31:28.531+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:31:28.871 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:31:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:31:28.531+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:31:28.871 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:31:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:31:28.531+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:31:29.200 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:31:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:31:29.002+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:29.002709+0000) 2024-02-20T09:31:29.200 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:31:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:31:28.869+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:31:29.201 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:31:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:31:28.869+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:31:29.201 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:31:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:31:28.869+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:31:29.201 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:31:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:31:29.117+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:29.118062+0000) 2024-02-20T09:31:29.201 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:31:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:31:29.110+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:31:29.201 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:31:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:31:29.110+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:31:29.201 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:31:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:31:29.110+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:31:29.202 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:31:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:31:29.110+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:31:29.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:31:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:31:29.198+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:29.199144+0000) 2024-02-20T09:31:29.836 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:31:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:31:29.565+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:31:29.836 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:31:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:31:29.565+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:31:29.837 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:31:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:31:29.565+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:31:30.104 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:31:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:31:30.002+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:30.002802+0000) 2024-02-20T09:31:30.104 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:31:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:31:29.835+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:31:30.104 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:31:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:31:29.835+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:31:30.105 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:31:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:31:29.835+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:31:30.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:31:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:31:30.117+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:30.118181+0000) 2024-02-20T09:31:30.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:31:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:31:30.102+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:31:30.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:31:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:31:30.102+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:31:30.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:31:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:31:30.102+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:31:30.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:31:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:31:30.102+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:31:30.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:31:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:31:30.201+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:30.201503+0000) 2024-02-20T09:31:30.748 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:31:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:31:30.517+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:31:30.749 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:31:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:31:30.517+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:31:30.749 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:31:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:31:30.517+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:31:31.003 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:31:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:31:30.858+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:31:31.004 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:31:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:31:30.858+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:31:31.004 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:31:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:31:30.858+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:31:31.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:31:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:31:31.117+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:31.118273+0000) 2024-02-20T09:31:31.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:31:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:31:31.002+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:31.002914+0000) 2024-02-20T09:31:31.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:31:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:31:31.141+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:31:31.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:31:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:31:31.141+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:31:31.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:31:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:31:31.141+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:31:31.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:31:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:31:31.141+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:31:31.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:31:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:31:31.203+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:31.203702+0000) 2024-02-20T09:31:31.896 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:31:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:31:31.563+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:31:31.896 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:31:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:31:31.563+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:31:31.897 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:31:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:31:31.563+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:31:32.180 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:31:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:31:32.002+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:32.003067+0000) 2024-02-20T09:31:32.180 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:31:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:31:31.895+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:31:32.180 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:31:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:31:31.895+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:31:32.180 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:31:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:31:31.895+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:31:32.181 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:31:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:31:32.118+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:32.118394+0000) 2024-02-20T09:31:32.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:31:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:31:32.178+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:31:32.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:31:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:31:32.178+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:31:32.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:31:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:31:32.178+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:31:32.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:31:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:31:32.178+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:31:32.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:31:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:31:32.205+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:32.205755+0000) 2024-02-20T09:31:32.892 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:31:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:31:32.524+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 72062 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:31:32.893 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:31:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:31:32.519+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:32.520239+0000) 2024-02-20T09:31:32.893 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:31:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:31:32.517+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:32.517992+0000) 2024-02-20T09:31:32.893 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:31:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:31:32.517+0000 7f8a5f86e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:32.517654+0000) 2024-02-20T09:31:32.894 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:31:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:31:32.591+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:31:32.894 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:31:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:31:32.591+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:31:32.894 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:31:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:31:32.591+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:31:33.208 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:31:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:31:33.002+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:33.003196+0000) 2024-02-20T09:31:33.208 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:31:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:31:33.133+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:31:33.208 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:31:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:31:33.133+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:31:33.208 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:31:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:31:33.133+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:31:33.209 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:31:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:31:33.133+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:31:33.209 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:31:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:31:32.891+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:31:33.209 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:31:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:31:32.891+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:31:33.209 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:31:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:31:32.891+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:31:33.209 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:31:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:31:33.118+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:33.118528+0000) 2024-02-20T09:31:33.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:31:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:31:33.206+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:33.206904+0000) 2024-02-20T09:31:33.851 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:31:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:31:33.580+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:31:33.851 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:31:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:31:33.580+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:31:33.851 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:31:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:31:33.580+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:31:34.104 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:31:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:31:34.002+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:34.003299+0000) 2024-02-20T09:31:34.105 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:31:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:31:33.850+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:31:34.105 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:31:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:31:33.850+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:31:34.105 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:31:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:31:33.850+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:31:34.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:31:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:31:34.118+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:34.118601+0000) 2024-02-20T09:31:34.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:31:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:31:34.103+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:31:34.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:31:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:31:34.103+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:31:34.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:31:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:31:34.103+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:31:34.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:31:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:31:34.103+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:31:34.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:31:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:31:34.209+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:34.210334+0000) 2024-02-20T09:31:34.826 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:31:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:31:34.563+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:31:34.827 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:31:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:31:34.563+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:31:34.827 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:31:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:31:34.563+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:31:35.112 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:31:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:31:35.003+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:35.003430+0000) 2024-02-20T09:31:35.113 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:31:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:31:34.825+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:31:35.113 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:31:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:31:34.825+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:31:35.113 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:31:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:31:34.825+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:31:35.166 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-02-20T09:31:35.181 INFO:tasks.cephadm:Teardown begin 2024-02-20T09:31:35.181 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/github.com_batrick_ceph_3d72bcfd7affbb4a923c191d212814b4934cfeaa/qa/tasks/cephadm.py", line 1807, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_batrick_ceph_3d72bcfd7affbb4a923c191d212814b4934cfeaa/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/github.com_batrick_ceph_3d72bcfd7affbb4a923c191d212814b4934cfeaa/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/github.com_batrick_ceph_3d72bcfd7affbb4a923c191d212814b4934cfeaa/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 450, in run r.execute() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 100, in execute self.client.exec_command(self.command) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/client.py", line 560, in exec_command chan = self._transport.open_session(timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 959, in open_session return self.open_channel( File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 1090, in open_channel raise e File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2159, in run ptype, m = self.packetizer.read_message() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 463, in read_message header = self.read_all(self.__block_size_in, check_rekey=True) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 308, in read_all x = self.__socket.recv(n) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/gevent/_socketcommon.py", line 660, in recv return self._sock.recv(*args) OSError: [Errno 113] No route to host 2024-02-20T09:31:35.188 DEBUG:teuthology.orchestra.run.smithi133:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-02-20T09:31:35.221 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi183.front.sepia.ceph.com' 2024-02-20T09:31:35.222 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi183.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-20T09:31:35.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:31:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:31:35.118+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:35.118695+0000) 2024-02-20T09:31:35.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:31:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:31:35.111+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:31:35.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:31:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:31:35.111+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:31:35.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:31:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:31:35.111+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:31:35.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:31:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:31:35.111+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:31:35.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:31:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:31:35.211+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:35.212261+0000) 2024-02-20T09:31:35.844 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:31:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:31:35.514+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:31:35.844 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:31:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:31:35.514+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:31:35.844 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:31:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:31:35.514+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:31:36.119 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:31:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:31:36.003+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:36.003528+0000) 2024-02-20T09:31:36.120 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:31:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:31:35.842+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:31:36.120 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:31:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:31:35.842+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:31:36.120 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:31:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:31:35.842+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:31:36.121 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:31:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:31:36.087+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:31:36.121 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:31:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:31:36.087+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:31:36.121 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:31:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:31:36.087+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:31:36.121 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:31:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:31:36.087+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:31:36.466 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:31:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:31:36.215+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:36.215419+0000) 2024-02-20T09:31:36.467 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:31:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:31:36.118+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:36.118816+0000) 2024-02-20T09:31:36.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:31:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:31:36.465+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:31:36.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:31:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:31:36.465+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:31:36.741 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:31:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:31:36.465+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:31:36.992 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:31:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:31:36.830+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:31:36.992 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:31:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:31:36.830+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:31:36.992 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:31:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:31:36.830+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:31:37.492 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:31:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:31:37.003+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:37.003618+0000) 2024-02-20T09:31:37.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:31:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:31:37.118+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:37.118932+0000) 2024-02-20T09:31:37.495 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:31:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:31:37.102+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:31:37.495 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:31:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:31:37.102+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:31:37.495 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:31:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:31:37.102+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:31:37.495 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:31:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:31:37.102+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:31:37.495 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:31:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:31:37.217+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:37.217390+0000) 2024-02-20T09:31:37.745 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:31:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:31:37.528+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:37.528862+0000) 2024-02-20T09:31:37.746 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:31:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:31:37.525+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:37.525455+0000) 2024-02-20T09:31:37.746 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:31:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:31:37.532+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 72101 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:31:37.747 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:31:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:31:37.493+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:31:37.747 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:31:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:31:37.493+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:31:37.747 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:31:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:31:37.493+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:31:37.747 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:31:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:31:37.524+0000 7f8a5f86e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:37.525367+0000) 2024-02-20T09:31:38.004 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:31:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:31:37.806+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:31:38.005 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:31:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:31:37.806+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:31:38.005 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:31:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:31:37.806+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:31:38.304 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.183 2024-02-20T09:31:38.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:31:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:31:38.118+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:38.119039+0000) 2024-02-20T09:31:38.493 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:31:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:31:38.003+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:38.003709+0000) 2024-02-20T09:31:38.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:31:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:31:38.076+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:31:38.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:31:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:31:38.076+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:31:38.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:31:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:31:38.076+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:31:38.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:31:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:31:38.076+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:31:38.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:31:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:31:38.218+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:38.218443+0000) 2024-02-20T09:31:38.822 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:31:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:31:38.517+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:31:38.822 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:31:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:31:38.517+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:31:38.822 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:31:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:31:38.517+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:31:39.083 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:31:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:31:39.003+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:39.003803+0000) 2024-02-20T09:31:39.083 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:31:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:31:38.821+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:31:39.084 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:31:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:31:38.821+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:31:39.084 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:31:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:31:38.821+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:31:39.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:31:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:31:39.118+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:39.119138+0000) 2024-02-20T09:31:39.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:31:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:31:39.081+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:31:39.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:31:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:31:39.081+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:31:39.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:31:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:31:39.081+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:31:39.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:31:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:31:39.082+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:31:39.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:31:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:31:39.220+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:39.220410+0000) 2024-02-20T09:31:39.826 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:31:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:31:39.531+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:31:39.826 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:31:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:31:39.531+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:31:39.827 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:31:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:31:39.531+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:31:40.120 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:31:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:31:40.003+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:40.003902+0000) 2024-02-20T09:31:40.120 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:31:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:31:39.825+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:31:40.121 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:31:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:31:39.825+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:31:40.121 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:31:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:31:39.825+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:31:40.121 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:31:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:31:40.052+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:31:40.121 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:31:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:31:40.052+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:31:40.121 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:31:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:31:40.052+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:31:40.121 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:31:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:31:40.052+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:31:40.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:31:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:31:40.223+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:40.223801+0000) 2024-02-20T09:31:40.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:31:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:31:40.118+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:40.119268+0000) 2024-02-20T09:31:40.749 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:31:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:31:40.571+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:31:40.750 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:31:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:31:40.571+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:31:40.750 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:31:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:31:40.571+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:31:41.005 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:31:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:31:40.824+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:31:41.005 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:31:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:31:40.824+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:31:41.005 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:31:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:31:40.824+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:31:41.492 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:31:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:31:41.003+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:41.004019+0000) 2024-02-20T09:31:41.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:31:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:31:41.119+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:41.119425+0000) 2024-02-20T09:31:41.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:31:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:31:41.072+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:31:41.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:31:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:31:41.072+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:31:41.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:31:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:31:41.072+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:31:41.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:31:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:31:41.072+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:31:41.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:31:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:31:41.225+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:41.226250+0000) 2024-02-20T09:31:41.992 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:31:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:31:41.812+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:31:41.992 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:31:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:31:41.812+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:31:41.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:31:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:31:41.812+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:31:41.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:31:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:31:41.570+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:31:41.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:31:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:31:41.570+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:31:41.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:31:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:31:41.570+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:31:42.492 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:31:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:31:42.003+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:42.004155+0000) 2024-02-20T09:31:42.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:31:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:31:42.119+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:42.119517+0000) 2024-02-20T09:31:42.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:31:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:31:42.075+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:31:42.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:31:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:31:42.075+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:31:42.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:31:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:31:42.075+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:31:42.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:31:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:31:42.075+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:31:42.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:31:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:31:42.229+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:42.229804+0000) 2024-02-20T09:31:42.856 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:31:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:31:42.540+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 72141 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:31:42.856 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:31:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:31:42.537+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:42.538261+0000) 2024-02-20T09:31:42.857 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:31:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:31:42.534+0000 7f8a5f86e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:42.534782+0000) 2024-02-20T09:31:42.857 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:31:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:31:42.538+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:31:42.857 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:31:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:31:42.538+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:31:42.857 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:31:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:31:42.538+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:31:42.858 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:31:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:31:42.533+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:42.534304+0000) 2024-02-20T09:31:42.858 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:31:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:31:42.854+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:31:42.858 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:31:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:31:42.854+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:31:42.858 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:31:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:31:42.854+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:31:43.120 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:31:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:31:43.003+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:43.004298+0000) 2024-02-20T09:31:43.121 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:31:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:31:43.062+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:31:43.121 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:31:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:31:43.062+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:31:43.121 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:31:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:31:43.062+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:31:43.121 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:31:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:31:43.062+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:31:43.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:31:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:31:43.119+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:43.119614+0000) 2024-02-20T09:31:43.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:31:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:31:43.230+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:43.230953+0000) 2024-02-20T09:31:43.890 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:31:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:31:43.581+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:31:43.922 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:31:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:31:43.581+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:31:43.922 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:31:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:31:43.581+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:12.223605+0000 (oldest deadline 2024-02-20T09:21:32.223605+0000) 2024-02-20T09:31:44.234 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:31:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:31:44.004+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:44.004409+0000) 2024-02-20T09:31:44.235 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:31:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:31:43.888+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:31:44.235 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:31:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:31:43.888+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:31:44.235 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:31:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:31:43.888+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:31:44.235 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:31:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:31:44.119+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:44.119693+0000) 2024-02-20T09:31:44.235 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:31:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:31:44.049+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:31:44.235 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:31:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:31:44.049+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:31:44.236 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:31:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:31:44.049+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:31:44.236 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:31:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:31:44.049+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:31:44.236 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:31:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:31:44.232+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:44.233064+0000) 2024-02-20T09:31:45.236 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:31:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:31:45.004+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:45.004504+0000) 2024-02-20T09:31:45.237 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:31:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:31:45.066+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:31:45.237 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:31:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:31:45.066+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:31:45.237 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:31:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:31:45.066+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:31:45.237 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:31:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:31:45.066+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:31:45.238 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:31:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:31:44.902+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:31:45.238 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:31:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:31:44.902+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:31:45.238 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:31:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:31:44.902+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:31:45.238 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:31:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:31:45.119+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:45.119786+0000) 2024-02-20T09:31:45.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:31:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:31:45.235+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:45.235550+0000) 2024-02-20T09:31:46.239 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:31:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:31:46.004+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:46.004631+0000) 2024-02-20T09:31:46.239 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:31:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:31:46.115+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:31:46.240 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:31:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:31:46.115+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:31:46.240 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:31:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:31:46.115+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:31:46.240 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:31:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:31:46.115+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:31:46.240 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:31:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:31:45.948+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:31:46.240 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:31:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:31:45.948+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:31:46.240 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:31:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:31:45.948+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:31:46.241 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:31:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:31:46.119+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:46.119874+0000) 2024-02-20T09:31:46.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:31:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:31:46.237+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:46.238201+0000) 2024-02-20T09:31:47.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:31:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:31:47.004+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:47.004724+0000) 2024-02-20T09:31:47.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:31:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:31:46.980+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:31:47.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:31:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:31:46.980+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:31:47.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:31:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:31:46.980+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:31:47.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:31:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:31:47.119+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:47.119991+0000) 2024-02-20T09:31:47.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:31:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:31:47.127+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:31:47.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:31:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:31:47.127+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:31:47.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:31:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:31:47.127+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:31:47.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:31:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:31:47.127+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:31:47.305 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi183.front.sepia.ceph.com' 2024-02-20T09:31:47.306 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi183.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-20T09:31:47.543 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:31:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:31:47.241+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:47.241738+0000) 2024-02-20T09:31:47.992 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:31:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:31:47.548+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 72180 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:31:47.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:31:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:31:47.541+0000 7f8a5f86e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:47.541917+0000) 2024-02-20T09:31:47.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:31:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:31:47.545+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:47.545548+0000) 2024-02-20T09:31:47.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:31:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:31:47.541+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:47.542073+0000) 2024-02-20T09:31:48.492 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:31:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:31:48.004+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:48.004808+0000) 2024-02-20T09:31:48.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:31:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:31:48.019+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:31:48.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:31:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:31:48.019+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:31:48.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:31:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:31:48.019+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:31:48.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:31:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:31:48.119+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:48.120103+0000) 2024-02-20T09:31:48.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:31:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:31:48.082+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:31:48.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:31:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:31:48.082+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:31:48.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:31:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:31:48.082+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:31:48.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:31:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:31:48.082+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:31:48.494 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:31:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:31:48.242+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:48.243005+0000) 2024-02-20T09:31:49.246 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:31:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:31:49.004+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:49.004985+0000) 2024-02-20T09:31:49.246 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:31:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:31:49.113+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:31:49.247 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:31:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:31:49.113+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:31:49.247 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:31:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:31:49.113+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:31:49.247 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:31:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:31:49.113+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:31:49.247 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:31:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:31:49.035+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:31:49.247 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:31:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:31:49.036+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:31:49.247 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:31:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:31:49.036+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:31:49.248 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:31:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:31:49.119+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:49.120178+0000) 2024-02-20T09:31:49.649 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:31:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:31:49.243+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:49.245141+0000) 2024-02-20T09:31:50.248 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:31:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:31:50.003+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:50.005098+0000) 2024-02-20T09:31:50.249 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:31:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:31:50.118+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:31:50.249 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:31:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:31:50.118+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:31:50.249 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:31:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:31:50.118+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:31:50.249 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:31:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:31:50.118+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:31:50.250 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:31:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:31:50.042+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:31:50.250 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:31:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:31:50.042+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:31:50.250 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:31:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:31:50.042+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:31:50.250 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:31:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:31:50.118+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:50.120314+0000) 2024-02-20T09:31:50.364 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.183 2024-02-20T09:31:50.649 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:31:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:31:50.246+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:50.247833+0000) 2024-02-20T09:31:51.251 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:31:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:31:51.003+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:51.005315+0000) 2024-02-20T09:31:51.252 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:31:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:31:51.086+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:31:51.252 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:31:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:31:51.086+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:31:51.252 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:31:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:31:51.086+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:31:51.252 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:31:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:31:51.119+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:51.120438+0000) 2024-02-20T09:31:51.252 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:31:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:31:51.070+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:31:51.252 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:31:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:31:51.070+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:31:51.253 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:31:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:31:51.070+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:31:51.253 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:31:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:31:51.070+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:31:51.649 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:31:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:31:51.249+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:51.250394+0000) 2024-02-20T09:31:52.254 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:31:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:31:52.004+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:52.005448+0000) 2024-02-20T09:31:52.254 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:31:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:31:52.021+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:31:52.254 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:31:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:31:52.021+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:31:52.255 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:31:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:31:52.021+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:31:52.255 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:31:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:31:52.021+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:31:52.255 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:31:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:31:52.119+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:52.120530+0000) 2024-02-20T09:31:52.255 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:31:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:31:52.121+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:31:52.255 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:31:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:31:52.121+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:31:52.255 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:31:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:31:52.121+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:31:52.551 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:31:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:31:52.251+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:52.252932+0000) 2024-02-20T09:31:52.980 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:31:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:31:52.556+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 72219 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:31:52.980 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:31:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:31:52.548+0000 7f8a5f86e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:52.550285+0000) 2024-02-20T09:31:52.981 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:31:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:31:52.548+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:52.550115+0000) 2024-02-20T09:31:52.981 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:31:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:31:52.551+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:52.552742+0000) 2024-02-20T09:31:52.981 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:31:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:31:52.977+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:31:52.981 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:31:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:31:52.977+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:31:52.981 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:31:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:31:52.977+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:31:52.981 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:31:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:31:52.977+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:31:53.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:31:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:31:53.004+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:53.005566+0000) 2024-02-20T09:31:53.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:31:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:31:53.119+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:53.120623+0000) 2024-02-20T09:31:53.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:31:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:31:53.151+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:31:53.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:31:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:31:53.151+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:31:53.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:31:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:31:53.151+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:31:53.650 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:31:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:31:53.253+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:53.254629+0000) 2024-02-20T09:31:54.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:31:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:31:54.004+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:54.005647+0000) 2024-02-20T09:31:54.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:31:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:31:53.965+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:31:54.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:31:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:31:53.965+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:31:54.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:31:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:31:53.965+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:31:54.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:31:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:31:53.965+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:31:54.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:31:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:31:54.119+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:54.120708+0000) 2024-02-20T09:31:54.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:31:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:31:54.167+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:31:54.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:31:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:31:54.167+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:31:54.244 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:31:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:31:54.167+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:31:54.650 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:31:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:31:54.255+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:54.256638+0000) 2024-02-20T09:31:55.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:31:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:31:55.004+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:55.005737+0000) 2024-02-20T09:31:55.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:31:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:31:55.119+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:55.120821+0000) 2024-02-20T09:31:55.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:31:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:31:55.193+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:31:55.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:31:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:31:55.193+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:31:55.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:31:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:31:55.193+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:31:55.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:31:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:31:54.978+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:31:55.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:31:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:31:54.978+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:31:55.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:31:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:31:54.978+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:31:55.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:31:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:31:54.978+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:31:55.650 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:31:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:31:55.257+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:55.258743+0000) 2024-02-20T09:31:56.262 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:31:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:31:56.004+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:56.005826+0000) 2024-02-20T09:31:56.262 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:31:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:31:55.993+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:31:56.262 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:31:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:31:55.993+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:31:56.262 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:31:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:31:55.993+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:31:56.263 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:31:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:31:55.993+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:31:56.263 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:31:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:31:56.119+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:56.120946+0000) 2024-02-20T09:31:56.263 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:31:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:31:56.191+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:31:56.263 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:31:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:31:56.191+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:31:56.263 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:31:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:31:56.191+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:31:56.650 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:31:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:31:56.259+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:56.261120+0000) 2024-02-20T09:31:57.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:31:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:31:57.004+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:57.005934+0000) 2024-02-20T09:31:57.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:31:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:31:57.119+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:57.121055+0000) 2024-02-20T09:31:57.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:31:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:31:57.184+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:31:57.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:31:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:31:57.184+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:31:57.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:31:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:31:57.184+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:31:57.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:31:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:31:56.957+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:31:57.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:31:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:31:56.957+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:31:57.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:31:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:31:56.957+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:31:57.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:31:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:31:56.957+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:31:57.560 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:31:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:31:57.261+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:57.263248+0000) 2024-02-20T09:31:57.968 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:31:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:31:57.564+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 72259 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:31:57.968 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:31:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:31:57.557+0000 7f8a5f86e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:57.559004+0000) 2024-02-20T09:31:57.968 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:31:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:31:57.557+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:57.559385+0000) 2024-02-20T09:31:57.969 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:31:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:31:57.560+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:57.561593+0000) 2024-02-20T09:31:57.969 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:31:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:31:57.965+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:31:57.969 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:31:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:31:57.965+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:31:58.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:31:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:31:58.004+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:58.006093+0000) 2024-02-20T09:31:58.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:31:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:31:57.965+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:31:58.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:31:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:31:57.965+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:31:58.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:31:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:31:58.119+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:58.121191+0000) 2024-02-20T09:31:58.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:31:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:31:58.138+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:31:58.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:31:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:31:58.138+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:31:58.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:31:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:31:58.138+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:27.140940+0000 (oldest deadline 2024-02-20T09:21:47.140940+0000) 2024-02-20T09:31:58.650 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:31:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:31:58.263+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:58.265273+0000) 2024-02-20T09:31:59.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:31:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:31:59.004+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:59.006188+0000) 2024-02-20T09:31:59.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:31:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:31:59.119+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:59.121292+0000) 2024-02-20T09:31:59.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:31:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:31:58.960+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:31:59.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:31:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:31:58.960+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:31:59.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:31:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:31:58.960+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:31:59.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:31:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:31:58.960+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:31:59.650 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:31:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:31:59.265+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:31:59.267207+0000) 2024-02-20T09:32:00.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:32:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:32:00.004+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:00.006338+0000) 2024-02-20T09:32:00.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:32:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:32:00.119+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:00.121381+0000) 2024-02-20T09:32:00.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:31:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:31:59.922+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:32:00.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:31:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:31:59.922+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:32:00.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:31:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:31:59.922+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:32:00.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:31:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:31:59.922+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:32:00.650 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:32:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:32:00.267+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:00.269355+0000) 2024-02-20T09:32:00.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:32:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:32:00.887+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:32:00.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:32:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:32:00.887+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:32:00.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:32:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:32:00.887+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:32:00.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:32:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:32:00.887+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:32:01.272 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:32:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:32:01.005+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:01.006509+0000) 2024-02-20T09:32:01.273 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:32:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:32:01.120+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:01.121492+0000) 2024-02-20T09:32:01.650 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:32:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:32:01.270+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:01.271499+0000) 2024-02-20T09:32:01.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:32:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:32:01.871+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:32:01.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:32:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:32:01.871+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:32:01.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:32:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:32:01.871+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:32:01.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:32:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:32:01.871+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:32:02.276 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:32:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:32:02.005+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:02.006644+0000) 2024-02-20T09:32:02.276 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:32:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:32:02.120+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:02.121588+0000) 2024-02-20T09:32:02.365 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi183.front.sepia.ceph.com' 2024-02-20T09:32:02.365 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi183.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-20T09:32:02.567 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:32:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:32:02.273+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:02.275099+0000) 2024-02-20T09:32:02.907 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:32:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:32:02.571+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 72298 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:32:02.908 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:32:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:32:02.565+0000 7f8a5f86e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:02.566436+0000) 2024-02-20T09:32:02.908 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:32:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:32:02.565+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:02.567368+0000) 2024-02-20T09:32:02.908 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:32:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:32:02.568+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:02.570359+0000) 2024-02-20T09:32:02.908 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:32:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:32:02.905+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:32:02.909 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:32:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:32:02.905+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:32:02.909 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:32:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:32:02.905+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:32:02.909 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:32:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:32:02.905+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:32:03.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:32:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:32:03.005+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:03.006778+0000) 2024-02-20T09:32:03.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:32:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:32:03.120+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:03.121710+0000) 2024-02-20T09:32:03.651 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:32:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:32:03.275+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:03.276477+0000) 2024-02-20T09:32:04.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:32:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:32:04.005+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:04.006873+0000) 2024-02-20T09:32:04.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:32:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:32:04.120+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:04.121846+0000) 2024-02-20T09:32:04.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:32:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:32:03.939+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:32:04.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:32:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:32:03.939+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:32:04.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:32:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:32:03.940+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:32:04.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:32:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:32:03.940+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:32:04.651 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:32:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:32:04.277+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:04.278717+0000) 2024-02-20T09:32:05.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:32:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:32:05.005+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:05.007003+0000) 2024-02-20T09:32:05.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:32:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:32:05.120+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:05.121946+0000) 2024-02-20T09:32:05.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:32:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:32:04.923+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:32:05.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:32:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:32:04.923+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:32:05.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:32:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:32:04.923+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:32:05.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:32:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:32:04.923+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:32:05.436 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.183 2024-02-20T09:32:05.651 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:32:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:32:05.280+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:05.282313+0000) 2024-02-20T09:32:05.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:32:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:32:05.885+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:32:05.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:32:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:32:05.885+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:32:05.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:32:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:32:05.885+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:32:05.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:32:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:32:05.885+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:32:06.286 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:32:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:32:06.005+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:06.007118+0000) 2024-02-20T09:32:06.286 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:32:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:32:06.120+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:06.122122+0000) 2024-02-20T09:32:06.651 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:32:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:32:06.283+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:06.284555+0000) 2024-02-20T09:32:06.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:32:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:32:06.767+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:31:46.482189+0000 (oldest deadline 2024-02-20T09:32:06.482189+0000) 2024-02-20T09:32:06.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:32:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:32:06.767+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:31:46.482189+0000 (oldest deadline 2024-02-20T09:32:06.482189+0000) 2024-02-20T09:32:06.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:32:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:32:06.767+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:31:46.482189+0000 (oldest deadline 2024-02-20T09:32:06.482189+0000) 2024-02-20T09:32:06.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:32:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:32:06.897+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:32:06.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:32:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:32:06.897+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:32:06.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:32:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:32:06.897+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:32:06.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:32:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:32:06.897+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:32:07.288 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:32:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:32:07.005+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:07.007239+0000) 2024-02-20T09:32:07.288 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:32:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:32:07.120+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:07.122293+0000) 2024-02-20T09:32:07.575 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:32:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:32:07.286+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:07.287395+0000) 2024-02-20T09:32:07.943 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:32:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:32:07.575+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:07.576890+0000) 2024-02-20T09:32:07.943 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:32:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:32:07.572+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:07.574143+0000) 2024-02-20T09:32:07.944 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:32:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:32:07.579+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 72338 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:32:07.944 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:32:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:32:07.572+0000 7f8a5f86e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:07.574275+0000) 2024-02-20T09:32:07.944 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:32:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:32:07.811+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:31:46.482189+0000 (oldest deadline 2024-02-20T09:32:06.482189+0000) 2024-02-20T09:32:07.944 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:32:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:32:07.811+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:31:46.482189+0000 (oldest deadline 2024-02-20T09:32:06.482189+0000) 2024-02-20T09:32:07.944 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:32:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:32:07.811+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:31:46.482189+0000 (oldest deadline 2024-02-20T09:32:06.482189+0000) 2024-02-20T09:32:08.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:32:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:32:08.121+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:08.122440+0000) 2024-02-20T09:32:08.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:32:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:32:08.005+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:08.007354+0000) 2024-02-20T09:32:08.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:32:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:32:07.940+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:32:08.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:32:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:32:07.940+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:32:08.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:32:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:32:07.940+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:32:08.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:32:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:32:07.941+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:32:08.437 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi183.front.sepia.ceph.com' 2024-02-20T09:32:08.437 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi183.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-20T09:32:08.508 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.183 2024-02-20T09:32:08.508 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-02-20T09:32:08.508 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-02-20T09:32:08.509 DEBUG:teuthology.orchestra.run.smithi133:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-02-20T09:32:08.523 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi183.front.sepia.ceph.com' 2024-02-20T09:32:08.524 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi183.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-20T09:32:08.651 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:32:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:32:08.287+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:08.288594+0000) 2024-02-20T09:32:08.983 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:32:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:32:08.839+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:31:46.482189+0000 (oldest deadline 2024-02-20T09:32:06.482189+0000) 2024-02-20T09:32:08.983 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:32:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:32:08.839+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:31:46.482189+0000 (oldest deadline 2024-02-20T09:32:06.482189+0000) 2024-02-20T09:32:08.984 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:32:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:32:08.839+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:31:46.482189+0000 (oldest deadline 2024-02-20T09:32:06.482189+0000) 2024-02-20T09:32:09.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:32:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:32:09.121+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:09.122517+0000) 2024-02-20T09:32:09.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:32:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:32:09.006+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:09.007441+0000) 2024-02-20T09:32:09.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:32:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:32:08.981+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:32:09.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:32:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:32:08.981+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:32:09.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:32:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:32:08.981+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:32:09.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:32:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:32:08.981+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:32:09.652 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:32:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:32:09.289+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:09.290634+0000) 2024-02-20T09:32:09.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:32:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:32:09.856+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:31:46.482189+0000 (oldest deadline 2024-02-20T09:32:06.482189+0000) 2024-02-20T09:32:09.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:32:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:32:09.856+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:31:46.482189+0000 (oldest deadline 2024-02-20T09:32:06.482189+0000) 2024-02-20T09:32:09.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:32:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:32:09.856+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:31:46.482189+0000 (oldest deadline 2024-02-20T09:32:06.482189+0000) 2024-02-20T09:32:10.294 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:32:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:32:10.006+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:10.007543+0000) 2024-02-20T09:32:10.294 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:32:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:32:10.121+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:10.122610+0000) 2024-02-20T09:32:10.294 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:32:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:32:10.004+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:32:10.295 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:32:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:32:10.004+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:32:10.295 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:32:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:32:10.004+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:32:10.295 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:32:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:32:10.004+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:32:10.652 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:32:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:32:10.291+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:10.292695+0000) 2024-02-20T09:32:10.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:32:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:32:10.845+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:31:46.482189+0000 (oldest deadline 2024-02-20T09:32:06.482189+0000) 2024-02-20T09:32:10.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:32:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:32:10.845+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:31:46.482189+0000 (oldest deadline 2024-02-20T09:32:06.482189+0000) 2024-02-20T09:32:10.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:32:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:32:10.845+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:31:46.482189+0000 (oldest deadline 2024-02-20T09:32:06.482189+0000) 2024-02-20T09:32:11.296 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:32:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:32:11.006+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:11.007641+0000) 2024-02-20T09:32:11.296 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:32:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:32:11.121+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:11.122705+0000) 2024-02-20T09:32:11.296 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:32:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:32:10.999+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:32:11.297 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:32:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:32:10.999+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:32:11.297 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:32:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:32:10.999+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:32:11.297 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:32:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:32:11.000+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:32:11.580 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.183 2024-02-20T09:32:11.652 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:32:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:32:11.293+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:11.295060+0000) 2024-02-20T09:32:11.974 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:32:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:32:11.816+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:31:46.482189+0000 (oldest deadline 2024-02-20T09:32:06.482189+0000) 2024-02-20T09:32:11.974 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:32:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:32:11.816+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:31:46.482189+0000 (oldest deadline 2024-02-20T09:32:06.482189+0000) 2024-02-20T09:32:11.974 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:32:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:32:11.816+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:31:46.482189+0000 (oldest deadline 2024-02-20T09:32:06.482189+0000) 2024-02-20T09:32:12.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:32:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:32:12.006+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:12.007735+0000) 2024-02-20T09:32:12.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:32:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:32:12.121+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:12.122814+0000) 2024-02-20T09:32:12.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:32:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:32:11.971+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:32:12.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:32:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:32:11.971+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:32:12.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:32:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:32:11.971+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:32:12.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:32:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:32:11.971+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:32:12.582 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:32:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:32:12.296+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:12.297456+0000) 2024-02-20T09:32:12.931 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:32:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:32:12.587+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 72377 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:32:12.932 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:32:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:32:12.581+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:12.582409+0000) 2024-02-20T09:32:12.932 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:32:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:32:12.580+0000 7f8a5f86e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:12.582117+0000) 2024-02-20T09:32:12.932 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:32:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:32:12.804+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:31:46.482189+0000 (oldest deadline 2024-02-20T09:32:06.482189+0000) 2024-02-20T09:32:12.933 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:32:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:32:12.804+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:31:46.482189+0000 (oldest deadline 2024-02-20T09:32:06.482189+0000) 2024-02-20T09:32:12.933 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:32:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:32:12.804+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:31:46.482189+0000 (oldest deadline 2024-02-20T09:32:06.482189+0000) 2024-02-20T09:32:12.933 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:32:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:32:12.583+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:12.584712+0000) 2024-02-20T09:32:12.933 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:32:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:32:12.929+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:32:12.933 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:32:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:32:12.929+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:32:12.933 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:32:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:32:12.929+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:32:13.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:32:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:32:12.929+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:32:13.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:32:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:32:13.006+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:13.007832+0000) 2024-02-20T09:32:13.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:32:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:32:13.121+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:13.122942+0000) 2024-02-20T09:32:13.652 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:32:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:32:13.297+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:13.298673+0000) 2024-02-20T09:32:13.919 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:32:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:32:13.832+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:31:46.482189+0000 (oldest deadline 2024-02-20T09:32:06.482189+0000) 2024-02-20T09:32:13.919 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:32:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:32:13.832+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:31:46.482189+0000 (oldest deadline 2024-02-20T09:32:06.482189+0000) 2024-02-20T09:32:13.920 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:32:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:32:13.832+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:31:46.482189+0000 (oldest deadline 2024-02-20T09:32:06.482189+0000) 2024-02-20T09:32:14.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:32:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:32:14.121+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:14.123031+0000) 2024-02-20T09:32:14.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:32:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:32:14.006+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:14.007918+0000) 2024-02-20T09:32:14.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:32:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:32:13.917+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:32:14.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:32:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:32:13.917+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:32:14.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:32:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:32:13.917+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:32:14.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:32:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:32:13.917+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:32:14.652 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:32:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:32:14.299+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:14.301080+0000) 2024-02-20T09:32:14.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:32:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:32:14.880+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:32:14.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:32:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:32:14.880+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:32:14.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:32:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:32:14.880+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:21:40.433787+0000 (oldest deadline 2024-02-20T09:22:00.433787+0000) 2024-02-20T09:32:14.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:32:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:32:14.880+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:32:14.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:32:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:32:14.797+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:31:46.482189+0000 (oldest deadline 2024-02-20T09:32:06.482189+0000) 2024-02-20T09:32:14.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:32:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:32:14.797+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:31:46.482189+0000 (oldest deadline 2024-02-20T09:32:06.482189+0000) 2024-02-20T09:32:14.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:32:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:32:14.797+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:31:46.482189+0000 (oldest deadline 2024-02-20T09:32:06.482189+0000) 2024-02-20T09:32:15.304 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:32:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:32:15.006+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:15.008042+0000) 2024-02-20T09:32:15.304 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:32:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:32:15.121+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:15.123122+0000) 2024-02-20T09:32:15.652 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:32:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:32:15.301+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:15.303149+0000) 2024-02-20T09:32:15.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:32:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:32:15.875+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:32:15.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:32:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:32:15.825+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:31:46.482189+0000 (oldest deadline 2024-02-20T09:32:06.482189+0000) 2024-02-20T09:32:15.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:32:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:32:15.825+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:31:46.482189+0000 (oldest deadline 2024-02-20T09:32:06.482189+0000) 2024-02-20T09:32:15.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:32:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:32:15.825+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:31:46.482189+0000 (oldest deadline 2024-02-20T09:32:06.482189+0000) 2024-02-20T09:32:16.307 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:32:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:32:16.006+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:16.008159+0000) 2024-02-20T09:32:16.308 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:32:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:32:16.121+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:16.123242+0000) 2024-02-20T09:32:16.652 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:32:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:32:16.305+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:16.306659+0000) 2024-02-20T09:32:16.919 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:32:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:32:16.800+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:31:46.482189+0000 (oldest deadline 2024-02-20T09:32:06.482189+0000) 2024-02-20T09:32:16.919 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:32:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:32:16.800+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:31:46.482189+0000 (oldest deadline 2024-02-20T09:32:06.482189+0000) 2024-02-20T09:32:16.920 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:32:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:32:16.800+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:31:46.482189+0000 (oldest deadline 2024-02-20T09:32:06.482189+0000) 2024-02-20T09:32:17.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:32:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:32:16.917+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:32:17.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:32:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:32:17.121+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:17.123334+0000) 2024-02-20T09:32:17.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:32:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:32:17.006+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:17.008267+0000) 2024-02-20T09:32:17.590 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:32:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:32:17.308+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:17.310065+0000) 2024-02-20T09:32:17.871 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:32:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:32:17.595+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 72417 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:32:17.871 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:32:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:32:17.590+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:17.592264+0000) 2024-02-20T09:32:17.871 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:32:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:32:17.869+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:32:17.872 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:32:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:32:17.589+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:17.590415+0000) 2024-02-20T09:32:17.872 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:32:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:32:17.588+0000 7f8a5f86e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:17.590098+0000) 2024-02-20T09:32:17.872 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:32:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:32:17.808+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:31:46.482189+0000 (oldest deadline 2024-02-20T09:32:06.482189+0000) 2024-02-20T09:32:17.872 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:32:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:32:17.808+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:31:46.482189+0000 (oldest deadline 2024-02-20T09:32:06.482189+0000) 2024-02-20T09:32:17.872 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:32:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:32:17.808+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:31:46.482189+0000 (oldest deadline 2024-02-20T09:32:06.482189+0000) 2024-02-20T09:32:18.124 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:32:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:32:18.006+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:18.008363+0000) 2024-02-20T09:32:18.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:32:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:32:18.310+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:18.312079+0000) 2024-02-20T09:32:18.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:32:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:32:18.122+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:18.123546+0000) 2024-02-20T09:32:18.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:32:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:32:18.863+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:32:18.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:32:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:32:18.765+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:31:46.482189+0000 (oldest deadline 2024-02-20T09:32:06.482189+0000) 2024-02-20T09:32:18.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:32:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:32:18.765+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:31:46.482189+0000 (oldest deadline 2024-02-20T09:32:06.482189+0000) 2024-02-20T09:32:18.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:32:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:32:18.765+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:31:46.482189+0000 (oldest deadline 2024-02-20T09:32:06.482189+0000) 2024-02-20T09:32:19.315 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:32:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:32:19.007+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:19.008443+0000) 2024-02-20T09:32:19.316 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:32:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:32:19.123+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:19.123675+0000) 2024-02-20T09:32:19.316 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:32:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:32:19.130+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:31:58.900551+0000 (oldest deadline 2024-02-20T09:32:18.900551+0000) 2024-02-20T09:32:19.316 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:32:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:32:19.130+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:31:58.900551+0000 (oldest deadline 2024-02-20T09:32:18.900551+0000) 2024-02-20T09:32:19.316 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:32:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:32:19.130+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:31:58.900551+0000 (oldest deadline 2024-02-20T09:32:18.900551+0000) 2024-02-20T09:32:19.653 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:32:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:32:19.313+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:19.314565+0000) 2024-02-20T09:32:19.906 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:32:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:32:19.767+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:31:46.482189+0000 (oldest deadline 2024-02-20T09:32:06.482189+0000) 2024-02-20T09:32:19.907 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:32:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:32:19.767+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:31:46.482189+0000 (oldest deadline 2024-02-20T09:32:06.482189+0000) 2024-02-20T09:32:19.907 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:32:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:32:19.767+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:31:46.482189+0000 (oldest deadline 2024-02-20T09:32:06.482189+0000) 2024-02-20T09:32:20.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:32:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:32:20.007+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:20.008543+0000) 2024-02-20T09:32:20.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:32:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:32:19.904+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:32:20.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:32:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:32:20.111+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:31:58.900551+0000 (oldest deadline 2024-02-20T09:32:18.900551+0000) 2024-02-20T09:32:20.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:32:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:32:20.111+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:31:58.900551+0000 (oldest deadline 2024-02-20T09:32:18.900551+0000) 2024-02-20T09:32:20.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:32:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:32:20.111+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:31:58.900551+0000 (oldest deadline 2024-02-20T09:32:18.900551+0000) 2024-02-20T09:32:20.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:32:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:32:20.122+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:20.123793+0000) 2024-02-20T09:32:20.581 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi183.front.sepia.ceph.com' 2024-02-20T09:32:20.582 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi183.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-20T09:32:20.653 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:32:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:32:20.315+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:20.317121+0000) 2024-02-20T09:32:20.932 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:32:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:32:20.811+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:31:46.482189+0000 (oldest deadline 2024-02-20T09:32:06.482189+0000) 2024-02-20T09:32:20.932 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:32:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:32:20.811+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:31:46.482189+0000 (oldest deadline 2024-02-20T09:32:06.482189+0000) 2024-02-20T09:32:20.932 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:32:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:32:20.811+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:31:46.482189+0000 (oldest deadline 2024-02-20T09:32:06.482189+0000) 2024-02-20T09:32:21.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:32:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:32:21.008+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:21.008661+0000) 2024-02-20T09:32:21.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:32:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:32:20.930+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:32:21.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:32:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:32:21.114+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:31:58.900551+0000 (oldest deadline 2024-02-20T09:32:18.900551+0000) 2024-02-20T09:32:21.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:32:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:32:21.114+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:31:58.900551+0000 (oldest deadline 2024-02-20T09:32:18.900551+0000) 2024-02-20T09:32:21.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:32:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:32:21.114+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:31:58.900551+0000 (oldest deadline 2024-02-20T09:32:18.900551+0000) 2024-02-20T09:32:21.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:32:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:32:21.122+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:21.123918+0000) 2024-02-20T09:32:21.653 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:32:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:32:21.319+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:21.319604+0000) 2024-02-20T09:32:21.923 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:32:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:32:21.840+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:31:46.482189+0000 (oldest deadline 2024-02-20T09:32:06.482189+0000) 2024-02-20T09:32:21.923 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:32:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:32:21.840+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:31:46.482189+0000 (oldest deadline 2024-02-20T09:32:06.482189+0000) 2024-02-20T09:32:21.924 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:32:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:32:21.840+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:31:46.482189+0000 (oldest deadline 2024-02-20T09:32:06.482189+0000) 2024-02-20T09:32:22.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:32:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:32:22.008+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:22.008814+0000) 2024-02-20T09:32:22.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:32:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:32:21.921+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:32:22.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:32:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:32:22.091+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:31:58.900551+0000 (oldest deadline 2024-02-20T09:32:18.900551+0000) 2024-02-20T09:32:22.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:32:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:32:22.091+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:31:58.900551+0000 (oldest deadline 2024-02-20T09:32:18.900551+0000) 2024-02-20T09:32:22.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:32:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:32:22.091+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:31:58.900551+0000 (oldest deadline 2024-02-20T09:32:18.900551+0000) 2024-02-20T09:32:22.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:32:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:32:22.122+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:22.124009+0000) 2024-02-20T09:32:22.598 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:32:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:32:22.322+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:22.323407+0000) 2024-02-20T09:32:22.948 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:32:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:32:22.603+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 72456 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:32:22.949 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:32:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:32:22.596+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:22.598040+0000) 2024-02-20T09:32:22.949 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:32:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:32:22.599+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:22.600994+0000) 2024-02-20T09:32:22.949 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:32:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:32:22.946+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:32:22.950 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:32:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:32:22.596+0000 7f8a5f86e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:22.597825+0000) 2024-02-20T09:32:22.950 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:32:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:32:22.838+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:31:46.482189+0000 (oldest deadline 2024-02-20T09:32:06.482189+0000) 2024-02-20T09:32:22.950 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:32:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:32:22.838+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:31:46.482189+0000 (oldest deadline 2024-02-20T09:32:06.482189+0000) 2024-02-20T09:32:22.950 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:32:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:32:22.838+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:31:46.482189+0000 (oldest deadline 2024-02-20T09:32:06.482189+0000) 2024-02-20T09:32:23.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:32:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:32:23.007+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:23.008930+0000) 2024-02-20T09:32:23.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:32:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:32:23.122+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:23.124143+0000) 2024-02-20T09:32:23.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:32:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:32:23.128+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:31:58.900551+0000 (oldest deadline 2024-02-20T09:32:18.900551+0000) 2024-02-20T09:32:23.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:32:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:32:23.128+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:31:58.900551+0000 (oldest deadline 2024-02-20T09:32:18.900551+0000) 2024-02-20T09:32:23.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:32:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:32:23.128+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:31:58.900551+0000 (oldest deadline 2024-02-20T09:32:18.900551+0000) 2024-02-20T09:32:23.644 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.183 2024-02-20T09:32:23.653 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:32:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:32:23.323+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:23.324619+0000) 2024-02-20T09:32:23.920 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:32:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:32:23.813+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:31:46.482189+0000 (oldest deadline 2024-02-20T09:32:06.482189+0000) 2024-02-20T09:32:23.920 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:32:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:32:23.813+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:31:46.482189+0000 (oldest deadline 2024-02-20T09:32:06.482189+0000) 2024-02-20T09:32:23.920 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:32:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:32:23.813+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:31:46.482189+0000 (oldest deadline 2024-02-20T09:32:06.482189+0000) 2024-02-20T09:32:24.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:32:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:32:24.008+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:24.009044+0000) 2024-02-20T09:32:24.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:32:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:32:23.917+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:32:24.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:32:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:32:24.123+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:24.124324+0000) 2024-02-20T09:32:24.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:32:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:32:24.145+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:31:58.900551+0000 (oldest deadline 2024-02-20T09:32:18.900551+0000) 2024-02-20T09:32:24.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:32:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:32:24.145+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:31:58.900551+0000 (oldest deadline 2024-02-20T09:32:18.900551+0000) 2024-02-20T09:32:24.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:32:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:32:24.145+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:31:58.900551+0000 (oldest deadline 2024-02-20T09:32:18.900551+0000) 2024-02-20T09:32:24.653 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:32:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:32:24.326+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:24.326585+0000) 2024-02-20T09:32:24.928 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:32:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:32:24.857+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:31:46.482189+0000 (oldest deadline 2024-02-20T09:32:06.482189+0000) 2024-02-20T09:32:24.928 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:32:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:32:24.858+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:31:46.482189+0000 (oldest deadline 2024-02-20T09:32:06.482189+0000) 2024-02-20T09:32:24.928 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:32:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:32:24.858+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:31:46.482189+0000 (oldest deadline 2024-02-20T09:32:06.482189+0000) 2024-02-20T09:32:25.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:32:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:32:25.008+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:25.009184+0000) 2024-02-20T09:32:25.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:32:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:32:24.926+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:32:25.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:32:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:32:25.096+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:31:58.900551+0000 (oldest deadline 2024-02-20T09:32:18.900551+0000) 2024-02-20T09:32:25.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:32:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:32:25.096+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:31:58.900551+0000 (oldest deadline 2024-02-20T09:32:18.900551+0000) 2024-02-20T09:32:25.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:32:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:32:25.096+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:31:58.900551+0000 (oldest deadline 2024-02-20T09:32:18.900551+0000) 2024-02-20T09:32:25.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:32:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:32:25.124+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:25.124483+0000) 2024-02-20T09:32:25.653 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:32:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:32:25.328+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:25.328890+0000) 2024-02-20T09:32:25.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:32:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:32:25.889+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:32:25.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:32:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:32:25.871+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:31:46.482189+0000 (oldest deadline 2024-02-20T09:32:06.482189+0000) 2024-02-20T09:32:25.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:32:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:32:25.871+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:31:46.482189+0000 (oldest deadline 2024-02-20T09:32:06.482189+0000) 2024-02-20T09:32:25.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:32:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:32:25.871+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:31:46.482189+0000 (oldest deadline 2024-02-20T09:32:06.482189+0000) 2024-02-20T09:32:26.333 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:32:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:32:26.008+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:26.009301+0000) 2024-02-20T09:32:26.334 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:32:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:32:26.075+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:31:58.900551+0000 (oldest deadline 2024-02-20T09:32:18.900551+0000) 2024-02-20T09:32:26.334 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:32:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:32:26.075+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:31:58.900551+0000 (oldest deadline 2024-02-20T09:32:18.900551+0000) 2024-02-20T09:32:26.334 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:32:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:32:26.075+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:31:58.900551+0000 (oldest deadline 2024-02-20T09:32:18.900551+0000) 2024-02-20T09:32:26.334 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:32:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:32:26.124+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:26.124614+0000) 2024-02-20T09:32:26.653 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:32:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:32:26.332+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:26.332509+0000) 2024-02-20T09:32:26.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:32:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:32:26.855+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:32:26.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:32:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:32:26.897+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:31:46.482189+0000 (oldest deadline 2024-02-20T09:32:06.482189+0000) 2024-02-20T09:32:26.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:32:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:32:26.897+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:31:46.482189+0000 (oldest deadline 2024-02-20T09:32:06.482189+0000) 2024-02-20T09:32:26.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:32:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:32:26.897+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:31:46.482189+0000 (oldest deadline 2024-02-20T09:32:06.482189+0000) 2024-02-20T09:32:27.337 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:32:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:32:27.009+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:27.009413+0000) 2024-02-20T09:32:27.337 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:32:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:32:27.080+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:31:58.900551+0000 (oldest deadline 2024-02-20T09:32:18.900551+0000) 2024-02-20T09:32:27.338 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:32:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:32:27.080+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:31:58.900551+0000 (oldest deadline 2024-02-20T09:32:18.900551+0000) 2024-02-20T09:32:27.338 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:32:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:32:27.080+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:31:58.900551+0000 (oldest deadline 2024-02-20T09:32:18.900551+0000) 2024-02-20T09:32:27.338 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:32:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:32:27.124+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:27.124705+0000) 2024-02-20T09:32:27.607 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:32:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:32:27.335+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:27.336205+0000) 2024-02-20T09:32:27.933 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:32:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:32:27.612+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 72496 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:32:27.933 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:32:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:32:27.605+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:27.606144+0000) 2024-02-20T09:32:27.934 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:32:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:32:27.608+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:27.608886+0000) 2024-02-20T09:32:27.934 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:32:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:32:27.831+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:32:27.934 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:32:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:32:27.605+0000 7f8a5f86e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:27.606061+0000) 2024-02-20T09:32:27.934 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:32:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:32:27.932+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:31:46.482189+0000 (oldest deadline 2024-02-20T09:32:06.482189+0000) 2024-02-20T09:32:27.935 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:32:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:32:27.932+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:31:46.482189+0000 (oldest deadline 2024-02-20T09:32:06.482189+0000) 2024-02-20T09:32:27.935 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:32:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:32:27.932+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:31:46.482189+0000 (oldest deadline 2024-02-20T09:32:06.482189+0000) 2024-02-20T09:32:28.081 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:32:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:32:28.009+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:28.009541+0000) 2024-02-20T09:32:28.081 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:32:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:32:28.078+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:31:58.900551+0000 (oldest deadline 2024-02-20T09:32:18.900551+0000) 2024-02-20T09:32:28.082 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:32:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:32:28.078+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:31:58.900551+0000 (oldest deadline 2024-02-20T09:32:18.900551+0000) 2024-02-20T09:32:28.082 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:32:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:32:28.078+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:31:58.900551+0000 (oldest deadline 2024-02-20T09:32:18.900551+0000) 2024-02-20T09:32:28.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:32:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:32:28.124+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:28.124828+0000) 2024-02-20T09:32:28.654 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:32:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:32:28.337+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:28.337531+0000) 2024-02-20T09:32:28.923 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:32:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:32:28.878+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:32:29.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:32:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:32:29.071+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:31:58.900551+0000 (oldest deadline 2024-02-20T09:32:18.900551+0000) 2024-02-20T09:32:29.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:32:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:32:29.071+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:31:58.900551+0000 (oldest deadline 2024-02-20T09:32:18.900551+0000) 2024-02-20T09:32:29.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:32:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:32:29.071+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:31:58.900551+0000 (oldest deadline 2024-02-20T09:32:18.900551+0000) 2024-02-20T09:32:29.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:32:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:32:29.124+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:29.124900+0000) 2024-02-20T09:32:29.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:32:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:32:28.922+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:31:46.482189+0000 (oldest deadline 2024-02-20T09:32:06.482189+0000) 2024-02-20T09:32:29.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:32:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:32:28.922+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:31:46.482189+0000 (oldest deadline 2024-02-20T09:32:06.482189+0000) 2024-02-20T09:32:29.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:32:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:32:28.922+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:31:46.482189+0000 (oldest deadline 2024-02-20T09:32:06.482189+0000) 2024-02-20T09:32:29.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:32:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:32:29.009+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:29.009619+0000) 2024-02-20T09:32:29.654 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:32:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:32:29.339+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:29.339647+0000) 2024-02-20T09:32:29.965 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:32:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:32:29.832+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:32:30.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:32:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:32:29.963+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:31:46.482189+0000 (oldest deadline 2024-02-20T09:32:06.482189+0000) 2024-02-20T09:32:30.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:32:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:32:29.963+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:31:46.482189+0000 (oldest deadline 2024-02-20T09:32:06.482189+0000) 2024-02-20T09:32:30.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:32:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:32:29.963+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:31:46.482189+0000 (oldest deadline 2024-02-20T09:32:06.482189+0000) 2024-02-20T09:32:30.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:32:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:32:30.009+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:30.009707+0000) 2024-02-20T09:32:30.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:32:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:32:30.060+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:31:58.900551+0000 (oldest deadline 2024-02-20T09:32:18.900551+0000) 2024-02-20T09:32:30.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:32:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:32:30.060+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:31:58.900551+0000 (oldest deadline 2024-02-20T09:32:18.900551+0000) 2024-02-20T09:32:30.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:32:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:32:30.060+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:31:58.900551+0000 (oldest deadline 2024-02-20T09:32:18.900551+0000) 2024-02-20T09:32:30.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:32:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:32:30.124+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:30.125003+0000) 2024-02-20T09:32:30.654 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:32:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:32:30.341+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:30.341761+0000) 2024-02-20T09:32:30.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:32:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:32:30.796+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:32:31.345 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:32:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:32:31.079+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:31:58.900551+0000 (oldest deadline 2024-02-20T09:32:18.900551+0000) 2024-02-20T09:32:31.345 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:32:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:32:31.079+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:31:58.900551+0000 (oldest deadline 2024-02-20T09:32:18.900551+0000) 2024-02-20T09:32:31.345 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:32:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:32:31.079+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:31:58.900551+0000 (oldest deadline 2024-02-20T09:32:18.900551+0000) 2024-02-20T09:32:31.345 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:32:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:32:31.124+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:31.125147+0000) 2024-02-20T09:32:31.346 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:32:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:32:31.003+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:31:46.482189+0000 (oldest deadline 2024-02-20T09:32:06.482189+0000) 2024-02-20T09:32:31.346 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:32:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:32:31.003+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:31:46.482189+0000 (oldest deadline 2024-02-20T09:32:06.482189+0000) 2024-02-20T09:32:31.346 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:32:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:32:31.003+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:31:46.482189+0000 (oldest deadline 2024-02-20T09:32:06.482189+0000) 2024-02-20T09:32:31.346 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:32:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:32:31.009+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:31.009829+0000) 2024-02-20T09:32:31.654 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:32:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:32:31.343+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:31.344088+0000) 2024-02-20T09:32:31.959 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:32:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:32:31.807+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:32:32.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:32:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:32:32.041+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:31:58.900551+0000 (oldest deadline 2024-02-20T09:32:18.900551+0000) 2024-02-20T09:32:32.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:32:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:32:32.041+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:31:58.900551+0000 (oldest deadline 2024-02-20T09:32:18.900551+0000) 2024-02-20T09:32:32.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:32:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:32:32.041+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:31:58.900551+0000 (oldest deadline 2024-02-20T09:32:18.900551+0000) 2024-02-20T09:32:32.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:32:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:32:32.124+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:32.125261+0000) 2024-02-20T09:32:32.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:32:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:32:31.958+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:31:46.482189+0000 (oldest deadline 2024-02-20T09:32:06.482189+0000) 2024-02-20T09:32:32.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:32:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:32:31.958+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:31:46.482189+0000 (oldest deadline 2024-02-20T09:32:06.482189+0000) 2024-02-20T09:32:32.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:32:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:32:31.958+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:31:46.482189+0000 (oldest deadline 2024-02-20T09:32:06.482189+0000) 2024-02-20T09:32:32.244 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:32:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:32:32.009+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:32.009948+0000) 2024-02-20T09:32:32.614 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:32:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:32:32.346+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:32.346393+0000) 2024-02-20T09:32:32.956 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:32:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:32:32.613+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:32.614321+0000) 2024-02-20T09:32:32.956 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:32:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:32:32.621+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 72535 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:32:32.957 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:32:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:32:32.616+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:32.616809+0000) 2024-02-20T09:32:32.957 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:32:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:32:32.783+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:32:32.957 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:32:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:32:32.613+0000 7f8a5f86e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:32.613827+0000) 2024-02-20T09:32:32.957 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:32:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:32:32.954+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:31:46.482189+0000 (oldest deadline 2024-02-20T09:32:06.482189+0000) 2024-02-20T09:32:32.957 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:32:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:32:32.954+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:31:46.482189+0000 (oldest deadline 2024-02-20T09:32:06.482189+0000) 2024-02-20T09:32:32.957 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:32:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:32:32.954+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:31:46.482189+0000 (oldest deadline 2024-02-20T09:32:06.482189+0000) 2024-02-20T09:32:33.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:32:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:32:33.009+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:33.010069+0000) 2024-02-20T09:32:33.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:32:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:32:33.028+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:31:58.900551+0000 (oldest deadline 2024-02-20T09:32:18.900551+0000) 2024-02-20T09:32:33.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:32:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:32:33.028+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:31:58.900551+0000 (oldest deadline 2024-02-20T09:32:18.900551+0000) 2024-02-20T09:32:33.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:32:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:32:33.028+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:31:58.900551+0000 (oldest deadline 2024-02-20T09:32:18.900551+0000) 2024-02-20T09:32:33.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:32:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:32:33.125+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:33.125392+0000) 2024-02-20T09:32:33.654 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:32:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:32:33.347+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:33.347612+0000) 2024-02-20T09:32:33.913 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:32:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:32:33.749+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:32:34.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:32:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:32:33.912+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:31:46.482189+0000 (oldest deadline 2024-02-20T09:32:06.482189+0000) 2024-02-20T09:32:34.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:32:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:32:33.912+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:31:46.482189+0000 (oldest deadline 2024-02-20T09:32:06.482189+0000) 2024-02-20T09:32:34.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:32:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:32:33.912+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:31:46.482189+0000 (oldest deadline 2024-02-20T09:32:06.482189+0000) 2024-02-20T09:32:34.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:32:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:32:34.009+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:34.010186+0000) 2024-02-20T09:32:34.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:32:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:32:34.025+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:31:58.900551+0000 (oldest deadline 2024-02-20T09:32:18.900551+0000) 2024-02-20T09:32:34.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:32:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:32:34.025+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:31:58.900551+0000 (oldest deadline 2024-02-20T09:32:18.900551+0000) 2024-02-20T09:32:34.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:32:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:32:34.025+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:31:58.900551+0000 (oldest deadline 2024-02-20T09:32:18.900551+0000) 2024-02-20T09:32:34.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:32:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:32:34.125+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:34.125480+0000) 2024-02-20T09:32:34.654 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:32:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:32:34.350+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:34.351136+0000) 2024-02-20T09:32:34.926 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:32:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:32:34.754+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:32:35.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:32:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:32:34.989+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:31:58.900551+0000 (oldest deadline 2024-02-20T09:32:18.900551+0000) 2024-02-20T09:32:35.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:32:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:32:34.989+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:31:58.900551+0000 (oldest deadline 2024-02-20T09:32:18.900551+0000) 2024-02-20T09:32:35.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:32:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:32:34.989+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:31:58.900551+0000 (oldest deadline 2024-02-20T09:32:18.900551+0000) 2024-02-20T09:32:35.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:32:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:32:35.125+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:35.125569+0000) 2024-02-20T09:32:35.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:32:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:32:34.925+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:31:46.482189+0000 (oldest deadline 2024-02-20T09:32:06.482189+0000) 2024-02-20T09:32:35.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:32:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:32:34.925+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:31:46.482189+0000 (oldest deadline 2024-02-20T09:32:06.482189+0000) 2024-02-20T09:32:35.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:32:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:32:34.925+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:31:46.482189+0000 (oldest deadline 2024-02-20T09:32:06.482189+0000) 2024-02-20T09:32:35.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:32:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:32:35.009+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:35.010316+0000) 2024-02-20T09:32:35.645 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi183.front.sepia.ceph.com' 2024-02-20T09:32:35.645 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi183.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-20T09:32:35.654 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:32:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:32:35.352+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:35.353079+0000) 2024-02-20T09:32:35.969 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:32:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:32:35.788+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:32:36.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:32:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:32:35.968+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:31:46.482189+0000 (oldest deadline 2024-02-20T09:32:06.482189+0000) 2024-02-20T09:32:36.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:32:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:32:35.968+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:31:46.482189+0000 (oldest deadline 2024-02-20T09:32:06.482189+0000) 2024-02-20T09:32:36.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:32:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:32:35.968+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:31:46.482189+0000 (oldest deadline 2024-02-20T09:32:06.482189+0000) 2024-02-20T09:32:36.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:32:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:32:36.010+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:36.010431+0000) 2024-02-20T09:32:36.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:32:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:32:36.028+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:31:58.900551+0000 (oldest deadline 2024-02-20T09:32:18.900551+0000) 2024-02-20T09:32:36.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:32:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:32:36.028+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:31:58.900551+0000 (oldest deadline 2024-02-20T09:32:18.900551+0000) 2024-02-20T09:32:36.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:32:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:32:36.028+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:31:58.900551+0000 (oldest deadline 2024-02-20T09:32:18.900551+0000) 2024-02-20T09:32:36.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:32:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:32:36.125+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:36.125679+0000) 2024-02-20T09:32:36.654 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:32:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:32:36.355+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:36.355589+0000) 2024-02-20T09:32:36.964 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:32:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:32:36.794+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:32:37.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:32:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:32:36.962+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:31:46.482189+0000 (oldest deadline 2024-02-20T09:32:06.482189+0000) 2024-02-20T09:32:37.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:32:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:32:36.962+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:31:46.482189+0000 (oldest deadline 2024-02-20T09:32:06.482189+0000) 2024-02-20T09:32:37.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:32:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:32:36.962+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:31:46.482189+0000 (oldest deadline 2024-02-20T09:32:06.482189+0000) 2024-02-20T09:32:37.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:32:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:32:37.010+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:37.010521+0000) 2024-02-20T09:32:37.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:32:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:32:37.070+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:31:58.900551+0000 (oldest deadline 2024-02-20T09:32:18.900551+0000) 2024-02-20T09:32:37.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:32:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:32:37.070+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:31:58.900551+0000 (oldest deadline 2024-02-20T09:32:18.900551+0000) 2024-02-20T09:32:37.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:32:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:32:37.070+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:31:58.900551+0000 (oldest deadline 2024-02-20T09:32:18.900551+0000) 2024-02-20T09:32:37.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:32:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:32:37.125+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:37.125850+0000) 2024-02-20T09:32:37.623 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:32:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:32:37.357+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:37.357810+0000) 2024-02-20T09:32:37.930 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:32:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:32:37.629+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 72575 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:32:37.930 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:32:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:32:37.622+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:37.622605+0000) 2024-02-20T09:32:37.931 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:32:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:32:37.622+0000 7f8a5f86e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:37.623026+0000) 2024-02-20T09:32:37.931 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:32:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:32:37.928+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:31:46.482189+0000 (oldest deadline 2024-02-20T09:32:06.482189+0000) 2024-02-20T09:32:37.931 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:32:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:32:37.928+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:31:46.482189+0000 (oldest deadline 2024-02-20T09:32:06.482189+0000) 2024-02-20T09:32:37.931 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:32:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:32:37.928+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:31:46.482189+0000 (oldest deadline 2024-02-20T09:32:06.482189+0000) 2024-02-20T09:32:37.931 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:32:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:32:37.624+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:37.625342+0000) 2024-02-20T09:32:37.931 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:32:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:32:37.798+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:32:16.988591+0000 (oldest deadline 2024-02-20T09:32:36.988591+0000) 2024-02-20T09:32:37.932 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:32:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:32:37.798+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:32:16.988591+0000 (oldest deadline 2024-02-20T09:32:36.988591+0000) 2024-02-20T09:32:37.932 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:32:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:32:37.798+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:32:16.988591+0000 (oldest deadline 2024-02-20T09:32:36.988591+0000) 2024-02-20T09:32:37.932 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:32:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:32:37.798+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:32:38.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:32:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:32:38.010+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:38.010635+0000) 2024-02-20T09:32:38.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:32:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:32:38.098+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:31:58.900551+0000 (oldest deadline 2024-02-20T09:32:18.900551+0000) 2024-02-20T09:32:38.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:32:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:32:38.098+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:31:58.900551+0000 (oldest deadline 2024-02-20T09:32:18.900551+0000) 2024-02-20T09:32:38.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:32:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:32:38.098+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:31:58.900551+0000 (oldest deadline 2024-02-20T09:32:18.900551+0000) 2024-02-20T09:32:38.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:32:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:32:38.125+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:38.125997+0000) 2024-02-20T09:32:38.655 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:32:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:32:38.358+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:38.359038+0000) 2024-02-20T09:32:38.720 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.183 2024-02-20T09:32:38.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:32:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:32:38.900+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:31:46.482189+0000 (oldest deadline 2024-02-20T09:32:06.482189+0000) 2024-02-20T09:32:38.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:32:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:32:38.900+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:31:46.482189+0000 (oldest deadline 2024-02-20T09:32:06.482189+0000) 2024-02-20T09:32:38.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:32:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:32:38.900+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:31:46.482189+0000 (oldest deadline 2024-02-20T09:32:06.482189+0000) 2024-02-20T09:32:38.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:32:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:32:38.757+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:32:16.988591+0000 (oldest deadline 2024-02-20T09:32:36.988591+0000) 2024-02-20T09:32:38.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:32:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:32:38.757+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:32:16.988591+0000 (oldest deadline 2024-02-20T09:32:36.988591+0000) 2024-02-20T09:32:38.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:32:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:32:38.757+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:32:16.988591+0000 (oldest deadline 2024-02-20T09:32:36.988591+0000) 2024-02-20T09:32:38.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:32:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:32:38.757+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:32:39.362 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:32:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:32:39.010+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:39.010800+0000) 2024-02-20T09:32:39.363 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:32:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:32:39.121+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:31:58.900551+0000 (oldest deadline 2024-02-20T09:32:18.900551+0000) 2024-02-20T09:32:39.363 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:32:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:32:39.121+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:31:58.900551+0000 (oldest deadline 2024-02-20T09:32:18.900551+0000) 2024-02-20T09:32:39.363 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:32:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:32:39.121+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:31:58.900551+0000 (oldest deadline 2024-02-20T09:32:18.900551+0000) 2024-02-20T09:32:39.363 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:32:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:32:39.125+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:39.126128+0000) 2024-02-20T09:32:39.655 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:32:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:32:39.361+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:39.361621+0000) 2024-02-20T09:32:39.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:32:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:32:39.871+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:31:46.482189+0000 (oldest deadline 2024-02-20T09:32:06.482189+0000) 2024-02-20T09:32:39.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:32:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:32:39.871+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:31:46.482189+0000 (oldest deadline 2024-02-20T09:32:06.482189+0000) 2024-02-20T09:32:39.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:32:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:32:39.871+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:31:46.482189+0000 (oldest deadline 2024-02-20T09:32:06.482189+0000) 2024-02-20T09:32:39.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:32:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:32:39.775+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:32:16.988591+0000 (oldest deadline 2024-02-20T09:32:36.988591+0000) 2024-02-20T09:32:39.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:32:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:32:39.775+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:32:16.988591+0000 (oldest deadline 2024-02-20T09:32:36.988591+0000) 2024-02-20T09:32:39.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:32:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:32:39.775+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:32:16.988591+0000 (oldest deadline 2024-02-20T09:32:36.988591+0000) 2024-02-20T09:32:39.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:32:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:32:39.775+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:32:40.365 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:32:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:32:40.010+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:40.010961+0000) 2024-02-20T09:32:40.365 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:32:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:32:40.125+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:40.126249+0000) 2024-02-20T09:32:40.365 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:32:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:32:40.131+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:31:58.900551+0000 (oldest deadline 2024-02-20T09:32:18.900551+0000) 2024-02-20T09:32:40.365 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:32:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:32:40.131+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:31:58.900551+0000 (oldest deadline 2024-02-20T09:32:18.900551+0000) 2024-02-20T09:32:40.365 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:32:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:32:40.131+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:31:58.900551+0000 (oldest deadline 2024-02-20T09:32:18.900551+0000) 2024-02-20T09:32:40.655 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:32:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:32:40.363+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:40.363736+0000) 2024-02-20T09:32:40.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:32:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:32:40.872+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:31:46.482189+0000 (oldest deadline 2024-02-20T09:32:06.482189+0000) 2024-02-20T09:32:40.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:32:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:32:40.872+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:31:46.482189+0000 (oldest deadline 2024-02-20T09:32:06.482189+0000) 2024-02-20T09:32:40.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:32:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:32:40.872+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:31:46.482189+0000 (oldest deadline 2024-02-20T09:32:06.482189+0000) 2024-02-20T09:32:40.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:32:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:32:40.743+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:32:16.988591+0000 (oldest deadline 2024-02-20T09:32:36.988591+0000) 2024-02-20T09:32:40.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:32:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:32:40.743+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:32:16.988591+0000 (oldest deadline 2024-02-20T09:32:36.988591+0000) 2024-02-20T09:32:40.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:32:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:32:40.743+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:32:16.988591+0000 (oldest deadline 2024-02-20T09:32:36.988591+0000) 2024-02-20T09:32:40.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:32:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:32:40.743+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:32:41.367 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:32:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:32:41.010+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:41.011116+0000) 2024-02-20T09:32:41.367 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:32:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:32:41.125+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:41.126376+0000) 2024-02-20T09:32:41.368 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:32:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:32:41.154+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:31:58.900551+0000 (oldest deadline 2024-02-20T09:32:18.900551+0000) 2024-02-20T09:32:41.368 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:32:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:32:41.155+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:31:58.900551+0000 (oldest deadline 2024-02-20T09:32:18.900551+0000) 2024-02-20T09:32:41.368 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:32:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:32:41.155+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:31:58.900551+0000 (oldest deadline 2024-02-20T09:32:18.900551+0000) 2024-02-20T09:32:41.655 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:32:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:32:41.365+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:41.366318+0000) 2024-02-20T09:32:41.721 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi183.front.sepia.ceph.com' 2024-02-20T09:32:41.721 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi183.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-20T09:32:41.788 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.183 2024-02-20T09:32:41.788 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-02-20T09:32:41.788 INFO:tasks.cephadm:Archiving crash dumps... 2024-02-20T09:32:41.791 DEBUG:teuthology.misc:Transferring archived files from smithi133:/var/lib/ceph/4537327c-cfc4-11ee-95bb-87774f69a715/crash to /home/teuthworker/archive/pdonnell-2024-02-20_07:23:03-fs:upgrade:mds_upgrade_sequence-wip-batrick-testing-20240220.022152-distro-default-smithi/7568541/remote/smithi133/crash 2024-02-20T09:32:41.791 DEBUG:teuthology.orchestra.run.smithi133:> sudo tar c -f - -C /var/lib/ceph/4537327c-cfc4-11ee-95bb-87774f69a715/crash -- . 2024-02-20T09:32:41.871 DEBUG:teuthology.misc:Transferring archived files from smithi183:/var/lib/ceph/4537327c-cfc4-11ee-95bb-87774f69a715/crash to /home/teuthworker/archive/pdonnell-2024-02-20_07:23:03-fs:upgrade:mds_upgrade_sequence-wip-batrick-testing-20240220.022152-distro-default-smithi/7568541/remote/smithi183/crash 2024-02-20T09:32:41.872 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi183.front.sepia.ceph.com' 2024-02-20T09:32:41.873 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi183.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-20T09:32:41.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:32:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:32:41.839+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:31:46.482189+0000 (oldest deadline 2024-02-20T09:32:06.482189+0000) 2024-02-20T09:32:41.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:32:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:32:41.839+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:31:46.482189+0000 (oldest deadline 2024-02-20T09:32:06.482189+0000) 2024-02-20T09:32:41.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:32:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:32:41.839+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:31:46.482189+0000 (oldest deadline 2024-02-20T09:32:06.482189+0000) 2024-02-20T09:32:41.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:32:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:32:41.777+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:32:16.988591+0000 (oldest deadline 2024-02-20T09:32:36.988591+0000) 2024-02-20T09:32:41.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:32:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:32:41.777+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:32:16.988591+0000 (oldest deadline 2024-02-20T09:32:36.988591+0000) 2024-02-20T09:32:41.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:32:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:32:41.777+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:32:16.988591+0000 (oldest deadline 2024-02-20T09:32:36.988591+0000) 2024-02-20T09:32:41.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:32:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:32:41.777+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:32:42.369 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:32:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:32:42.010+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:42.011291+0000) 2024-02-20T09:32:42.370 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:32:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:32:42.126+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:42.126445+0000) 2024-02-20T09:32:42.370 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:32:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:32:42.180+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:31:58.900551+0000 (oldest deadline 2024-02-20T09:32:18.900551+0000) 2024-02-20T09:32:42.370 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:32:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:32:42.180+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:31:58.900551+0000 (oldest deadline 2024-02-20T09:32:18.900551+0000) 2024-02-20T09:32:42.370 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:32:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:32:42.180+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:31:58.900551+0000 (oldest deadline 2024-02-20T09:32:18.900551+0000) 2024-02-20T09:32:42.632 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:32:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:32:42.368+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:42.368543+0000) 2024-02-20T09:32:42.992 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:32:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:32:42.630+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:42.631207+0000) 2024-02-20T09:32:42.993 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:32:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:32:42.637+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 72614 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:32:42.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:32:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:32:42.630+0000 7f8a5f86e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:42.630941+0000) 2024-02-20T09:32:42.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:32:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:32:42.798+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:31:46.482189+0000 (oldest deadline 2024-02-20T09:32:06.482189+0000) 2024-02-20T09:32:42.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:32:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:32:42.798+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:31:46.482189+0000 (oldest deadline 2024-02-20T09:32:06.482189+0000) 2024-02-20T09:32:42.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:32:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:32:42.798+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:31:46.482189+0000 (oldest deadline 2024-02-20T09:32:06.482189+0000) 2024-02-20T09:32:42.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:32:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:32:42.634+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:42.634894+0000) 2024-02-20T09:32:42.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:32:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:32:42.781+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:32:16.988591+0000 (oldest deadline 2024-02-20T09:32:36.988591+0000) 2024-02-20T09:32:42.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:32:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:32:42.781+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:32:16.988591+0000 (oldest deadline 2024-02-20T09:32:36.988591+0000) 2024-02-20T09:32:42.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:32:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:32:42.781+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:32:16.988591+0000 (oldest deadline 2024-02-20T09:32:36.988591+0000) 2024-02-20T09:32:42.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:32:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:32:42.781+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:32:43.371 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:32:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:32:43.011+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:43.011441+0000) 2024-02-20T09:32:43.371 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:32:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:32:43.126+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:43.126549+0000) 2024-02-20T09:32:43.371 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:32:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:32:43.212+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:31:58.900551+0000 (oldest deadline 2024-02-20T09:32:18.900551+0000) 2024-02-20T09:32:43.371 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:32:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:32:43.212+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:31:58.900551+0000 (oldest deadline 2024-02-20T09:32:18.900551+0000) 2024-02-20T09:32:43.372 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:32:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:32:43.212+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:31:58.900551+0000 (oldest deadline 2024-02-20T09:32:18.900551+0000) 2024-02-20T09:32:43.655 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:32:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:32:43.369+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:43.369990+0000) 2024-02-20T09:32:43.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:32:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:32:43.748+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:31:46.482189+0000 (oldest deadline 2024-02-20T09:32:06.482189+0000) 2024-02-20T09:32:43.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:32:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:32:43.748+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:31:46.482189+0000 (oldest deadline 2024-02-20T09:32:06.482189+0000) 2024-02-20T09:32:43.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:32:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:32:43.748+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:31:46.482189+0000 (oldest deadline 2024-02-20T09:32:06.482189+0000) 2024-02-20T09:32:43.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:32:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:32:43.783+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:32:16.988591+0000 (oldest deadline 2024-02-20T09:32:36.988591+0000) 2024-02-20T09:32:43.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:32:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:32:43.783+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:32:16.988591+0000 (oldest deadline 2024-02-20T09:32:36.988591+0000) 2024-02-20T09:32:43.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:32:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:32:43.783+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:32:16.988591+0000 (oldest deadline 2024-02-20T09:32:36.988591+0000) 2024-02-20T09:32:43.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:32:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:32:43.784+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:32:44.255 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:32:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:32:44.011+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:44.011522+0000) 2024-02-20T09:32:44.255 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:32:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:32:44.126+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:44.126650+0000) 2024-02-20T09:32:44.255 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:32:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:32:44.253+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:31:58.900551+0000 (oldest deadline 2024-02-20T09:32:18.900551+0000) 2024-02-20T09:32:44.255 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:32:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:32:44.253+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:31:58.900551+0000 (oldest deadline 2024-02-20T09:32:18.900551+0000) 2024-02-20T09:32:44.256 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:32:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:32:44.253+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:31:58.900551+0000 (oldest deadline 2024-02-20T09:32:18.900551+0000) 2024-02-20T09:32:44.656 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:32:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:32:44.371+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:44.372230+0000) 2024-02-20T09:32:44.924 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.183 2024-02-20T09:32:44.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:32:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:32:44.790+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:31:46.482189+0000 (oldest deadline 2024-02-20T09:32:06.482189+0000) 2024-02-20T09:32:44.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:32:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:32:44.790+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:31:46.482189+0000 (oldest deadline 2024-02-20T09:32:06.482189+0000) 2024-02-20T09:32:44.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:32:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:32:44.790+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:31:46.482189+0000 (oldest deadline 2024-02-20T09:32:06.482189+0000) 2024-02-20T09:32:44.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:32:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:32:44.829+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:32:16.988591+0000 (oldest deadline 2024-02-20T09:32:36.988591+0000) 2024-02-20T09:32:44.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:32:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:32:44.829+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:32:16.988591+0000 (oldest deadline 2024-02-20T09:32:36.988591+0000) 2024-02-20T09:32:44.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:32:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:32:44.829+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:32:16.988591+0000 (oldest deadline 2024-02-20T09:32:36.988591+0000) 2024-02-20T09:32:44.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:32:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:32:44.829+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:32:45.262 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:32:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:32:45.011+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:45.011618+0000) 2024-02-20T09:32:45.262 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:32:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:32:45.126+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:45.126750+0000) 2024-02-20T09:32:45.262 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:32:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:32:45.260+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:31:58.900551+0000 (oldest deadline 2024-02-20T09:32:18.900551+0000) 2024-02-20T09:32:45.656 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:32:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:32:45.375+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:45.376040+0000) 2024-02-20T09:32:45.656 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:32:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:32:45.260+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:31:58.900551+0000 (oldest deadline 2024-02-20T09:32:18.900551+0000) 2024-02-20T09:32:45.656 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:32:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:32:45.260+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:31:58.900551+0000 (oldest deadline 2024-02-20T09:32:18.900551+0000) 2024-02-20T09:32:45.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:32:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:32:45.834+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:32:16.988591+0000 (oldest deadline 2024-02-20T09:32:36.988591+0000) 2024-02-20T09:32:45.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:32:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:32:45.834+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:32:16.988591+0000 (oldest deadline 2024-02-20T09:32:36.988591+0000) 2024-02-20T09:32:45.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:32:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:32:45.834+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:32:16.988591+0000 (oldest deadline 2024-02-20T09:32:36.988591+0000) 2024-02-20T09:32:45.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:32:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:32:45.834+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:32:45.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:32:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:32:45.750+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:31:46.482189+0000 (oldest deadline 2024-02-20T09:32:06.482189+0000) 2024-02-20T09:32:45.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:32:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:32:45.750+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:31:46.482189+0000 (oldest deadline 2024-02-20T09:32:06.482189+0000) 2024-02-20T09:32:45.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:32:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:32:45.750+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:31:46.482189+0000 (oldest deadline 2024-02-20T09:32:06.482189+0000) 2024-02-20T09:32:46.275 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:32:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:32:46.011+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:46.011723+0000) 2024-02-20T09:32:46.276 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:32:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:32:46.126+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:46.126878+0000) 2024-02-20T09:32:46.656 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:32:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:32:46.378+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:46.378560+0000) 2024-02-20T09:32:46.656 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:32:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:32:46.274+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:31:58.900551+0000 (oldest deadline 2024-02-20T09:32:18.900551+0000) 2024-02-20T09:32:46.656 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:32:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:32:46.274+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:31:58.900551+0000 (oldest deadline 2024-02-20T09:32:18.900551+0000) 2024-02-20T09:32:46.656 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:32:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:32:46.274+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:31:58.900551+0000 (oldest deadline 2024-02-20T09:32:18.900551+0000) 2024-02-20T09:32:46.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:32:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:32:46.880+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:32:16.988591+0000 (oldest deadline 2024-02-20T09:32:36.988591+0000) 2024-02-20T09:32:46.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:32:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:32:46.880+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:32:16.988591+0000 (oldest deadline 2024-02-20T09:32:36.988591+0000) 2024-02-20T09:32:46.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:32:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:32:46.880+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:32:16.988591+0000 (oldest deadline 2024-02-20T09:32:36.988591+0000) 2024-02-20T09:32:46.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:32:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:32:46.880+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:32:46.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:32:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:32:46.794+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:31:46.482189+0000 (oldest deadline 2024-02-20T09:32:06.482189+0000) 2024-02-20T09:32:46.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:32:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:32:46.794+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:31:46.482189+0000 (oldest deadline 2024-02-20T09:32:06.482189+0000) 2024-02-20T09:32:46.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:32:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:32:46.794+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:31:46.482189+0000 (oldest deadline 2024-02-20T09:32:06.482189+0000) 2024-02-20T09:32:47.312 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:32:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:32:47.010+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:47.011921+0000) 2024-02-20T09:32:47.312 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:32:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:32:47.125+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:47.126990+0000) 2024-02-20T09:32:47.313 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:32:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:32:47.310+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:31:58.900551+0000 (oldest deadline 2024-02-20T09:32:18.900551+0000) 2024-02-20T09:32:47.640 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:32:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:32:47.379+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:47.380779+0000) 2024-02-20T09:32:47.640 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:32:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:32:47.310+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:31:58.900551+0000 (oldest deadline 2024-02-20T09:32:18.900551+0000) 2024-02-20T09:32:47.640 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:32:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:32:47.310+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:31:58.900551+0000 (oldest deadline 2024-02-20T09:32:18.900551+0000) 2024-02-20T09:32:47.894 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:32:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:32:47.644+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 72654 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:32:47.894 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:32:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:32:47.641+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:47.642580+0000) 2024-02-20T09:32:47.895 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:32:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:32:47.637+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:47.639242+0000) 2024-02-20T09:32:47.895 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:32:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:32:47.637+0000 7f8a5f86e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:47.638987+0000) 2024-02-20T09:32:47.895 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:32:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:32:47.799+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:31:46.482189+0000 (oldest deadline 2024-02-20T09:32:06.482189+0000) 2024-02-20T09:32:47.895 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:32:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:32:47.799+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:31:46.482189+0000 (oldest deadline 2024-02-20T09:32:06.482189+0000) 2024-02-20T09:32:47.895 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:32:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:32:47.799+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:31:46.482189+0000 (oldest deadline 2024-02-20T09:32:06.482189+0000) 2024-02-20T09:32:48.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:32:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:32:48.010+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:48.012130+0000) 2024-02-20T09:32:48.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:32:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:32:48.125+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:48.127099+0000) 2024-02-20T09:32:48.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:32:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:32:47.891+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:32:16.988591+0000 (oldest deadline 2024-02-20T09:32:36.988591+0000) 2024-02-20T09:32:48.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:32:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:32:47.891+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:32:16.988591+0000 (oldest deadline 2024-02-20T09:32:36.988591+0000) 2024-02-20T09:32:48.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:32:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:32:47.891+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:32:16.988591+0000 (oldest deadline 2024-02-20T09:32:36.988591+0000) 2024-02-20T09:32:48.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:32:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:32:47.891+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:32:48.656 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:32:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:32:48.380+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:48.382062+0000) 2024-02-20T09:32:48.656 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:32:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:32:48.344+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:31:58.900551+0000 (oldest deadline 2024-02-20T09:32:18.900551+0000) 2024-02-20T09:32:48.656 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:32:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:32:48.344+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:31:58.900551+0000 (oldest deadline 2024-02-20T09:32:18.900551+0000) 2024-02-20T09:32:48.657 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:32:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:32:48.344+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:31:58.900551+0000 (oldest deadline 2024-02-20T09:32:18.900551+0000) 2024-02-20T09:32:48.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:32:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:32:48.898+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:32:16.988591+0000 (oldest deadline 2024-02-20T09:32:36.988591+0000) 2024-02-20T09:32:48.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:32:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:32:48.898+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:32:16.988591+0000 (oldest deadline 2024-02-20T09:32:36.988591+0000) 2024-02-20T09:32:48.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:32:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:32:48.898+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:32:16.988591+0000 (oldest deadline 2024-02-20T09:32:36.988591+0000) 2024-02-20T09:32:48.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:32:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:32:48.899+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:32:48.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:32:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:32:48.754+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:31:46.482189+0000 (oldest deadline 2024-02-20T09:32:06.482189+0000) 2024-02-20T09:32:48.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:32:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:32:48.754+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:31:46.482189+0000 (oldest deadline 2024-02-20T09:32:06.482189+0000) 2024-02-20T09:32:48.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:32:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:32:48.754+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:31:46.482189+0000 (oldest deadline 2024-02-20T09:32:06.482189+0000) 2024-02-20T09:32:49.363 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:32:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:32:49.010+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:49.012330+0000) 2024-02-20T09:32:49.363 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:32:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:32:49.125+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:49.127201+0000) 2024-02-20T09:32:49.364 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:32:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:32:49.360+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:31:58.900551+0000 (oldest deadline 2024-02-20T09:32:18.900551+0000) 2024-02-20T09:32:49.364 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:32:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:32:49.360+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:31:58.900551+0000 (oldest deadline 2024-02-20T09:32:18.900551+0000) 2024-02-20T09:32:49.364 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:32:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:32:49.360+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:31:58.900551+0000 (oldest deadline 2024-02-20T09:32:18.900551+0000) 2024-02-20T09:32:49.656 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:32:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:32:49.383+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:49.384544+0000) 2024-02-20T09:32:49.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:32:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:32:49.862+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:32:16.988591+0000 (oldest deadline 2024-02-20T09:32:36.988591+0000) 2024-02-20T09:32:49.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:32:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:32:49.862+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:32:16.988591+0000 (oldest deadline 2024-02-20T09:32:36.988591+0000) 2024-02-20T09:32:49.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:32:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:32:49.862+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:32:16.988591+0000 (oldest deadline 2024-02-20T09:32:36.988591+0000) 2024-02-20T09:32:49.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:32:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:32:49.862+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:32:49.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:32:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:32:49.796+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:31:46.482189+0000 (oldest deadline 2024-02-20T09:32:06.482189+0000) 2024-02-20T09:32:49.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:32:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:32:49.796+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:31:46.482189+0000 (oldest deadline 2024-02-20T09:32:06.482189+0000) 2024-02-20T09:32:49.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:32:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:32:49.796+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:31:46.482189+0000 (oldest deadline 2024-02-20T09:32:06.482189+0000) 2024-02-20T09:32:50.324 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:32:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:32:50.011+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:50.012505+0000) 2024-02-20T09:32:50.324 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:32:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:32:50.125+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:50.127301+0000) 2024-02-20T09:32:50.324 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:32:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:32:50.320+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:31:58.900551+0000 (oldest deadline 2024-02-20T09:32:18.900551+0000) 2024-02-20T09:32:50.656 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:32:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:32:50.385+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:50.386601+0000) 2024-02-20T09:32:50.657 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:32:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:32:50.320+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:31:58.900551+0000 (oldest deadline 2024-02-20T09:32:18.900551+0000) 2024-02-20T09:32:50.657 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:32:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:32:50.320+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:31:58.900551+0000 (oldest deadline 2024-02-20T09:32:18.900551+0000) 2024-02-20T09:32:50.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:32:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:32:50.774+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:31:46.482189+0000 (oldest deadline 2024-02-20T09:32:06.482189+0000) 2024-02-20T09:32:50.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:32:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:32:50.774+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:31:46.482189+0000 (oldest deadline 2024-02-20T09:32:06.482189+0000) 2024-02-20T09:32:50.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:32:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:32:50.774+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:31:46.482189+0000 (oldest deadline 2024-02-20T09:32:06.482189+0000) 2024-02-20T09:32:50.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:32:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:32:50.900+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:32:16.988591+0000 (oldest deadline 2024-02-20T09:32:36.988591+0000) 2024-02-20T09:32:50.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:32:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:32:50.901+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:32:16.988591+0000 (oldest deadline 2024-02-20T09:32:36.988591+0000) 2024-02-20T09:32:50.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:32:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:32:50.901+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:32:16.988591+0000 (oldest deadline 2024-02-20T09:32:36.988591+0000) 2024-02-20T09:32:50.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:32:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:32:50.901+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:32:51.294 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:32:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:32:51.011+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:51.012631+0000) 2024-02-20T09:32:51.295 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:32:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:32:51.126+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:51.127454+0000) 2024-02-20T09:32:51.295 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:32:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:32:51.292+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:31:58.900551+0000 (oldest deadline 2024-02-20T09:32:18.900551+0000) 2024-02-20T09:32:51.295 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:32:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:32:51.292+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:31:58.900551+0000 (oldest deadline 2024-02-20T09:32:18.900551+0000) 2024-02-20T09:32:51.295 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:32:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:32:51.292+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:31:58.900551+0000 (oldest deadline 2024-02-20T09:32:18.900551+0000) 2024-02-20T09:32:51.656 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:32:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:32:51.387+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:51.388649+0000) 2024-02-20T09:32:51.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:32:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:32:51.724+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:31:46.482189+0000 (oldest deadline 2024-02-20T09:32:06.482189+0000) 2024-02-20T09:32:51.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:32:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:32:51.724+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:31:46.482189+0000 (oldest deadline 2024-02-20T09:32:06.482189+0000) 2024-02-20T09:32:51.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:32:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:32:51.724+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:31:46.482189+0000 (oldest deadline 2024-02-20T09:32:06.482189+0000) 2024-02-20T09:32:51.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:32:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:32:51.903+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:32:16.988591+0000 (oldest deadline 2024-02-20T09:32:36.988591+0000) 2024-02-20T09:32:51.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:32:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:32:51.903+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:32:16.988591+0000 (oldest deadline 2024-02-20T09:32:36.988591+0000) 2024-02-20T09:32:51.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:32:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:32:51.903+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:32:16.988591+0000 (oldest deadline 2024-02-20T09:32:36.988591+0000) 2024-02-20T09:32:51.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:32:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:32:51.903+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:32:52.392 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:32:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:32:52.011+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:52.012744+0000) 2024-02-20T09:32:52.392 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:32:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:32:52.126+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:52.127575+0000) 2024-02-20T09:32:52.392 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:32:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:32:52.243+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:31:58.900551+0000 (oldest deadline 2024-02-20T09:32:18.900551+0000) 2024-02-20T09:32:52.392 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:32:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:32:52.243+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:31:58.900551+0000 (oldest deadline 2024-02-20T09:32:18.900551+0000) 2024-02-20T09:32:52.393 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:32:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:32:52.243+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:31:58.900551+0000 (oldest deadline 2024-02-20T09:32:18.900551+0000) 2024-02-20T09:32:52.648 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:32:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:32:52.389+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:52.391095+0000) 2024-02-20T09:32:52.992 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:32:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:32:52.653+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 72693 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:32:52.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:32:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:32:52.646+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:52.648200+0000) 2024-02-20T09:32:52.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:32:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:32:52.649+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:52.650474+0000) 2024-02-20T09:32:52.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:32:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:32:52.869+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:32:16.988591+0000 (oldest deadline 2024-02-20T09:32:36.988591+0000) 2024-02-20T09:32:52.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:32:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:32:52.869+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:32:16.988591+0000 (oldest deadline 2024-02-20T09:32:36.988591+0000) 2024-02-20T09:32:52.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:32:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:32:52.869+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:32:16.988591+0000 (oldest deadline 2024-02-20T09:32:36.988591+0000) 2024-02-20T09:32:52.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:32:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:32:52.869+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:32:52.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:32:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:32:52.646+0000 7f8a5f86e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:52.647407+0000) 2024-02-20T09:32:52.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:32:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:32:52.742+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:31:46.482189+0000 (oldest deadline 2024-02-20T09:32:06.482189+0000) 2024-02-20T09:32:52.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:32:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:32:52.742+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:31:46.482189+0000 (oldest deadline 2024-02-20T09:32:06.482189+0000) 2024-02-20T09:32:52.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:32:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:32:52.742+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:31:46.482189+0000 (oldest deadline 2024-02-20T09:32:06.482189+0000) 2024-02-20T09:32:53.394 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:32:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:32:53.011+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:53.012897+0000) 2024-02-20T09:32:53.394 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:32:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:32:53.126+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:53.127671+0000) 2024-02-20T09:32:53.394 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:32:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:32:53.218+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:31:58.900551+0000 (oldest deadline 2024-02-20T09:32:18.900551+0000) 2024-02-20T09:32:53.394 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:32:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:32:53.218+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:31:58.900551+0000 (oldest deadline 2024-02-20T09:32:18.900551+0000) 2024-02-20T09:32:53.394 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:32:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:32:53.218+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:31:58.900551+0000 (oldest deadline 2024-02-20T09:32:18.900551+0000) 2024-02-20T09:32:53.656 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:32:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:32:53.391+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:53.392895+0000) 2024-02-20T09:32:53.925 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi183.front.sepia.ceph.com' 2024-02-20T09:32:53.926 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi183.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-20T09:32:53.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:32:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:32:53.715+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:31:46.482189+0000 (oldest deadline 2024-02-20T09:32:06.482189+0000) 2024-02-20T09:32:53.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:32:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:32:53.715+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:31:46.482189+0000 (oldest deadline 2024-02-20T09:32:06.482189+0000) 2024-02-20T09:32:53.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:32:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:32:53.715+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:31:46.482189+0000 (oldest deadline 2024-02-20T09:32:06.482189+0000) 2024-02-20T09:32:53.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:32:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:32:53.882+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:32:16.988591+0000 (oldest deadline 2024-02-20T09:32:36.988591+0000) 2024-02-20T09:32:53.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:32:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:32:53.882+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:32:16.988591+0000 (oldest deadline 2024-02-20T09:32:36.988591+0000) 2024-02-20T09:32:53.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:32:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:32:53.882+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:32:16.988591+0000 (oldest deadline 2024-02-20T09:32:36.988591+0000) 2024-02-20T09:32:53.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:32:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:32:53.882+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:32:54.397 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:32:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:32:54.011+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:54.012978+0000) 2024-02-20T09:32:54.397 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:32:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:32:54.126+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:54.127740+0000) 2024-02-20T09:32:54.398 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:32:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:32:54.203+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:31:58.900551+0000 (oldest deadline 2024-02-20T09:32:18.900551+0000) 2024-02-20T09:32:54.398 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:32:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:32:54.203+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:31:58.900551+0000 (oldest deadline 2024-02-20T09:32:18.900551+0000) 2024-02-20T09:32:54.398 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:32:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:32:54.203+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:31:58.900551+0000 (oldest deadline 2024-02-20T09:32:18.900551+0000) 2024-02-20T09:32:54.656 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:32:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:32:54.394+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:54.396251+0000) 2024-02-20T09:32:54.912 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:32:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:32:54.725+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:31:46.482189+0000 (oldest deadline 2024-02-20T09:32:06.482189+0000) 2024-02-20T09:32:54.912 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:32:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:32:54.725+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:31:46.482189+0000 (oldest deadline 2024-02-20T09:32:06.482189+0000) 2024-02-20T09:32:54.912 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:32:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:32:54.725+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:31:46.482189+0000 (oldest deadline 2024-02-20T09:32:06.482189+0000) 2024-02-20T09:32:55.184 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:32:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:32:55.011+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:55.013093+0000) 2024-02-20T09:32:55.185 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:32:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:32:55.126+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:55.127825+0000) 2024-02-20T09:32:55.185 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:32:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:32:55.181+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:31:58.900551+0000 (oldest deadline 2024-02-20T09:32:18.900551+0000) 2024-02-20T09:32:55.185 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:32:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:32:54.909+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:32:16.988591+0000 (oldest deadline 2024-02-20T09:32:36.988591+0000) 2024-02-20T09:32:55.185 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:32:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:32:54.909+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:32:16.988591+0000 (oldest deadline 2024-02-20T09:32:36.988591+0000) 2024-02-20T09:32:55.186 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:32:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:32:54.909+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:32:16.988591+0000 (oldest deadline 2024-02-20T09:32:36.988591+0000) 2024-02-20T09:32:55.186 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:32:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:32:54.909+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:32:55.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:32:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:32:55.398+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:55.399757+0000) 2024-02-20T09:32:55.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:32:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:32:55.181+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:31:58.900551+0000 (oldest deadline 2024-02-20T09:32:18.900551+0000) 2024-02-20T09:32:55.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:32:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:32:55.181+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:31:58.900551+0000 (oldest deadline 2024-02-20T09:32:18.900551+0000) 2024-02-20T09:32:55.961 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:32:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:32:55.736+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:31:46.482189+0000 (oldest deadline 2024-02-20T09:32:06.482189+0000) 2024-02-20T09:32:55.961 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:32:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:32:55.736+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:31:46.482189+0000 (oldest deadline 2024-02-20T09:32:06.482189+0000) 2024-02-20T09:32:55.961 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:32:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:32:55.736+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:31:46.482189+0000 (oldest deadline 2024-02-20T09:32:06.482189+0000) 2024-02-20T09:32:56.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:32:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:32:56.011+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:56.013184+0000) 2024-02-20T09:32:56.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:32:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:32:56.126+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:56.127911+0000) 2024-02-20T09:32:56.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:32:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:32:56.155+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:31:58.900551+0000 (oldest deadline 2024-02-20T09:32:18.900551+0000) 2024-02-20T09:32:56.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:32:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:32:56.155+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:31:58.900551+0000 (oldest deadline 2024-02-20T09:32:18.900551+0000) 2024-02-20T09:32:56.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:32:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:32:56.155+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:31:58.900551+0000 (oldest deadline 2024-02-20T09:32:18.900551+0000) 2024-02-20T09:32:56.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:32:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:32:55.958+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:32:16.988591+0000 (oldest deadline 2024-02-20T09:32:36.988591+0000) 2024-02-20T09:32:56.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:32:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:32:55.958+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:32:16.988591+0000 (oldest deadline 2024-02-20T09:32:36.988591+0000) 2024-02-20T09:32:56.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:32:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:32:55.958+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:32:16.988591+0000 (oldest deadline 2024-02-20T09:32:36.988591+0000) 2024-02-20T09:32:56.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:32:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:32:55.958+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:32:56.657 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:32:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:32:56.401+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:56.403276+0000) 2024-02-20T09:32:56.988 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.183 2024-02-20T09:32:56.990 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:32:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:32:56.699+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:31:46.482189+0000 (oldest deadline 2024-02-20T09:32:06.482189+0000) 2024-02-20T09:32:56.990 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:32:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:32:56.699+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:31:46.482189+0000 (oldest deadline 2024-02-20T09:32:06.482189+0000) 2024-02-20T09:32:56.990 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:32:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:32:56.699+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:31:46.482189+0000 (oldest deadline 2024-02-20T09:32:06.482189+0000) 2024-02-20T09:32:57.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:32:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:32:57.011+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:57.013300+0000) 2024-02-20T09:32:57.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:32:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:32:57.126+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:57.128040+0000) 2024-02-20T09:32:57.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:32:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:32:57.163+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:31:58.900551+0000 (oldest deadline 2024-02-20T09:32:18.900551+0000) 2024-02-20T09:32:57.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:32:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:32:57.163+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:31:58.900551+0000 (oldest deadline 2024-02-20T09:32:18.900551+0000) 2024-02-20T09:32:57.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:32:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:32:57.163+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:31:58.900551+0000 (oldest deadline 2024-02-20T09:32:18.900551+0000) 2024-02-20T09:32:57.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:32:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:32:56.988+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:32:16.988591+0000 (oldest deadline 2024-02-20T09:32:36.988591+0000) 2024-02-20T09:32:57.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:32:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:32:56.988+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:32:16.988591+0000 (oldest deadline 2024-02-20T09:32:36.988591+0000) 2024-02-20T09:32:57.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:32:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:32:56.988+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:32:16.988591+0000 (oldest deadline 2024-02-20T09:32:36.988591+0000) 2024-02-20T09:32:57.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:32:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:32:56.988+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:32:57.657 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:32:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:32:57.654+0000 7f8a5f86e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:57.655865+0000) 2024-02-20T09:32:57.658 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:32:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:32:57.405+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:57.406779+0000) 2024-02-20T09:32:57.658 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:32:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:32:57.654+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:57.656028+0000) 2024-02-20T09:32:57.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:32:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:32:57.657+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:57.659079+0000) 2024-02-20T09:32:57.993 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:32:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:32:57.661+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 72733 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:32:57.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:32:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:32:57.700+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:31:46.482189+0000 (oldest deadline 2024-02-20T09:32:06.482189+0000) 2024-02-20T09:32:57.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:32:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:32:57.700+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:31:46.482189+0000 (oldest deadline 2024-02-20T09:32:06.482189+0000) 2024-02-20T09:32:57.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:32:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:32:57.700+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:31:46.482189+0000 (oldest deadline 2024-02-20T09:32:06.482189+0000) 2024-02-20T09:32:58.409 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:32:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:32:58.012+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:58.013441+0000) 2024-02-20T09:32:58.410 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:32:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:32:58.002+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:32:16.988591+0000 (oldest deadline 2024-02-20T09:32:36.988591+0000) 2024-02-20T09:32:58.410 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:32:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:32:58.002+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:32:16.988591+0000 (oldest deadline 2024-02-20T09:32:36.988591+0000) 2024-02-20T09:32:58.410 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:32:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:32:58.002+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:32:16.988591+0000 (oldest deadline 2024-02-20T09:32:36.988591+0000) 2024-02-20T09:32:58.410 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:32:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:32:58.002+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:32:58.410 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:32:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:32:58.126+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:58.128173+0000) 2024-02-20T09:32:58.410 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:32:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:32:58.136+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:31:58.900551+0000 (oldest deadline 2024-02-20T09:32:18.900551+0000) 2024-02-20T09:32:58.411 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:32:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:32:58.136+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:31:58.900551+0000 (oldest deadline 2024-02-20T09:32:18.900551+0000) 2024-02-20T09:32:58.411 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:32:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:32:58.136+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:31:58.900551+0000 (oldest deadline 2024-02-20T09:32:18.900551+0000) 2024-02-20T09:32:58.662 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:32:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:32:58.407+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:58.408625+0000) 2024-02-20T09:32:58.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:32:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:32:58.660+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:31:46.482189+0000 (oldest deadline 2024-02-20T09:32:06.482189+0000) 2024-02-20T09:32:58.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:32:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:32:58.660+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:31:46.482189+0000 (oldest deadline 2024-02-20T09:32:06.482189+0000) 2024-02-20T09:32:58.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:32:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:32:58.660+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:31:46.482189+0000 (oldest deadline 2024-02-20T09:32:06.482189+0000) 2024-02-20T09:32:59.413 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:32:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:32:59.012+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:59.013523+0000) 2024-02-20T09:32:59.413 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:32:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:32:59.126+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:59.128273+0000) 2024-02-20T09:32:59.413 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:32:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:32:59.155+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:31:58.900551+0000 (oldest deadline 2024-02-20T09:32:18.900551+0000) 2024-02-20T09:32:59.414 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:32:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:32:59.155+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:31:58.900551+0000 (oldest deadline 2024-02-20T09:32:18.900551+0000) 2024-02-20T09:32:59.414 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:32:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:32:59.155+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:31:58.900551+0000 (oldest deadline 2024-02-20T09:32:18.900551+0000) 2024-02-20T09:32:59.414 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:32:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:32:59.017+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:32:16.988591+0000 (oldest deadline 2024-02-20T09:32:36.988591+0000) 2024-02-20T09:32:59.414 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:32:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:32:59.017+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:32:16.988591+0000 (oldest deadline 2024-02-20T09:32:36.988591+0000) 2024-02-20T09:32:59.414 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:32:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:32:59.017+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:32:16.988591+0000 (oldest deadline 2024-02-20T09:32:36.988591+0000) 2024-02-20T09:32:59.414 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:32:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:32:59.017+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:32:59.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:32:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:32:59.410+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:32:59.412029+0000) 2024-02-20T09:32:59.742 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:32:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:32:59.626+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:31:46.482189+0000 (oldest deadline 2024-02-20T09:32:06.482189+0000) 2024-02-20T09:32:59.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:32:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:32:59.626+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:31:46.482189+0000 (oldest deadline 2024-02-20T09:32:06.482189+0000) 2024-02-20T09:32:59.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:32:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:32:59.626+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:31:46.482189+0000 (oldest deadline 2024-02-20T09:32:06.482189+0000) 2024-02-20T09:33:00.416 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:33:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:33:00.012+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:00.013616+0000) 2024-02-20T09:33:00.417 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:33:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:33:00.127+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:00.128393+0000) 2024-02-20T09:33:00.417 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:33:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:33:00.182+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:31:58.900551+0000 (oldest deadline 2024-02-20T09:32:18.900551+0000) 2024-02-20T09:33:00.417 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:33:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:33:00.182+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:31:58.900551+0000 (oldest deadline 2024-02-20T09:32:18.900551+0000) 2024-02-20T09:33:00.417 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:33:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:33:00.182+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:31:58.900551+0000 (oldest deadline 2024-02-20T09:32:18.900551+0000) 2024-02-20T09:33:00.417 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:33:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:33:00.010+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:32:16.988591+0000 (oldest deadline 2024-02-20T09:32:36.988591+0000) 2024-02-20T09:33:00.417 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:33:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:33:00.010+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:32:16.988591+0000 (oldest deadline 2024-02-20T09:32:36.988591+0000) 2024-02-20T09:33:00.418 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:33:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:33:00.010+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:32:16.988591+0000 (oldest deadline 2024-02-20T09:32:36.988591+0000) 2024-02-20T09:33:00.418 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:33:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:33:00.010+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:33:00.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:33:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:33:00.414+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:00.415462+0000) 2024-02-20T09:33:00.742 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:33:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:33:00.628+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:31:46.482189+0000 (oldest deadline 2024-02-20T09:32:06.482189+0000) 2024-02-20T09:33:00.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:33:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:33:00.628+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:31:46.482189+0000 (oldest deadline 2024-02-20T09:32:06.482189+0000) 2024-02-20T09:33:00.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:33:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:33:00.628+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:31:46.482189+0000 (oldest deadline 2024-02-20T09:32:06.482189+0000) 2024-02-20T09:33:01.420 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:33:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:33:01.012+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:01.013708+0000) 2024-02-20T09:33:01.420 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:33:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:33:01.127+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:01.128503+0000) 2024-02-20T09:33:01.420 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:33:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:33:01.146+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:31:58.900551+0000 (oldest deadline 2024-02-20T09:32:18.900551+0000) 2024-02-20T09:33:01.420 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:33:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:33:01.146+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:31:58.900551+0000 (oldest deadline 2024-02-20T09:32:18.900551+0000) 2024-02-20T09:33:01.420 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:33:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:33:01.146+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:31:58.900551+0000 (oldest deadline 2024-02-20T09:32:18.900551+0000) 2024-02-20T09:33:01.421 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:33:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:33:01.057+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:32:16.988591+0000 (oldest deadline 2024-02-20T09:32:36.988591+0000) 2024-02-20T09:33:01.421 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:33:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:33:01.057+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:32:16.988591+0000 (oldest deadline 2024-02-20T09:32:36.988591+0000) 2024-02-20T09:33:01.421 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:33:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:33:01.057+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:32:16.988591+0000 (oldest deadline 2024-02-20T09:32:36.988591+0000) 2024-02-20T09:33:01.421 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:33:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:33:01.057+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:33:01.421 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:33:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:33:01.417+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:01.418893+0000) 2024-02-20T09:33:01.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:33:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:33:01.677+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:31:46.482189+0000 (oldest deadline 2024-02-20T09:32:06.482189+0000) 2024-02-20T09:33:01.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:33:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:33:01.677+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:31:46.482189+0000 (oldest deadline 2024-02-20T09:32:06.482189+0000) 2024-02-20T09:33:01.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:33:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:33:01.677+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:31:46.482189+0000 (oldest deadline 2024-02-20T09:32:06.482189+0000) 2024-02-20T09:33:02.423 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:33:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:33:02.012+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:02.013827+0000) 2024-02-20T09:33:02.424 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:33:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:33:02.077+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:32:16.988591+0000 (oldest deadline 2024-02-20T09:32:36.988591+0000) 2024-02-20T09:33:02.424 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:33:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:33:02.077+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:32:16.988591+0000 (oldest deadline 2024-02-20T09:32:36.988591+0000) 2024-02-20T09:33:02.424 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:33:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:33:02.077+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:32:16.988591+0000 (oldest deadline 2024-02-20T09:32:36.988591+0000) 2024-02-20T09:33:02.424 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:33:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:33:02.078+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:33:02.424 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:33:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:33:02.127+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:02.128601+0000) 2024-02-20T09:33:02.424 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:33:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:33:02.185+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:31:58.900551+0000 (oldest deadline 2024-02-20T09:32:18.900551+0000) 2024-02-20T09:33:02.425 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:33:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:33:02.185+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:31:58.900551+0000 (oldest deadline 2024-02-20T09:32:18.900551+0000) 2024-02-20T09:33:02.425 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:33:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:33:02.185+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:31:58.900551+0000 (oldest deadline 2024-02-20T09:32:18.900551+0000) 2024-02-20T09:33:02.678 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:33:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:33:02.669+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 72772 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:33:02.679 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:33:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:33:02.662+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:02.664080+0000) 2024-02-20T09:33:02.679 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:33:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:33:02.421+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:02.422406+0000) 2024-02-20T09:33:02.679 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:33:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:33:02.665+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:02.666633+0000) 2024-02-20T09:33:02.679 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:33:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:33:02.640+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:31:46.482189+0000 (oldest deadline 2024-02-20T09:32:06.482189+0000) 2024-02-20T09:33:02.680 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:33:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:33:02.640+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:31:46.482189+0000 (oldest deadline 2024-02-20T09:32:06.482189+0000) 2024-02-20T09:33:02.680 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:33:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:33:02.640+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:31:46.482189+0000 (oldest deadline 2024-02-20T09:32:06.482189+0000) 2024-02-20T09:33:02.680 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:33:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:33:02.662+0000 7f8a5f86e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:02.663966+0000) 2024-02-20T09:33:03.425 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:33:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:33:03.012+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:03.013924+0000) 2024-02-20T09:33:03.425 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:33:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:33:03.032+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:32:16.988591+0000 (oldest deadline 2024-02-20T09:32:36.988591+0000) 2024-02-20T09:33:03.425 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:33:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:33:03.032+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:32:16.988591+0000 (oldest deadline 2024-02-20T09:32:36.988591+0000) 2024-02-20T09:33:03.426 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:33:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:33:03.032+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:32:16.988591+0000 (oldest deadline 2024-02-20T09:32:36.988591+0000) 2024-02-20T09:33:03.426 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:33:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:33:03.032+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:33:03.426 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:33:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:33:03.127+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:03.128709+0000) 2024-02-20T09:33:03.426 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:33:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:33:03.231+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:31:58.900551+0000 (oldest deadline 2024-02-20T09:32:18.900551+0000) 2024-02-20T09:33:03.426 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:33:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:33:03.231+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:31:58.900551+0000 (oldest deadline 2024-02-20T09:32:18.900551+0000) 2024-02-20T09:33:03.426 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:33:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:33:03.231+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:31:58.900551+0000 (oldest deadline 2024-02-20T09:32:18.900551+0000) 2024-02-20T09:33:03.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:33:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:33:03.422+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:03.423907+0000) 2024-02-20T09:33:03.742 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:33:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:33:03.591+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:31:46.482189+0000 (oldest deadline 2024-02-20T09:32:06.482189+0000) 2024-02-20T09:33:03.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:33:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:33:03.591+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:31:46.482189+0000 (oldest deadline 2024-02-20T09:32:06.482189+0000) 2024-02-20T09:33:03.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:33:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:33:03.591+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:31:46.482189+0000 (oldest deadline 2024-02-20T09:32:06.482189+0000) 2024-02-20T09:33:04.003 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:33:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:33:03.998+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:32:16.988591+0000 (oldest deadline 2024-02-20T09:32:36.988591+0000) 2024-02-20T09:33:04.003 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:33:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:33:03.998+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:32:16.988591+0000 (oldest deadline 2024-02-20T09:32:36.988591+0000) 2024-02-20T09:33:04.004 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:33:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:33:03.998+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:32:16.988591+0000 (oldest deadline 2024-02-20T09:32:36.988591+0000) 2024-02-20T09:33:04.004 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:33:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:33:03.998+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:33:04.264 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:33:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:33:04.012+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:04.014033+0000) 2024-02-20T09:33:04.265 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:33:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:33:04.127+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:04.128797+0000) 2024-02-20T09:33:04.265 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:33:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:33:04.261+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:31:58.900551+0000 (oldest deadline 2024-02-20T09:32:18.900551+0000) 2024-02-20T09:33:04.265 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:33:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:33:04.261+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:31:58.900551+0000 (oldest deadline 2024-02-20T09:32:18.900551+0000) 2024-02-20T09:33:04.265 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:33:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:33:04.261+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:31:58.900551+0000 (oldest deadline 2024-02-20T09:32:18.900551+0000) 2024-02-20T09:33:04.637 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:33:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:33:04.424+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:04.425981+0000) 2024-02-20T09:33:04.962 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:33:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:33:04.634+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:31:46.482189+0000 (oldest deadline 2024-02-20T09:32:06.482189+0000) 2024-02-20T09:33:04.962 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:33:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:33:04.634+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:31:46.482189+0000 (oldest deadline 2024-02-20T09:32:06.482189+0000) 2024-02-20T09:33:04.963 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:33:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:33:04.634+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:31:46.482189+0000 (oldest deadline 2024-02-20T09:32:06.482189+0000) 2024-02-20T09:33:05.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:33:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:33:05.012+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:05.014174+0000) 2024-02-20T09:33:05.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:33:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:33:05.127+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:05.128886+0000) 2024-02-20T09:33:05.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:33:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:33:04.960+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:32:16.988591+0000 (oldest deadline 2024-02-20T09:32:36.988591+0000) 2024-02-20T09:33:05.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:33:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:33:04.960+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:32:16.988591+0000 (oldest deadline 2024-02-20T09:32:36.988591+0000) 2024-02-20T09:33:05.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:33:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:33:04.960+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:32:16.988591+0000 (oldest deadline 2024-02-20T09:32:36.988591+0000) 2024-02-20T09:33:05.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:33:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:33:04.960+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:33:05.642 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:33:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:33:05.427+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:05.428552+0000) 2024-02-20T09:33:05.642 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:33:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:33:05.281+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:31:58.900551+0000 (oldest deadline 2024-02-20T09:32:18.900551+0000) 2024-02-20T09:33:05.642 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:33:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:33:05.281+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:31:58.900551+0000 (oldest deadline 2024-02-20T09:32:18.900551+0000) 2024-02-20T09:33:05.642 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:33:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:33:05.281+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:31:58.900551+0000 (oldest deadline 2024-02-20T09:32:18.900551+0000) 2024-02-20T09:33:05.951 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:33:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:33:05.639+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:31:46.482189+0000 (oldest deadline 2024-02-20T09:32:06.482189+0000) 2024-02-20T09:33:05.951 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:33:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:33:05.639+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:31:46.482189+0000 (oldest deadline 2024-02-20T09:32:06.482189+0000) 2024-02-20T09:33:05.951 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:33:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:33:05.639+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:31:46.482189+0000 (oldest deadline 2024-02-20T09:32:06.482189+0000) 2024-02-20T09:33:06.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:33:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:33:06.012+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:06.014273+0000) 2024-02-20T09:33:06.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:33:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:33:06.127+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:06.129038+0000) 2024-02-20T09:33:06.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:33:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:33:05.948+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:32:16.988591+0000 (oldest deadline 2024-02-20T09:32:36.988591+0000) 2024-02-20T09:33:06.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:33:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:33:05.948+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:32:16.988591+0000 (oldest deadline 2024-02-20T09:32:36.988591+0000) 2024-02-20T09:33:06.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:33:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:33:05.948+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:32:16.988591+0000 (oldest deadline 2024-02-20T09:32:36.988591+0000) 2024-02-20T09:33:06.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:33:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:33:05.948+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:33:06.676 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:33:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:33:06.429+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:06.431070+0000) 2024-02-20T09:33:06.676 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:33:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:33:06.263+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:31:58.900551+0000 (oldest deadline 2024-02-20T09:32:18.900551+0000) 2024-02-20T09:33:06.676 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:33:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:33:06.263+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:31:58.900551+0000 (oldest deadline 2024-02-20T09:32:18.900551+0000) 2024-02-20T09:33:06.676 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:33:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:33:06.263+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:31:58.900551+0000 (oldest deadline 2024-02-20T09:32:18.900551+0000) 2024-02-20T09:33:06.974 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:33:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:33:06.673+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:31:46.482189+0000 (oldest deadline 2024-02-20T09:32:06.482189+0000) 2024-02-20T09:33:06.974 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:33:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:33:06.673+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:31:46.482189+0000 (oldest deadline 2024-02-20T09:32:06.482189+0000) 2024-02-20T09:33:06.974 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:33:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:33:06.673+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:31:46.482189+0000 (oldest deadline 2024-02-20T09:32:06.482189+0000) 2024-02-20T09:33:07.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:33:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:33:07.127+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:07.129164+0000) 2024-02-20T09:33:07.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:33:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:33:07.013+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:07.014405+0000) 2024-02-20T09:33:07.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:33:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:33:06.971+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:32:16.988591+0000 (oldest deadline 2024-02-20T09:32:36.988591+0000) 2024-02-20T09:33:07.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:33:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:33:06.971+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:32:16.988591+0000 (oldest deadline 2024-02-20T09:32:36.988591+0000) 2024-02-20T09:33:07.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:33:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:33:06.971+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:32:16.988591+0000 (oldest deadline 2024-02-20T09:32:36.988591+0000) 2024-02-20T09:33:07.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:33:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:33:06.971+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:33:07.673 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:33:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:33:07.432+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:07.433644+0000) 2024-02-20T09:33:07.673 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:33:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:33:07.256+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:31:58.900551+0000 (oldest deadline 2024-02-20T09:32:18.900551+0000) 2024-02-20T09:33:07.673 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:33:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:33:07.256+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:31:58.900551+0000 (oldest deadline 2024-02-20T09:32:18.900551+0000) 2024-02-20T09:33:07.673 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:33:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:33:07.256+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:31:58.900551+0000 (oldest deadline 2024-02-20T09:32:18.900551+0000) 2024-02-20T09:33:07.958 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:33:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:33:07.670+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:07.672281+0000) 2024-02-20T09:33:07.958 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:33:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:33:07.678+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 72812 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:33:07.959 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:33:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:33:07.670+0000 7f8a5f86e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:07.672135+0000) 2024-02-20T09:33:07.959 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:33:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:33:07.717+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:31:46.482189+0000 (oldest deadline 2024-02-20T09:32:06.482189+0000) 2024-02-20T09:33:07.959 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:33:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:33:07.717+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:31:46.482189+0000 (oldest deadline 2024-02-20T09:32:06.482189+0000) 2024-02-20T09:33:07.959 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:33:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:33:07.717+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:31:46.482189+0000 (oldest deadline 2024-02-20T09:32:06.482189+0000) 2024-02-20T09:33:07.959 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:33:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:33:07.673+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:07.674951+0000) 2024-02-20T09:33:07.960 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:33:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:33:07.955+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:32:16.988591+0000 (oldest deadline 2024-02-20T09:32:36.988591+0000) 2024-02-20T09:33:07.960 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:33:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:33:07.955+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:32:16.988591+0000 (oldest deadline 2024-02-20T09:32:36.988591+0000) 2024-02-20T09:33:07.960 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:33:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:33:07.955+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:32:16.988591+0000 (oldest deadline 2024-02-20T09:32:36.988591+0000) 2024-02-20T09:33:08.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:33:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:33:08.013+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:08.014499+0000) 2024-02-20T09:33:08.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:33:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:33:07.956+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:33:08.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:33:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:33:08.127+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:08.129279+0000) 2024-02-20T09:33:08.678 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:33:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:33:08.433+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:08.435077+0000) 2024-02-20T09:33:08.679 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:33:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:33:08.272+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:31:58.900551+0000 (oldest deadline 2024-02-20T09:32:18.900551+0000) 2024-02-20T09:33:08.679 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:33:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:33:08.272+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:31:58.900551+0000 (oldest deadline 2024-02-20T09:32:18.900551+0000) 2024-02-20T09:33:08.679 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:33:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:33:08.272+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:31:58.900551+0000 (oldest deadline 2024-02-20T09:32:18.900551+0000) 2024-02-20T09:33:08.960 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:33:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:33:08.676+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:31:46.482189+0000 (oldest deadline 2024-02-20T09:32:06.482189+0000) 2024-02-20T09:33:08.961 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:33:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:33:08.676+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:31:46.482189+0000 (oldest deadline 2024-02-20T09:32:06.482189+0000) 2024-02-20T09:33:08.961 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:33:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:33:08.676+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:31:46.482189+0000 (oldest deadline 2024-02-20T09:32:06.482189+0000) 2024-02-20T09:33:08.989 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi183.front.sepia.ceph.com' 2024-02-20T09:33:08.990 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi183.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-20T09:33:09.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:33:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:33:09.013+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:09.014593+0000) 2024-02-20T09:33:09.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:33:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:33:09.127+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:09.129383+0000) 2024-02-20T09:33:09.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:33:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:33:08.958+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:32:16.988591+0000 (oldest deadline 2024-02-20T09:32:36.988591+0000) 2024-02-20T09:33:09.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:33:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:33:08.958+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:32:16.988591+0000 (oldest deadline 2024-02-20T09:32:36.988591+0000) 2024-02-20T09:33:09.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:33:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:33:08.958+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:32:16.988591+0000 (oldest deadline 2024-02-20T09:32:36.988591+0000) 2024-02-20T09:33:09.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:33:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:33:08.958+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:33:09.716 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:33:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:33:09.435+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:09.437033+0000) 2024-02-20T09:33:09.716 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:33:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:33:09.301+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:31:58.900551+0000 (oldest deadline 2024-02-20T09:32:18.900551+0000) 2024-02-20T09:33:09.716 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:33:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:33:09.301+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:31:58.900551+0000 (oldest deadline 2024-02-20T09:32:18.900551+0000) 2024-02-20T09:33:09.716 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:33:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:33:09.301+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:31:58.900551+0000 (oldest deadline 2024-02-20T09:32:18.900551+0000) 2024-02-20T09:33:09.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:33:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:33:09.713+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:31:46.482189+0000 (oldest deadline 2024-02-20T09:32:06.482189+0000) 2024-02-20T09:33:09.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:33:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:33:09.713+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:31:46.482189+0000 (oldest deadline 2024-02-20T09:32:06.482189+0000) 2024-02-20T09:33:09.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:33:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:33:09.713+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:31:46.482189+0000 (oldest deadline 2024-02-20T09:32:06.482189+0000) 2024-02-20T09:33:09.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:33:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:33:09.956+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:32:16.988591+0000 (oldest deadline 2024-02-20T09:32:36.988591+0000) 2024-02-20T09:33:09.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:33:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:33:09.956+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:32:16.988591+0000 (oldest deadline 2024-02-20T09:32:36.988591+0000) 2024-02-20T09:33:09.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:33:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:33:09.956+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:32:16.988591+0000 (oldest deadline 2024-02-20T09:32:36.988591+0000) 2024-02-20T09:33:09.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:33:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:33:09.957+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:33:10.284 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:33:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:33:10.013+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:10.014681+0000) 2024-02-20T09:33:10.284 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:33:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:33:10.128+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:10.129500+0000) 2024-02-20T09:33:10.285 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:33:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:33:10.281+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:31:58.900551+0000 (oldest deadline 2024-02-20T09:32:18.900551+0000) 2024-02-20T09:33:10.285 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:33:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:33:10.281+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:31:58.900551+0000 (oldest deadline 2024-02-20T09:32:18.900551+0000) 2024-02-20T09:33:10.285 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:33:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:33:10.281+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:31:58.900551+0000 (oldest deadline 2024-02-20T09:32:18.900551+0000) 2024-02-20T09:33:10.699 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:33:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:33:10.439+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:10.440629+0000) 2024-02-20T09:33:10.980 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:33:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:33:10.696+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:31:46.482189+0000 (oldest deadline 2024-02-20T09:32:06.482189+0000) 2024-02-20T09:33:10.980 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:33:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:33:10.696+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:31:46.482189+0000 (oldest deadline 2024-02-20T09:32:06.482189+0000) 2024-02-20T09:33:10.980 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:33:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:33:10.696+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:31:46.482189+0000 (oldest deadline 2024-02-20T09:32:06.482189+0000) 2024-02-20T09:33:11.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:33:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:33:11.013+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:11.014791+0000) 2024-02-20T09:33:11.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:33:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:33:11.128+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:11.129602+0000) 2024-02-20T09:33:11.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:33:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:33:10.977+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:32:16.988591+0000 (oldest deadline 2024-02-20T09:32:36.988591+0000) 2024-02-20T09:33:11.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:33:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:33:10.977+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:32:16.988591+0000 (oldest deadline 2024-02-20T09:32:36.988591+0000) 2024-02-20T09:33:11.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:33:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:33:10.977+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:32:16.988591+0000 (oldest deadline 2024-02-20T09:32:36.988591+0000) 2024-02-20T09:33:11.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:33:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:33:10.977+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:33:11.728 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:33:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:33:11.441+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:11.442602+0000) 2024-02-20T09:33:11.728 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:33:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:33:11.318+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:31:58.900551+0000 (oldest deadline 2024-02-20T09:32:18.900551+0000) 2024-02-20T09:33:11.729 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:33:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:33:11.318+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:31:58.900551+0000 (oldest deadline 2024-02-20T09:32:18.900551+0000) 2024-02-20T09:33:11.729 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:33:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:33:11.318+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:31:58.900551+0000 (oldest deadline 2024-02-20T09:32:18.900551+0000) 2024-02-20T09:33:11.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:33:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:33:11.725+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:31:46.482189+0000 (oldest deadline 2024-02-20T09:32:06.482189+0000) 2024-02-20T09:33:11.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:33:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:33:11.725+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:31:46.482189+0000 (oldest deadline 2024-02-20T09:32:06.482189+0000) 2024-02-20T09:33:11.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:33:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:33:11.725+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:31:46.482189+0000 (oldest deadline 2024-02-20T09:32:06.482189+0000) 2024-02-20T09:33:11.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:33:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:33:11.974+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:32:16.988591+0000 (oldest deadline 2024-02-20T09:32:36.988591+0000) 2024-02-20T09:33:11.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:33:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:33:11.974+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:32:16.988591+0000 (oldest deadline 2024-02-20T09:32:36.988591+0000) 2024-02-20T09:33:11.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:33:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:33:11.974+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:32:16.988591+0000 (oldest deadline 2024-02-20T09:32:36.988591+0000) 2024-02-20T09:33:11.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:33:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:33:11.974+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:33:12.060 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.183 2024-02-20T09:33:12.306 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:33:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:33:12.013+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:12.014878+0000) 2024-02-20T09:33:12.306 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:33:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:33:12.128+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:12.129697+0000) 2024-02-20T09:33:12.306 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:33:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:33:12.303+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:31:58.900551+0000 (oldest deadline 2024-02-20T09:32:18.900551+0000) 2024-02-20T09:33:12.306 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:33:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:33:12.303+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:31:58.900551+0000 (oldest deadline 2024-02-20T09:32:18.900551+0000) 2024-02-20T09:33:12.306 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:33:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:33:12.303+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:31:58.900551+0000 (oldest deadline 2024-02-20T09:32:18.900551+0000) 2024-02-20T09:33:12.682 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:33:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:33:12.444+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:12.446247+0000) 2024-02-20T09:33:12.971 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:33:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:33:12.680+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:12.681448+0000) 2024-02-20T09:33:12.971 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:33:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:33:12.687+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 72851 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:33:12.972 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:33:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:33:12.684+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:12.685459+0000) 2024-02-20T09:33:12.972 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:33:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:33:12.968+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:32:16.988591+0000 (oldest deadline 2024-02-20T09:32:36.988591+0000) 2024-02-20T09:33:12.972 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:33:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:33:12.968+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:32:16.988591+0000 (oldest deadline 2024-02-20T09:32:36.988591+0000) 2024-02-20T09:33:12.973 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:33:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:33:12.680+0000 7f8a5f86e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:12.681558+0000) 2024-02-20T09:33:12.973 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:33:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:33:12.727+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:31:46.482189+0000 (oldest deadline 2024-02-20T09:32:06.482189+0000) 2024-02-20T09:33:12.973 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:33:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:33:12.727+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:31:46.482189+0000 (oldest deadline 2024-02-20T09:32:06.482189+0000) 2024-02-20T09:33:12.973 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:33:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:33:12.727+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:31:46.482189+0000 (oldest deadline 2024-02-20T09:32:06.482189+0000) 2024-02-20T09:33:13.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:33:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:33:13.013+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:13.014998+0000) 2024-02-20T09:33:13.242 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:33:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:33:13.128+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:13.129809+0000) 2024-02-20T09:33:13.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:33:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:33:12.968+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:32:16.988591+0000 (oldest deadline 2024-02-20T09:32:36.988591+0000) 2024-02-20T09:33:13.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:33:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:33:12.968+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:33:13.724 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:33:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:33:13.446+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:13.447601+0000) 2024-02-20T09:33:13.725 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:33:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:33:13.255+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:31:58.900551+0000 (oldest deadline 2024-02-20T09:32:18.900551+0000) 2024-02-20T09:33:13.725 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:33:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:33:13.255+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:31:58.900551+0000 (oldest deadline 2024-02-20T09:32:18.900551+0000) 2024-02-20T09:33:13.725 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:33:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:33:13.255+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:31:58.900551+0000 (oldest deadline 2024-02-20T09:32:18.900551+0000) 2024-02-20T09:33:13.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:33:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:33:13.722+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:31:46.482189+0000 (oldest deadline 2024-02-20T09:32:06.482189+0000) 2024-02-20T09:33:13.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:33:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:33:13.722+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:31:46.482189+0000 (oldest deadline 2024-02-20T09:32:06.482189+0000) 2024-02-20T09:33:13.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:33:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:33:13.722+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:31:46.482189+0000 (oldest deadline 2024-02-20T09:32:06.482189+0000) 2024-02-20T09:33:14.307 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:33:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:33:14.013+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:14.015072+0000) 2024-02-20T09:33:14.308 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:33:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:33:14.008+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:32:16.988591+0000 (oldest deadline 2024-02-20T09:32:36.988591+0000) 2024-02-20T09:33:14.308 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:33:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:33:14.008+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:32:16.988591+0000 (oldest deadline 2024-02-20T09:32:36.988591+0000) 2024-02-20T09:33:14.308 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:33:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:33:14.008+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:32:16.988591+0000 (oldest deadline 2024-02-20T09:32:36.988591+0000) 2024-02-20T09:33:14.308 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:33:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:33:14.008+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:33:14.309 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:33:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:33:14.128+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:14.129887+0000) 2024-02-20T09:33:14.309 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:33:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:33:14.305+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:31:58.900551+0000 (oldest deadline 2024-02-20T09:32:18.900551+0000) 2024-02-20T09:33:14.309 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:33:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:33:14.305+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:31:58.900551+0000 (oldest deadline 2024-02-20T09:32:18.900551+0000) 2024-02-20T09:33:14.309 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:33:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:33:14.305+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:31:58.900551+0000 (oldest deadline 2024-02-20T09:32:18.900551+0000) 2024-02-20T09:33:14.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:33:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:33:14.448+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:14.449682+0000) 2024-02-20T09:33:15.004 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:33:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:33:14.748+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:31:46.482189+0000 (oldest deadline 2024-02-20T09:32:06.482189+0000) 2024-02-20T09:33:15.004 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:33:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:33:14.748+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:31:46.482189+0000 (oldest deadline 2024-02-20T09:32:06.482189+0000) 2024-02-20T09:33:15.005 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:33:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:33:14.748+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:31:46.482189+0000 (oldest deadline 2024-02-20T09:32:06.482189+0000) 2024-02-20T09:33:15.005 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:33:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:33:14.970+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:32:16.988591+0000 (oldest deadline 2024-02-20T09:32:36.988591+0000) 2024-02-20T09:33:15.005 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:33:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:33:14.970+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:32:16.988591+0000 (oldest deadline 2024-02-20T09:32:36.988591+0000) 2024-02-20T09:33:15.005 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:33:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:33:14.970+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:32:16.988591+0000 (oldest deadline 2024-02-20T09:32:36.988591+0000) 2024-02-20T09:33:15.005 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:33:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:33:14.970+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:33:15.061 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi183.front.sepia.ceph.com' 2024-02-20T09:33:15.061 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi183.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-20T09:33:15.132 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.183 2024-02-20T09:33:15.132 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-02-20T09:33:15.132 INFO:tasks.cephadm:Checking cluster log for badness... 2024-02-20T09:33:15.133 DEBUG:teuthology.orchestra.run.smithi133:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/4537327c-cfc4-11ee-95bb-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v OSD_DOWN | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | head -n 1 2024-02-20T09:33:15.170 INFO:tasks.cephadm:Compressing logs... 2024-02-20T09:33:15.171 DEBUG:teuthology.orchestra.run.smithi133:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-02-20T09:33:15.213 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi183.front.sepia.ceph.com' 2024-02-20T09:33:15.214 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi183.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-20T09:33:15.237 INFO:teuthology.orchestra.run.smithi133.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2024-02-20T09:33:15.238 INFO:teuthology.orchestra.run.smithi133.stderr:gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-02-20T09:33:15.238 INFO:teuthology.orchestra.run.smithi133.stderr:gzip -5 --verbose -- /var/log/ceph/4537327c-cfc4-11ee-95bb-87774f69a715/ceph-mds.cephfs.smithi133.itlmbm.log 2024-02-20T09:33:15.239 INFO:teuthology.orchestra.run.smithi133.stderr:gzip -5 --verbose -- /var/log/ceph/4537327c-cfc4-11ee-95bb-87774f69a715/ceph-mon.smithi133.log 2024-02-20T09:33:15.239 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/4537327c-cfc4-11ee-95bb-87774f69a715/ceph.audit.log 2024-02-20T09:33:15.239 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/4537327c-cfc4-11ee-95bb-87774f69a715/ceph-mds.cephfs.smithi133.itlmbm.log: /var/log/ceph/4537327c-cfc4-11ee-95bb-87774f69a715/ceph-mon.smithi133.log: gzip -5 --verbose -- /var/log/ceph/4537327c-cfc4-11ee-95bb-87774f69a715/ceph-osd.1.log 2024-02-20T09:33:15.239 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/4537327c-cfc4-11ee-95bb-87774f69a715/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/4537327c-cfc4-11ee-95bb-87774f69a715/ceph-mds.cephfs.smithi133.cmofvy.log 2024-02-20T09:33:15.240 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/4537327c-cfc4-11ee-95bb-87774f69a715/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/4537327c-cfc4-11ee-95bb-87774f69a715/ceph-osd.0.log 2024-02-20T09:33:15.240 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/4537327c-cfc4-11ee-95bb-87774f69a715/ceph-mds.cephfs.smithi133.cmofvy.log: gzip -5 --verbose -- /var/log/ceph/4537327c-cfc4-11ee-95bb-87774f69a715/ceph.cephadm.log 2024-02-20T09:33:15.241 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/4537327c-cfc4-11ee-95bb-87774f69a715/ceph-osd.0.log: 90.0% -- replaced with /var/log/ceph/4537327c-cfc4-11ee-95bb-87774f69a715/ceph.audit.log.gz 2024-02-20T09:33:15.241 INFO:teuthology.orchestra.run.smithi133.stderr:gzip -5 --verbose -- /var/log/ceph/4537327c-cfc4-11ee-95bb-87774f69a715/ceph.log 2024-02-20T09:33:15.241 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/4537327c-cfc4-11ee-95bb-87774f69a715/ceph.cephadm.log: 81.5% -- replaced with /var/log/ceph/4537327c-cfc4-11ee-95bb-87774f69a715/ceph.cephadm.log.gz 2024-02-20T09:33:15.242 INFO:teuthology.orchestra.run.smithi133.stderr:gzip -5 --verbose -- /var/log/ceph/4537327c-cfc4-11ee-95bb-87774f69a715/ceph-osd.2.log 2024-02-20T09:33:15.243 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/4537327c-cfc4-11ee-95bb-87774f69a715/ceph.log: 85.8%gzip -5 --verbose -- /var/log/ceph/4537327c-cfc4-11ee-95bb-87774f69a715/ceph-volume.log 2024-02-20T09:33:15.243 INFO:teuthology.orchestra.run.smithi133.stderr: -- replaced with /var/log/ceph/4537327c-cfc4-11ee-95bb-87774f69a715/ceph.log.gz 2024-02-20T09:33:15.247 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/4537327c-cfc4-11ee-95bb-87774f69a715/ceph-osd.2.log: 93.3% -- replaced with /var/log/ceph/cephadm.log.gz 2024-02-20T09:33:15.256 INFO:teuthology.orchestra.run.smithi133.stderr:gzip -5 --verbose -- /var/log/ceph/4537327c-cfc4-11ee-95bb-87774f69a715/ceph-mgr.smithi133.mrvcbp.log 2024-02-20T09:33:15.263 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:33:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:33:15.128+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:15.130020+0000) 2024-02-20T09:33:15.263 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:33:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:33:15.260+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:31:58.900551+0000 (oldest deadline 2024-02-20T09:32:18.900551+0000) 2024-02-20T09:33:15.264 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:33:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:33:15.013+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:15.015203+0000) 2024-02-20T09:33:15.266 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/4537327c-cfc4-11ee-95bb-87774f69a715/ceph-volume.log: 90.8% -- replaced with /var/log/ceph/4537327c-cfc4-11ee-95bb-87774f69a715/ceph-volume.log.gz 2024-02-20T09:33:15.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:33:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:33:15.450+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:15.451716+0000) 2024-02-20T09:33:15.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:33:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:33:15.260+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:31:58.900551+0000 (oldest deadline 2024-02-20T09:32:18.900551+0000) 2024-02-20T09:33:15.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:33:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:33:15.260+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:31:58.900551+0000 (oldest deadline 2024-02-20T09:32:18.900551+0000) 2024-02-20T09:33:16.005 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:33:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:33:15.753+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:31:46.482189+0000 (oldest deadline 2024-02-20T09:32:06.482189+0000) 2024-02-20T09:33:16.005 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:33:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:33:15.753+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:31:46.482189+0000 (oldest deadline 2024-02-20T09:32:06.482189+0000) 2024-02-20T09:33:16.006 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:33:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:33:15.753+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:31:46.482189+0000 (oldest deadline 2024-02-20T09:32:06.482189+0000) 2024-02-20T09:33:16.006 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:33:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:33:15.974+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:32:16.988591+0000 (oldest deadline 2024-02-20T09:32:36.988591+0000) 2024-02-20T09:33:16.006 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:33:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:33:15.974+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:32:16.988591+0000 (oldest deadline 2024-02-20T09:32:36.988591+0000) 2024-02-20T09:33:16.006 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:33:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:33:15.974+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:32:16.988591+0000 (oldest deadline 2024-02-20T09:32:36.988591+0000) 2024-02-20T09:33:16.006 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:33:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:33:15.974+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:33:16.302 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:33:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:33:16.128+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:16.130141+0000) 2024-02-20T09:33:16.302 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:33:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:33:16.298+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:31:58.900551+0000 (oldest deadline 2024-02-20T09:32:18.900551+0000) 2024-02-20T09:33:16.302 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:33:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:33:16.298+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:31:58.900551+0000 (oldest deadline 2024-02-20T09:32:18.900551+0000) 2024-02-20T09:33:16.302 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:33:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:33:16.013+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:16.015331+0000) 2024-02-20T09:33:16.611 INFO:teuthology.orchestra.run.smithi133.stderr:/var/log/ceph/4537327c-cfc4-11ee-95bb-87774f69a715/ceph-mgr.smithi133.mrvcbp.log: 95.5% -- replaced with /var/log/ceph/4537327c-cfc4-11ee-95bb-87774f69a715/ceph-mds.cephfs.smithi133.itlmbm.log.gz 2024-02-20T09:33:16.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:33:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:33:16.298+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:31:58.900551+0000 (oldest deadline 2024-02-20T09:32:18.900551+0000) 2024-02-20T09:33:16.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:33:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:33:16.453+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:16.454744+0000) 2024-02-20T09:33:17.005 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:33:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:33:16.778+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:31:46.482189+0000 (oldest deadline 2024-02-20T09:32:06.482189+0000) 2024-02-20T09:33:17.005 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:33:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:33:16.778+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:31:46.482189+0000 (oldest deadline 2024-02-20T09:32:06.482189+0000) 2024-02-20T09:33:17.005 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:33:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:33:16.778+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:31:46.482189+0000 (oldest deadline 2024-02-20T09:32:06.482189+0000) 2024-02-20T09:33:17.006 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:33:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:33:16.969+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:32:16.988591+0000 (oldest deadline 2024-02-20T09:32:36.988591+0000) 2024-02-20T09:33:17.006 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:33:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:33:16.969+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:32:16.988591+0000 (oldest deadline 2024-02-20T09:32:36.988591+0000) 2024-02-20T09:33:17.006 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:33:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:33:16.969+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:32:16.988591+0000 (oldest deadline 2024-02-20T09:32:36.988591+0000) 2024-02-20T09:33:17.006 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:33:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:33:16.969+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:33:17.239 INFO:teuthology.orchestra.run.smithi133.stderr:gzip: /var/log/ceph/4537327c-cfc4-11ee-95bb-87774f69a715/ceph-osd.0.log: file size changed while zipping 2024-02-20T09:33:17.240 INFO:teuthology.orchestra.run.smithi133.stderr: 92.4% -- replaced with /var/log/ceph/4537327c-cfc4-11ee-95bb-87774f69a715/ceph-osd.0.log.gz 2024-02-20T09:33:17.277 INFO:teuthology.orchestra.run.smithi133.stderr:gzip: /var/log/ceph/4537327c-cfc4-11ee-95bb-87774f69a715/ceph-mgr.smithi133.mrvcbp.log: file size changed while zipping 2024-02-20T09:33:17.277 INFO:teuthology.orchestra.run.smithi133.stderr: 90.0% -- replaced with /var/log/ceph/4537327c-cfc4-11ee-95bb-87774f69a715/ceph-mgr.smithi133.mrvcbp.log.gz 2024-02-20T09:33:17.328 INFO:teuthology.orchestra.run.smithi133.stderr:gzip: /var/log/ceph/4537327c-cfc4-11ee-95bb-87774f69a715/ceph-osd.2.log: file size changed while zipping 2024-02-20T09:33:17.328 INFO:teuthology.orchestra.run.smithi133.stderr: 92.6% -- replaced with /var/log/ceph/4537327c-cfc4-11ee-95bb-87774f69a715/ceph-osd.2.log.gz 2024-02-20T09:33:17.342 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:33:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:33:17.014+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:17.015421+0000) 2024-02-20T09:33:17.342 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:33:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:33:17.128+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:17.130267+0000) 2024-02-20T09:33:17.342 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:33:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:33:17.339+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:31:58.900551+0000 (oldest deadline 2024-02-20T09:32:18.900551+0000) 2024-02-20T09:33:17.477 INFO:teuthology.orchestra.run.smithi133.stderr:gzip: /var/log/ceph/4537327c-cfc4-11ee-95bb-87774f69a715/ceph-osd.1.log: file size changed while zipping 2024-02-20T09:33:17.477 INFO:teuthology.orchestra.run.smithi133.stderr: 92.4% -- replaced with /var/log/ceph/4537327c-cfc4-11ee-95bb-87774f69a715/ceph-osd.1.log.gz 2024-02-20T09:33:17.691 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:33:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:33:17.456+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:17.457553+0000) 2024-02-20T09:33:17.692 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:33:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:33:17.340+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:31:58.900551+0000 (oldest deadline 2024-02-20T09:32:18.900551+0000) 2024-02-20T09:33:17.692 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:33:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:33:17.340+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:31:58.900551+0000 (oldest deadline 2024-02-20T09:32:18.900551+0000) 2024-02-20T09:33:17.964 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:33:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:33:17.689+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:17.690471+0000) 2024-02-20T09:33:17.965 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:33:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:33:17.692+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:17.694376+0000) 2024-02-20T09:33:17.965 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:33:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:33:17.962+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:32:16.988591+0000 (oldest deadline 2024-02-20T09:32:36.988591+0000) 2024-02-20T09:33:17.965 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:33:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:33:17.689+0000 7f8a5f86e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:17.690802+0000) 2024-02-20T09:33:17.965 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:33:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:33:17.824+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:31:46.482189+0000 (oldest deadline 2024-02-20T09:32:06.482189+0000) 2024-02-20T09:33:17.966 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:33:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:33:17.824+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:31:46.482189+0000 (oldest deadline 2024-02-20T09:32:06.482189+0000) 2024-02-20T09:33:17.966 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:33:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:33:17.824+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:31:46.482189+0000 (oldest deadline 2024-02-20T09:32:06.482189+0000) 2024-02-20T09:33:17.966 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:33:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:33:17.695+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 72891 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:33:18.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:33:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:33:18.014+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:18.015546+0000) 2024-02-20T09:33:18.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:33:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:33:18.129+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:18.130416+0000) 2024-02-20T09:33:18.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:33:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:33:17.962+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:32:16.988591+0000 (oldest deadline 2024-02-20T09:32:36.988591+0000) 2024-02-20T09:33:18.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:33:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:33:17.962+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:32:16.988591+0000 (oldest deadline 2024-02-20T09:32:36.988591+0000) 2024-02-20T09:33:18.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:33:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:33:17.962+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:33:18.272 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.183 2024-02-20T09:33:18.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:33:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:33:18.458+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:18.459416+0000) 2024-02-20T09:33:18.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:33:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:33:18.313+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:31:58.900551+0000 (oldest deadline 2024-02-20T09:32:18.900551+0000) 2024-02-20T09:33:18.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:33:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:33:18.313+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:31:58.900551+0000 (oldest deadline 2024-02-20T09:32:18.900551+0000) 2024-02-20T09:33:18.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:33:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:33:18.313+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:31:58.900551+0000 (oldest deadline 2024-02-20T09:32:18.900551+0000) 2024-02-20T09:33:19.005 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:33:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:33:18.802+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:31:46.482189+0000 (oldest deadline 2024-02-20T09:32:06.482189+0000) 2024-02-20T09:33:19.005 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:33:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:33:18.802+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:31:46.482189+0000 (oldest deadline 2024-02-20T09:32:06.482189+0000) 2024-02-20T09:33:19.005 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:33:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:33:18.802+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:31:46.482189+0000 (oldest deadline 2024-02-20T09:32:06.482189+0000) 2024-02-20T09:33:19.006 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:33:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:33:18.991+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:32:16.988591+0000 (oldest deadline 2024-02-20T09:32:36.988591+0000) 2024-02-20T09:33:19.006 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:33:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:33:18.991+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:32:16.988591+0000 (oldest deadline 2024-02-20T09:32:36.988591+0000) 2024-02-20T09:33:19.006 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:33:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:33:18.991+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:32:16.988591+0000 (oldest deadline 2024-02-20T09:32:36.988591+0000) 2024-02-20T09:33:19.006 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:33:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:33:18.991+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:33:19.361 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:33:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:33:19.014+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:19.015630+0000) 2024-02-20T09:33:19.361 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:33:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:33:19.129+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:19.130502+0000) 2024-02-20T09:33:19.361 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:33:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:33:19.358+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:31:58.900551+0000 (oldest deadline 2024-02-20T09:32:18.900551+0000) 2024-02-20T09:33:19.361 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:33:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:33:19.358+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:31:58.900551+0000 (oldest deadline 2024-02-20T09:32:18.900551+0000) 2024-02-20T09:33:19.362 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:33:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:33:19.358+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:31:58.900551+0000 (oldest deadline 2024-02-20T09:32:18.900551+0000) 2024-02-20T09:33:19.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:33:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:33:19.460+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:19.461880+0000) 2024-02-20T09:33:20.005 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:33:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:33:19.753+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:31:46.482189+0000 (oldest deadline 2024-02-20T09:32:06.482189+0000) 2024-02-20T09:33:20.005 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:33:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:33:19.753+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:31:46.482189+0000 (oldest deadline 2024-02-20T09:32:06.482189+0000) 2024-02-20T09:33:20.005 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:33:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:33:19.753+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:31:46.482189+0000 (oldest deadline 2024-02-20T09:32:06.482189+0000) 2024-02-20T09:33:20.006 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:33:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:33:19.994+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:32:16.988591+0000 (oldest deadline 2024-02-20T09:32:36.988591+0000) 2024-02-20T09:33:20.006 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:33:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:33:19.994+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:32:16.988591+0000 (oldest deadline 2024-02-20T09:32:36.988591+0000) 2024-02-20T09:33:20.006 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:33:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:33:19.994+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:32:16.988591+0000 (oldest deadline 2024-02-20T09:32:36.988591+0000) 2024-02-20T09:33:20.006 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:33:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:33:19.994+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:33:20.391 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:33:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:33:20.014+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:20.015759+0000) 2024-02-20T09:33:20.391 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:33:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:33:20.129+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:20.130613+0000) 2024-02-20T09:33:20.391 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:33:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:33:20.388+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:31:58.900551+0000 (oldest deadline 2024-02-20T09:32:18.900551+0000) 2024-02-20T09:33:20.392 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:33:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:33:20.388+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:31:58.900551+0000 (oldest deadline 2024-02-20T09:32:18.900551+0000) 2024-02-20T09:33:20.392 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:33:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:33:20.388+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:31:58.900551+0000 (oldest deadline 2024-02-20T09:32:18.900551+0000) 2024-02-20T09:33:20.721 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:33:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:33:20.462+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:20.464021+0000) 2024-02-20T09:33:20.995 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:33:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:33:20.718+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:31:46.482189+0000 (oldest deadline 2024-02-20T09:32:06.482189+0000) 2024-02-20T09:33:20.995 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:33:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:33:20.718+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:31:46.482189+0000 (oldest deadline 2024-02-20T09:32:06.482189+0000) 2024-02-20T09:33:20.996 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:33:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:33:20.718+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:31:46.482189+0000 (oldest deadline 2024-02-20T09:32:06.482189+0000) 2024-02-20T09:33:20.996 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:33:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:33:20.961+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:32:16.988591+0000 (oldest deadline 2024-02-20T09:32:36.988591+0000) 2024-02-20T09:33:20.996 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:33:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:33:20.961+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:32:16.988591+0000 (oldest deadline 2024-02-20T09:32:36.988591+0000) 2024-02-20T09:33:20.996 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:33:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:33:20.961+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:32:16.988591+0000 (oldest deadline 2024-02-20T09:32:36.988591+0000) 2024-02-20T09:33:20.997 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:33:20 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:33:20.961+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:33:21.409 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:33:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:33:21.129+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:21.130717+0000) 2024-02-20T09:33:21.410 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:33:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:33:21.014+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:21.015881+0000) 2024-02-20T09:33:21.721 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:33:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:33:21.467+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:21.467581+0000) 2024-02-20T09:33:21.721 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:33:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:33:21.430+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:31:58.900551+0000 (oldest deadline 2024-02-20T09:32:18.900551+0000) 2024-02-20T09:33:21.721 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:33:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:33:21.430+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:31:58.900551+0000 (oldest deadline 2024-02-20T09:32:18.900551+0000) 2024-02-20T09:33:21.722 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:33:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:33:21.430+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:31:58.900551+0000 (oldest deadline 2024-02-20T09:32:18.900551+0000) 2024-02-20T09:33:21.982 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:33:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:33:21.719+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:31:46.482189+0000 (oldest deadline 2024-02-20T09:32:06.482189+0000) 2024-02-20T09:33:21.982 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:33:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:33:21.719+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:31:46.482189+0000 (oldest deadline 2024-02-20T09:32:06.482189+0000) 2024-02-20T09:33:21.982 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:33:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:33:21.719+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:31:46.482189+0000 (oldest deadline 2024-02-20T09:32:06.482189+0000) 2024-02-20T09:33:22.011 INFO:teuthology.orchestra.run.smithi133.stderr:gzip: /var/log/ceph/4537327c-cfc4-11ee-95bb-87774f69a715/ceph-mon.smithi133.log: file size changed while zipping 2024-02-20T09:33:22.011 INFO:teuthology.orchestra.run.smithi133.stderr: 97.3% -- replaced with /var/log/ceph/4537327c-cfc4-11ee-95bb-87774f69a715/ceph-mon.smithi133.log.gz 2024-02-20T09:33:22.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:33:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:33:22.015+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:22.015979+0000) 2024-02-20T09:33:22.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:33:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:33:22.130+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:22.130829+0000) 2024-02-20T09:33:22.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:33:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:33:21.980+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:32:16.988591+0000 (oldest deadline 2024-02-20T09:32:36.988591+0000) 2024-02-20T09:33:22.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:33:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:33:21.980+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:32:16.988591+0000 (oldest deadline 2024-02-20T09:32:36.988591+0000) 2024-02-20T09:33:22.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:33:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:33:21.980+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:32:16.988591+0000 (oldest deadline 2024-02-20T09:32:36.988591+0000) 2024-02-20T09:33:22.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:33:21 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:33:21.980+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:33:22.699 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:33:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:33:22.469+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:22.469826+0000) 2024-02-20T09:33:22.699 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:33:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:33:22.458+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:31:58.900551+0000 (oldest deadline 2024-02-20T09:32:18.900551+0000) 2024-02-20T09:33:22.700 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:33:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:33:22.458+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:31:58.900551+0000 (oldest deadline 2024-02-20T09:32:18.900551+0000) 2024-02-20T09:33:22.700 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:33:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:33:22.458+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:31:58.900551+0000 (oldest deadline 2024-02-20T09:32:18.900551+0000) 2024-02-20T09:33:22.955 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:33:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:33:22.701+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:22.702101+0000) 2024-02-20T09:33:22.955 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:33:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:33:22.697+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:22.698316+0000) 2024-02-20T09:33:22.956 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:33:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:33:22.697+0000 7f8a5f86e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:22.698190+0000) 2024-02-20T09:33:22.956 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:33:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:33:22.738+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:31:46.482189+0000 (oldest deadline 2024-02-20T09:32:06.482189+0000) 2024-02-20T09:33:22.956 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:33:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:33:22.739+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:31:46.482189+0000 (oldest deadline 2024-02-20T09:32:06.482189+0000) 2024-02-20T09:33:22.956 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:33:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:33:22.739+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:31:46.482189+0000 (oldest deadline 2024-02-20T09:32:06.482189+0000) 2024-02-20T09:33:22.956 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:33:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:33:22.705+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 72930 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:33:23.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:33:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:33:23.015+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:23.016106+0000) 2024-02-20T09:33:23.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:33:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:33:23.130+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:23.130957+0000) 2024-02-20T09:33:23.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:33:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:33:22.953+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:32:16.988591+0000 (oldest deadline 2024-02-20T09:32:36.988591+0000) 2024-02-20T09:33:23.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:33:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:33:22.953+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:32:16.988591+0000 (oldest deadline 2024-02-20T09:32:36.988591+0000) 2024-02-20T09:33:23.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:33:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:33:22.953+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:32:16.988591+0000 (oldest deadline 2024-02-20T09:32:36.988591+0000) 2024-02-20T09:33:23.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:33:22 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:33:22.953+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:33:23.737 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:33:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:33:23.471+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:23.472105+0000) 2024-02-20T09:33:23.737 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:33:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:33:23.483+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:31:58.900551+0000 (oldest deadline 2024-02-20T09:32:18.900551+0000) 2024-02-20T09:33:23.738 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:33:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:33:23.483+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:31:58.900551+0000 (oldest deadline 2024-02-20T09:32:18.900551+0000) 2024-02-20T09:33:23.738 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:33:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:33:23.483+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:31:58.900551+0000 (oldest deadline 2024-02-20T09:32:18.900551+0000) 2024-02-20T09:33:23.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:33:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:33:23.735+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:31:46.482189+0000 (oldest deadline 2024-02-20T09:32:06.482189+0000) 2024-02-20T09:33:23.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:33:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:33:23.735+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:31:46.482189+0000 (oldest deadline 2024-02-20T09:32:06.482189+0000) 2024-02-20T09:33:23.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:33:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:33:23.735+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:31:46.482189+0000 (oldest deadline 2024-02-20T09:32:06.482189+0000) 2024-02-20T09:33:23.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:33:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:33:23.933+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:32:16.988591+0000 (oldest deadline 2024-02-20T09:32:36.988591+0000) 2024-02-20T09:33:23.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:33:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:33:23.933+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:32:16.988591+0000 (oldest deadline 2024-02-20T09:32:36.988591+0000) 2024-02-20T09:33:23.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:33:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:33:23.933+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:32:16.988591+0000 (oldest deadline 2024-02-20T09:32:36.988591+0000) 2024-02-20T09:33:23.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:33:23 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:33:23.934+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:33:24.409 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:33:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:33:24.130+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:24.131051+0000) 2024-02-20T09:33:24.410 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:33:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:33:24.015+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:24.016210+0000) 2024-02-20T09:33:24.707 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:33:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:33:24.475+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:24.475639+0000) 2024-02-20T09:33:24.707 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:33:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:33:24.484+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:31:58.900551+0000 (oldest deadline 2024-02-20T09:32:18.900551+0000) 2024-02-20T09:33:24.707 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:33:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:33:24.484+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:31:58.900551+0000 (oldest deadline 2024-02-20T09:32:18.900551+0000) 2024-02-20T09:33:24.707 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:33:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:33:24.484+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:31:58.900551+0000 (oldest deadline 2024-02-20T09:32:18.900551+0000) 2024-02-20T09:33:24.963 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:33:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:33:24.705+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:31:46.482189+0000 (oldest deadline 2024-02-20T09:32:06.482189+0000) 2024-02-20T09:33:24.963 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:33:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:33:24.705+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:31:46.482189+0000 (oldest deadline 2024-02-20T09:32:06.482189+0000) 2024-02-20T09:33:24.964 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:33:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:33:24.705+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:31:46.482189+0000 (oldest deadline 2024-02-20T09:32:06.482189+0000) 2024-02-20T09:33:25.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:33:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:33:25.015+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:25.016325+0000) 2024-02-20T09:33:25.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:33:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:33:25.130+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:25.131189+0000) 2024-02-20T09:33:25.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:33:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:33:24.961+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:32:16.988591+0000 (oldest deadline 2024-02-20T09:32:36.988591+0000) 2024-02-20T09:33:25.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:33:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:33:24.961+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:32:16.988591+0000 (oldest deadline 2024-02-20T09:32:36.988591+0000) 2024-02-20T09:33:25.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:33:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:33:24.961+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:32:16.988591+0000 (oldest deadline 2024-02-20T09:32:36.988591+0000) 2024-02-20T09:33:25.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:33:24 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:33:24.961+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:33:25.350 INFO:teuthology.orchestra.run.smithi133.stderr: 92.5% -- replaced with /var/log/ceph/4537327c-cfc4-11ee-95bb-87774f69a715/ceph-mds.cephfs.smithi133.cmofvy.log.gz 2024-02-20T09:33:25.351 INFO:teuthology.orchestra.run.smithi133.stderr: 2024-02-20T09:33:25.351 INFO:teuthology.orchestra.run.smithi133.stderr:real 0m10.126s 2024-02-20T09:33:25.351 INFO:teuthology.orchestra.run.smithi133.stderr:user 0m25.797s 2024-02-20T09:33:25.352 INFO:teuthology.orchestra.run.smithi133.stderr:sys 0m0.713s 2024-02-20T09:33:25.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:33:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:33:25.478+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:25.479193+0000) 2024-02-20T09:33:25.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:33:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:33:25.477+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:31:58.900551+0000 (oldest deadline 2024-02-20T09:32:18.900551+0000) 2024-02-20T09:33:25.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:33:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:33:25.477+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:31:58.900551+0000 (oldest deadline 2024-02-20T09:32:18.900551+0000) 2024-02-20T09:33:25.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:33:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:33:25.477+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:31:58.900551+0000 (oldest deadline 2024-02-20T09:32:18.900551+0000) 2024-02-20T09:33:26.005 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:33:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:33:25.754+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:31:46.482189+0000 (oldest deadline 2024-02-20T09:32:06.482189+0000) 2024-02-20T09:33:26.006 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:33:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:33:25.754+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:31:46.482189+0000 (oldest deadline 2024-02-20T09:32:06.482189+0000) 2024-02-20T09:33:26.006 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:33:25 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:33:25.754+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:31:46.482189+0000 (oldest deadline 2024-02-20T09:32:06.482189+0000) 2024-02-20T09:33:26.410 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:33:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:33:26.016+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:26.016441+0000) 2024-02-20T09:33:26.410 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:33:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:33:26.130+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:26.131314+0000) 2024-02-20T09:33:26.411 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:33:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:33:26.005+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:32:16.988591+0000 (oldest deadline 2024-02-20T09:32:36.988591+0000) 2024-02-20T09:33:26.411 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:33:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:33:26.005+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:32:16.988591+0000 (oldest deadline 2024-02-20T09:32:36.988591+0000) 2024-02-20T09:33:26.411 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:33:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:33:26.005+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:32:16.988591+0000 (oldest deadline 2024-02-20T09:32:36.988591+0000) 2024-02-20T09:33:26.411 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:33:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:33:26.005+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:33:26.724 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:33:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:33:26.480+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:26.481280+0000) 2024-02-20T09:33:26.725 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:33:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:33:26.518+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:31:58.900551+0000 (oldest deadline 2024-02-20T09:32:18.900551+0000) 2024-02-20T09:33:26.725 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:33:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:33:26.518+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:31:58.900551+0000 (oldest deadline 2024-02-20T09:32:18.900551+0000) 2024-02-20T09:33:26.725 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:33:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:33:26.518+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:31:58.900551+0000 (oldest deadline 2024-02-20T09:32:18.900551+0000) 2024-02-20T09:33:26.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:33:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:33:26.966+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:32:16.988591+0000 (oldest deadline 2024-02-20T09:32:36.988591+0000) 2024-02-20T09:33:26.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:33:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:33:26.966+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:32:16.988591+0000 (oldest deadline 2024-02-20T09:32:36.988591+0000) 2024-02-20T09:33:26.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:33:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:33:26.966+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:32:16.988591+0000 (oldest deadline 2024-02-20T09:32:36.988591+0000) 2024-02-20T09:33:26.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:33:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:33:26.966+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:33:26.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:33:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:33:26.723+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:31:46.482189+0000 (oldest deadline 2024-02-20T09:32:06.482189+0000) 2024-02-20T09:33:26.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:33:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:33:26.723+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:31:46.482189+0000 (oldest deadline 2024-02-20T09:32:06.482189+0000) 2024-02-20T09:33:26.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:33:26 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:33:26.723+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:31:46.482189+0000 (oldest deadline 2024-02-20T09:32:06.482189+0000) 2024-02-20T09:33:27.273 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi183.front.sepia.ceph.com' 2024-02-20T09:33:27.274 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi183.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-20T09:33:27.410 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:33:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:33:27.016+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:27.016529+0000) 2024-02-20T09:33:27.410 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:33:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:33:27.131+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:27.131433+0000) 2024-02-20T09:33:27.696 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:33:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:33:27.484+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:27.484905+0000) 2024-02-20T09:33:27.697 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:33:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:33:27.547+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:31:58.900551+0000 (oldest deadline 2024-02-20T09:32:18.900551+0000) 2024-02-20T09:33:27.697 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:33:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:33:27.547+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:31:58.900551+0000 (oldest deadline 2024-02-20T09:32:18.900551+0000) 2024-02-20T09:33:27.697 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:33:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:33:27.547+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:31:58.900551+0000 (oldest deadline 2024-02-20T09:32:18.900551+0000) 2024-02-20T09:33:27.956 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:33:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:33:27.707+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:27.707754+0000) 2024-02-20T09:33:27.957 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:33:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:33:27.713+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 72970 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:33:27.957 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:33:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:33:27.710+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:27.710636+0000) 2024-02-20T09:33:27.957 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:33:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:33:27.955+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:32:16.988591+0000 (oldest deadline 2024-02-20T09:32:36.988591+0000) 2024-02-20T09:33:27.957 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:33:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:33:27.955+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:32:16.988591+0000 (oldest deadline 2024-02-20T09:32:36.988591+0000) 2024-02-20T09:33:27.958 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:33:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:33:27.955+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:32:16.988591+0000 (oldest deadline 2024-02-20T09:32:36.988591+0000) 2024-02-20T09:33:27.958 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:33:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:33:27.695+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:31:46.482189+0000 (oldest deadline 2024-02-20T09:32:06.482189+0000) 2024-02-20T09:33:27.958 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:33:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:33:27.695+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:31:46.482189+0000 (oldest deadline 2024-02-20T09:32:06.482189+0000) 2024-02-20T09:33:27.958 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:33:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:33:27.695+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:31:46.482189+0000 (oldest deadline 2024-02-20T09:32:06.482189+0000) 2024-02-20T09:33:27.958 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:33:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:33:27.706+0000 7f8a5f86e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:27.707178+0000) 2024-02-20T09:33:28.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:33:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:33:28.016+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:28.016670+0000) 2024-02-20T09:33:28.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:33:27 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:33:27.955+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:33:28.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:33:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:33:28.131+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:28.131576+0000) 2024-02-20T09:33:28.735 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:33:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:33:28.486+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:28.486662+0000) 2024-02-20T09:33:28.736 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:33:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:33:28.502+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:31:58.900551+0000 (oldest deadline 2024-02-20T09:32:18.900551+0000) 2024-02-20T09:33:28.736 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:33:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:33:28.502+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:31:58.900551+0000 (oldest deadline 2024-02-20T09:32:18.900551+0000) 2024-02-20T09:33:28.736 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:33:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:33:28.502+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:31:58.900551+0000 (oldest deadline 2024-02-20T09:32:18.900551+0000) 2024-02-20T09:33:28.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:33:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:33:28.733+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:31:46.482189+0000 (oldest deadline 2024-02-20T09:32:06.482189+0000) 2024-02-20T09:33:28.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:33:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:33:28.733+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:31:46.482189+0000 (oldest deadline 2024-02-20T09:32:06.482189+0000) 2024-02-20T09:33:28.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:33:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:33:28.733+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:31:46.482189+0000 (oldest deadline 2024-02-20T09:32:06.482189+0000) 2024-02-20T09:33:29.410 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:33:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:33:29.016+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:29.016751+0000) 2024-02-20T09:33:29.411 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:33:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:33:29.131+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:29.131709+0000) 2024-02-20T09:33:29.411 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:33:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:33:28.996+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:32:16.988591+0000 (oldest deadline 2024-02-20T09:32:36.988591+0000) 2024-02-20T09:33:29.411 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:33:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:33:28.996+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:32:16.988591+0000 (oldest deadline 2024-02-20T09:32:36.988591+0000) 2024-02-20T09:33:29.411 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:33:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:33:28.996+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:32:16.988591+0000 (oldest deadline 2024-02-20T09:32:36.988591+0000) 2024-02-20T09:33:29.411 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:33:28 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:33:28.996+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:33:29.708 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:33:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:33:29.488+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:29.489280+0000) 2024-02-20T09:33:29.708 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:33:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:33:29.528+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:31:58.900551+0000 (oldest deadline 2024-02-20T09:32:18.900551+0000) 2024-02-20T09:33:29.708 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:33:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:33:29.528+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:31:58.900551+0000 (oldest deadline 2024-02-20T09:32:18.900551+0000) 2024-02-20T09:33:29.708 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:33:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:33:29.528+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:31:58.900551+0000 (oldest deadline 2024-02-20T09:32:18.900551+0000) 2024-02-20T09:33:29.996 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:33:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:33:29.706+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:31:46.482189+0000 (oldest deadline 2024-02-20T09:32:06.482189+0000) 2024-02-20T09:33:29.997 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:33:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:33:29.706+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:31:46.482189+0000 (oldest deadline 2024-02-20T09:32:06.482189+0000) 2024-02-20T09:33:29.997 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:33:29 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:33:29.706+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:31:46.482189+0000 (oldest deadline 2024-02-20T09:32:06.482189+0000) 2024-02-20T09:33:30.332 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.183 2024-02-20T09:33:30.410 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:33:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:33:30.016+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:30.016920+0000) 2024-02-20T09:33:30.411 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:33:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:33:30.131+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:30.131905+0000) 2024-02-20T09:33:30.411 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:33:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:33:30.004+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:32:16.988591+0000 (oldest deadline 2024-02-20T09:32:36.988591+0000) 2024-02-20T09:33:30.411 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:33:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:33:30.004+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:32:16.988591+0000 (oldest deadline 2024-02-20T09:32:36.988591+0000) 2024-02-20T09:33:30.412 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:33:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:33:30.004+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:32:16.988591+0000 (oldest deadline 2024-02-20T09:32:36.988591+0000) 2024-02-20T09:33:30.412 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:33:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:33:30.004+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:33:30.722 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:33:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:33:30.491+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:30.491688+0000) 2024-02-20T09:33:30.723 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:33:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:33:30.576+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:31:58.900551+0000 (oldest deadline 2024-02-20T09:32:18.900551+0000) 2024-02-20T09:33:30.723 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:33:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:33:30.576+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:31:58.900551+0000 (oldest deadline 2024-02-20T09:32:18.900551+0000) 2024-02-20T09:33:30.723 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:33:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:33:30.576+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:31:58.900551+0000 (oldest deadline 2024-02-20T09:32:18.900551+0000) 2024-02-20T09:33:30.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:33:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:33:30.721+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:31:46.482189+0000 (oldest deadline 2024-02-20T09:32:06.482189+0000) 2024-02-20T09:33:30.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:33:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:33:30.721+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:31:46.482189+0000 (oldest deadline 2024-02-20T09:32:06.482189+0000) 2024-02-20T09:33:30.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:33:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:33:30.721+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:31:46.482189+0000 (oldest deadline 2024-02-20T09:32:06.482189+0000) 2024-02-20T09:33:30.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:33:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:33:30.969+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:32:16.988591+0000 (oldest deadline 2024-02-20T09:32:36.988591+0000) 2024-02-20T09:33:30.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:33:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:33:30.969+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:32:16.988591+0000 (oldest deadline 2024-02-20T09:32:36.988591+0000) 2024-02-20T09:33:30.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:33:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:33:30.969+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:32:16.988591+0000 (oldest deadline 2024-02-20T09:32:36.988591+0000) 2024-02-20T09:33:30.994 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:33:30 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:33:30.969+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:33:31.410 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:33:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:33:31.016+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:31.017041+0000) 2024-02-20T09:33:31.411 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:33:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:33:31.131+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:31.132084+0000) 2024-02-20T09:33:31.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:33:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:33:31.494+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:31.495352+0000) 2024-02-20T09:33:31.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:33:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:33:31.582+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:31:58.900551+0000 (oldest deadline 2024-02-20T09:32:18.900551+0000) 2024-02-20T09:33:31.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:33:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:33:31.582+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:31:58.900551+0000 (oldest deadline 2024-02-20T09:32:18.900551+0000) 2024-02-20T09:33:31.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:33:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:33:31.582+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:31:58.900551+0000 (oldest deadline 2024-02-20T09:32:18.900551+0000) 2024-02-20T09:33:32.006 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:33:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:33:31.744+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:31:46.482189+0000 (oldest deadline 2024-02-20T09:32:06.482189+0000) 2024-02-20T09:33:32.006 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:33:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:33:31.744+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:31:46.482189+0000 (oldest deadline 2024-02-20T09:32:06.482189+0000) 2024-02-20T09:33:32.006 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:33:31 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:33:31.744+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:31:46.482189+0000 (oldest deadline 2024-02-20T09:32:06.482189+0000) 2024-02-20T09:33:32.410 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:33:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:33:32.131+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:32.132286+0000) 2024-02-20T09:33:32.411 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:33:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:33:32.016+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:32.017167+0000) 2024-02-20T09:33:32.411 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:33:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:33:32.016+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:32:16.988591+0000 (oldest deadline 2024-02-20T09:32:36.988591+0000) 2024-02-20T09:33:32.411 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:33:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:33:32.016+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:32:16.988591+0000 (oldest deadline 2024-02-20T09:32:36.988591+0000) 2024-02-20T09:33:32.411 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:33:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:33:32.016+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:32:16.988591+0000 (oldest deadline 2024-02-20T09:32:36.988591+0000) 2024-02-20T09:33:32.412 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:33:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:33:32.016+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:33:32.704 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:33:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:33:32.498+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:32.498956+0000) 2024-02-20T09:33:32.704 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:33:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:33:32.605+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:31:58.900551+0000 (oldest deadline 2024-02-20T09:32:18.900551+0000) 2024-02-20T09:33:32.704 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:33:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:33:32.605+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:31:58.900551+0000 (oldest deadline 2024-02-20T09:32:18.900551+0000) 2024-02-20T09:33:32.704 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:33:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:33:32.605+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:31:58.900551+0000 (oldest deadline 2024-02-20T09:32:18.900551+0000) 2024-02-20T09:33:32.984 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:33:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:33:32.717+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:32.718203+0000) 2024-02-20T09:33:32.985 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:33:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:33:32.715+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:32.715865+0000) 2024-02-20T09:33:32.985 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:33:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:33:32.702+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:31:46.482189+0000 (oldest deadline 2024-02-20T09:32:06.482189+0000) 2024-02-20T09:33:32.986 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:33:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:33:32.702+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:31:46.482189+0000 (oldest deadline 2024-02-20T09:32:06.482189+0000) 2024-02-20T09:33:32.986 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:33:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:33:32.702+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:31:46.482189+0000 (oldest deadline 2024-02-20T09:32:06.482189+0000) 2024-02-20T09:33:32.986 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:33:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:33:32.714+0000 7f8a5f86e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:32.715307+0000) 2024-02-20T09:33:32.986 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:33:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:33:32.722+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 73009 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:33:33.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:33:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:33:33.016+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:33.017271+0000) 2024-02-20T09:33:33.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:33:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:33:33.132+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:33.132407+0000) 2024-02-20T09:33:33.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:33:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:33:32.982+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:32:16.988591+0000 (oldest deadline 2024-02-20T09:32:36.988591+0000) 2024-02-20T09:33:33.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:33:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:33:32.982+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:32:16.988591+0000 (oldest deadline 2024-02-20T09:32:36.988591+0000) 2024-02-20T09:33:33.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:33:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:33:32.982+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:32:16.988591+0000 (oldest deadline 2024-02-20T09:32:36.988591+0000) 2024-02-20T09:33:33.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:33:32 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:33:32.982+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:33:33.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:33:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:33:33.499+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:33.500255+0000) 2024-02-20T09:33:33.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:33:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:33:33.650+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:31:58.900551+0000 (oldest deadline 2024-02-20T09:32:18.900551+0000) 2024-02-20T09:33:33.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:33:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:33:33.650+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:31:58.900551+0000 (oldest deadline 2024-02-20T09:32:18.900551+0000) 2024-02-20T09:33:33.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:33:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:33:33.650+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:31:58.900551+0000 (oldest deadline 2024-02-20T09:32:18.900551+0000) 2024-02-20T09:33:34.006 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:33:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:33:33.747+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:31:46.482189+0000 (oldest deadline 2024-02-20T09:32:06.482189+0000) 2024-02-20T09:33:34.007 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:33:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:33:33.747+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:31:46.482189+0000 (oldest deadline 2024-02-20T09:32:06.482189+0000) 2024-02-20T09:33:34.007 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:33:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:33:33.747+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:31:46.482189+0000 (oldest deadline 2024-02-20T09:32:06.482189+0000) 2024-02-20T09:33:34.007 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:33:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:33:33.971+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:32:16.988591+0000 (oldest deadline 2024-02-20T09:32:36.988591+0000) 2024-02-20T09:33:34.007 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:33:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:33:33.971+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:32:16.988591+0000 (oldest deadline 2024-02-20T09:32:36.988591+0000) 2024-02-20T09:33:34.007 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:33:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:33:33.971+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:32:16.988591+0000 (oldest deadline 2024-02-20T09:32:36.988591+0000) 2024-02-20T09:33:34.007 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:33:33 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:33:33.971+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:33:34.411 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:33:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:33:34.016+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:34.017374+0000) 2024-02-20T09:33:34.411 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:33:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:33:34.132+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:34.132442+0000) 2024-02-20T09:33:34.727 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:33:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:33:34.503+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:34.503717+0000) 2024-02-20T09:33:34.728 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:33:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:33:34.609+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:31:58.900551+0000 (oldest deadline 2024-02-20T09:32:18.900551+0000) 2024-02-20T09:33:34.728 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:33:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:33:34.609+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:31:58.900551+0000 (oldest deadline 2024-02-20T09:32:18.900551+0000) 2024-02-20T09:33:34.728 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:33:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:33:34.609+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:31:58.900551+0000 (oldest deadline 2024-02-20T09:32:18.900551+0000) 2024-02-20T09:33:34.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:33:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:33:34.726+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:31:46.482189+0000 (oldest deadline 2024-02-20T09:32:06.482189+0000) 2024-02-20T09:33:34.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:33:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:33:34.726+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:31:46.482189+0000 (oldest deadline 2024-02-20T09:32:06.482189+0000) 2024-02-20T09:33:34.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:33:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:33:34.726+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:31:46.482189+0000 (oldest deadline 2024-02-20T09:32:06.482189+0000) 2024-02-20T09:33:34.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:33:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:33:34.949+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:32:16.988591+0000 (oldest deadline 2024-02-20T09:32:36.988591+0000) 2024-02-20T09:33:34.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:33:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:33:34.949+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:32:16.988591+0000 (oldest deadline 2024-02-20T09:32:36.988591+0000) 2024-02-20T09:33:34.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:33:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:33:34.949+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:32:16.988591+0000 (oldest deadline 2024-02-20T09:32:36.988591+0000) 2024-02-20T09:33:34.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:33:34 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:33:34.950+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:33:35.411 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:33:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:33:35.017+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:35.017498+0000) 2024-02-20T09:33:35.411 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:33:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:33:35.132+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:35.132558+0000) 2024-02-20T09:33:35.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:33:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:33:35.506+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:35.507152+0000) 2024-02-20T09:33:35.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:33:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:33:35.574+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:31:58.900551+0000 (oldest deadline 2024-02-20T09:32:18.900551+0000) 2024-02-20T09:33:35.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:33:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:33:35.574+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:31:58.900551+0000 (oldest deadline 2024-02-20T09:32:18.900551+0000) 2024-02-20T09:33:35.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:33:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:33:35.574+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:31:58.900551+0000 (oldest deadline 2024-02-20T09:32:18.900551+0000) 2024-02-20T09:33:36.006 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:33:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:33:35.773+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:31:46.482189+0000 (oldest deadline 2024-02-20T09:32:06.482189+0000) 2024-02-20T09:33:36.007 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:33:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:33:35.773+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:31:46.482189+0000 (oldest deadline 2024-02-20T09:32:06.482189+0000) 2024-02-20T09:33:36.007 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:33:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:33:35.773+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:31:46.482189+0000 (oldest deadline 2024-02-20T09:32:06.482189+0000) 2024-02-20T09:33:36.007 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:33:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:33:35.949+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:32:16.988591+0000 (oldest deadline 2024-02-20T09:32:36.988591+0000) 2024-02-20T09:33:36.007 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:33:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:33:35.949+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:32:16.988591+0000 (oldest deadline 2024-02-20T09:32:36.988591+0000) 2024-02-20T09:33:36.007 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:33:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:33:35.949+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:32:16.988591+0000 (oldest deadline 2024-02-20T09:32:36.988591+0000) 2024-02-20T09:33:36.007 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:33:35 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:33:35.949+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:33:36.411 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:33:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:33:36.017+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:36.017619+0000) 2024-02-20T09:33:36.411 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:33:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:33:36.132+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:36.132666+0000) 2024-02-20T09:33:36.743 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:33:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:33:36.510+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:36.510585+0000) 2024-02-20T09:33:36.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:33:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:33:36.531+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:31:58.900551+0000 (oldest deadline 2024-02-20T09:32:18.900551+0000) 2024-02-20T09:33:36.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:33:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:33:36.531+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:31:58.900551+0000 (oldest deadline 2024-02-20T09:32:18.900551+0000) 2024-02-20T09:33:36.744 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:33:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:33:36.531+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:31:58.900551+0000 (oldest deadline 2024-02-20T09:32:18.900551+0000) 2024-02-20T09:33:37.007 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:33:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:33:36.764+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:31:46.482189+0000 (oldest deadline 2024-02-20T09:32:06.482189+0000) 2024-02-20T09:33:37.007 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:33:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:33:36.764+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:31:46.482189+0000 (oldest deadline 2024-02-20T09:32:06.482189+0000) 2024-02-20T09:33:37.008 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:33:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:33:36.764+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:31:46.482189+0000 (oldest deadline 2024-02-20T09:32:06.482189+0000) 2024-02-20T09:33:37.008 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:33:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:33:36.915+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:32:16.988591+0000 (oldest deadline 2024-02-20T09:32:36.988591+0000) 2024-02-20T09:33:37.008 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:33:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:33:36.915+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:32:16.988591+0000 (oldest deadline 2024-02-20T09:32:36.988591+0000) 2024-02-20T09:33:37.009 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:33:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:33:36.915+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:32:16.988591+0000 (oldest deadline 2024-02-20T09:32:36.988591+0000) 2024-02-20T09:33:37.009 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:33:36 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:33:36.915+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:33:37.411 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:33:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:33:37.017+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:37.017708+0000) 2024-02-20T09:33:37.411 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:33:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:33:37.132+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:37.132761+0000) 2024-02-20T09:33:37.724 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:33:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:33:37.513+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:37.514148+0000) 2024-02-20T09:33:37.725 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:33:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:33:37.511+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:31:58.900551+0000 (oldest deadline 2024-02-20T09:32:18.900551+0000) 2024-02-20T09:33:37.725 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:33:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:33:37.511+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:31:58.900551+0000 (oldest deadline 2024-02-20T09:32:18.900551+0000) 2024-02-20T09:33:37.725 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:33:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:33:37.511+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:31:58.900551+0000 (oldest deadline 2024-02-20T09:32:18.900551+0000) 2024-02-20T09:33:37.725 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:33:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:33:37.723+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:37.723610+0000) 2024-02-20T09:33:37.992 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:33:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:33:37.730+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 73049 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:33:37.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:33:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:33:37.726+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:37.727203+0000) 2024-02-20T09:33:37.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:33:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:33:37.888+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:32:16.988591+0000 (oldest deadline 2024-02-20T09:32:36.988591+0000) 2024-02-20T09:33:37.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:33:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:33:37.888+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:32:16.988591+0000 (oldest deadline 2024-02-20T09:32:36.988591+0000) 2024-02-20T09:33:37.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:33:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:33:37.888+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:32:16.988591+0000 (oldest deadline 2024-02-20T09:32:36.988591+0000) 2024-02-20T09:33:37.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:33:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:33:37.889+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:33:37.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:33:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:33:37.723+0000 7f8a5f86e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:37.723666+0000) 2024-02-20T09:33:37.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:33:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:33:37.774+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:31:46.482189+0000 (oldest deadline 2024-02-20T09:32:06.482189+0000) 2024-02-20T09:33:37.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:33:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:33:37.774+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:31:46.482189+0000 (oldest deadline 2024-02-20T09:32:06.482189+0000) 2024-02-20T09:33:37.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:33:37 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:33:37.774+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:31:46.482189+0000 (oldest deadline 2024-02-20T09:32:06.482189+0000) 2024-02-20T09:33:38.411 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:33:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:33:38.017+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:38.017797+0000) 2024-02-20T09:33:38.411 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:33:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:33:38.132+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:38.132875+0000) 2024-02-20T09:33:38.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:33:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:33:38.515+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:38.515589+0000) 2024-02-20T09:33:38.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:33:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:33:38.510+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:31:58.900551+0000 (oldest deadline 2024-02-20T09:32:18.900551+0000) 2024-02-20T09:33:38.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:33:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:33:38.510+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:31:58.900551+0000 (oldest deadline 2024-02-20T09:32:18.900551+0000) 2024-02-20T09:33:38.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:33:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:33:38.510+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:31:58.900551+0000 (oldest deadline 2024-02-20T09:32:18.900551+0000) 2024-02-20T09:33:39.007 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:33:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:33:38.786+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:31:46.482189+0000 (oldest deadline 2024-02-20T09:32:06.482189+0000) 2024-02-20T09:33:39.007 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:33:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:33:38.786+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:31:46.482189+0000 (oldest deadline 2024-02-20T09:32:06.482189+0000) 2024-02-20T09:33:39.007 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:33:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:33:38.786+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:31:46.482189+0000 (oldest deadline 2024-02-20T09:32:06.482189+0000) 2024-02-20T09:33:39.008 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:33:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:33:38.932+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:32:16.988591+0000 (oldest deadline 2024-02-20T09:32:36.988591+0000) 2024-02-20T09:33:39.008 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:33:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:33:38.932+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:32:16.988591+0000 (oldest deadline 2024-02-20T09:32:36.988591+0000) 2024-02-20T09:33:39.008 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:33:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:33:38.932+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:32:16.988591+0000 (oldest deadline 2024-02-20T09:32:36.988591+0000) 2024-02-20T09:33:39.008 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:33:38 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:33:38.932+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:33:39.411 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:33:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:33:39.017+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:39.017892+0000) 2024-02-20T09:33:39.411 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:33:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:33:39.132+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:39.132976+0000) 2024-02-20T09:33:39.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:33:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:33:39.517+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:39.517673+0000) 2024-02-20T09:33:39.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:33:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:33:39.499+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:31:58.900551+0000 (oldest deadline 2024-02-20T09:32:18.900551+0000) 2024-02-20T09:33:39.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:33:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:33:39.499+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:31:58.900551+0000 (oldest deadline 2024-02-20T09:32:18.900551+0000) 2024-02-20T09:33:39.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:33:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:33:39.499+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:31:58.900551+0000 (oldest deadline 2024-02-20T09:32:18.900551+0000) 2024-02-20T09:33:40.007 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:33:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:33:39.791+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:31:46.482189+0000 (oldest deadline 2024-02-20T09:32:06.482189+0000) 2024-02-20T09:33:40.007 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:33:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:33:39.791+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:31:46.482189+0000 (oldest deadline 2024-02-20T09:32:06.482189+0000) 2024-02-20T09:33:40.007 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:33:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:33:39.791+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:31:46.482189+0000 (oldest deadline 2024-02-20T09:32:06.482189+0000) 2024-02-20T09:33:40.008 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:33:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:33:39.957+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:32:16.988591+0000 (oldest deadline 2024-02-20T09:32:36.988591+0000) 2024-02-20T09:33:40.008 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:33:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:33:39.957+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:32:16.988591+0000 (oldest deadline 2024-02-20T09:32:36.988591+0000) 2024-02-20T09:33:40.008 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:33:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:33:39.957+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:32:16.988591+0000 (oldest deadline 2024-02-20T09:32:36.988591+0000) 2024-02-20T09:33:40.008 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:33:39 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:33:39.957+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:33:40.411 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:33:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:33:40.017+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:40.017986+0000) 2024-02-20T09:33:40.412 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:33:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:33:40.132+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:40.133100+0000) 2024-02-20T09:33:40.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:33:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:33:40.520+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:40.520465+0000) 2024-02-20T09:33:40.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:33:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:33:40.461+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:31:58.900551+0000 (oldest deadline 2024-02-20T09:32:18.900551+0000) 2024-02-20T09:33:40.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:33:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:33:40.461+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:31:58.900551+0000 (oldest deadline 2024-02-20T09:32:18.900551+0000) 2024-02-20T09:33:40.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:33:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:33:40.461+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:31:58.900551+0000 (oldest deadline 2024-02-20T09:32:18.900551+0000) 2024-02-20T09:33:41.007 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:33:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:33:40.770+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:31:46.482189+0000 (oldest deadline 2024-02-20T09:32:06.482189+0000) 2024-02-20T09:33:41.007 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:33:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:33:40.770+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:31:46.482189+0000 (oldest deadline 2024-02-20T09:32:06.482189+0000) 2024-02-20T09:33:41.008 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:33:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:33:40.770+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:31:46.482189+0000 (oldest deadline 2024-02-20T09:32:06.482189+0000) 2024-02-20T09:33:41.008 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:33:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:33:40.933+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:32:16.988591+0000 (oldest deadline 2024-02-20T09:32:36.988591+0000) 2024-02-20T09:33:41.008 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:33:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:33:40.933+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:32:16.988591+0000 (oldest deadline 2024-02-20T09:32:36.988591+0000) 2024-02-20T09:33:41.008 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:33:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:33:40.933+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:32:16.988591+0000 (oldest deadline 2024-02-20T09:32:36.988591+0000) 2024-02-20T09:33:41.008 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:33:40 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:33:40.933+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:33:41.411 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:33:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:33:41.132+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:41.133187+0000) 2024-02-20T09:33:41.412 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:33:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:33:41.017+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:41.018098+0000) 2024-02-20T09:33:41.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:33:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:33:41.522+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:41.523087+0000) 2024-02-20T09:33:41.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:33:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:33:41.436+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:31:58.900551+0000 (oldest deadline 2024-02-20T09:32:18.900551+0000) 2024-02-20T09:33:41.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:33:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:33:41.436+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:31:58.900551+0000 (oldest deadline 2024-02-20T09:32:18.900551+0000) 2024-02-20T09:33:41.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:33:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:33:41.436+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:31:58.900551+0000 (oldest deadline 2024-02-20T09:32:18.900551+0000) 2024-02-20T09:33:42.007 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:33:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:33:41.806+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:31:46.482189+0000 (oldest deadline 2024-02-20T09:32:06.482189+0000) 2024-02-20T09:33:42.008 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:33:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:33:41.806+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:31:46.482189+0000 (oldest deadline 2024-02-20T09:32:06.482189+0000) 2024-02-20T09:33:42.008 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:33:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:33:41.806+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:31:46.482189+0000 (oldest deadline 2024-02-20T09:32:06.482189+0000) 2024-02-20T09:33:42.008 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:33:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:33:41.940+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:32:16.988591+0000 (oldest deadline 2024-02-20T09:32:36.988591+0000) 2024-02-20T09:33:42.008 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:33:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:33:41.940+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:32:16.988591+0000 (oldest deadline 2024-02-20T09:32:36.988591+0000) 2024-02-20T09:33:42.008 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:33:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:33:41.940+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:32:16.988591+0000 (oldest deadline 2024-02-20T09:32:36.988591+0000) 2024-02-20T09:33:42.008 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:33:41 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:33:41.940+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:33:42.333 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi183.front.sepia.ceph.com' 2024-02-20T09:33:42.334 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi183.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-20T09:33:42.411 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:33:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:33:42.017+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:42.018215+0000) 2024-02-20T09:33:42.412 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:33:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:33:42.132+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:42.133283+0000) 2024-02-20T09:33:42.733 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:33:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:33:42.524+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:42.525178+0000) 2024-02-20T09:33:42.733 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:33:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:33:42.474+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:31:58.900551+0000 (oldest deadline 2024-02-20T09:32:18.900551+0000) 2024-02-20T09:33:42.733 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:33:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:33:42.474+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:31:58.900551+0000 (oldest deadline 2024-02-20T09:32:18.900551+0000) 2024-02-20T09:33:42.734 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:33:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:33:42.474+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:31:58.900551+0000 (oldest deadline 2024-02-20T09:32:18.900551+0000) 2024-02-20T09:33:42.990 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:33:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:33:42.738+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 73088 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:33:42.990 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:33:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:33:42.732+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:42.732740+0000) 2024-02-20T09:33:42.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:33:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:33:42.735+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:42.735604+0000) 2024-02-20T09:33:42.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:33:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:33:42.988+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:32:16.988591+0000 (oldest deadline 2024-02-20T09:32:36.988591+0000) 2024-02-20T09:33:42.991 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:33:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:33:42.988+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:32:16.988591+0000 (oldest deadline 2024-02-20T09:32:36.988591+0000) 2024-02-20T09:33:42.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:33:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:33:42.732+0000 7f8a5f86e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:42.733156+0000) 2024-02-20T09:33:42.991 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:33:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:33:42.789+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:31:46.482189+0000 (oldest deadline 2024-02-20T09:32:06.482189+0000) 2024-02-20T09:33:42.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:33:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:33:42.789+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:31:46.482189+0000 (oldest deadline 2024-02-20T09:32:06.482189+0000) 2024-02-20T09:33:42.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:33:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:33:42.789+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:31:46.482189+0000 (oldest deadline 2024-02-20T09:32:06.482189+0000) 2024-02-20T09:33:43.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:33:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:33:43.017+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:43.018334+0000) 2024-02-20T09:33:43.243 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:33:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:33:43.133+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:43.133439+0000) 2024-02-20T09:33:43.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:33:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:33:42.988+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:32:16.988591+0000 (oldest deadline 2024-02-20T09:32:36.988591+0000) 2024-02-20T09:33:43.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:33:42 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:33:42.988+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:33:43.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:33:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:33:43.526+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:43.527036+0000) 2024-02-20T09:33:43.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:33:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:33:43.438+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:31:58.900551+0000 (oldest deadline 2024-02-20T09:32:18.900551+0000) 2024-02-20T09:33:43.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:33:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:33:43.438+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:31:58.900551+0000 (oldest deadline 2024-02-20T09:32:18.900551+0000) 2024-02-20T09:33:43.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:33:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:33:43.438+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:31:58.900551+0000 (oldest deadline 2024-02-20T09:32:18.900551+0000) 2024-02-20T09:33:44.007 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:33:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:33:43.819+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:31:46.482189+0000 (oldest deadline 2024-02-20T09:32:06.482189+0000) 2024-02-20T09:33:44.007 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:33:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:33:43.819+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:31:46.482189+0000 (oldest deadline 2024-02-20T09:32:06.482189+0000) 2024-02-20T09:33:44.008 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:33:43 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:33:43.819+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:31:46.482189+0000 (oldest deadline 2024-02-20T09:32:06.482189+0000) 2024-02-20T09:33:44.411 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:33:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:33:44.018+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:44.018440+0000) 2024-02-20T09:33:44.412 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:33:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:33:44.132+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:44.133542+0000) 2024-02-20T09:33:44.412 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:33:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:33:44.021+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:32:16.988591+0000 (oldest deadline 2024-02-20T09:32:36.988591+0000) 2024-02-20T09:33:44.412 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:33:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:33:44.021+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:32:16.988591+0000 (oldest deadline 2024-02-20T09:32:36.988591+0000) 2024-02-20T09:33:44.413 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:33:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:33:44.021+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:32:16.988591+0000 (oldest deadline 2024-02-20T09:32:36.988591+0000) 2024-02-20T09:33:44.413 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:33:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:33:44.021+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:33:44.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:33:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:33:44.528+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:44.529444+0000) 2024-02-20T09:33:44.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:33:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:33:44.472+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:31:58.900551+0000 (oldest deadline 2024-02-20T09:32:18.900551+0000) 2024-02-20T09:33:44.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:33:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:33:44.472+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:31:58.900551+0000 (oldest deadline 2024-02-20T09:32:18.900551+0000) 2024-02-20T09:33:44.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:33:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:33:44.472+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:31:58.900551+0000 (oldest deadline 2024-02-20T09:32:18.900551+0000) 2024-02-20T09:33:45.007 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:33:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:33:44.830+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:31:46.482189+0000 (oldest deadline 2024-02-20T09:32:06.482189+0000) 2024-02-20T09:33:45.008 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:33:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:33:44.830+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:31:46.482189+0000 (oldest deadline 2024-02-20T09:32:06.482189+0000) 2024-02-20T09:33:45.008 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:33:44 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:33:44.830+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:31:46.482189+0000 (oldest deadline 2024-02-20T09:32:06.482189+0000) 2024-02-20T09:33:45.404 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.183 2024-02-20T09:33:45.412 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:33:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:33:45.017+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:45.018531+0000) 2024-02-20T09:33:45.412 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:33:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:33:45.132+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:45.133636+0000) 2024-02-20T09:33:45.413 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:33:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:33:45.011+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:32:16.988591+0000 (oldest deadline 2024-02-20T09:32:36.988591+0000) 2024-02-20T09:33:45.413 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:33:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:33:45.011+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:32:16.988591+0000 (oldest deadline 2024-02-20T09:32:36.988591+0000) 2024-02-20T09:33:45.413 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:33:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:33:45.011+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:32:16.988591+0000 (oldest deadline 2024-02-20T09:32:36.988591+0000) 2024-02-20T09:33:45.413 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:33:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:33:45.011+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:33:45.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:33:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:33:45.530+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:45.531443+0000) 2024-02-20T09:33:45.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:33:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:33:45.522+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:31:58.900551+0000 (oldest deadline 2024-02-20T09:32:18.900551+0000) 2024-02-20T09:33:45.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:33:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:33:45.522+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:31:58.900551+0000 (oldest deadline 2024-02-20T09:32:18.900551+0000) 2024-02-20T09:33:45.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:33:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:33:45.522+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:31:58.900551+0000 (oldest deadline 2024-02-20T09:32:18.900551+0000) 2024-02-20T09:33:46.008 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:33:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:33:45.789+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:31:46.482189+0000 (oldest deadline 2024-02-20T09:32:06.482189+0000) 2024-02-20T09:33:46.008 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:33:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:33:45.789+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:31:46.482189+0000 (oldest deadline 2024-02-20T09:32:06.482189+0000) 2024-02-20T09:33:46.008 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:33:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:33:45.789+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:31:46.482189+0000 (oldest deadline 2024-02-20T09:32:06.482189+0000) 2024-02-20T09:33:46.008 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:33:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:33:45.970+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:32:16.988591+0000 (oldest deadline 2024-02-20T09:32:36.988591+0000) 2024-02-20T09:33:46.008 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:33:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:33:45.970+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:32:16.988591+0000 (oldest deadline 2024-02-20T09:32:36.988591+0000) 2024-02-20T09:33:46.009 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:33:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:33:45.970+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:32:16.988591+0000 (oldest deadline 2024-02-20T09:32:36.988591+0000) 2024-02-20T09:33:46.009 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:33:45 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:33:45.970+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:33:46.412 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:33:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:33:46.017+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:46.018618+0000) 2024-02-20T09:33:46.413 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:33:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:33:46.132+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:46.133724+0000) 2024-02-20T09:33:46.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:33:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:33:46.532+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:46.533510+0000) 2024-02-20T09:33:46.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:33:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:33:46.553+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:31:58.900551+0000 (oldest deadline 2024-02-20T09:32:18.900551+0000) 2024-02-20T09:33:46.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:33:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:33:46.553+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:31:58.900551+0000 (oldest deadline 2024-02-20T09:32:18.900551+0000) 2024-02-20T09:33:46.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:33:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:33:46.553+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:31:58.900551+0000 (oldest deadline 2024-02-20T09:32:18.900551+0000) 2024-02-20T09:33:47.008 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:33:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:33:46.838+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:31:46.482189+0000 (oldest deadline 2024-02-20T09:32:06.482189+0000) 2024-02-20T09:33:47.008 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:33:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:33:46.839+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:31:46.482189+0000 (oldest deadline 2024-02-20T09:32:06.482189+0000) 2024-02-20T09:33:47.008 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:33:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:33:46.839+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:31:46.482189+0000 (oldest deadline 2024-02-20T09:32:06.482189+0000) 2024-02-20T09:33:47.008 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:33:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:33:46.956+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:32:16.988591+0000 (oldest deadline 2024-02-20T09:32:36.988591+0000) 2024-02-20T09:33:47.009 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:33:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:33:46.956+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:32:16.988591+0000 (oldest deadline 2024-02-20T09:32:36.988591+0000) 2024-02-20T09:33:47.009 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:33:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:33:46.956+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:32:16.988591+0000 (oldest deadline 2024-02-20T09:32:36.988591+0000) 2024-02-20T09:33:47.009 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:33:46 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:33:46.956+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:33:47.412 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:33:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:33:47.132+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:47.133815+0000) 2024-02-20T09:33:47.412 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:33:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:33:47.017+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:47.018712+0000) 2024-02-20T09:33:47.741 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:33:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:33:47.534+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:47.535640+0000) 2024-02-20T09:33:47.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:33:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:33:47.566+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:31:58.900551+0000 (oldest deadline 2024-02-20T09:32:18.900551+0000) 2024-02-20T09:33:47.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:33:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:33:47.566+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:31:58.900551+0000 (oldest deadline 2024-02-20T09:32:18.900551+0000) 2024-02-20T09:33:47.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:33:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:33:47.566+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:31:58.900551+0000 (oldest deadline 2024-02-20T09:32:18.900551+0000) 2024-02-20T09:33:47.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:33:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:33:47.739+0000 7f8e5da8d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:47.740429+0000) 2024-02-20T09:33:47.992 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:33:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:33:47.746+0000 7fa8301f8700 -1 mon.smithi133@0(probing) e2 get_health_metrics reporting 73128 slow ops, oldest is log(1 entries from seq 273 at 2024-02-20T07:56:45.595378+0000) 2024-02-20T09:33:47.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:33:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:33:47.741+0000 7f010465f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:47.742749+0000) 2024-02-20T09:33:47.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:33:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:33:47.961+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:32:16.988591+0000 (oldest deadline 2024-02-20T09:32:36.988591+0000) 2024-02-20T09:33:47.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:33:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:33:47.962+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:32:16.988591+0000 (oldest deadline 2024-02-20T09:32:36.988591+0000) 2024-02-20T09:33:47.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:33:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:33:47.962+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:32:16.988591+0000 (oldest deadline 2024-02-20T09:32:36.988591+0000) 2024-02-20T09:33:47.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:33:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:33:47.962+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:33:47.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:33:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:33:47.739+0000 7f8a5f86e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:47.740887+0000) 2024-02-20T09:33:47.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:33:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:33:47.867+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:31:46.482189+0000 (oldest deadline 2024-02-20T09:32:06.482189+0000) 2024-02-20T09:33:47.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:33:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:33:47.867+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:31:46.482189+0000 (oldest deadline 2024-02-20T09:32:06.482189+0000) 2024-02-20T09:33:47.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:33:47 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:33:47.867+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:31:46.482189+0000 (oldest deadline 2024-02-20T09:32:06.482189+0000) 2024-02-20T09:33:48.405 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi183.front.sepia.ceph.com' 2024-02-20T09:33:48.406 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi183.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-20T09:33:48.412 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:33:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:33:48.017+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:48.018821+0000) 2024-02-20T09:33:48.412 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:33:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:33:48.132+0000 7f8e50ead700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:48.133901+0000) 2024-02-20T09:33:48.480 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.183 2024-02-20T09:33:48.480 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-02-20T09:33:48.481 INFO:tasks.cephadm:Removing cluster... 2024-02-20T09:33:48.481 DEBUG:teuthology.orchestra.run.smithi133:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 4537327c-cfc4-11ee-95bb-87774f69a715 --force 2024-02-20T09:33:48.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:33:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:33:48.535+0000 7f00f627c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:48.537330+0000) 2024-02-20T09:33:48.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:33:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:33:48.603+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:31:58.900551+0000 (oldest deadline 2024-02-20T09:32:18.900551+0000) 2024-02-20T09:33:48.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:33:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:33:48.604+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:31:58.900551+0000 (oldest deadline 2024-02-20T09:32:18.900551+0000) 2024-02-20T09:33:48.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:33:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:33:48.604+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:31:58.900551+0000 (oldest deadline 2024-02-20T09:32:18.900551+0000) 2024-02-20T09:33:49.008 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:33:48 smithi133 systemd[1]: Stopping Ceph mon.smithi133 for 4537327c-cfc4-11ee-95bb-87774f69a715... 2024-02-20T09:33:49.009 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:33:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:33:48.819+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:31:46.482189+0000 (oldest deadline 2024-02-20T09:32:06.482189+0000) 2024-02-20T09:33:49.009 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:33:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:33:48.819+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:31:46.482189+0000 (oldest deadline 2024-02-20T09:32:06.482189+0000) 2024-02-20T09:33:49.009 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:33:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:33:48.819+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:31:46.482189+0000 (oldest deadline 2024-02-20T09:32:06.482189+0000) 2024-02-20T09:33:49.009 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:33:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:33:48.966+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:32:16.988591+0000 (oldest deadline 2024-02-20T09:32:36.988591+0000) 2024-02-20T09:33:49.009 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:33:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:33:48.966+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:32:16.988591+0000 (oldest deadline 2024-02-20T09:32:36.988591+0000) 2024-02-20T09:33:49.009 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:33:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:33:48.966+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:32:16.988591+0000 (oldest deadline 2024-02-20T09:32:36.988591+0000) 2024-02-20T09:33:49.009 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:33:48 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:33:48.966+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:33:49.492 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:33:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:33:49.017+0000 7f8a4fc88700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-20T08:33:49.018860+0000) 2024-02-20T09:33:49.493 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:33:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:33:49.039+0000 7fa8331fe700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi133 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-02-20T09:33:49.493 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:33:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133[29113]: 2024-02-20T09:33:49.039+0000 7fa8331fe700 -1 mon.smithi133@0(probing) e2 *** Got Signal Terminated *** 2024-02-20T09:33:49.849 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:33:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:33:49.561+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:31:58.900551+0000 (oldest deadline 2024-02-20T09:32:18.900551+0000) 2024-02-20T09:33:49.849 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:33:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:33:49.561+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:31:58.900551+0000 (oldest deadline 2024-02-20T09:32:18.900551+0000) 2024-02-20T09:33:49.849 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:33:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:33:49.561+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:31:58.900551+0000 (oldest deadline 2024-02-20T09:32:18.900551+0000) 2024-02-20T09:33:50.227 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:33:49 smithi133 podman[93485]: 2024-02-20 09:33:49.936011573 +0000 UTC m=+1.026725810 container died fe7046a20c8704d4c5e4b4c19b43a3423bc6705e3d978aa30e3796944d81371e (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, RELEASE=quincy-c80e716, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240214, org.label-schema.name=CentOS Stream 8 Base Image) 2024-02-20T09:33:50.227 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:33:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:33:49.846+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:31:46.482189+0000 (oldest deadline 2024-02-20T09:32:06.482189+0000) 2024-02-20T09:33:50.227 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:33:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:33:49.846+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:31:46.482189+0000 (oldest deadline 2024-02-20T09:32:06.482189+0000) 2024-02-20T09:33:50.228 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:33:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:33:49.846+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:31:46.482189+0000 (oldest deadline 2024-02-20T09:32:06.482189+0000) 2024-02-20T09:33:50.228 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:33:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:33:49.996+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:32:16.988591+0000 (oldest deadline 2024-02-20T09:32:36.988591+0000) 2024-02-20T09:33:50.228 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:33:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:33:49.996+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:32:16.988591+0000 (oldest deadline 2024-02-20T09:32:36.988591+0000) 2024-02-20T09:33:50.228 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:33:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:33:49.996+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:32:16.988591+0000 (oldest deadline 2024-02-20T09:32:36.988591+0000) 2024-02-20T09:33:50.228 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:33:49 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:33:49.996+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:33:50.492 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:33:50 smithi133 podman[93485]: 2024-02-20 09:33:50.408288169 +0000 UTC m=+1.499002414 container cleanup fe7046a20c8704d4c5e4b4c19b43a3423bc6705e3d978aa30e3796944d81371e (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, RELEASE=quincy-c80e716, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240214, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True) 2024-02-20T09:33:50.492 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:33:50 smithi133 bash[93485]: ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133 2024-02-20T09:33:50.809 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:33:50 smithi133 podman[93499]: 2024-02-20 09:33:50.676460565 +0000 UTC m=+0.745086583 container remove fe7046a20c8704d4c5e4b4c19b43a3423bc6705e3d978aa30e3796944d81371e (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-4537327c-cfc4-11ee-95bb-87774f69a715-mon-smithi133, GIT_CLEAN=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, CEPH_POINT_RELEASE=, RELEASE=quincy-c80e716, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240214, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, ceph=True) 2024-02-20T09:33:50.809 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:33:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:33:50.611+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:31:58.900551+0000 (oldest deadline 2024-02-20T09:32:18.900551+0000) 2024-02-20T09:33:50.810 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:33:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:33:50.611+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:31:58.900551+0000 (oldest deadline 2024-02-20T09:32:18.900551+0000) 2024-02-20T09:33:50.810 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:33:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:33:50.611+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:31:58.900551+0000 (oldest deadline 2024-02-20T09:32:18.900551+0000) 2024-02-20T09:33:51.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:33:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:33:50.861+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:31:46.482189+0000 (oldest deadline 2024-02-20T09:32:06.482189+0000) 2024-02-20T09:33:51.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:33:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:33:50.861+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:31:46.482189+0000 (oldest deadline 2024-02-20T09:32:06.482189+0000) 2024-02-20T09:33:51.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:33:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:33:50.861+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:31:46.482189+0000 (oldest deadline 2024-02-20T09:32:06.482189+0000) 2024-02-20T09:33:51.243 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:33:50 smithi133 systemd[1]: ceph-4537327c-cfc4-11ee-95bb-87774f69a715@mon.smithi133.service: Deactivated successfully. 2024-02-20T09:33:51.243 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:33:50 smithi133 systemd[1]: Stopped Ceph mon.smithi133 for 4537327c-cfc4-11ee-95bb-87774f69a715. 2024-02-20T09:33:51.243 INFO:journalctl@ceph.mon.smithi133.smithi133.stdout:Feb 20 09:33:50 smithi133 systemd[1]: ceph-4537327c-cfc4-11ee-95bb-87774f69a715@mon.smithi133.service: Consumed 25min 29.331s CPU time. 2024-02-20T09:33:51.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:33:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:33:50.960+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:32:16.988591+0000 (oldest deadline 2024-02-20T09:32:36.988591+0000) 2024-02-20T09:33:51.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:33:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:33:50.960+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:32:16.988591+0000 (oldest deadline 2024-02-20T09:32:36.988591+0000) 2024-02-20T09:33:51.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:33:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:33:50.960+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:32:16.988591+0000 (oldest deadline 2024-02-20T09:32:36.988591+0000) 2024-02-20T09:33:51.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:33:50 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:33:50.960+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:33:51.902 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:33:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:33:51.594+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:31:58.900551+0000 (oldest deadline 2024-02-20T09:32:18.900551+0000) 2024-02-20T09:33:51.902 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:33:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:33:51.594+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:31:58.900551+0000 (oldest deadline 2024-02-20T09:32:18.900551+0000) 2024-02-20T09:33:51.902 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:33:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:33:51.594+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:31:58.900551+0000 (oldest deadline 2024-02-20T09:32:18.900551+0000) 2024-02-20T09:33:52.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:33:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:33:51.900+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:31:46.482189+0000 (oldest deadline 2024-02-20T09:32:06.482189+0000) 2024-02-20T09:33:52.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:33:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:33:51.900+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:31:46.482189+0000 (oldest deadline 2024-02-20T09:32:06.482189+0000) 2024-02-20T09:33:52.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:33:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:33:51.900+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:31:46.482189+0000 (oldest deadline 2024-02-20T09:32:06.482189+0000) 2024-02-20T09:33:52.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:33:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:33:51.955+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:32:16.988591+0000 (oldest deadline 2024-02-20T09:32:36.988591+0000) 2024-02-20T09:33:52.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:33:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:33:51.955+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:32:16.988591+0000 (oldest deadline 2024-02-20T09:32:36.988591+0000) 2024-02-20T09:33:52.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:33:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:33:51.955+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:32:16.988591+0000 (oldest deadline 2024-02-20T09:32:36.988591+0000) 2024-02-20T09:33:52.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:33:51 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:33:51.956+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:33:52.865 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:33:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:33:52.599+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:31:58.900551+0000 (oldest deadline 2024-02-20T09:32:18.900551+0000) 2024-02-20T09:33:52.865 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:33:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:33:52.599+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:31:58.900551+0000 (oldest deadline 2024-02-20T09:32:18.900551+0000) 2024-02-20T09:33:52.865 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:33:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:33:52.599+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:31:58.900551+0000 (oldest deadline 2024-02-20T09:32:18.900551+0000) 2024-02-20T09:33:53.187 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:33:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:33:52.863+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:31:46.482189+0000 (oldest deadline 2024-02-20T09:32:06.482189+0000) 2024-02-20T09:33:53.187 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:33:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:33:52.863+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:31:46.482189+0000 (oldest deadline 2024-02-20T09:32:06.482189+0000) 2024-02-20T09:33:53.187 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:33:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:33:52.863+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:31:46.482189+0000 (oldest deadline 2024-02-20T09:32:06.482189+0000) 2024-02-20T09:33:53.187 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:33:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:33:52.984+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:32:16.988591+0000 (oldest deadline 2024-02-20T09:32:36.988591+0000) 2024-02-20T09:33:53.188 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:33:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:33:52.984+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:32:16.988591+0000 (oldest deadline 2024-02-20T09:32:36.988591+0000) 2024-02-20T09:33:53.188 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:33:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:33:52.984+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:32:16.988591+0000 (oldest deadline 2024-02-20T09:32:36.988591+0000) 2024-02-20T09:33:53.188 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:33:52 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:33:52.984+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:33:53.767 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:33:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:33:53.564+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:31:58.900551+0000 (oldest deadline 2024-02-20T09:32:18.900551+0000) 2024-02-20T09:33:53.767 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:33:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:33:53.564+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:31:58.900551+0000 (oldest deadline 2024-02-20T09:32:18.900551+0000) 2024-02-20T09:33:53.767 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:33:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:33:53.564+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:31:58.900551+0000 (oldest deadline 2024-02-20T09:32:18.900551+0000) 2024-02-20T09:33:54.057 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:33:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:33:53.827+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:31:46.482189+0000 (oldest deadline 2024-02-20T09:32:06.482189+0000) 2024-02-20T09:33:54.057 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:33:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:33:53.827+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:31:46.482189+0000 (oldest deadline 2024-02-20T09:32:06.482189+0000) 2024-02-20T09:33:54.057 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:33:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:33:53.827+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:31:46.482189+0000 (oldest deadline 2024-02-20T09:32:06.482189+0000) 2024-02-20T09:33:54.058 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:33:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:33:53.998+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:32:16.988591+0000 (oldest deadline 2024-02-20T09:32:36.988591+0000) 2024-02-20T09:33:54.058 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:33:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:33:53.998+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:32:16.988591+0000 (oldest deadline 2024-02-20T09:32:36.988591+0000) 2024-02-20T09:33:54.058 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:33:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:33:53.998+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:32:16.988591+0000 (oldest deadline 2024-02-20T09:32:36.988591+0000) 2024-02-20T09:33:54.058 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:33:53 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:33:53.998+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:33:54.721 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:33:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:33:54.589+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:31:58.900551+0000 (oldest deadline 2024-02-20T09:32:18.900551+0000) 2024-02-20T09:33:54.722 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:33:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:33:54.589+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:31:58.900551+0000 (oldest deadline 2024-02-20T09:32:18.900551+0000) 2024-02-20T09:33:54.722 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:33:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:33:54.589+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:31:58.900551+0000 (oldest deadline 2024-02-20T09:32:18.900551+0000) 2024-02-20T09:33:54.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:33:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:33:54.784+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:31:46.482189+0000 (oldest deadline 2024-02-20T09:32:06.482189+0000) 2024-02-20T09:33:54.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:33:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:33:54.784+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:31:46.482189+0000 (oldest deadline 2024-02-20T09:32:06.482189+0000) 2024-02-20T09:33:54.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:33:54 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:33:54.784+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:31:46.482189+0000 (oldest deadline 2024-02-20T09:32:06.482189+0000) 2024-02-20T09:33:55.336 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:33:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:33:55.012+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:32:16.988591+0000 (oldest deadline 2024-02-20T09:32:36.988591+0000) 2024-02-20T09:33:55.336 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:33:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:33:55.012+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:32:16.988591+0000 (oldest deadline 2024-02-20T09:32:36.988591+0000) 2024-02-20T09:33:55.336 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:33:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:33:55.012+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:32:16.988591+0000 (oldest deadline 2024-02-20T09:32:36.988591+0000) 2024-02-20T09:33:55.336 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:33:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:33:55.012+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:33:55.888 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:33:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:33:55.764+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:31:46.482189+0000 (oldest deadline 2024-02-20T09:32:06.482189+0000) 2024-02-20T09:33:55.888 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:33:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:33:55.764+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:31:46.482189+0000 (oldest deadline 2024-02-20T09:32:06.482189+0000) 2024-02-20T09:33:55.888 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:33:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:33:55.764+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:31:46.482189+0000 (oldest deadline 2024-02-20T09:32:06.482189+0000) 2024-02-20T09:33:55.889 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:33:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:33:55.625+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:31:58.900551+0000 (oldest deadline 2024-02-20T09:32:18.900551+0000) 2024-02-20T09:33:55.889 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:33:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:33:55.625+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:31:58.900551+0000 (oldest deadline 2024-02-20T09:32:18.900551+0000) 2024-02-20T09:33:55.889 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:33:55 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:33:55.625+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:31:58.900551+0000 (oldest deadline 2024-02-20T09:32:18.900551+0000) 2024-02-20T09:33:56.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:33:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:33:56.040+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:32:16.988591+0000 (oldest deadline 2024-02-20T09:32:36.988591+0000) 2024-02-20T09:33:56.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:33:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:33:56.040+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:32:16.988591+0000 (oldest deadline 2024-02-20T09:32:36.988591+0000) 2024-02-20T09:33:56.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:33:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:33:56.040+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:32:16.988591+0000 (oldest deadline 2024-02-20T09:32:36.988591+0000) 2024-02-20T09:33:56.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:33:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:33:56.040+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:33:56.992 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:33:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:33:56.814+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:31:46.482189+0000 (oldest deadline 2024-02-20T09:32:06.482189+0000) 2024-02-20T09:33:56.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:33:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:33:56.814+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:31:46.482189+0000 (oldest deadline 2024-02-20T09:32:06.482189+0000) 2024-02-20T09:33:56.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:33:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:33:56.814+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:31:46.482189+0000 (oldest deadline 2024-02-20T09:32:06.482189+0000) 2024-02-20T09:33:56.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:33:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:33:56.623+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:31:58.900551+0000 (oldest deadline 2024-02-20T09:32:18.900551+0000) 2024-02-20T09:33:56.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:33:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:33:56.623+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:31:58.900551+0000 (oldest deadline 2024-02-20T09:32:18.900551+0000) 2024-02-20T09:33:56.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:33:56 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:33:56.623+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:31:58.900551+0000 (oldest deadline 2024-02-20T09:32:18.900551+0000) 2024-02-20T09:33:57.361 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:33:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:33:57.014+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:32:16.988591+0000 (oldest deadline 2024-02-20T09:32:36.988591+0000) 2024-02-20T09:33:57.362 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:33:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:33:57.014+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:32:16.988591+0000 (oldest deadline 2024-02-20T09:32:36.988591+0000) 2024-02-20T09:33:57.362 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:33:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:33:57.014+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:32:16.988591+0000 (oldest deadline 2024-02-20T09:32:36.988591+0000) 2024-02-20T09:33:57.362 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:33:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:33:57.014+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:33:57.936 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:33:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:33:57.765+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:31:46.482189+0000 (oldest deadline 2024-02-20T09:32:06.482189+0000) 2024-02-20T09:33:57.937 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:33:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:33:57.765+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:31:46.482189+0000 (oldest deadline 2024-02-20T09:32:06.482189+0000) 2024-02-20T09:33:57.937 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:33:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:33:57.765+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:31:46.482189+0000 (oldest deadline 2024-02-20T09:32:06.482189+0000) 2024-02-20T09:33:57.937 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:33:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:33:57.655+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:31:58.900551+0000 (oldest deadline 2024-02-20T09:32:18.900551+0000) 2024-02-20T09:33:57.937 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:33:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:33:57.655+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:31:58.900551+0000 (oldest deadline 2024-02-20T09:32:18.900551+0000) 2024-02-20T09:33:57.937 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:33:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:33:57.655+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:31:58.900551+0000 (oldest deadline 2024-02-20T09:32:18.900551+0000) 2024-02-20T09:33:58.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:33:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:33:57.970+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:32:16.988591+0000 (oldest deadline 2024-02-20T09:32:36.988591+0000) 2024-02-20T09:33:58.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:33:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:33:57.970+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:32:16.988591+0000 (oldest deadline 2024-02-20T09:32:36.988591+0000) 2024-02-20T09:33:58.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:33:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:33:57.970+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:32:16.988591+0000 (oldest deadline 2024-02-20T09:32:36.988591+0000) 2024-02-20T09:33:58.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:33:57 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:33:57.970+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:33:58.641 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:33:58 smithi133 systemd[1]: Stopping Ceph osd.0 for 4537327c-cfc4-11ee-95bb-87774f69a715... 2024-02-20T09:33:58.992 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:33:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:33:58.638+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:31:58.900551+0000 (oldest deadline 2024-02-20T09:32:18.900551+0000) 2024-02-20T09:33:58.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:33:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:33:58.638+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:31:58.900551+0000 (oldest deadline 2024-02-20T09:32:18.900551+0000) 2024-02-20T09:33:58.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:33:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:33:58.638+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:31:58.900551+0000 (oldest deadline 2024-02-20T09:32:18.900551+0000) 2024-02-20T09:33:58.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:33:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:33:58.697+0000 7f8a5e06b700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-02-20T09:33:58.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:33:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:33:58.697+0000 7f8a5e06b700 -1 osd.0 46 *** Got signal Terminated *** 2024-02-20T09:33:58.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:33:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:33:58.697+0000 7f8a5e06b700 -1 osd.0 46 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-20T09:33:58.993 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:33:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:33:58.790+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:31:46.482189+0000 (oldest deadline 2024-02-20T09:32:06.482189+0000) 2024-02-20T09:33:58.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:33:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:33:58.790+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:31:46.482189+0000 (oldest deadline 2024-02-20T09:32:06.482189+0000) 2024-02-20T09:33:58.994 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:33:58 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:33:58.790+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:31:46.482189+0000 (oldest deadline 2024-02-20T09:32:06.482189+0000) 2024-02-20T09:33:59.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:33:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:33:59.008+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:32:16.988591+0000 (oldest deadline 2024-02-20T09:32:36.988591+0000) 2024-02-20T09:33:59.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:33:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:33:59.008+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:32:16.988591+0000 (oldest deadline 2024-02-20T09:32:36.988591+0000) 2024-02-20T09:33:59.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:33:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:33:59.008+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:32:16.988591+0000 (oldest deadline 2024-02-20T09:32:36.988591+0000) 2024-02-20T09:33:59.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:33:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:33:59.008+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:33:59.970 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:33:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:33:59.770+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:31:46.482189+0000 (oldest deadline 2024-02-20T09:32:06.482189+0000) 2024-02-20T09:33:59.970 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:33:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:33:59.770+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:31:46.482189+0000 (oldest deadline 2024-02-20T09:32:06.482189+0000) 2024-02-20T09:33:59.970 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:33:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:33:59.770+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:31:46.482189+0000 (oldest deadline 2024-02-20T09:32:06.482189+0000) 2024-02-20T09:33:59.971 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:33:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:33:59.657+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:31:58.900551+0000 (oldest deadline 2024-02-20T09:32:18.900551+0000) 2024-02-20T09:33:59.971 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:33:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:33:59.657+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:31:58.900551+0000 (oldest deadline 2024-02-20T09:32:18.900551+0000) 2024-02-20T09:33:59.971 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:33:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:33:59.657+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:31:58.900551+0000 (oldest deadline 2024-02-20T09:32:18.900551+0000) 2024-02-20T09:34:00.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:33:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:33:59.968+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:32:16.988591+0000 (oldest deadline 2024-02-20T09:32:36.988591+0000) 2024-02-20T09:34:00.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:33:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:33:59.968+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:32:16.988591+0000 (oldest deadline 2024-02-20T09:32:36.988591+0000) 2024-02-20T09:34:00.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:33:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:33:59.968+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:32:16.988591+0000 (oldest deadline 2024-02-20T09:32:36.988591+0000) 2024-02-20T09:34:00.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:33:59 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:33:59.968+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:34:00.988 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:34:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:34:00.768+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:31:46.482189+0000 (oldest deadline 2024-02-20T09:32:06.482189+0000) 2024-02-20T09:34:00.988 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:34:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:34:00.768+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:31:46.482189+0000 (oldest deadline 2024-02-20T09:32:06.482189+0000) 2024-02-20T09:34:00.989 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:34:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:34:00.768+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:31:46.482189+0000 (oldest deadline 2024-02-20T09:32:06.482189+0000) 2024-02-20T09:34:00.989 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:34:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:34:00.617+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:31:58.900551+0000 (oldest deadline 2024-02-20T09:32:18.900551+0000) 2024-02-20T09:34:00.989 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:34:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:34:00.617+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:31:58.900551+0000 (oldest deadline 2024-02-20T09:32:18.900551+0000) 2024-02-20T09:34:00.990 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:34:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:34:00.617+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:31:58.900551+0000 (oldest deadline 2024-02-20T09:32:18.900551+0000) 2024-02-20T09:34:01.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:34:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:34:00.986+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:32:16.988591+0000 (oldest deadline 2024-02-20T09:32:36.988591+0000) 2024-02-20T09:34:01.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:34:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:34:00.986+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:32:16.988591+0000 (oldest deadline 2024-02-20T09:32:36.988591+0000) 2024-02-20T09:34:01.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:34:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:34:00.986+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:32:16.988591+0000 (oldest deadline 2024-02-20T09:32:36.988591+0000) 2024-02-20T09:34:01.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:34:00 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:34:00.986+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:34:01.781 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:34:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:34:01.599+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:31:58.900551+0000 (oldest deadline 2024-02-20T09:32:18.900551+0000) 2024-02-20T09:34:01.781 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:34:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:34:01.599+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:31:58.900551+0000 (oldest deadline 2024-02-20T09:32:18.900551+0000) 2024-02-20T09:34:01.781 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:34:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:34:01.599+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:31:58.900551+0000 (oldest deadline 2024-02-20T09:32:18.900551+0000) 2024-02-20T09:34:02.242 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:34:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:34:01.778+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:31:46.482189+0000 (oldest deadline 2024-02-20T09:32:06.482189+0000) 2024-02-20T09:34:02.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:34:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:34:01.778+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:31:46.482189+0000 (oldest deadline 2024-02-20T09:32:06.482189+0000) 2024-02-20T09:34:02.243 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:34:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:34:01.778+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:31:46.482189+0000 (oldest deadline 2024-02-20T09:32:06.482189+0000) 2024-02-20T09:34:02.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:34:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:34:01.970+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:32:16.988591+0000 (oldest deadline 2024-02-20T09:32:36.988591+0000) 2024-02-20T09:34:02.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:34:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:34:01.970+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:32:16.988591+0000 (oldest deadline 2024-02-20T09:32:36.988591+0000) 2024-02-20T09:34:02.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:34:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:34:01.970+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:32:16.988591+0000 (oldest deadline 2024-02-20T09:32:36.988591+0000) 2024-02-20T09:34:02.244 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:34:01 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:34:01.970+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:34:02.983 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:34:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:34:02.797+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:31:46.482189+0000 (oldest deadline 2024-02-20T09:32:06.482189+0000) 2024-02-20T09:34:02.984 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:34:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:34:02.797+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:31:46.482189+0000 (oldest deadline 2024-02-20T09:32:06.482189+0000) 2024-02-20T09:34:02.984 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:34:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0[47366]: 2024-02-20T09:34:02.797+0000 7f8a57663700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:31:46.482189+0000 (oldest deadline 2024-02-20T09:32:06.482189+0000) 2024-02-20T09:34:02.984 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:34:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:34:02.581+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:31:58.900551+0000 (oldest deadline 2024-02-20T09:32:18.900551+0000) 2024-02-20T09:34:02.985 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:34:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:34:02.581+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:31:58.900551+0000 (oldest deadline 2024-02-20T09:32:18.900551+0000) 2024-02-20T09:34:02.985 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:34:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:34:02.581+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:31:58.900551+0000 (oldest deadline 2024-02-20T09:32:18.900551+0000) 2024-02-20T09:34:03.242 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:34:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:34:02.981+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:32:16.988591+0000 (oldest deadline 2024-02-20T09:32:36.988591+0000) 2024-02-20T09:34:03.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:34:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:34:02.981+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:32:16.988591+0000 (oldest deadline 2024-02-20T09:32:36.988591+0000) 2024-02-20T09:34:03.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:34:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:34:02.981+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:32:16.988591+0000 (oldest deadline 2024-02-20T09:32:36.988591+0000) 2024-02-20T09:34:03.243 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:34:02 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:34:02.981+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:34:03.992 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:34:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:34:03.593+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:31:58.900551+0000 (oldest deadline 2024-02-20T09:32:18.900551+0000) 2024-02-20T09:34:04.393 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:34:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:34:03.593+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:31:58.900551+0000 (oldest deadline 2024-02-20T09:32:18.900551+0000) 2024-02-20T09:34:04.393 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:34:03 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:34:03.593+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:31:58.900551+0000 (oldest deadline 2024-02-20T09:32:18.900551+0000) 2024-02-20T09:34:04.485 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:34:04 smithi133 podman[94333]: 2024-02-20 09:34:04.239026687 +0000 UTC m=+5.650478292 container died eee5e0d5322fffe315038b895d3d8566945b1c04928c6a1457e88983a5ad4539 (image=quay.ceph.io/ceph-ci/ceph@sha256:479114d4af9dd6c9b2771de62c79e08a4e4ba3cecb535f7d44dca6fc106bb04b, name=ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0, ceph=True, org.label-schema.build-date=20240214, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, RELEASE=quincy-c80e716) 2024-02-20T09:34:04.486 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:34:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:34:04.017+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:32:16.988591+0000 (oldest deadline 2024-02-20T09:32:36.988591+0000) 2024-02-20T09:34:04.486 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:34:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:34:04.017+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:32:16.988591+0000 (oldest deadline 2024-02-20T09:32:36.988591+0000) 2024-02-20T09:34:04.486 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:34:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:34:04.017+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:32:16.988591+0000 (oldest deadline 2024-02-20T09:32:36.988591+0000) 2024-02-20T09:34:04.486 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:34:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:34:04.017+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:34:04.742 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:34:04 smithi133 podman[94333]: 2024-02-20 09:34:04.57347644 +0000 UTC m=+5.984928044 container cleanup eee5e0d5322fffe315038b895d3d8566945b1c04928c6a1457e88983a5ad4539 (image=quay.ceph.io/ceph-ci/ceph@sha256:479114d4af9dd6c9b2771de62c79e08a4e4ba3cecb535f7d44dca6fc106bb04b, name=ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.build-date=20240214, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, RELEASE=quincy-c80e716, ceph=True) 2024-02-20T09:34:04.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:34:04 smithi133 bash[94333]: ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0 2024-02-20T09:34:04.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:34:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:34:04.574+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:31:58.900551+0000 (oldest deadline 2024-02-20T09:32:18.900551+0000) 2024-02-20T09:34:04.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:34:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:34:04.574+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:31:58.900551+0000 (oldest deadline 2024-02-20T09:32:18.900551+0000) 2024-02-20T09:34:04.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:34:04 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:34:04.574+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:31:58.900551+0000 (oldest deadline 2024-02-20T09:32:18.900551+0000) 2024-02-20T09:34:05.052 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:34:04 smithi133 podman[94352]: 2024-02-20 09:34:04.774830551 +0000 UTC m=+0.529495048 container remove eee5e0d5322fffe315038b895d3d8566945b1c04928c6a1457e88983a5ad4539 (image=quay.ceph.io/ceph-ci/ceph@sha256:479114d4af9dd6c9b2771de62c79e08a4e4ba3cecb535f7d44dca6fc106bb04b, name=ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240214, GIT_CLEAN=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, RELEASE=quincy-c80e716, CEPH_POINT_RELEASE=, ceph=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-02-20T09:34:05.374 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:34:05 smithi133 podman[94448]: 2024-02-20 09:34:05.087888402 +0000 UTC m=+0.018529205 image pull 10b0c032c913004d1f4f6c0f2496f5bfddfde4aeddb489f4129f49e49277ea3e quay.ceph.io/ceph-ci/ceph@sha256:479114d4af9dd6c9b2771de62c79e08a4e4ba3cecb535f7d44dca6fc106bb04b 2024-02-20T09:34:05.374 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:34:05 smithi133 podman[94448]: 2024-02-20 09:34:05.284005832 +0000 UTC m=+0.214646620 container create 4d86849997eb8c9549cfe669073ccce8bb76ff25823712a1c79bf23c35e99faa (image=quay.ceph.io/ceph-ci/ceph@sha256:479114d4af9dd6c9b2771de62c79e08a4e4ba3cecb535f7d44dca6fc106bb04b, name=ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0-deactivate, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240214, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=quincy-c80e716) 2024-02-20T09:34:05.374 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:34:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:34:05.049+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:32:16.988591+0000 (oldest deadline 2024-02-20T09:32:36.988591+0000) 2024-02-20T09:34:05.374 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:34:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:34:05.049+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:32:16.988591+0000 (oldest deadline 2024-02-20T09:32:36.988591+0000) 2024-02-20T09:34:05.375 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:34:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:34:05.049+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:32:16.988591+0000 (oldest deadline 2024-02-20T09:32:36.988591+0000) 2024-02-20T09:34:05.375 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:34:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:34:05.049+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:34:05.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:34:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:34:05.610+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:31:58.900551+0000 (oldest deadline 2024-02-20T09:32:18.900551+0000) 2024-02-20T09:34:05.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:34:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:34:05.610+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:31:58.900551+0000 (oldest deadline 2024-02-20T09:32:18.900551+0000) 2024-02-20T09:34:05.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:34:05 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:34:05.610+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:31:58.900551+0000 (oldest deadline 2024-02-20T09:32:18.900551+0000) 2024-02-20T09:34:05.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:34:05 smithi133 podman[94448]: 2024-02-20 09:34:05.494881445 +0000 UTC m=+0.425522242 container init 4d86849997eb8c9549cfe669073ccce8bb76ff25823712a1c79bf23c35e99faa (image=quay.ceph.io/ceph-ci/ceph@sha256:479114d4af9dd6c9b2771de62c79e08a4e4ba3cecb535f7d44dca6fc106bb04b, name=ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0-deactivate, maintainer=Guillaume Abrioux , ceph=True, GIT_BRANCH=HEAD, RELEASE=quincy-c80e716, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240214, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2) 2024-02-20T09:34:05.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:34:05 smithi133 podman[94448]: 2024-02-20 09:34:05.498593814 +0000 UTC m=+0.429234609 container start 4d86849997eb8c9549cfe669073ccce8bb76ff25823712a1c79bf23c35e99faa (image=quay.ceph.io/ceph-ci/ceph@sha256:479114d4af9dd6c9b2771de62c79e08a4e4ba3cecb535f7d44dca6fc106bb04b, name=ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0-deactivate, ceph=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, RELEASE=quincy-c80e716, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.build-date=20240214, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=) 2024-02-20T09:34:05.743 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:34:05 smithi133 podman[94448]: 2024-02-20 09:34:05.586314256 +0000 UTC m=+0.516955053 container attach 4d86849997eb8c9549cfe669073ccce8bb76ff25823712a1c79bf23c35e99faa (image=quay.ceph.io/ceph-ci/ceph@sha256:479114d4af9dd6c9b2771de62c79e08a4e4ba3cecb535f7d44dca6fc106bb04b, name=ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0-deactivate, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.build-date=20240214, ceph=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, RELEASE=quincy-c80e716, maintainer=Guillaume Abrioux ) 2024-02-20T09:34:06.057 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:34:05 smithi133 podman[94448]: 2024-02-20 09:34:05.775550326 +0000 UTC m=+0.706191110 container died 4d86849997eb8c9549cfe669073ccce8bb76ff25823712a1c79bf23c35e99faa (image=quay.ceph.io/ceph-ci/ceph@sha256:479114d4af9dd6c9b2771de62c79e08a4e4ba3cecb535f7d44dca6fc106bb04b, name=ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.build-date=20240214, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.vendor=CentOS, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, RELEASE=quincy-c80e716, ceph=True) 2024-02-20T09:34:06.316 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:34:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:34:06.058+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:32:16.988591+0000 (oldest deadline 2024-02-20T09:32:36.988591+0000) 2024-02-20T09:34:06.316 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:34:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:34:06.058+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:32:16.988591+0000 (oldest deadline 2024-02-20T09:32:36.988591+0000) 2024-02-20T09:34:06.316 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:34:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:34:06.058+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:32:16.988591+0000 (oldest deadline 2024-02-20T09:32:36.988591+0000) 2024-02-20T09:34:06.317 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:34:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:34:06.058+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:34:06.583 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:34:06 smithi133 systemd[1]: Stopping Ceph osd.1 for 4537327c-cfc4-11ee-95bb-87774f69a715... 2024-02-20T09:34:06.584 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:34:06 smithi133 podman[94474]: 2024-02-20 09:34:06.315044797 +0000 UTC m=+0.534367363 container remove 4d86849997eb8c9549cfe669073ccce8bb76ff25823712a1c79bf23c35e99faa (image=quay.ceph.io/ceph-ci/ceph@sha256:479114d4af9dd6c9b2771de62c79e08a4e4ba3cecb535f7d44dca6fc106bb04b, name=ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-0-deactivate, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=quincy-c80e716, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.build-date=20240214, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd) 2024-02-20T09:34:06.584 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:34:06 smithi133 systemd[1]: ceph-4537327c-cfc4-11ee-95bb-87774f69a715@osd.0.service: Deactivated successfully. 2024-02-20T09:34:06.584 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:34:06 smithi133 systemd[1]: Stopped Ceph osd.0 for 4537327c-cfc4-11ee-95bb-87774f69a715. 2024-02-20T09:34:06.584 INFO:journalctl@ceph.osd.0.smithi133.stdout:Feb 20 09:34:06 smithi133 systemd[1]: ceph-4537327c-cfc4-11ee-95bb-87774f69a715@osd.0.service: Consumed 30.483s CPU time. 2024-02-20T09:34:06.992 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:34:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:34:06.792+0000 7f010265b700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-02-20T09:34:06.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:34:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:34:06.792+0000 7f010265b700 -1 osd.1 46 *** Got signal Terminated *** 2024-02-20T09:34:06.993 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:34:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:34:06.792+0000 7f010265b700 -1 osd.1 46 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-20T09:34:06.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:34:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:34:06.580+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:31:58.900551+0000 (oldest deadline 2024-02-20T09:32:18.900551+0000) 2024-02-20T09:34:06.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:34:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:34:06.580+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:31:58.900551+0000 (oldest deadline 2024-02-20T09:32:18.900551+0000) 2024-02-20T09:34:06.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:34:06 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:34:06.580+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:31:58.900551+0000 (oldest deadline 2024-02-20T09:32:18.900551+0000) 2024-02-20T09:34:07.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:34:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:34:07.024+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:32:16.988591+0000 (oldest deadline 2024-02-20T09:32:36.988591+0000) 2024-02-20T09:34:07.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:34:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:34:07.024+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:32:16.988591+0000 (oldest deadline 2024-02-20T09:32:36.988591+0000) 2024-02-20T09:34:07.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:34:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:34:07.024+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:32:16.988591+0000 (oldest deadline 2024-02-20T09:32:36.988591+0000) 2024-02-20T09:34:07.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:34:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:34:07.024+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:34:07.992 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:34:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:34:07.550+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:31:58.900551+0000 (oldest deadline 2024-02-20T09:32:18.900551+0000) 2024-02-20T09:34:07.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:34:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:34:07.550+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:31:58.900551+0000 (oldest deadline 2024-02-20T09:32:18.900551+0000) 2024-02-20T09:34:07.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:34:07 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:34:07.550+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:31:58.900551+0000 (oldest deadline 2024-02-20T09:32:18.900551+0000) 2024-02-20T09:34:08.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:34:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:34:08.074+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:32:16.988591+0000 (oldest deadline 2024-02-20T09:32:36.988591+0000) 2024-02-20T09:34:08.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:34:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:34:08.074+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:32:16.988591+0000 (oldest deadline 2024-02-20T09:32:36.988591+0000) 2024-02-20T09:34:08.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:34:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:34:08.074+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:32:16.988591+0000 (oldest deadline 2024-02-20T09:32:36.988591+0000) 2024-02-20T09:34:08.493 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:34:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:34:08.074+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:34:08.992 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:34:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:34:08.515+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:31:58.900551+0000 (oldest deadline 2024-02-20T09:32:18.900551+0000) 2024-02-20T09:34:08.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:34:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:34:08.515+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:31:58.900551+0000 (oldest deadline 2024-02-20T09:32:18.900551+0000) 2024-02-20T09:34:08.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:34:08 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:34:08.515+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:31:58.900551+0000 (oldest deadline 2024-02-20T09:32:18.900551+0000) 2024-02-20T09:34:09.485 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:34:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:34:09.030+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:32:16.988591+0000 (oldest deadline 2024-02-20T09:32:36.988591+0000) 2024-02-20T09:34:09.485 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:34:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:34:09.030+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:32:16.988591+0000 (oldest deadline 2024-02-20T09:32:36.988591+0000) 2024-02-20T09:34:09.486 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:34:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:34:09.030+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:32:16.988591+0000 (oldest deadline 2024-02-20T09:32:36.988591+0000) 2024-02-20T09:34:09.486 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:34:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:34:09.030+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:34:09.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:34:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:34:09.483+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:31:58.900551+0000 (oldest deadline 2024-02-20T09:32:18.900551+0000) 2024-02-20T09:34:09.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:34:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:34:09.483+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:31:58.900551+0000 (oldest deadline 2024-02-20T09:32:18.900551+0000) 2024-02-20T09:34:09.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:34:09 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:34:09.483+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:31:58.900551+0000 (oldest deadline 2024-02-20T09:32:18.900551+0000) 2024-02-20T09:34:10.488 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:34:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:34:10.039+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:32:16.988591+0000 (oldest deadline 2024-02-20T09:32:36.988591+0000) 2024-02-20T09:34:10.488 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:34:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:34:10.039+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:32:16.988591+0000 (oldest deadline 2024-02-20T09:32:36.988591+0000) 2024-02-20T09:34:10.488 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:34:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:34:10.039+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:32:16.988591+0000 (oldest deadline 2024-02-20T09:32:36.988591+0000) 2024-02-20T09:34:10.488 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:34:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:34:10.039+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:34:10.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:34:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:34:10.486+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:31:58.900551+0000 (oldest deadline 2024-02-20T09:32:18.900551+0000) 2024-02-20T09:34:10.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:34:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:34:10.486+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:31:58.900551+0000 (oldest deadline 2024-02-20T09:32:18.900551+0000) 2024-02-20T09:34:10.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:34:10 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:34:10.486+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:31:58.900551+0000 (oldest deadline 2024-02-20T09:32:18.900551+0000) 2024-02-20T09:34:11.477 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:34:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:34:11.069+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:32:16.988591+0000 (oldest deadline 2024-02-20T09:32:36.988591+0000) 2024-02-20T09:34:11.477 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:34:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:34:11.069+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:32:16.988591+0000 (oldest deadline 2024-02-20T09:32:36.988591+0000) 2024-02-20T09:34:11.477 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:34:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:34:11.069+0000 7f00fdc57700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:32:16.988591+0000 (oldest deadline 2024-02-20T09:32:36.988591+0000) 2024-02-20T09:34:11.477 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:34:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1[55173]: 2024-02-20T09:34:11.069+0000 7f00fdc57700 -1 osd.1 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.14486.14:248 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=214593b,omap-rm-keys in=47059b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-02-20T09:34:11.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:34:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:34:11.475+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:31:58.900551+0000 (oldest deadline 2024-02-20T09:32:18.900551+0000) 2024-02-20T09:34:11.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:34:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:34:11.475+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:31:58.900551+0000 (oldest deadline 2024-02-20T09:32:18.900551+0000) 2024-02-20T09:34:11.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:34:11 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:34:11.475+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:31:58.900551+0000 (oldest deadline 2024-02-20T09:32:18.900551+0000) 2024-02-20T09:34:12.521 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:34:12 smithi133 podman[94555]: 2024-02-20 09:34:12.250649466 +0000 UTC m=+5.566708766 container died aa50c25fdf11e0374d149bf5c80765c54115dcdf5333af9b9dd8cce21d101f1a (image=quay.ceph.io/ceph-ci/ceph@sha256:479114d4af9dd6c9b2771de62c79e08a4e4ba3cecb535f7d44dca6fc106bb04b, name=ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240214, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, RELEASE=quincy-c80e716, ceph=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux ) 2024-02-20T09:34:12.521 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:34:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:34:12.456+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:31:58.900551+0000 (oldest deadline 2024-02-20T09:32:18.900551+0000) 2024-02-20T09:34:12.521 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:34:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:34:12.456+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:31:58.900551+0000 (oldest deadline 2024-02-20T09:32:18.900551+0000) 2024-02-20T09:34:12.521 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:34:12 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:34:12.456+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:31:58.900551+0000 (oldest deadline 2024-02-20T09:32:18.900551+0000) 2024-02-20T09:34:12.912 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:34:12 smithi133 podman[94555]: 2024-02-20 09:34:12.618757883 +0000 UTC m=+5.934817185 container cleanup aa50c25fdf11e0374d149bf5c80765c54115dcdf5333af9b9dd8cce21d101f1a (image=quay.ceph.io/ceph-ci/ceph@sha256:479114d4af9dd6c9b2771de62c79e08a4e4ba3cecb535f7d44dca6fc106bb04b, name=ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=quincy-c80e716, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240214, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux ) 2024-02-20T09:34:12.913 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:34:12 smithi133 bash[94555]: ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1 2024-02-20T09:34:13.228 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:34:12 smithi133 podman[94586]: 2024-02-20 09:34:12.911578329 +0000 UTC m=+0.651410095 container remove aa50c25fdf11e0374d149bf5c80765c54115dcdf5333af9b9dd8cce21d101f1a (image=quay.ceph.io/ceph-ci/ceph@sha256:479114d4af9dd6c9b2771de62c79e08a4e4ba3cecb535f7d44dca6fc106bb04b, name=ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.build-date=20240214, GIT_CLEAN=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, RELEASE=quincy-c80e716, io.buildah.version=1.33.5) 2024-02-20T09:34:13.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:34:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:34:13.464+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:31:58.900551+0000 (oldest deadline 2024-02-20T09:32:18.900551+0000) 2024-02-20T09:34:13.491 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:34:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:34:13.464+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:31:58.900551+0000 (oldest deadline 2024-02-20T09:32:18.900551+0000) 2024-02-20T09:34:13.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:34:13 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:34:13.464+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:31:58.900551+0000 (oldest deadline 2024-02-20T09:32:18.900551+0000) 2024-02-20T09:34:13.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:34:13 smithi133 podman[94672]: 2024-02-20 09:34:13.129741833 +0000 UTC m=+0.018289367 image pull 10b0c032c913004d1f4f6c0f2496f5bfddfde4aeddb489f4129f49e49277ea3e quay.ceph.io/ceph-ci/ceph@sha256:479114d4af9dd6c9b2771de62c79e08a4e4ba3cecb535f7d44dca6fc106bb04b 2024-02-20T09:34:13.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:34:13 smithi133 podman[94672]: 2024-02-20 09:34:13.237443352 +0000 UTC m=+0.125990883 container create 566ff8be1cbb88bf17b86556b8109b963fcc4ac8336e62faf5098ef52612ccc8 (image=quay.ceph.io/ceph-ci/ceph@sha256:479114d4af9dd6c9b2771de62c79e08a4e4ba3cecb535f7d44dca6fc106bb04b, name=ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1-deactivate, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=quincy-c80e716, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240214, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, io.buildah.version=1.33.5) 2024-02-20T09:34:13.492 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:34:13 smithi133 podman[94672]: 2024-02-20 09:34:13.490233977 +0000 UTC m=+0.378781515 container init 566ff8be1cbb88bf17b86556b8109b963fcc4ac8336e62faf5098ef52612ccc8 (image=quay.ceph.io/ceph-ci/ceph@sha256:479114d4af9dd6c9b2771de62c79e08a4e4ba3cecb535f7d44dca6fc106bb04b, name=ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.build-date=20240214, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, RELEASE=quincy-c80e716, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, ceph=True) 2024-02-20T09:34:13.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:34:13 smithi133 podman[94672]: 2024-02-20 09:34:13.493317542 +0000 UTC m=+0.381865080 container start 566ff8be1cbb88bf17b86556b8109b963fcc4ac8336e62faf5098ef52612ccc8 (image=quay.ceph.io/ceph-ci/ceph@sha256:479114d4af9dd6c9b2771de62c79e08a4e4ba3cecb535f7d44dca6fc106bb04b, name=ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1-deactivate, GIT_BRANCH=HEAD, ceph=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240214, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=quincy-c80e716) 2024-02-20T09:34:13.742 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:34:13 smithi133 podman[94672]: 2024-02-20 09:34:13.598389477 +0000 UTC m=+0.486937016 container attach 566ff8be1cbb88bf17b86556b8109b963fcc4ac8336e62faf5098ef52612ccc8 (image=quay.ceph.io/ceph-ci/ceph@sha256:479114d4af9dd6c9b2771de62c79e08a4e4ba3cecb535f7d44dca6fc106bb04b, name=ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1-deactivate, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=quincy-c80e716, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240214, org.label-schema.schema-version=1.0) 2024-02-20T09:34:14.052 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:34:13 smithi133 podman[94672]: 2024-02-20 09:34:13.778937181 +0000 UTC m=+0.667484712 container died 566ff8be1cbb88bf17b86556b8109b963fcc4ac8336e62faf5098ef52612ccc8 (image=quay.ceph.io/ceph-ci/ceph@sha256:479114d4af9dd6c9b2771de62c79e08a4e4ba3cecb535f7d44dca6fc106bb04b, name=ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1-deactivate, CEPH_POINT_RELEASE=, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240214, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, RELEASE=quincy-c80e716) 2024-02-20T09:34:14.312 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:34:14 smithi133 podman[94697]: 2024-02-20 09:34:14.276607705 +0000 UTC m=+0.492742236 container remove 566ff8be1cbb88bf17b86556b8109b963fcc4ac8336e62faf5098ef52612ccc8 (image=quay.ceph.io/ceph-ci/ceph@sha256:479114d4af9dd6c9b2771de62c79e08a4e4ba3cecb535f7d44dca6fc106bb04b, name=ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-1-deactivate, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240214, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, RELEASE=quincy-c80e716, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, ceph=True) 2024-02-20T09:34:14.313 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:34:14 smithi133 systemd[1]: ceph-4537327c-cfc4-11ee-95bb-87774f69a715@osd.1.service: Deactivated successfully. 2024-02-20T09:34:14.313 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:34:14 smithi133 systemd[1]: Stopped Ceph osd.1 for 4537327c-cfc4-11ee-95bb-87774f69a715. 2024-02-20T09:34:14.313 INFO:journalctl@ceph.osd.1.smithi133.stdout:Feb 20 09:34:14 smithi133 systemd[1]: ceph-4537327c-cfc4-11ee-95bb-87774f69a715@osd.1.service: Consumed 46.662s CPU time. 2024-02-20T09:34:14.723 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:34:14 smithi133 systemd[1]: Stopping Ceph osd.2 for 4537327c-cfc4-11ee-95bb-87774f69a715... 2024-02-20T09:34:14.723 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:34:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:34:14.484+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:31:58.900551+0000 (oldest deadline 2024-02-20T09:32:18.900551+0000) 2024-02-20T09:34:14.723 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:34:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:34:14.484+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:31:58.900551+0000 (oldest deadline 2024-02-20T09:32:18.900551+0000) 2024-02-20T09:34:14.723 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:34:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:34:14.484+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:31:58.900551+0000 (oldest deadline 2024-02-20T09:32:18.900551+0000) 2024-02-20T09:34:14.723 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:34:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:34:14.720+0000 7f8e5c28a700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-02-20T09:34:14.724 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:34:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:34:14.720+0000 7f8e5c28a700 -1 osd.2 46 *** Got signal Terminated *** 2024-02-20T09:34:14.724 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:34:14 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:34:14.720+0000 7f8e5c28a700 -1 osd.2 46 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-20T09:34:15.992 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:34:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:34:15.498+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:31:58.900551+0000 (oldest deadline 2024-02-20T09:32:18.900551+0000) 2024-02-20T09:34:15.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:34:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:34:15.498+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:31:58.900551+0000 (oldest deadline 2024-02-20T09:32:18.900551+0000) 2024-02-20T09:34:15.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:34:15 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:34:15.498+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:31:58.900551+0000 (oldest deadline 2024-02-20T09:32:18.900551+0000) 2024-02-20T09:34:16.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:34:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:34:16.466+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:31:58.900551+0000 (oldest deadline 2024-02-20T09:32:18.900551+0000) 2024-02-20T09:34:16.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:34:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:34:16.466+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:31:58.900551+0000 (oldest deadline 2024-02-20T09:32:18.900551+0000) 2024-02-20T09:34:16.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:34:16 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:34:16.466+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:31:58.900551+0000 (oldest deadline 2024-02-20T09:32:18.900551+0000) 2024-02-20T09:34:17.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:34:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:34:17.475+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:31:58.900551+0000 (oldest deadline 2024-02-20T09:32:18.900551+0000) 2024-02-20T09:34:17.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:34:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:34:17.475+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:31:58.900551+0000 (oldest deadline 2024-02-20T09:32:18.900551+0000) 2024-02-20T09:34:17.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:34:17 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:34:17.475+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:31:58.900551+0000 (oldest deadline 2024-02-20T09:32:18.900551+0000) 2024-02-20T09:34:18.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:34:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:34:18.460+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:31:58.900551+0000 (oldest deadline 2024-02-20T09:32:18.900551+0000) 2024-02-20T09:34:18.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:34:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:34:18.460+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:31:58.900551+0000 (oldest deadline 2024-02-20T09:32:18.900551+0000) 2024-02-20T09:34:18.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:34:18 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:34:18.460+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:31:58.900551+0000 (oldest deadline 2024-02-20T09:32:18.900551+0000) 2024-02-20T09:34:19.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:34:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:34:19.423+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6804 osd.3 ever on either front or back, first ping sent 2024-02-20T09:31:58.900551+0000 (oldest deadline 2024-02-20T09:32:18.900551+0000) 2024-02-20T09:34:19.742 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:34:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:34:19.423+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6812 osd.4 ever on either front or back, first ping sent 2024-02-20T09:31:58.900551+0000 (oldest deadline 2024-02-20T09:32:18.900551+0000) 2024-02-20T09:34:19.743 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:34:19 smithi133 ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2[62532]: 2024-02-20T09:34:19.423+0000 7f8e58888700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.183:6820 osd.5 ever on either front or back, first ping sent 2024-02-20T09:31:58.900551+0000 (oldest deadline 2024-02-20T09:32:18.900551+0000) 2024-02-20T09:34:20.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:34:20 smithi133 podman[94776]: 2024-02-20 09:34:20.23751607 +0000 UTC m=+5.623875525 container died 91e34ab8741fd805d016d943020aa90b0eba0940fa5282eac566a6cf3715543a (image=quay.ceph.io/ceph-ci/ceph@sha256:479114d4af9dd6c9b2771de62c79e08a4e4ba3cecb535f7d44dca6fc106bb04b, name=ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240214, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, RELEASE=quincy-c80e716, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2) 2024-02-20T09:34:20.992 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:34:20 smithi133 podman[94776]: 2024-02-20 09:34:20.57095322 +0000 UTC m=+5.957312672 container cleanup 91e34ab8741fd805d016d943020aa90b0eba0940fa5282eac566a6cf3715543a (image=quay.ceph.io/ceph-ci/ceph@sha256:479114d4af9dd6c9b2771de62c79e08a4e4ba3cecb535f7d44dca6fc106bb04b, name=ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2, RELEASE=quincy-c80e716, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240214, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS) 2024-02-20T09:34:20.992 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:34:20 smithi133 bash[94776]: ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2 2024-02-20T09:34:20.993 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:34:20 smithi133 podman[94793]: 2024-02-20 09:34:20.73033509 +0000 UTC m=+0.489100437 container remove 91e34ab8741fd805d016d943020aa90b0eba0940fa5282eac566a6cf3715543a (image=quay.ceph.io/ceph-ci/ceph@sha256:479114d4af9dd6c9b2771de62c79e08a4e4ba3cecb535f7d44dca6fc106bb04b, name=ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240214, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, RELEASE=quincy-c80e716, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, ceph=True) 2024-02-20T09:34:21.492 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:34:21 smithi133 podman[94883]: 2024-02-20 09:34:21.122725547 +0000 UTC m=+0.111342131 container create 8bb4f4530c108268fa644364ad127c5f38831d5b98848a8d90a55d8d76a7d79c (image=quay.ceph.io/ceph-ci/ceph@sha256:479114d4af9dd6c9b2771de62c79e08a4e4ba3cecb535f7d44dca6fc106bb04b, name=ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2-deactivate, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240214, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, ceph=True, GIT_CLEAN=True, RELEASE=quincy-c80e716, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2) 2024-02-20T09:34:21.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:34:21 smithi133 podman[94883]: 2024-02-20 09:34:21.029820669 +0000 UTC m=+0.018437266 image pull 10b0c032c913004d1f4f6c0f2496f5bfddfde4aeddb489f4129f49e49277ea3e quay.ceph.io/ceph-ci/ceph@sha256:479114d4af9dd6c9b2771de62c79e08a4e4ba3cecb535f7d44dca6fc106bb04b 2024-02-20T09:34:21.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:34:21 smithi133 podman[94883]: 2024-02-20 09:34:21.283170693 +0000 UTC m=+0.271787284 container init 8bb4f4530c108268fa644364ad127c5f38831d5b98848a8d90a55d8d76a7d79c (image=quay.ceph.io/ceph-ci/ceph@sha256:479114d4af9dd6c9b2771de62c79e08a4e4ba3cecb535f7d44dca6fc106bb04b, name=ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2-deactivate, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, RELEASE=quincy-c80e716, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240214, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True) 2024-02-20T09:34:21.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:34:21 smithi133 podman[94883]: 2024-02-20 09:34:21.286721388 +0000 UTC m=+0.275337982 container start 8bb4f4530c108268fa644364ad127c5f38831d5b98848a8d90a55d8d76a7d79c (image=quay.ceph.io/ceph-ci/ceph@sha256:479114d4af9dd6c9b2771de62c79e08a4e4ba3cecb535f7d44dca6fc106bb04b, name=ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, ceph=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, RELEASE=quincy-c80e716, org.label-schema.build-date=20240214) 2024-02-20T09:34:21.493 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:34:21 smithi133 podman[94883]: 2024-02-20 09:34:21.349842392 +0000 UTC m=+0.338458986 container attach 8bb4f4530c108268fa644364ad127c5f38831d5b98848a8d90a55d8d76a7d79c (image=quay.ceph.io/ceph-ci/ceph@sha256:479114d4af9dd6c9b2771de62c79e08a4e4ba3cecb535f7d44dca6fc106bb04b, name=ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2-deactivate, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, RELEASE=quincy-c80e716, ceph=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240214) 2024-02-20T09:34:21.862 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:34:21 smithi133 podman[94883]: 2024-02-20 09:34:21.57763322 +0000 UTC m=+0.566249812 container died 8bb4f4530c108268fa644364ad127c5f38831d5b98848a8d90a55d8d76a7d79c (image=quay.ceph.io/ceph-ci/ceph@sha256:479114d4af9dd6c9b2771de62c79e08a4e4ba3cecb535f7d44dca6fc106bb04b, name=ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2-deactivate, org.label-schema.build-date=20240214, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_CLEAN=True, io.buildah.version=1.33.5, RELEASE=quincy-c80e716, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux ) 2024-02-20T09:34:22.124 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:34:22 smithi133 podman[94909]: 2024-02-20 09:34:22.102824697 +0000 UTC m=+0.518243891 container remove 8bb4f4530c108268fa644364ad127c5f38831d5b98848a8d90a55d8d76a7d79c (image=quay.ceph.io/ceph-ci/ceph@sha256:479114d4af9dd6c9b2771de62c79e08a4e4ba3cecb535f7d44dca6fc106bb04b, name=ceph-4537327c-cfc4-11ee-95bb-87774f69a715-osd-2-deactivate, RELEASE=quincy-c80e716, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240214, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_CLEAN=True) 2024-02-20T09:34:22.466 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:34:22 smithi133 systemd[1]: ceph-4537327c-cfc4-11ee-95bb-87774f69a715@osd.2.service: Deactivated successfully. 2024-02-20T09:34:22.466 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:34:22 smithi133 systemd[1]: Stopped Ceph osd.2 for 4537327c-cfc4-11ee-95bb-87774f69a715. 2024-02-20T09:34:22.466 INFO:journalctl@ceph.osd.2.smithi133.stdout:Feb 20 09:34:22 smithi133 systemd[1]: ceph-4537327c-cfc4-11ee-95bb-87774f69a715@osd.2.service: Consumed 29.919s CPU time. 2024-02-20T09:34:23.088 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi183.front.sepia.ceph.com' 2024-02-20T09:34:23.089 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi183.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-20T09:34:26.144 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.183 2024-02-20T09:34:35.146 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi183.front.sepia.ceph.com' 2024-02-20T09:34:35.147 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi183.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-20T09:34:38.208 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.183 2024-02-20T09:34:50.209 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi183.front.sepia.ceph.com' 2024-02-20T09:34:50.211 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi183.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-20T09:34:53.276 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.183 2024-02-20T09:34:56.277 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi183.front.sepia.ceph.com' 2024-02-20T09:34:56.277 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi183.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-20T09:34:56.348 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.183 2024-02-20T09:34:56.348 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-02-20T09:34:56.349 INFO:tasks.cephadm:Teardown complete 2024-02-20T09:34:56.349 ERROR:teuthology.run_tasks:Manager failed: cephadm Traceback (most recent call last): File "/home/teuthworker/src/github.com_batrick_ceph_3d72bcfd7affbb4a923c191d212814b4934cfeaa/qa/tasks/cephadm.py", line 1553, in distribute_config_and_admin_keyring yield File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/github.com_batrick_ceph_3d72bcfd7affbb4a923c191d212814b4934cfeaa/qa/tasks/cephadm.py", line 1807, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_batrick_ceph_3d72bcfd7affbb4a923c191d212814b4934cfeaa/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/github.com_batrick_ceph_3d72bcfd7affbb4a923c191d212814b4934cfeaa/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/github.com_batrick_ceph_3d72bcfd7affbb4a923c191d212814b4934cfeaa/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 450, in run r.execute() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 100, in execute self.client.exec_command(self.command) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/client.py", line 560, in exec_command chan = self._transport.open_session(timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 959, in open_session return self.open_channel( File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 1090, in open_channel raise e File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2159, in run ptype, m = self.packetizer.read_message() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 463, in read_message header = self.read_all(self.__block_size_in, check_rekey=True) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 308, in read_all x = self.__socket.recv(n) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/gevent/_socketcommon.py", line 660, in recv return self._sock.recv(*args) OSError: [Errno 113] No route to host During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_batrick_ceph_3d72bcfd7affbb4a923c191d212814b4934cfeaa/qa/tasks/cephadm.py", line 1810, in task log.info('Teardown begin') File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_batrick_ceph_3d72bcfd7affbb4a923c191d212814b4934cfeaa/qa/tasks/cephadm.py", line 1686, in initialize_config yield File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_batrick_ceph_3d72bcfd7affbb4a923c191d212814b4934cfeaa/qa/tasks/cephadm.py", line 142, in download_cephadm _rm_cluster(ctx, cluster_name) File "/home/teuthworker/src/github.com_batrick_ceph_3d72bcfd7affbb4a923c191d212814b4934cfeaa/qa/tasks/cephadm.py", line 310, in _rm_cluster ctx.cluster.run(args=[ File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi183 2024-02-20T09:34:56.372 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-02-20T09:34:56.386 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/task/install/__init__.py", line 626, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_batrick_ceph_3d72bcfd7affbb4a923c191d212814b4934cfeaa/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/github.com_batrick_ceph_3d72bcfd7affbb4a923c191d212814b4934cfeaa/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/github.com_batrick_ceph_3d72bcfd7affbb4a923c191d212814b4934cfeaa/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 450, in run r.execute() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 100, in execute self.client.exec_command(self.command) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/client.py", line 560, in exec_command chan = self._transport.open_session(timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 959, in open_session return self.open_channel( File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 1090, in open_channel raise e File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2159, in run ptype, m = self.packetizer.read_message() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 463, in read_message header = self.read_all(self.__block_size_in, check_rekey=True) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 308, in read_all x = self.__socket.recv(n) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/gevent/_socketcommon.py", line 660, in recv return self._sock.recv(*args) OSError: [Errno 113] No route to host 2024-02-20T09:34:56.402 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-02-20T09:34:56.402 DEBUG:teuthology.orchestra.run.smithi133:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-02-20T09:34:56.425 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi183.front.sepia.ceph.com' 2024-02-20T09:34:56.425 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi183.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-20T09:34:59.488 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.183 2024-02-20T09:35:08.489 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi183.front.sepia.ceph.com' 2024-02-20T09:35:08.490 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi183.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-20T09:35:11.548 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.183 2024-02-20T09:35:23.549 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi183.front.sepia.ceph.com' 2024-02-20T09:35:23.550 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi183.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-20T09:35:26.620 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.183 2024-02-20T09:35:29.622 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi183.front.sepia.ceph.com' 2024-02-20T09:35:29.622 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi183.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-20T09:35:29.692 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.183 2024-02-20T09:35:29.692 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-02-20T09:35:29.739 ERROR:teuthology.run_tasks:Manager failed: install Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/task/install/util.py", line 148, in ship_utilities yield File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/task/install/__init__.py", line 626, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_batrick_ceph_3d72bcfd7affbb4a923c191d212814b4934cfeaa/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/github.com_batrick_ceph_3d72bcfd7affbb4a923c191d212814b4934cfeaa/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/github.com_batrick_ceph_3d72bcfd7affbb4a923c191d212814b4934cfeaa/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 450, in run r.execute() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 100, in execute self.client.exec_command(self.command) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/client.py", line 560, in exec_command chan = self._transport.open_session(timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 959, in open_session return self.open_channel( File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 1090, in open_channel raise e File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2159, in run ptype, m = self.packetizer.read_message() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 463, in read_message header = self.read_all(self.__block_size_in, check_rekey=True) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 308, in read_all x = self.__socket.recv(n) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/gevent/_socketcommon.py", line 660, in recv return self._sock.recv(*args) OSError: [Errno 113] No route to host During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/task/install/__init__.py", line 626, in task yield File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/task/install/__init__.py", line 220, in install yield File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/task/install/util.py", line 150, in ship_utilities _remove_utilities(ctx, filenames) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/task/install/util.py", line 121, in _remove_utilities ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi183 2024-02-20T09:35:29.760 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-02-20T09:35:29.771 INFO:teuthology.task.clock:Checking final clock skew... 2024-02-20T09:35:29.772 DEBUG:teuthology.orchestra.run.smithi133:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-02-20T09:35:29.781 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi183.front.sepia.ceph.com' 2024-02-20T09:35:29.782 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi183.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-20T09:35:29.796 INFO:teuthology.orchestra.run.smithi133.stderr:bash: line 1: ntpq: command not found 2024-02-20T09:35:29.802 INFO:teuthology.orchestra.run.smithi133.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-02-20T09:35:29.802 INFO:teuthology.orchestra.run.smithi133.stdout:=============================================================================== 2024-02-20T09:35:29.802 INFO:teuthology.orchestra.run.smithi133.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 28 +711us[ +711us] +/- 130ms 2024-02-20T09:35:29.802 INFO:teuthology.orchestra.run.smithi133.stdout:^* hv02.front.sepia.ceph.com 3 6 377 30 -943us[ -956us] +/- 71ms 2024-02-20T09:35:29.802 INFO:teuthology.orchestra.run.smithi133.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 27 +813us[ +813us] +/- 90ms 2024-02-20T09:35:29.802 INFO:teuthology.orchestra.run.smithi133.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-02-20T09:35:32.860 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.183 2024-02-20T09:35:41.861 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi183.front.sepia.ceph.com' 2024-02-20T09:35:41.862 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi183.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-20T09:35:44.928 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.183 2024-02-20T09:35:56.929 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi183.front.sepia.ceph.com' 2024-02-20T09:35:56.930 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi183.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-20T09:35:59.996 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.183 2024-02-20T09:36:02.997 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi183.front.sepia.ceph.com' 2024-02-20T09:36:02.998 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi183.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-20T09:36:03.068 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.183 2024-02-20T09:36:03.068 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-02-20T09:36:03.069 ERROR:teuthology.run_tasks:Manager failed: clock Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/task/clock.py", line 66, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_batrick_ceph_3d72bcfd7affbb4a923c191d212814b4934cfeaa/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/github.com_batrick_ceph_3d72bcfd7affbb4a923c191d212814b4934cfeaa/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/github.com_batrick_ceph_3d72bcfd7affbb4a923c191d212814b4934cfeaa/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 450, in run r.execute() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 100, in execute self.client.exec_command(self.command) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/client.py", line 560, in exec_command chan = self._transport.open_session(timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 959, in open_session return self.open_channel( File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 1090, in open_channel raise e File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2159, in run ptype, m = self.packetizer.read_message() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 463, in read_message header = self.read_all(self.__block_size_in, check_rekey=True) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 308, in read_all x = self.__socket.recv(n) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/gevent/_socketcommon.py", line 660, in recv return self._sock.recv(*args) OSError: [Errno 113] No route to host During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/task/clock.py", line 72, in task cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi183 2024-02-20T09:36:03.086 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-02-20T09:36:03.098 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-02-20T09:36:03.098 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-02-20T09:36:03.115 DEBUG:teuthology.orchestra.run.smithi133:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-02-20T09:36:03.205 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi183.front.sepia.ceph.com' 2024-02-20T09:36:03.206 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi183.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-20T09:36:06.268 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.183 2024-02-20T09:36:15.269 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi183.front.sepia.ceph.com' 2024-02-20T09:36:15.270 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi183.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-20T09:36:18.336 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.183 2024-02-20T09:36:30.337 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi183.front.sepia.ceph.com' 2024-02-20T09:36:30.338 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi183.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-20T09:36:33.408 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.183 2024-02-20T09:36:36.409 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi183.front.sepia.ceph.com' 2024-02-20T09:36:36.410 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi183.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-20T09:36:36.476 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.183 2024-02-20T09:36:36.476 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-02-20T09:36:36.477 ERROR:teuthology.run_tasks:Manager failed: selinux Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/task/__init__.py", line 136, in __exit__ self.teardown() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/task/selinux.py", line 165, in teardown self.archive_log() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/task/selinux.py", line 195, in archive_log self.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi183 2024-02-20T09:36:36.485 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-02-20T09:36:36.501 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-02-20T09:36:36.513 INFO:teuthology.task.internal:Duration was 6952.134802 seconds 2024-02-20T09:36:36.514 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-02-20T09:36:36.526 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-02-20T09:36:36.526 DEBUG:teuthology.orchestra.run.smithi133:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-02-20T09:36:36.532 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi183.front.sepia.ceph.com' 2024-02-20T09:36:36.533 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi183.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-20T09:36:36.580 INFO:teuthology.orchestra.run.smithi133.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-02-20T09:36:39.616 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.183 2024-02-20T09:36:48.616 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi183.front.sepia.ceph.com' 2024-02-20T09:36:48.617 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi183.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-20T09:36:51.680 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.183 2024-02-20T09:37:03.681 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi183.front.sepia.ceph.com' 2024-02-20T09:37:03.682 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi183.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-20T09:37:06.752 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.183 2024-02-20T09:37:09.753 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi183.front.sepia.ceph.com' 2024-02-20T09:37:09.754 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi183.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-20T09:37:09.820 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.183 2024-02-20T09:37:09.820 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-02-20T09:37:09.821 ERROR:teuthology.run_tasks:Manager failed: internal.syslog Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/task/internal/syslog.py", line 76, in syslog yield File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_batrick_ceph_3d72bcfd7affbb4a923c191d212814b4934cfeaa/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/github.com_batrick_ceph_3d72bcfd7affbb4a923c191d212814b4934cfeaa/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/github.com_batrick_ceph_3d72bcfd7affbb4a923c191d212814b4934cfeaa/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 450, in run r.execute() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 100, in execute self.client.exec_command(self.command) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/client.py", line 560, in exec_command chan = self._transport.open_session(timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 959, in open_session return self.open_channel( File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 1090, in open_channel raise e File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2159, in run ptype, m = self.packetizer.read_message() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 463, in read_message header = self.read_all(self.__block_size_in, check_rekey=True) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 308, in read_all x = self.__socket.recv(n) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/gevent/_socketcommon.py", line 660, in recv return self._sock.recv(*args) OSError: [Errno 113] No route to host During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/task/internal/syslog.py", line 85, in syslog cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi183 2024-02-20T09:37:09.838 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-02-20T09:37:09.852 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-02-20T09:37:09.852 DEBUG:teuthology.orchestra.run.smithi133:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-02-20T09:37:09.890 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi183.front.sepia.ceph.com' 2024-02-20T09:37:09.890 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi183.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-20T09:37:12.956 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.183 2024-02-20T09:37:21.957 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi183.front.sepia.ceph.com' 2024-02-20T09:37:21.958 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi183.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-20T09:37:25.020 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.183 2024-02-20T09:37:37.021 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi183.front.sepia.ceph.com' 2024-02-20T09:37:37.022 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi183.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-20T09:37:40.096 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.183 2024-02-20T09:37:43.097 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi183.front.sepia.ceph.com' 2024-02-20T09:37:43.098 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi183.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-20T09:37:43.168 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.183 2024-02-20T09:37:43.168 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-02-20T09:37:43.169 ERROR:teuthology.run_tasks:Manager failed: internal.sudo Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/task/internal/__init__.py", line 452, in sudo yield File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_batrick_ceph_3d72bcfd7affbb4a923c191d212814b4934cfeaa/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/github.com_batrick_ceph_3d72bcfd7affbb4a923c191d212814b4934cfeaa/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/github.com_batrick_ceph_3d72bcfd7affbb4a923c191d212814b4934cfeaa/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 450, in run r.execute() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 100, in execute self.client.exec_command(self.command) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/client.py", line 560, in exec_command chan = self._transport.open_session(timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 959, in open_session return self.open_channel( File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 1090, in open_channel raise e File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2159, in run ptype, m = self.packetizer.read_message() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 463, in read_message header = self.read_all(self.__block_size_in, check_rekey=True) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 308, in read_all x = self.__socket.recv(n) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/gevent/_socketcommon.py", line 660, in recv return self._sock.recv(*args) OSError: [Errno 113] No route to host During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/task/internal/__init__.py", line 455, in sudo ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi183 2024-02-20T09:37:43.185 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-02-20T09:37:43.200 DEBUG:teuthology.orchestra.run.smithi133:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-02-20T09:37:43.207 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi183.front.sepia.ceph.com' 2024-02-20T09:37:43.208 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi183.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-20T09:37:43.235 INFO:teuthology.orchestra.run.smithi133.stdout:kernel.core_pattern = core 2024-02-20T09:37:46.268 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.183 2024-02-20T09:37:55.269 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi183.front.sepia.ceph.com' 2024-02-20T09:37:55.270 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi183.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-20T09:37:58.336 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.183 2024-02-20T09:38:10.337 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi183.front.sepia.ceph.com' 2024-02-20T09:38:10.338 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi183.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-20T09:38:13.408 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.183 2024-02-20T09:38:16.409 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi183.front.sepia.ceph.com' 2024-02-20T09:38:16.409 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi183.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-20T09:38:16.476 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.183 2024-02-20T09:38:16.476 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-02-20T09:38:16.477 ERROR:teuthology.run_tasks:Manager failed: internal.coredump Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/task/internal/__init__.py", line 488, in coredump yield File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_batrick_ceph_3d72bcfd7affbb4a923c191d212814b4934cfeaa/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/github.com_batrick_ceph_3d72bcfd7affbb4a923c191d212814b4934cfeaa/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/github.com_batrick_ceph_3d72bcfd7affbb4a923c191d212814b4934cfeaa/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 450, in run r.execute() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 100, in execute self.client.exec_command(self.command) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/client.py", line 560, in exec_command chan = self._transport.open_session(timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 959, in open_session return self.open_channel( File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 1090, in open_channel raise e File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2159, in run ptype, m = self.packetizer.read_message() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 463, in read_message header = self.read_all(self.__block_size_in, check_rekey=True) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 308, in read_all x = self.__socket.recv(n) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/gevent/_socketcommon.py", line 660, in recv return self._sock.recv(*args) OSError: [Errno 113] No route to host During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/task/internal/__init__.py", line 492, in coredump cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi183 2024-02-20T09:38:16.493 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-02-20T09:38:16.505 INFO:teuthology.task.internal:Transferring archived files... 2024-02-20T09:38:16.506 DEBUG:teuthology.misc:Transferring archived files from smithi133:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/pdonnell-2024-02-20_07:23:03-fs:upgrade:mds_upgrade_sequence-wip-batrick-testing-20240220.022152-distro-default-smithi/7568541/remote/smithi133 2024-02-20T09:38:16.507 DEBUG:teuthology.orchestra.run.smithi133:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-02-20T09:38:16.549 DEBUG:teuthology.misc:Transferring archived files from smithi183:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/pdonnell-2024-02-20_07:23:03-fs:upgrade:mds_upgrade_sequence-wip-batrick-testing-20240220.022152-distro-default-smithi/7568541/remote/smithi183 2024-02-20T09:38:16.549 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi183.front.sepia.ceph.com' 2024-02-20T09:38:16.550 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi183.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-20T09:38:19.612 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.183 2024-02-20T09:38:28.613 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi183.front.sepia.ceph.com' 2024-02-20T09:38:28.614 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi183.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-20T09:38:31.676 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.183 2024-02-20T09:38:43.677 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi183.front.sepia.ceph.com' 2024-02-20T09:38:43.678 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi183.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-20T09:38:46.748 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.183 2024-02-20T09:38:49.749 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi183.front.sepia.ceph.com' 2024-02-20T09:38:49.750 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi183.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-20T09:38:49.820 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.183 2024-02-20T09:38:49.820 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-02-20T09:38:49.820 ERROR:teuthology.run_tasks:Manager failed: internal.archive Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/task/internal/__init__.py", line 392, in archive yield File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_batrick_ceph_3d72bcfd7affbb4a923c191d212814b4934cfeaa/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/github.com_batrick_ceph_3d72bcfd7affbb4a923c191d212814b4934cfeaa/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/github.com_batrick_ceph_3d72bcfd7affbb4a923c191d212814b4934cfeaa/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 450, in run r.execute() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 100, in execute self.client.exec_command(self.command) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/client.py", line 560, in exec_command chan = self._transport.open_session(timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 959, in open_session return self.open_channel( File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 1090, in open_channel raise e File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2159, in run ptype, m = self.packetizer.read_message() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 463, in read_message header = self.read_all(self.__block_size_in, check_rekey=True) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 308, in read_all x = self.__socket.recv(n) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/gevent/_socketcommon.py", line 660, in recv return self._sock.recv(*args) OSError: [Errno 113] No route to host During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/task/internal/__init__.py", line 418, in archive misc.pull_directory(rem, archive_dir, path, maybe_compress) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/misc.py", line 748, in pull_directory r = remote.get_tar_stream(remotedir, sudo=True, compress=False) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/remote.py", line 678, in get_tar_stream return self.run(args=args, wait=False, stdout=run.PIPE) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi183 2024-02-20T09:38:49.838 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-02-20T09:38:49.856 INFO:teuthology.task.internal:Not uploading archives. 2024-02-20T09:38:49.856 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-02-20T09:38:49.868 INFO:teuthology.task.internal:Tidying up after the test... 2024-02-20T09:38:49.868 DEBUG:teuthology.orchestra.run.smithi133:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-02-20T09:38:49.875 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi183.front.sepia.ceph.com' 2024-02-20T09:38:49.876 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi183.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-20T09:38:49.891 INFO:teuthology.orchestra.run.smithi133.stdout: 265295 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Feb 20 09:34 /home/ubuntu/cephtest 2024-02-20T09:38:49.892 INFO:teuthology.orchestra.run.smithi133.stdout: 265296 4 drwxr-xr-x 4 ubuntu ubuntu 4096 Feb 20 09:37 /home/ubuntu/cephtest/archive 2024-02-20T09:38:49.892 INFO:teuthology.orchestra.run.smithi133.stdout: 265300 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Feb 20 07:40 /home/ubuntu/cephtest/archive/syslog 2024-02-20T09:38:49.892 INFO:teuthology.orchestra.run.smithi133.stdout: 265302 0 -rw-rw-rw- 1 ubuntu ubuntu 0 Feb 20 07:40 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-02-20T09:38:49.892 INFO:teuthology.orchestra.run.smithi133.stdout: 265301 0 -rw-rw-rw- 1 ubuntu ubuntu 0 Feb 20 07:40 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-02-20T09:38:49.892 INFO:teuthology.orchestra.run.smithi133.stdout: 402361 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Feb 20 09:36 /home/ubuntu/cephtest/archive/audit 2024-02-20T09:38:49.892 INFO:teuthology.orchestra.run.smithi133.stdout: 426294 180 -rw------- 1 ubuntu ubuntu 181759 Feb 20 09:36 /home/ubuntu/cephtest/archive/audit/audit.log.gz 2024-02-20T09:38:49.892 INFO:teuthology.orchestra.run.smithi133.stdout: 265380 360 -rwxr-xr-x 1 ubuntu ubuntu 367151 Feb 20 07:47 /home/ubuntu/cephtest/cephadm 2024-02-20T09:38:49.893 INFO:teuthology.orchestra.run.smithi133.stderr:rmdir: failed to remove '/home/ubuntu/cephtest': Directory not empty 2024-02-20T09:38:52.956 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.183 2024-02-20T09:39:01.957 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi183.front.sepia.ceph.com' 2024-02-20T09:39:01.958 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi183.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-20T09:39:05.020 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.183 2024-02-20T09:39:17.021 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi183.front.sepia.ceph.com' 2024-02-20T09:39:17.022 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi183.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-20T09:39:20.092 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.183 2024-02-20T09:39:23.093 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi183.front.sepia.ceph.com' 2024-02-20T09:39:23.093 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi183.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-20T09:39:23.164 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.183 2024-02-20T09:39:23.164 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-02-20T09:39:23.165 ERROR:teuthology.run_tasks:Manager failed: internal.base Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/task/internal/__init__.py", line 48, in base yield File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_batrick_ceph_3d72bcfd7affbb4a923c191d212814b4934cfeaa/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/github.com_batrick_ceph_3d72bcfd7affbb4a923c191d212814b4934cfeaa/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/github.com_batrick_ceph_3d72bcfd7affbb4a923c191d212814b4934cfeaa/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 450, in run r.execute() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 100, in execute self.client.exec_command(self.command) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/client.py", line 560, in exec_command chan = self._transport.open_session(timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 959, in open_session return self.open_channel( File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 1090, in open_channel raise e File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2159, in run ptype, m = self.packetizer.read_message() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 463, in read_message header = self.read_all(self.__block_size_in, check_rekey=True) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 308, in read_all x = self.__socket.recv(n) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/gevent/_socketcommon.py", line 660, in recv return self._sock.recv(*args) OSError: [Errno 113] No route to host During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/task/internal/__init__.py", line 54, in base ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi183 2024-02-20T09:39:23.181 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-02-20T09:39:23.195 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-02-20T09:39:23.207 DEBUG:teuthology.run_tasks:Exception was not quenched, exiting: OSError: [Errno 113] No route to host 2024-02-20T09:39:23.207 INFO:teuthology.nuke:Checking targets against current locks 2024-02-20T09:39:23.237 DEBUG:teuthology.nuke:shortname: smithi133 2024-02-20T09:39:23.237 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-02-20T09:39:23.261 DEBUG:teuthology.nuke:shortname: smithi183 2024-02-20T09:39:23.261 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-02-20T09:39:23.265 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi133.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2024-02-20_07:23:03-fs:upgrade:mds_upgrade_sequence-wip-batrick-testing-20240220.022152-distro-default-smithi/7568541', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-02-20 07:32:27.090664', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLkjOQOC8d7yhry/bHm5pROTzCvqDQM+A7aVk/xdrmEzKhMlapDJ8ADcoYOUv7PtjtrLT3zeo0UbgMBXaX5hyO8='} 2024-02-20T09:39:23.292 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi183.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2024-02-20_07:23:03-fs:upgrade:mds_upgrade_sequence-wip-batrick-testing-20240220.022152-distro-default-smithi/7568541', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-02-20 07:32:27.088682', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEgZmL9WQMs3I6qqnHNxQrqtYMMn1nYmIxZNjI4HotJ6cvNlZcY79uT1EOzikzEjiglVbto/qdQL+mWRmUzefIU='} 2024-02-20T09:39:23.321 INFO:teuthology.orchestra.console.smithi133:Power off 2024-02-20T09:39:23.321 DEBUG:teuthology.orchestra.console.smithi133:pexpect command: ipmitool -H smithi133.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-02-20T09:39:23.350 DEBUG:teuthology.orchestra.console.smithi133:power off output: Chassis Power Control: Down/Off 2024-02-20T09:39:23.350 DEBUG:teuthology.orchestra.console.smithi133:pexpect command: ipmitool -H smithi133.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-20T09:39:23.368 INFO:teuthology.orchestra.console.smithi183:Power off 2024-02-20T09:39:23.369 DEBUG:teuthology.orchestra.console.smithi183:pexpect command: ipmitool -H smithi183.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-02-20T09:39:23.379 DEBUG:teuthology.orchestra.console.smithi133:check power output: Chassis Power is on 2024-02-20T09:39:23.393 DEBUG:teuthology.orchestra.console.smithi183:power off output: Chassis Power Control: Down/Off 2024-02-20T09:39:23.393 DEBUG:teuthology.orchestra.console.smithi183:pexpect command: ipmitool -H smithi183.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-20T09:39:23.420 DEBUG:teuthology.orchestra.console.smithi183:check power output: Chassis Power is on 2024-02-20T09:39:27.381 DEBUG:teuthology.orchestra.console.smithi133:pexpect command: ipmitool -H smithi133.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-20T09:39:27.421 DEBUG:teuthology.orchestra.console.smithi183:pexpect command: ipmitool -H smithi183.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-20T09:39:27.492 DEBUG:teuthology.orchestra.console.smithi133:check power output: Chassis Power is on 2024-02-20T09:39:27.532 DEBUG:teuthology.orchestra.console.smithi183:check power output: Chassis Power is on 2024-02-20T09:39:31.496 DEBUG:teuthology.orchestra.console.smithi133:pexpect command: ipmitool -H smithi133.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-20T09:39:31.533 DEBUG:teuthology.orchestra.console.smithi183:pexpect command: ipmitool -H smithi183.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-20T09:39:31.608 DEBUG:teuthology.orchestra.console.smithi133:check power output: Chassis Power is off 2024-02-20T09:39:31.644 DEBUG:teuthology.orchestra.console.smithi183:check power output: Chassis Power is off 2024-02-20T09:39:31.710 INFO:teuthology.orchestra.console.smithi133:Power off completed 2024-02-20T09:39:31.745 INFO:teuthology.orchestra.console.smithi183:Power off completed 2024-02-20T09:39:31.852 INFO:teuthology.run:Summary data: description: fs:upgrade:mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 0a 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client/kclient 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} duration: 6952.134801864624 failure_reason: 'Command failed on smithi133 with status 1: ''sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4537327c-cfc4-11ee-95bb-87774f69a715 -e sha1=91ff50e903dcc99ee3ecf9523050878eb452da5c -- bash -c \''ceph versions | jq -e \''"\''"\''.mgr | keys\''"\''"\'' | grep $sha1\''''' flavor: default owner: scheduled_pdonnell@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=3743dfb8b3974a0ea422048339080efc status: fail success: false 2024-02-20T09:39:31.853 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-02-20T09:39:31.943 INFO:teuthology.run:FAIL